Real-time workpiece quality detection method, device, equipment, medium and product based on multimodal data fusion

Abstract

The present invention discloses a method, device, equipment, medium and product for real-time detection of workpiece quality based on multi-modal data fusion, belonging to the field of artificial intelligence technology. The method includes: obtaining an image of a target workpiece by using the camera, and obtaining time series operation parameters corresponding to the production equipment during the production of the target workpiece by using the sensor; inputting the image and the time series operation parameters into a pre-trained detection model to obtain a quality detection result output by the detection model; mapping the image feature vector and the time feature vector to the same dimension by using the detection model, calculating the attention weight between the two modalities by using the attention mechanism, and performing feature fusion on the image feature vector and the time feature vector by using the attention weight; and performing quality detection on the target workpiece based on the fused features to obtain a quality detection result. The present invention can improve the accuracy of real-time detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a real-time workpiece quality detection method, device, equipment, medium and product based on multimodal data fusion. Background Art

[0002] Workpiece quality detection is an essential part in the workpiece manufacturing process. Through workpiece quality detection, defects such as cracks, pores, deformation, etc. on the workpiece surface can be detected.

[0003] In the related art, a neural network model is trained based on the workpiece surface image to detect the quality of the workpiece. However, only analyzing the workpiece quality through images has poor detection accuracy. Summary of the Invention

[0004] The present invention provides a real-time workpiece quality detection method, device, equipment, medium and product based on multimodal data fusion. The technical solutions are as follows:

[0005] On the one hand, a real-time workpiece quality detection method based on multimodal data fusion is provided, which is applied to an edge device in a detection system. The detection system further includes a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the method includes:

[0006] Obtaining an image of a target workpiece by using the camera, and obtaining time series operating parameters corresponding to the production equipment during the production of the target workpiece by using the sensor;

[0007] Inputting the image and the time series operating parameters into a pre-trained detection model to obtain a quality detection result output by the detection model;

[0008] The processing process of the detection model for the input information includes:

[0009] Extracting features from the image by using an image processing module to obtain an image feature vector;

[0010] Capturing time-dependent features of the time series operating parameters by using a long short-term memory module to obtain a time feature vector;

[0011] Mapping the image feature vector and the time feature vector to the same dimension by using a multimodal feature fusion module, calculating the attention weights between the two modalities by using an attention mechanism, and fusing the image feature vector and the time feature vector by using the attention weights;

[0012] Use a classification module to perform quality inspection on the target workpiece based on the fused features to obtain a quality inspection result.

[0013] On the other hand, a real-time workpiece quality detection device based on multimodal data fusion is provided, which is applied to an edge device in a detection system. The detection system further includes a camera and a sensor connected to the edge device; the camera is used to perform real-time monitoring on the workpieces produced on a production line, and the sensor is used to perform real-time monitoring on the operating parameters of production equipment; the device includes:

[0014] An acquisition unit, configured to use the camera to acquire an image of a target workpiece, and use the sensor to acquire time-series operating parameters corresponding to the production equipment during the production of the target workpiece;

[0015] A detection unit, configured to input the image and the time-series operating parameters into a pre-trained detection model to obtain a quality detection result output by the detection model;

[0016] The processing process of the detection model for the input information includes:

[0017] Use an image processing module to extract features from the image to obtain an image feature vector;

[0018] Use a long short-term memory module to capture time-dependent features from the time-series operating parameters to obtain a time feature vector;

[0019] Use a multimodal feature fusion module to map the image feature vector and the time feature vector to the same dimension, and use an attention mechanism to calculate the attention weights between the two modalities, and use the attention weights to perform feature fusion on the image feature vector and the time feature vector;

[0020] Use a classification module to perform quality inspection on the target workpiece based on the fused features to obtain a quality inspection result.

[0021] On the other hand, a computer device is provided. The computer device includes a memory and a processor. The memory is used to store a computer program, and the processor is used to execute the computer program stored on the memory to implement the steps of the above-mentioned real-time workpiece quality detection method based on multimodal data fusion.

[0022] On the other hand, a computer-readable storage medium is provided. The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned real-time workpiece quality detection method based on multimodal data fusion are implemented.

[0023] On the other hand, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps of the above-mentioned real-time workpiece quality detection method based on multimodal data fusion.

[0024] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0025] A detection system is pre-set on the production line side. A camera monitors the workpieces produced on the production line in real time, and a sensor monitors the operating parameters of the production equipment in real time. A trained detection model is pre-set in the edge device. The image of the target workpiece is obtained by the camera and the time series operating parameters of the production equipment during the production of the target workpiece are obtained by the sensor. The edge device inputs the image and the time series operating parameters into the detection model, so as to be able to detect the quality of the workpieces on the production line in real time. Moreover, by fusing the workpiece image information at the product end and the equipment operation information at the production end, the information complementarity between the production end and the product end can be realized, thereby improving the accuracy of real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 is a flowchart of a real-time workpiece quality detection method based on multimodal data fusion provided by an embodiment of the present invention;

[0028] Figure 2 is a schematic structural diagram of a detection model provided by an embodiment of the present invention;

[0029] Figure 3 is another schematic structural diagram of a detection model provided by an embodiment of the present invention;

[0030] Figure 4 is a structural diagram of a real-time workpiece quality detection device based on multimodal data fusion provided by an embodiment of the present invention;

[0031] Figure 5 is a hardware architecture diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0033] As described above, a neural network model is trained using the workpiece surface image to perform quality inspection on the workpiece. However, relying solely on static visual images for quality inspection results in a single feature and poor accuracy of quality inspection.

[0034] The inventive concept of the present invention lies in: considering that for the production equipment used to produce workpieces, its operating parameters can reflect the dynamic changes during the production process. For example, abnormal fluctuations in operating parameters such as current and voltage may cause damage to the workpiece and affect the quality of the workpiece. Therefore, static unstructured image data and dynamic structured operating parameter data can be fused to jointly detect the quality of the workpiece from both static and dynamic perspectives, so as to solve the defect of relying solely on static visual images for quality inspection.

[0035] Furthermore, in the related art, quality inspection of workpieces is performed before leaving the factory. This is based on the default that the production equipment is normal. However, by adding the operating parameters of the production equipment as the input for workpiece quality inspection, the present invention can not only enhance the quality inspection results based on the operating parameters, but also determine the time period when an abnormality occurs, and then adjust the operating parameters of the production equipment in reverse according to the quality inspection results to improve the production quality of the workpiece.

[0036] The following describes the specific implementation of the above concept.

[0037] Please refer to Figure 1 , a real-time workpiece quality detection method based on multi-modal data fusion provided by an embodiment of the present invention is applied to an edge device in a detection system. The detection system further includes a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the method includes:

[0038] Step 100, obtaining an image of a target workpiece using the camera, and obtaining time-series operating parameters of the production equipment corresponding to the production of the target workpiece using the sensor;

[0039] Step 102: Input the image and the time series operation parameters into a pre-trained detection model to obtain the quality detection result output by the detection model. The processing process of the detection model for the input information includes: using an image processing module to extract features from the image to obtain an image feature vector; using a long short-term memory module to capture time-dependent features of the time series operation parameters to obtain a time feature vector; using a multi-modal feature fusion module to map the image feature vector and the time feature vector to the same dimension, and using an attention mechanism to calculate the attention weights between the two modalities, and using the attention weights to perform feature fusion on the image feature vector and the time feature vector; using a classification module to perform quality detection on the target workpiece based on the fused features to obtain a quality detection result.

[0040] In an embodiment of the present invention, a detection system is pre-set on the production line side. A camera is used to monitor the workpieces produced on the production line in real time, and a sensor is used to monitor the operation parameters of the production equipment in real time. A pre-trained detection model is pre-set in the edge device. The camera is used to obtain an image of the target workpiece, and the sensor is used to obtain the time series operation parameters of the production equipment during the production of the target workpiece. The edge device inputs the image and the time series operation parameters into the detection model, so as to be able to perform real-time detection on the quality of the workpieces on the production line. Moreover, by fusing the workpiece image information at the product end and the equipment operation information at the production end, the information complementarity between the production end and the product end can be realized, thereby improving the accuracy of real-time detection.

[0041] The following describes Figure 1 the execution manners of the steps shown.

[0042] First, for step 100, use the camera to obtain an image of the target workpiece, and use the sensor to obtain the time series operation parameters corresponding to the production equipment during the production of the target workpiece.

[0043] Workpiece production usually faces multiple production stages. In each production stage, corresponding production equipment is required for production. The workpiece quality detection can be performed after all production stages are completed, or can be performed after each production stage is completed.

[0044] In an embodiment of the present invention, the camera can be set on the production line side to take images of each workpiece on the production line, so as to perform quality detection on each workpiece by using the images.

[0045] In addition, the operating parameters of the production equipment can indicate whether there are faults in the production equipment during the workpiece production process and whether they cause fluctuations in the operating parameters, thereby affecting the quality of the workpiece. Therefore, using the operating parameters of the production equipment for workpiece quality inspection can provide complementary information to the workpiece image information, thereby improving the accuracy of workpiece quality inspection.

[0046] It should be noted that this operating parameter is not the production parameter of the workpiece, but the state parameter exhibited by the production equipment. Among them, the operating parameters of the production equipment can be directly measured by sensors, and the operating parameters can include at least one of current, voltage, temperature, vibration frequency, and operating speed.

[0047] Since each stage of the workpiece requires a period of time to complete the production of that stage, the obtained operating parameters correspond to the workpiece during the entire production period. Then, time-series operating parameters can be obtained in real time through sensors.

[0048] Taking the workpiece quality inspection after each production stage is completed as an example, when the workpiece is completed in this production stage, an image of the workpiece is taken by a camera. Since the start time point when the workpiece enters this production stage and the end time point when this production stage is completed can be known, the operating parameters of the production equipment at each time point can be obtained through sensors, and then the operating parameters of the production equipment at each time point from the start time point to the end time point are extracted, that is, time-series operating parameters.

[0049] Since the edge device is set on the production line side, the image taken by the camera of the target workpiece and the operating parameters sent by the sensor can be quickly obtained, reducing the data transmission delay, thereby improving the real-time response ability.

[0050] Then, for step 102, the image and the time-series operating parameters are input into a pre-trained detection model to obtain the quality inspection result output by the detection model.

[0051] In the embodiment of the present invention, the detection model needs to be pre-trained and pre-set in the edge device, and then the real-time detection is realized by using the detection model.

[0052] The detection model will be described first below.

[0053] In an embodiment of the present invention, please refer to Figure 2 , the detection model at least includes: an image processing module, a long short-term memory module, a multi-modal feature fusion module, and a classification module;

[0054] Image processing module. In one implementation, it can be implemented using a Convolutional Neural Network (CNN), combined with a Residual Network and a Feature Pyramid Network to perform multi-scale feature extraction on the image and then enhance the features, so as to improve the sensitivity of the detection model to defects of different scales and output an image feature vector F image . The image feature vector has a fixed length and is used to represent the global features of the image.

[0055] Long Short-Term Memory (LSTM) module. In one implementation, the LSTM module is implemented by a Long Short-Term Memory Neural Network (LSTM) and is used for dynamic time modeling to capture time-dependent features. A Variational Autoencoder (VAE) is introduced to reduce the dimensionality of high-dimensional time series and extract the main feature patterns, and output a time feature vector F for representing the dynamic behavior pattern time .

[0056] Multi-modal feature fusion module, which is used to capture the correlation between the image feature vector and the time feature vector. Specifically, first map the image feature vector and the time feature vector to the same dimension through a fully connected layer to achieve feature alignment for subsequent fusion, then use the attention mechanism to calculate the attention weights between the two modalities (one modality is image features and the other modality is time features), and finally use the attention weights for feature fusion to obtain the fused feature F fusion .

[0057] Classification module, which can be implemented by a Multi-Layer Perceptron (MLP). The MLP uses the fused feature to calculate the probability value and confidence level of the workpiece having quality problems

[0058] It can be seen that the detection module can jointly detect the quality of the workpiece based on the image features on the workpiece side and the time operation features on the production equipment side. The data on both sides form a complementary relationship, thus improving the detection accuracy

[0059] Regarding the above-mentioned modules included in the detection model, the training method of the detection model includes:

[0060] Obtain a plurality of training samples; the training samples include: sample images of sample workpieces acquired based on the camera, time series sample operation parameters of the production equipment during the production of the sample workpieces acquired using the sensor, and the training samples further include: whether the sample workpieces have quality problems

[0061] Use the sample images and the time series sample operation parameters as inputs and whether the sample workpieces have quality problems as outputs to train the detection model using a plurality of training samples

[0062] Among them, the sample workpieces cover workpieces with similar structures but different types; and the sample workpieces with quality problems cover multiple types of quality problems

[0063] Whether the sample workpiece as the output in the training samples has quality problems can include two identifications. One identification is that the sample workpiece has quality problems, and the other identification is that the sample workpiece does not have quality problems. In the embodiments of the present invention, when the sample workpiece has quality problems, the quality problems can be surface defects. In one implementation, the quality problems can be multiple types of surface defects. The types of surface defects can include at least one of cracks, peeling, deformation, and holes. By covering multiple types of quality problems for the sample workpiece used to train the detection model, the detection model can adaptively detect new types of quality problems that occur in the workpiece.

[0064] In addition, generally, the workpieces produced on the same production line are of the same type. However, in order to improve the utilization rate of the production line, workpieces of similar types are also produced. For example, straight screws and cross screws. In the embodiments of the present invention, the sample workpiece can cover workpieces with similar structures but different types. Thus, the detection model trained therefrom can adaptively detect quality problems of workpieces with similar structures but different types.

[0065] Furthermore, considering that when the camera takes images of the workpiece, it will be interfered by environmental parameters, affecting the image quality, and the quality of the image will affect the accuracy of the workpiece quality detection; similarly, if the time-series operation parameters of the production equipment fluctuate greatly, it indicates that the probability of the production equipment having a failure is relatively high, which will increase the probability of the workpiece having quality problems. Therefore, the fluctuation of the time-series operation parameters will affect the accuracy during the workpiece quality detection. Based on this, please refer to Figure 3 , the detection module can further include: a first quality self-assessment module and a second quality self-assessment module;

[0066] Specifically, the processing process of the detection model for the input information further includes:

[0067] Using the first quality self-assessment module to perform quality scoring on the image to obtain a first score value;

[0068] Using the second quality self-assessment module to perform quality scoring on the time-series operation parameters to obtain a second score value;

[0069] Using the multi-modal feature fusion module to calculate the attention weights between the two modalities by combining the first score value and the second score value.

[0070] Among them, the first score value and the second score value can be normalized numerical values.

[0071] In the embodiment of the present invention, since the attention weights of the image features and the time features are already learned and fixed after the above detection model is trained, however, considering that in the actual application process of the detection model, the image is interfered by environmental parameters, for example, the environmental parameters may include illumination, granularity, etc., then if images of different qualities are all subjected to feature fusion according to the learned attention weights and the time feature vectors, it will affect the detection accuracy. Therefore, the first score and the second score can be mapped to modulation coefficients, and the modulation coefficients are multiplied by the attention weights to obtain the updated attention weights, and the updated attention weights are used for feature fusion, so that the contribution of low-quality modal data is reduced, thereby improving the detection accuracy.

[0072] In one implementation, the attention weight before update is:

[0073] A image =Softmax(Q·K T image ),A time =Softmax(Q·K T time )

[0074] where A image is the attention weight of the image features before update, and A time is the attention weight of the time features before update; Q is the query vector; K T image is the key feature of the image modality; K T time is the key feature of the time modality;

[0075] The attention weight after update is:

[0076] W image =Softmax(A image ×θ(q image )),W time =Softmax(A time ·θ(q time ))

[0077] where W image is the attention weight of the image features after update, and W time is the attention weight of the time features after update; θ(·) is a non-linear function; q image is the first score, and q time is the second score, and both the first score and the second score are in the range of [0, 1];

[0078] The fused feature F fusion after feature fusion is:

[0079] F fusion = W image ·F image + W time ·F time

[0080] It can be seen that in the embodiments of the present invention, the attention weights for feature fusion are dynamically updated based on the quality of the current input information, thereby ensuring the long-term performance and robustness of the detection model.

[0081] Specifically, performing quality scoring on the image includes: determining at least one of image sharpness and / or signal-to-noise ratio based on the image, and obtaining a first score according to the image sharpness and / or the signal-to-noise ratio; the first score is positively correlated with the image sharpness / the signal-to-noise ratio;

[0082] Performing quality scoring on the time series operating parameters includes: determining the degree of fluctuation of the time series operating parameters based on the time series previous operating parameters respectively corresponding to the production equipment during the production of several previous workpieces; and obtaining a second score according to the degree of fluctuation; the second score is positively correlated with the degree of fluctuation; the previous workpieces are the workpieces produced prior to the target workpiece on the production line, and the previous workpieces and the target workpiece are workpieces of the same type.

[0083] In the embodiments of the present invention, under normal circumstances, when the production equipment on the production line produces workpieces of the same type, the operating parameters of the production equipment are the same. Therefore, when determining the degree of fluctuation of the time series operating parameters, the time series previous operating parameters of the same type of previous workpieces are used for determination. Among them, the degree of fluctuation can be calculated by means of variance, standard deviation, etc.

[0084] When the detection model further includes the above-mentioned first quality self-evaluation module and second quality self-evaluation module, the training process is still implemented using multiple training samples, so as to perform quality scoring on the two-modal data input through the two quality self-evaluation modules respectively, so as to realize the dynamic adjustment of the attention weights, thereby adapting to the input data under different qualities and ensuring the accuracy of the detection results.

[0085] In one implementation, after obtaining the image and the time series operating parameters, before inputting the image and the time series operating parameters into the detection model, it may further include: performing denoising, enhancement, and normalization processing on the image to improve the analysis accuracy; and / or, performing detrending processing and smoothing processing on the time series operating parameters to remove the global trend interference in the data through the detrending processing, and reducing the noise interference through the smoothing processing to improve the data stability; in addition, data normalization processing may also be performed on the time series operating parameters to unify the scale and distribution of the time series, facilitating the fusion with the image features.

[0086] In the embodiment of the present invention, after the detection model is trained, the detection model is pre-set in the edge device. Each workpiece on the production line can be used as a target workpiece to be detected. Whenever an image and time series operation parameters of the target workpiece are obtained, the edge device inputs the image and time series operation parameters into the detection model to obtain the quality detection result output by the detection model.

[0087] Furthermore, considering that in practical applications, the quality inspection of workpieces is carried out before leaving the factory. If the quality problem of the workpiece is caused by the failure of the production equipment, then a large number of workpieces may have quality problems. In order to avoid the quality problems of a large number of workpieces, in the embodiment of the present invention, the edge device is used to perform real-time detection on the workpieces on the production line. The obtained quality detection result in real time can be used to adjust the operation parameters of the production equipment to improve the quality of the subsequent workpiece production on the production line.

[0088] Specifically, after obtaining the quality detection result, it further includes: if the quality detection result indicates that the target workpiece has a quality problem, then use the quality detection results of several adjacent previous workpieces and subsequent workpieces to determine whether the quality problem of the target workpiece is caused by the operation parameters; if so, locate the target operation parameter that causes this result from the operation parameters to perform an online adjustment on the target operation parameter.

[0089] Among them, the types of the previous workpieces and the subsequent workpieces are the same as the type of the target workpiece.

[0090] If the quality problem of the workpiece is caused by fluctuations in the operation parameters of the production equipment, then its subsequent workpieces are likely to have quality problems. Based on this, when it is detected that the target workpiece has a quality problem, the quality detection results of several adjacent previous workpieces and several adjacent subsequent workpieces can be used together to determine whether the quality problem of the workpiece is caused by the operation parameters. More specifically, it can be determined in the following way:

[0091] If the quality detection results of several adjacent previous workpieces are that they do not have quality problems, and the quality detection results of several adjacent subsequent workpieces are that they have quality problems, then calculate the first mean value of each operation parameter based on the time series operation parameters corresponding to several adjacent previous workpieces; and calculate the second mean value of each operation parameter based on the time series operation parameters corresponding to the target workpiece and several adjacent subsequent workpieces; if there is a target operation parameter and the difference between the first mean value and the second mean value is greater than the set threshold, then determine that the quality problem of the target workpiece is caused by the target operation parameter.

[0092] When making an online adjustment to the target operating parameter, the target operating parameter can be adjusted in the direction of the corresponding first mean value so that the difference between the mean value of the adjusted target operating parameter and the first mean value is not greater than the set threshold value.

[0093] It can be seen that the embodiment of the present invention can not only realize the workpiece quality detection, but also locate the time point of the abnormality and the abnormal operating parameter based on the quality detection result, and then improve the workpiece production quality as a whole by adjusting the production equipment on the production line.

[0094] Please refer to Figure 4 , the embodiment of the present invention provides a real-time workpiece quality detection device based on multi-modal data fusion, which is applied to an edge device in a detection system. The detection system further includes a camera and a sensor connected to the edge device; the camera is used for real-time monitoring of the workpieces produced on the production line, and the sensor is used for real-time monitoring of the operating parameters of the production equipment; the device includes:

[0095] An acquisition unit 400, configured to acquire an image of a target workpiece by using the camera, and acquire time-series operating parameters corresponding to the production equipment during the production of the target workpiece by using the sensor;

[0096] A detection unit 402, configured to input the image and the time-series operating parameters into a pre-trained detection model to obtain a quality detection result output by the detection model;

[0097] The processing process of the detection model for the input information includes:

[0098] Using an image processing module to extract features from the image to obtain an image feature vector;

[0099] Using a long short-term memory module to capture time-dependent features of the time-series operating parameters to obtain a time feature vector;

[0100] Using a multi-modal feature fusion module to map the image feature vector and the time feature vector to the same dimension, calculating the attention weight between the two modalities by using an attention mechanism, and fusing the image feature vector and the time feature vector by using the attention weight;

[0101] Using a classification module to perform quality detection on the target workpiece based on the fused features to obtain a quality detection result.

[0102] In an embodiment of the present invention, the processing process of the detection model for the input information further includes:

[0103] Using a first quality self-evaluation module to perform quality scoring on the image to obtain a first score;

[0104] Use the second quality self - assessment module to perform quality scoring on the time - series operation parameters to obtain a second score;

[0105] Use the multi - modal feature fusion module to calculate the attention weights between two modalities by combining the first score and the second score.

[0106] In an embodiment of the present invention, performing quality scoring on the image includes: determining at least one of image sharpness and / or signal - to - noise ratio based on the image, and obtaining a first score according to the image sharpness and / or the signal - to - noise ratio; the first score is positively correlated with the image sharpness / the signal - to - noise ratio;

[0107] Performing quality scoring on the time - series operation parameters includes: determining the degree of fluctuation of the time - series operation parameters based on the time - series pre - operation parameters respectively corresponding to the production equipment during the production of a number of previous workpieces; and obtaining a second score according to the degree of fluctuation; the second score is positively correlated with the degree of fluctuation; the previous workpieces are the workpieces produced prior to the target workpiece on the production line, and the previous workpieces and the target workpiece are of the same type of workpiece.

[0108] In an embodiment of the present invention, the training method of the detection model includes:

[0109] Obtain a plurality of training samples; the training samples include: sample images of sample workpieces acquired based on the camera, time - series sample operation parameters of the production equipment during the production of the sample workpieces acquired by using the sensor, and the training samples further include: whether the sample workpieces have quality problems;

[0110] Use the sample images and the time - series sample operation parameters as inputs, and whether the sample workpieces have quality problems as outputs to train the detection model by using a plurality of training samples;

[0111] Among them, the sample workpieces cover workpieces with similar structures but different types; and the sample workpieces with quality problems cover multiple types of quality problems.

[0112] In an embodiment of the present invention, the device may further include:

[0113] A positioning and adjustment unit, configured to, when the quality inspection result is that the target workpiece has a quality problem, use the quality inspection results of several adjacent previous workpieces and several adjacent subsequent workpieces to determine whether the quality problem of the target workpiece is caused by the operation parameters; if so, locate the target operation parameter that causes the result from the operation parameters to perform online adjustment on the target operation parameter.

[0114] In one embodiment of the present invention, when the positioning and adjustment unit determines whether the quality problem of the target workpiece is caused by operating parameters, it specifically includes: if the quality inspection results of several adjacent previous workpieces do not have quality problems, and the quality inspection results of several adjacent subsequent workpieces have quality problems, then based on the time-series operating parameters corresponding to several adjacent previous workpieces, calculate the first mean value of each operating parameter; and based on the time-series operating parameters corresponding to the target workpiece and several adjacent subsequent workpieces, calculate the second mean value of each operating parameter; if the difference between the first mean value and the second mean value of the target operating parameter is greater than the set threshold, then determine whether the quality problem of the target workpiece is caused by the target operating parameter.

[0115] It should be noted that: the workpiece quality real-time detection device based on multi-modal data fusion provided in the above embodiment is only illustrated by dividing the above-mentioned functional modules. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the workpiece quality real-time detection device based on multi-modal data fusion provided in the above embodiment and the embodiment of the workpiece quality real-time detection method based on multi-modal data fusion belong to the same concept. For the specific implementation process, please refer to the method embodiment, which will not be elaborated here.

[0116] An embodiment of the present application also provides a computer device, please refer to Figure 5 , the computer device includes a processor and a memory, and at least one instruction, at least one program, a code set or an instruction set is stored in the memory. The at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement the workpiece quality real-time detection method based on multi-modal data fusion provided in each of the above method embodiments.

[0117] An embodiment of the present application also provides a computer-readable storage medium, on which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement the workpiece quality real-time detection method based on multi-modal data fusion provided in each of the above method embodiments.

[0118] An embodiment of the present application also provides a computer program product, which includes a computer program. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the workpiece quality real-time detection method based on multi-modal data fusion described in any one of the above embodiments.

[0119] For the convenience of description, when describing the above system or device, it is divided into various modules or units according to functions for separate description. Of course, when implementing this application, the functions of each unit can be realized in the same or multiple software and / or hardware.

[0120] From the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0121] Finally, it should also be noted that in this article, relational terms such as first, second, third, and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the said element.

[0122] The above are only the preferred embodiments of this application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A real-time workpiece quality detection method based on multi-modal data fusion, characterized in that, An edge device applied to a detection system, the detection system further including a camera and a sensor connected to the edge device; the camera is used for real-time monitoring of workpieces produced on a production line, and the sensor is used for real-time monitoring of operating parameters of production equipment; the method includes: Obtaining an image of a target workpiece by using the camera, and obtaining time-series operating parameters corresponding to the production equipment during the production of the target workpiece by using the sensor; Inputting the image and the time-series operating parameters into a pre-trained detection model to obtain a quality detection result output by the detection model; The processing process of the detection model for the input information includes: using an image processing module to extract features from the image to obtain an image feature vector; using a long short-term memory module to capture time-dependent features from the time-series operating parameters to obtain a time feature vector; using a multi-modal feature fusion module to map the image feature vector and the time feature vector to the same dimension, and using an attention mechanism to calculate the attention weights between the two modalities, and using the attention weights to perform feature fusion on the image feature vector and the time feature vector; using a classification module to perform quality detection on the target workpiece based on the fused features to obtain a quality detection result; The attention weights for feature fusion are dynamically updated based on the quality of the current input information; the processing process of the detection model for the input information further includes: using a first quality self-evaluation module to determine at least one of image clarity and / or signal-to-noise ratio based on the image, and obtaining a first score according to the image clarity and / or the signal-to-noise ratio; the first score is positively correlated with the image clarity / the signal-to-noise ratio; using a second quality self-evaluation module to determine the degree of fluctuation of the time-series operating parameters based on the time-series previous operating parameters respectively corresponding to the production equipment during the production of several previous workpieces; and obtaining a second score according to the degree of fluctuation; the second score is positively correlated with the degree of fluctuation; the previous workpieces are workpieces produced on the production line prior to the target workpiece, and the previous workpieces and the target workpiece are of the same type; mapping the first score and the second score into modulation coefficients, and multiplying the modulation coefficients by the fixed attention weights trained in the detection model to obtain updated attention weights; so as to use the multi-modal feature fusion module to perform the feature fusion according to the updated attention weights, so that the contribution of low-quality modal data is reduced; After obtaining the quality detection result, it further includes: if the quality detection result is that the target workpiece has a quality problem, using the quality detection results of several adjacent previous workpieces and several adjacent subsequent workpieces to determine whether the quality problem of the target workpiece is caused by operating parameters; if so, locating the target operating parameter that causes the result from the operating parameters to perform online adjustment on the target operating parameter; Determining whether the quality problem of the target workpiece is caused by operating parameters includes: if the quality inspection results of several adjacent previous workpieces do not have quality problems, and the quality inspection results of several adjacent subsequent workpieces have quality problems, then based on the time series operating parameters corresponding to several adjacent previous workpieces, calculate the first mean value of each operating parameter; and based on the time series operating parameters corresponding to the target workpiece and several adjacent subsequent workpieces, calculate the second mean value of each operating parameter; if there is a difference between the first mean value and the second mean value of the target operating parameter that is greater than the set threshold, then determine whether the quality problem of the target workpiece is caused by the target operating parameter; When performing on-line adjustment of the target operating parameter, adjust the target operating parameter in the direction corresponding to the first mean value so that the difference between the mean value of the adjusted target operating parameter and the first mean value is not greater than the set threshold.

2. The method according to claim 1, wherein The training method of the detection model includes: Obtain a plurality of training samples; the training samples include: sample images of sample workpieces obtained based on the camera, time series sample operating parameters of the production equipment during the production of the sample workpieces obtained by using the sensor, and the training samples also include: whether the sample workpieces have quality problems; Use the sample image and the time series sample operating parameters as inputs, and whether the sample workpiece has a quality problem as an output to train the detection model with a plurality of training samples; Among them, the sample workpieces cover workpieces with similar structures but different types; and the sample workpieces with quality problems cover multiple types of quality problems.

3. An in - situ workpiece quality detection device based on multi - modal data fusion, characterized in that, Applied to an edge device in a detection system, the detection system further includes a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the device includes: An acquisition unit, configured to obtain an image of a target workpiece by using the camera, and obtain time series operating parameters corresponding to the production equipment during the production of the target workpiece by using the sensor; A detection unit, configured to input the image and the time series operating parameters into a pre-trained detection model to obtain a quality inspection result output by the detection model; The processing process of the detection model for the input information includes: using an image processing module to extract features from the image to obtain an image feature vector; using a long short-term memory module to capture time-dependent features of the time series operating parameters to obtain a time feature vector; using a multi-modal feature fusion module to map the image feature vector and the time feature vector to the same dimension, and using an attention mechanism to calculate the attention weights between the two modalities, and using the attention weights to perform feature fusion on the image feature vector and the time feature vector; using a classification module to perform quality inspection on the target workpiece based on the fused features to obtain a quality inspection result; The attention weights for feature fusion are dynamically updated based on the quality of the current input information; the process of the detection model for processing the input information further includes: using a first quality self-assessment module to determine at least one of image sharpness and / or signal-to-noise ratio based on the image, and obtaining a first score according to the image sharpness and / or the signal-to-noise ratio; the first score is positively correlated with the image sharpness / the signal-to-noise ratio; using a second quality self-assessment module to determine the degree of fluctuation of the time series operating parameters based on the time series pre-order operating parameters respectively corresponding to the production equipment during the production of several pre-order workpieces; and obtaining a second score according to the degree of fluctuation; the second score is positively correlated with the degree of fluctuation; the pre-order workpieces are the workpieces produced prior to the target workpiece on the production line, and the pre-order workpieces and the target workpiece are of the same type; mapping the first score and the second score to modulation coefficients, and multiplying the modulation coefficients by the trained fixed attention weights in the detection model to obtain updated attention weights; so as to use the multi-modal feature fusion module to perform the feature fusion according to the updated attention weights, so as to reduce the contribution of low-quality modal data; A positioning and adjustment unit, configured to, when the quality detection result is that the target workpiece has a quality problem, use the quality detection results of several adjacent pre-order workpieces and several adjacent post-order workpieces to determine whether the quality problem of the target workpiece is caused by operating parameters; if so, locate the target operating parameter that causes the result from the operating parameters to perform an online adjustment on the target operating parameter; When the positioning and adjustment unit executes to determine whether the quality problem of the target workpiece is caused by operating parameters, it specifically includes: if the quality detection results of several adjacent pre-order workpieces are that there are no quality problems, and the quality detection results of several adjacent post-order workpieces are that there are quality problems, then calculate the first mean of each operating parameter based on the time series operating parameters corresponding to the several adjacent pre-order workpieces; and calculate the second mean of each operating parameter based on the time series operating parameters corresponding to the target workpiece and the several adjacent post-order workpieces; if there is a difference between the first mean and the second mean of the target operating parameter that is greater than a set threshold, then determine whether the quality problem of the target workpiece is caused by the target operating parameter; when performing an online adjustment on the target operating parameter, adjust the target operating parameter in the direction corresponding to the first mean, so that the difference between the mean of the adjusted target operating parameter and the first mean is not greater than the set threshold.

4. A computer device, characterized in that, The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored on the memory to implement the steps of the method according to any one of claims 1-2 above.

5. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-2.

6. A computer program product, characterized in that, Including a computer program, which when executed by a processor implements the steps of the method according to any one of claims 1-2.

Images