Workpiece quality real-time detection method, device and equipment based on multi-modal data fusion, medium and product

Abstract

The invention discloses a workpiece quality real-time detection method, device and equipment based on multi-modal data fusion, a medium and a product, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring an image of a target workpiece by using the camera, and acquiring time sequence operation parameters corresponding to production equipment during production of the target workpiece by using the sensor; inputting the image and the time sequence operation parameter into a pre-trained detection model to obtain a quality detection result output by the detection model; using a detection model to map the image feature vector and the time feature vector to the same dimension, using an attention mechanism to calculate an attention weight between two modals, and using the attention weight to perform feature fusion on the image feature vector and the time feature vector; and carrying out quality detection on the target workpiece based on the fusion features to obtain a quality detection result. According to the invention, the accuracy of real-time detection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, medium and product for real-time detection of workpiece quality based on multimodal data fusion. Background Art

[0002] Workpiece quality inspection is an indispensable part of the workpiece manufacturing process. Through workpiece quality inspection, it is possible to detect whether there are defects such as cracks, pores, deformation, etc. on the workpiece surface.

[0003] In the related art, the quality of the workpiece is inspected by training a neural network model based on the workpiece surface image. However, the inspection accuracy is poor if the workpiece quality is only analyzed by image. Summary of the invention

[0004] The present invention provides a method, device, equipment, medium and product for real-time detection of workpiece quality based on multimodal data fusion. The technical solution is as follows: On the one hand, a real-time workpiece quality detection method based on multimodal data fusion is provided, which is applied to an edge device in a detection system, wherein the detection system further comprises a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the method comprises: Using the camera to acquire an image of a target workpiece, and using the sensor to acquire time series operating parameters corresponding to a production device during production of the target workpiece; Inputting the image and the time series operation parameters into a pre-trained detection model to obtain a quality detection result output by the detection model; The detection model processes the input information in the following steps: Using an image processing module to extract features from the image to obtain an image feature vector; Using a long short-term memory module to capture time-dependent characteristics of the time series operation parameters to obtain a time feature vector; The image feature vector and the time feature vector are mapped to the same dimension using a multimodal feature fusion module, and an attention mechanism is used to calculate the attention weight between the two modalities, and the image feature vector and the time feature vector are feature fused using the attention weight; The classification module is used to perform quality inspection on the target workpiece based on the fusion features to obtain a quality inspection result.

[0005] On the other hand, a workpiece quality real-time detection device based on multimodal data fusion is provided, which is applied to an edge device in a detection system, wherein the detection system further comprises a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the device comprises: an acquisition unit, configured to acquire an image of a target workpiece using the camera, and to acquire time series operating parameters corresponding to a production device during production of the target workpiece using the sensor; A detection unit, used for inputting the image and the time series operation parameters into a pre-trained detection model to obtain a quality detection result output by the detection model; The detection model processes the input information in the following steps: Using an image processing module to extract features from the image to obtain an image feature vector; Using a long short-term memory module to capture time-dependent characteristics of the time series operation parameters to obtain a time feature vector; The image feature vector and the time feature vector are mapped to the same dimension using a multimodal feature fusion module, and an attention mechanism is used to calculate the attention weight between the two modalities, and the image feature vector and the time feature vector are feature fused using the attention weight; The classification module is used to perform quality inspection on the target workpiece based on the fusion features to obtain a quality inspection result.

[0006] On the other hand, a computer device is provided, which includes a memory and a processor, the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the above-mentioned method for real-time workpiece quality detection based on multimodal data fusion.

[0007] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method for real-time detection of workpiece quality based on multimodal data fusion are implemented.

[0008] On the other hand, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for real-time detection of workpiece quality based on multimodal data fusion.

[0009] The technical solution provided by the present invention can at least bring the following beneficial effects: A detection system is pre-installed on the production line side, and the camera monitors the workpieces produced on the production line in real time, and the sensor monitors the operating parameters of the production equipment in real time. A trained detection model is pre-installed in the edge device, and the camera is used to obtain the image of the target workpiece and the sensor is used to obtain the time series operating parameters of the production equipment during the production of the target workpiece. The edge device inputs the image and time series operating parameters into the detection model, so that the quality of the workpieces on the production line can be detected in real time. In addition, the workpiece image information on the product side and the equipment operation information on the production side are integrated to achieve information complementarity between the production side and the product side, thereby improving the accuracy of real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 It is a flow chart of a method for real-time detection of workpiece quality based on multimodal data fusion provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a detection model structure provided by an embodiment of the present invention; Figure 3 is a schematic diagram of another detection model structure provided by an embodiment of the present invention; Figure 4 It is a structural diagram of a workpiece quality real-time detection device based on multi-modal data fusion provided by an embodiment of the present invention; Figure 5 It is a hardware architecture diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0013] As mentioned above, the workpiece surface image is used to train the neural network model to detect the quality of the workpiece. However, quality inspection only relies on static visual images, which have single features and lead to poor quality inspection accuracy.

[0014] The inventive concept of the present invention is that: considering that the operating parameters of the production equipment used to produce workpieces can reflect the dynamic changes in the production process, for example, abnormal fluctuations in operating parameters such as current and voltage may cause damage to the workpiece and affect the quality of the workpiece. Therefore, the static unstructured image data and the dynamic structured operating parameter data can be fused to jointly detect the quality of the workpiece from both static and dynamic perspectives to solve the defect of quality detection relying solely on static visual images.

[0015] Furthermore, in the related art, the quality inspection of the workpiece is performed before it leaves the factory. This assumes that the production equipment is normal, while the present invention adds the operating parameters of the production equipment as the input for the workpiece quality inspection, which can not only enhance the quality inspection results based on the operating parameters, but also can determine the time period when the abnormality occurs, and then reversely adjust the operating parameters of the production equipment according to the quality inspection results to improve the production quality of the workpiece.

[0016] The specific implementation of the above concept is described below.

[0017] Please refer to Figure 1 , a real-time workpiece quality detection method based on multimodal data fusion provided by an embodiment of the present invention is applied to an edge device in a detection system, wherein the detection system further comprises a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the method comprises: Step 100, using the camera to acquire an image of a target workpiece, and using the sensor to acquire time series operating parameters corresponding to a production device during production of the target workpiece; Step 102, input the image and the time series operating parameters into a pre-trained detection model to obtain the quality detection result output by the detection model; the detection model processes the input information including: using an image processing module to extract features from the image to obtain an image feature vector; using a long short-term memory module to capture time-dependent features of the time series operating parameters to obtain a time feature vector; using a multimodal feature fusion module to map the image feature vector and the time feature vector to the same dimension, and using an attention mechanism to calculate the attention weight between the two modalities, and using the attention weight to perform feature fusion on the image feature vector and the time feature vector; using a classification module to perform quality inspection on the target workpiece based on the fused features to obtain a quality inspection result.

[0018] In an embodiment of the present invention, a detection system is pre-set on the production line side, a camera is used to monitor the workpieces produced on the production line in real time, a sensor is used to monitor the operating parameters of the production equipment in real time, a trained detection model is pre-set in the edge device, the camera is used to obtain an image of the target workpiece, and the sensor is used to obtain the time series operating parameters of the production equipment during the production of the target workpiece, the edge device inputs the image and the time series operating parameters into the detection model, thereby being able to perform real-time detection of the quality of the workpieces on the production line, and the workpiece image information on the product end and the equipment operation information on the production end are integrated, so that information complementarity between the production end and the product end can be achieved, thereby improving the accuracy of real-time detection.

[0019] Described below Figure 1 How the various steps are performed.

[0020] First, with respect to step 100, the camera is used to acquire an image of a target workpiece, and the sensor is used to acquire time series operating parameters corresponding to the production equipment during the production of the target workpiece.

[0021] Workpiece production usually involves multiple production stages, and each production stage requires production to be carried out by the production equipment of the corresponding production stage. Workpiece quality inspection can be carried out after all production stages are completed, or after each production stage is completed.

[0022] In the embodiment of the present invention, a camera may be arranged on the production line side to capture an image of each workpiece on the production line, so as to implement quality inspection of each workpiece using the image.

[0023] In addition, the operating parameters of the production equipment can indicate whether there is a fault in the production equipment during the production of the workpiece and whether it causes fluctuations in the operating parameters, thereby affecting the quality of the workpiece. Therefore, using the operating parameters of the production equipment for workpiece quality inspection can provide complementary information to the workpiece image information, thereby improving the accuracy of workpiece quality inspection.

[0024] It should be noted that the operating parameters are not the production parameters of the workpiece, but the state parameters of the production equipment. The operating parameters of the production equipment can be directly measured by sensors, and the operating parameters may include at least one of current, voltage, temperature, vibration frequency and operating speed.

[0025] Since a workpiece requires a period of time to complete the production of each stage, the operating parameters obtained correspond to the workpiece during the entire production period. In this way, the time series operating parameters can be obtained in real time through sensors.

[0026] Taking the workpiece quality inspection after each production stage as an example, when the workpiece is completed in the production stage, the camera is used to capture the workpiece to obtain an image. Since the start time point when the workpiece enters the production stage and the end time point when the workpiece completes the production stage can be known, the operating parameters of the production equipment at each time point can be obtained through the sensor, and then the operating parameters of the production equipment at each time point from the start time point to the end time point are extracted, that is, the time series operating parameters.

[0027] Since the edge device is set on the production line side, it can quickly obtain the image of the target workpiece taken by the camera and the operating parameters sent by the sensor, reducing data transmission delay and thus improving real-time response capabilities.

[0028] Then, for step 102, the image and the time series operating parameters are input into a pre-trained detection model to obtain a quality detection result output by the detection model.

[0029] In the embodiment of the present invention, the detection model needs to be pre-trained and pre-installed in the edge device, and then the detection model is used to achieve real-time detection.

[0030] The detection model is first explained below.

[0031] In one embodiment of the present invention, please refer to Figure 2 ,The detection model at least includes: an image processing module, a long short-term memory module, a multimodal feature fusion module and a classification module; The image processing module can be implemented by using a convolutional neural network (CNN) and combining a residual network and a feature pyramid network to extract multi-scale features from the image and then perform feature enhancement to improve the sensitivity of the detection model to defects of different scales, and output the image feature vector F image . The image feature vector is of fixed length and is used to represent the global features of the image.

[0032] Long short-term memory module, in one implementation, the long short-term memory module is implemented by a long short-term memory neural network LSTM, which is used for dynamic time modeling to capture time-dependent features; a variational autoencoder is introduced to reduce the dimension of high-dimensional time series, extract the main feature patterns, and output a time feature vector F for characterizing the dynamic behavior pattern time .

[0033] The multimodal feature fusion module is used to capture the association between the image feature vector and the time feature vector. Specifically, the image feature vector and the time feature vector are first mapped to the same dimension through a fully connected layer to achieve feature alignment for subsequent fusion. Then, the attention mechanism is used to calculate the attention weight between the two modalities (one modality is the image feature and the other modality is the time feature). Finally, the attention weight is used to perform feature fusion to obtain the fused feature F. fusion .

[0034] The classification module can be implemented by a multi-layer perceptron, which uses fusion features to calculate the probability value and confidence level of the workpiece having quality problems.

[0035] It can be seen that the detection module can jointly realize the detection of workpiece quality based on the image features on the workpiece side and the time operation features on the production equipment side. The data on both sides form a complementary relationship, thereby improving the detection accuracy.

[0036] For the above modules included in the detection model, the training method of the detection model includes: Acquire a plurality of training samples; the training samples include: sample images of sample workpieces acquired by the camera, time series sample operation parameters of production equipment acquired by the sensor during the production of the sample workpieces, and whether the sample workpieces have quality problems; Taking the sample image and the time series sample operation parameters as input and taking whether the sample workpiece has quality problems as output, so as to train the detection model using a plurality of training samples; The sample workpieces cover workpieces with similar structures but different types; and the sample workpieces with quality problems cover multiple types of quality problems.

[0037] Whether the sample workpiece outputted in the training sample has quality problems may include two identifications, one identification indicating that the sample workpiece has quality problems, and the other identification indicating that the sample workpiece does not have quality problems. In an embodiment of the present invention, when the sample workpiece has quality problems, the quality problem may be a surface defect. In one implementation, the quality problem may be a plurality of types of surface defects. The types of surface defects may include: at least one of cracks, detachment, deformation, and holes. By covering a plurality of types of quality problems, the sample workpiece used to train the detection model can enable the detection model to adaptively detect new types of quality problems appearing in the workpiece.

[0038] In addition, in general, the workpieces produced by the same production line are of the same type, but in order to improve the utilization rate of the production line, similar types of workpieces are also produced. For example, a slotted screw and a cross screw. In the embodiment of the present invention, the sample workpieces can cover workpieces with similar structures but different types, so that the detection model trained thereby can adaptively detect quality problems of workpieces with similar structures but different types.

[0039] Furthermore, considering that the camera will be affected by environmental parameters when taking images of workpieces, affecting the image quality, and the quality of the image will affect the accuracy of workpiece quality detection; similarly, if the time series operating parameters of the production equipment fluctuate greatly, it means that the production equipment has a high probability of failure, which will increase the probability of quality problems in the workpiece. Therefore, the fluctuation of the time series operating parameters will affect the accuracy of workpiece quality detection. Based on this, please refer to Figure 3 , the detection module may also include: a first quality self-assessment module and a second quality self-assessment module; Specifically, the detection model processes the input information further including: Using a first quality self-assessment module to perform a quality score on the image to obtain a first score; Using a second quality self-assessment module to perform a quality score on the time series operation parameter to obtain a second score; The multimodal feature fusion module is used to calculate the attention weight between the two modalities in combination with the first score and the second score.

[0040] The first score and the second score may be normalized values.

[0041] In an embodiment of the present invention, after the above-mentioned detection model is trained, the attention weights for image features and time features have been learned and fixed, but considering that in the actual application of the detection model, the image is interfered by environmental parameters, for example, the environmental parameters may include illumination, granularity, etc., then images of different qualities are all subjected to feature fusion according to the learned attention weights and time feature vectors, which will affect the detection accuracy. Therefore, the first score and the second score can be mapped to a modulation coefficient, and the modulation coefficient is multiplied by the attention weight to obtain an updated attention weight, and the updated attention weight is used for feature fusion, so that the contribution of low-quality modal data is reduced, thereby improving the detection accuracy.

[0042] In one implementation, the attention weight before updating is: A image =Softmax(Q·K T image ), A time =Softmax(Q·KT time ) Among them, A image is the attention weight of the image feature before updating, A time is the attention weight of the temporal feature before updating; Q is the query vector; K T image is the key feature of the image modality; K T time is the key feature of the time mode; The updated attention weights are: W image =Softmax(A image ×θ(q image )),W time =Softmax(A time ·θ(q time )) Among them, W image is the attention weight of the updated image feature, W time is the attention weight of the updated temporal feature; θ(·) is a nonlinear function; q image is the first score, q time is the second score, and both the first score and the second score are in the range of [0,1]; The fused feature F after feature fusion fusion for: F fusion = W image ·F image + W time ·F time It can be seen that in the embodiment of the present invention, the attention weights for feature fusion are dynamically updated based on the quality of the current input information, thereby ensuring the long-term performance and robustness of the detection model.

[0043] Specifically, scoring the quality of the image includes: determining at least one of image clarity and / or signal-to-noise ratio based on the image, and obtaining a first score according to the image clarity and / or the signal-to-noise ratio; the first score is positively correlated with the image clarity / the signal-to-noise ratio; Performing a quality score on a time series operation parameter includes: determining the degree of fluctuation of the time series operation parameter based on the time series predecessor operation parameters respectively corresponding to production equipment of a number of predecessor workpieces during production; and deriving a second score according to the degree of fluctuation; the second score is positively correlated with the degree of fluctuation; the predecessor workpiece is a workpiece produced on the production line before the target workpiece, and the predecessor workpiece and the target workpiece are workpieces of the same type.

[0044] In the embodiment of the present invention, under normal circumstances, when the production equipment on the production line produces the same type of workpieces, the operating parameters of the production equipment are the same. Therefore, when determining the degree of fluctuation of the time series operating parameters, the previous operating parameters of the time series of the previous workpieces of the same type are used to determine. The degree of fluctuation can be calculated by variance, standard deviation, etc.

[0045] When the detection model also includes the above-mentioned first quality self-assessment module and the second quality self-assessment module, the training process is still implemented using multiple training samples, so that the two modal input data are respectively scored for quality through the two quality self-assessment modules, so as to achieve dynamic adjustment of the attention weight, thereby adapting to input data of different qualities and ensuring the accuracy of the detection results.

[0046] In one implementation, after obtaining the image and time series operating parameters and before inputting the image and time series operating parameters into the detection model, it may also include: denoising, enhancing and normalizing the image to improve the analysis accuracy; and / or, detrending and smoothing the time series operating parameters to remove the global trend interference in the data through detrending, and reducing noise interference through smoothing to improve data stability; in addition, data normalization may also be performed on the time series operating parameters to unify the scale and distribution of the time series to facilitate fusion with image features.

[0047] In an embodiment of the present invention, after the detection model training is completed, the detection model is preset in the edge device. Each workpiece on the production line can be used as a target workpiece to be detected. Whenever the image and time series operating parameters of the target workpiece are obtained, the edge device inputs the image and time series operating parameters into the detection model to obtain the quality inspection results output by the detection model.

[0048] Furthermore, considering that in actual applications, the quality inspection of workpieces is carried out before leaving the factory, if the quality problem of the workpiece is caused by a failure of the production equipment, then a large number of workpieces may have quality problems. In order to avoid quality problems of a large number of workpieces, in an embodiment of the present invention, edge devices are used to perform real-time inspection of workpieces on the production line. The real-time quality inspection results can be used to adjust the operating parameters of the production equipment to improve the quality of subsequent workpiece production on the production line.

[0049] Specifically, after obtaining the quality inspection result, it also includes: if the quality inspection result shows that the target workpiece has a quality problem, then using the quality inspection results of several adjacent preceding workpieces and succeeding workpieces to determine whether the quality problem of the target workpiece is caused by operating parameters; if so, locating the target operating parameters that cause the result from the operating parameters to adjust the target operating parameters online.

[0050] The types of the preceding workpiece and the succeeding workpiece are both the same as the type of the target workpiece.

[0051] If the quality problem of a workpiece is caused by fluctuations in the operating parameters of the production equipment, then the subsequent workpieces are likely to have quality problems. Based on this, when the target workpiece is detected to have a quality problem, the quality inspection results of several adjacent previous workpieces and several adjacent subsequent workpieces can be used to jointly determine whether the quality problem of the workpiece is caused by the operating parameters. More specifically, it can be determined in the following way: If the quality inspection results of several adjacent preceding workpieces are that there are no quality problems, and the quality inspection results of several adjacent subsequent workpieces are that there are quality problems, then based on the time series operating parameters corresponding to the several adjacent preceding workpieces, a first mean of each operating parameter is calculated; and based on the time series operating parameters corresponding to the target workpiece and the several adjacent subsequent workpieces, a second mean of each operating parameter is calculated; if there is a target operating parameter, and the difference between the first mean and the second mean is greater than a set threshold, it is determined that the quality problem of the target workpiece is caused by the target operating parameter.

[0052] When the target operating parameter is adjusted online, the target operating parameter may be adjusted in a direction corresponding to the first mean value, so that the difference between the mean value of the adjusted target operating parameter and the first mean value is not greater than the set threshold value.

[0053] It can be seen that the embodiments of the present invention can not only realize workpiece quality inspection, but also locate the time point and abnormal operating parameters of the abnormality based on the quality inspection results, and then improve the overall workpiece production quality by adjusting the production equipment on the production line.

[0054] Please refer to Figure 4 The embodiment of the present invention provides a workpiece quality real-time detection device based on multimodal data fusion, which is applied to an edge device in a detection system, wherein the detection system further comprises a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the device comprises: An acquisition unit 400 is used to acquire an image of a target workpiece using the camera, and to acquire time series operation parameters corresponding to a production device during production of the target workpiece using the sensor; A detection unit 402, used for inputting the image and the time series operation parameter into a pre-trained detection model to obtain a quality detection result output by the detection model; The detection model processes the input information in the following steps: Using an image processing module to extract features from the image to obtain an image feature vector; Using a long short-term memory module to capture time-dependent characteristics of the time series running parameters to obtain a time feature vector; The image feature vector and the time feature vector are mapped to the same dimension using a multimodal feature fusion module, and an attention mechanism is used to calculate the attention weight between the two modalities, and the image feature vector and the time feature vector are feature fused using the attention weight; The classification module is used to perform quality inspection on the target workpiece based on the fusion features to obtain a quality inspection result.

[0055] In one embodiment of the present invention, the process of processing the input information by the detection model further includes: Using a first quality self-assessment module to perform a quality score on the image to obtain a first score; Using a second quality self-assessment module to perform a quality score on the time series operation parameter to obtain a second score; The multimodal feature fusion module is used to calculate the attention weight between the two modalities in combination with the first score and the second score.

[0056] In one embodiment of the present invention, the image quality is scored, including: determining at least one of image clarity and / or signal-to-noise ratio based on the image, and obtaining a first score according to the image clarity and / or the signal-to-noise ratio; the first score is positively correlated with the image clarity / the signal-to-noise ratio; Performing a quality score on a time series operation parameter includes: determining the degree of fluctuation of the time series operation parameter based on the time series predecessor operation parameters respectively corresponding to production equipment of a number of predecessor workpieces during production; and deriving a second score according to the degree of fluctuation; the second score is positively correlated with the degree of fluctuation; the predecessor workpiece is a workpiece produced on the production line before the target workpiece, and the predecessor workpiece and the target workpiece are workpieces of the same type.

[0057] In one embodiment of the present invention, the training method of the detection model includes: Acquire a plurality of training samples; the training samples include: sample images of sample workpieces acquired by the camera, time series sample operation parameters of production equipment acquired by the sensor during the production of the sample workpieces, and whether the sample workpieces have quality problems; Taking the sample image and the time series sample operation parameters as input and taking whether the sample workpiece has quality problems as output, so as to train the detection model using a plurality of training samples; The sample workpieces cover workpieces with similar structures but different types; and the sample workpieces with quality problems cover multiple types of quality problems.

[0058] In one embodiment of the present invention, the device may further include: A positioning and adjustment unit is used to determine whether the quality problem of the target workpiece is caused by operating parameters by using the quality inspection results of several adjacent preceding workpieces and several adjacent subsequent workpieces when the quality inspection result shows that the target workpiece has a quality problem; if so, locate the target operating parameters that cause the result from the operating parameters to adjust the target operating parameters online.

[0059] In one embodiment of the present invention, when the positioning adjustment unit determines whether the quality problem of the target workpiece is caused by the operating parameters, it specifically includes: if the quality inspection results of several adjacent preceding workpieces are that there are no quality problems, and the quality inspection results of several adjacent subsequent workpieces are that there are quality problems, then based on the time series operating parameters corresponding to the several adjacent preceding workpieces, a first mean of each operating parameter is calculated; and based on the time series operating parameters corresponding to the target workpiece and the several adjacent subsequent workpieces, a second mean of each operating parameter is calculated; if the difference between the first mean and the second mean of the target operating parameter is greater than a set threshold, it is determined whether the quality problem of the target workpiece is caused by the target operating parameter.

[0060] It should be noted that the real-time workpiece quality detection device based on multimodal data fusion provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the real-time workpiece quality detection device based on multimodal data fusion provided in the above embodiment and the real-time workpiece quality detection method embodiment based on multimodal data fusion belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0061] The embodiment of the present application also provides a computer device, please refer to Figure 5 The computer device includes a processor and a memory, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the real-time workpiece quality detection method based on multimodal data fusion provided by the above-mentioned method embodiments.

[0062] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the real-time workpiece quality detection method based on multimodal data fusion provided in the above-mentioned method embodiments.

[0063] An embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the real-time workpiece quality detection method based on multimodal data fusion described in any of the above embodiments.

[0064] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0065] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0066] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0067] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A real-time workpiece quality detection method based on multimodal data fusion, characterized in that: An edge device used in a detection system, the detection system further comprising a camera and a sensor connected to the edge device; the camera is used to monitor workpieces produced on a production line in real time, and the sensor is used to monitor operating parameters of production equipment in real time; the method comprises: Using the camera to acquire an image of a target workpiece, and using the sensor to acquire time series operating parameters corresponding to a production device during production of the target workpiece; Inputting the image and the time series operation parameters into a pre-trained detection model to obtain a quality detection result output by the detection model; The detection model processes the input information in the following steps: Using an image processing module to extract features from the image to obtain an image feature vector; Using a long short-term memory module to capture time-dependent characteristics of the time series running parameters to obtain a time feature vector; The image feature vector and the time feature vector are mapped to the same dimension using a multimodal feature fusion module, and an attention mechanism is used to calculate the attention weight between the two modalities, and the image feature vector and the time feature vector are feature fused using the attention weight; The classification module is used to perform quality inspection on the target workpiece based on the fusion features to obtain a quality inspection result.

2. The method according to claim 1, characterized in that The detection model further includes: Using a first quality self-assessment module to perform a quality score on the image to obtain a first score; Using a second quality self-assessment module to perform a quality score on the time series operation parameter to obtain a second score; The multimodal feature fusion module is used to calculate the attention weight between the two modalities in combination with the first score and the second score.

3. The method according to claim 2, characterized in that Scoring the quality of the image, comprising: determining at least one of image clarity and / or signal-to-noise ratio based on the image, and obtaining a first score according to the image clarity and / or the signal-to-noise ratio; the first score is positively correlated with the image clarity / the signal-to-noise ratio; Performing a quality score on a time series operation parameter includes: determining a fluctuation degree of the time series operation parameter based on the time series predecessor operation parameters respectively corresponding to production equipment of a number of predecessor workpieces during production; and deriving a second score according to the fluctuation degree; the second score is positively correlated with the fluctuation degree; the predecessor workpiece is a workpiece produced on the production line before the target workpiece, and the predecessor workpiece and the target workpiece are workpieces of the same type.

4. The method according to claim 2, characterized in that: The training method of the detection model includes: Acquire a plurality of training samples; the training samples include: sample images of sample workpieces acquired by the camera, time series sample operation parameters of production equipment acquired by the sensor during the production of the sample workpieces, and whether the sample workpieces have quality problems; Taking the sample image and the time series sample operation parameters as input and taking whether the sample workpiece has quality problems as output, so as to train the detection model using a plurality of training samples; The sample workpieces cover workpieces with similar structures but different types; and the sample workpieces with quality problems cover multiple types of quality problems.

5. The method according to any one of claims 1 to 4, characterized in that: After obtaining the quality test results, it also includes: If the quality inspection result shows that the target workpiece has a quality problem, the quality inspection results of several adjacent preceding workpieces and several adjacent succeeding workpieces are used to determine whether the quality problem of the target workpiece is caused by the operating parameters; if so, the target operating parameters that cause the result are located from the operating parameters so as to adjust the target operating parameters online.

6. The method according to claim 5, characterized in that Determining whether the quality problem of the target workpiece is caused by an operating parameter includes: If the quality inspection results of several adjacent preceding workpieces are that there is no quality problem, and the quality inspection results of several adjacent subsequent workpieces are that there is a quality problem, then based on the time series operation parameters corresponding to the several adjacent preceding workpieces, a first mean value of each operation parameter is calculated; and calculating a second mean value of each operating parameter based on the time series operating parameters corresponding to the target workpiece and a plurality of adjacent subsequent workpieces; If the difference between the first mean value and the second mean value of the target operating parameter is greater than a set threshold, it is determined whether the quality problem of the target workpiece is caused by the target operating parameter.

7. A real-time workpiece quality detection device based on multimodal data fusion, characterized in that: An edge device used in a detection system, the detection system also includes a camera and a sensor connected to the edge device; the camera is used to monitor the workpieces produced on the production line in real time, and the sensor is used to monitor the operating parameters of the production equipment in real time; the device includes: an acquisition unit, configured to acquire an image of a target workpiece using the camera, and to acquire time series operating parameters corresponding to a production device during production of the target workpiece using the sensor; A detection unit, used for inputting the image and the time series operation parameters into a pre-trained detection model to obtain a quality detection result output by the detection model; The detection model processes the input information in the following steps: Using an image processing module to extract features from the image to obtain an image feature vector; Using a long short-term memory module to capture time-dependent characteristics of the time series running parameters to obtain a time feature vector; The image feature vector and the time feature vector are mapped to the same dimension using a multimodal feature fusion module, and an attention mechanism is used to calculate the attention weight between the two modalities, and the image feature vector and the time feature vector are feature fused using the attention weight; The classification module is used to perform quality inspection on the target workpiece based on the fusion features to obtain a quality inspection result.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any one of the methods described in claims 1-6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images