Discrete manufacturing product defect ai vision detection method based on multi-modal fusion

Abstract

The application relates to the technical field of product detection, in particular to a discrete manufacturing product defect AI visual detection method based on multi-modal fusion, which comprises the following steps: acquiring time-aligned visible light images and thermal infrared images of a target product; performing photometric distortion correction on the visible light images; performing radiation calibration correction on the thermal infrared images; performing space registration on the visible light images after photometric distortion correction and the thermal infrared images after radiation calibration correction to obtain registered visible light images and registered thermal infrared images; performing feature level fusion based on multi-level DWT and decision level fusion based on DS evidence theory on the registered visible light images and the registered thermal infrared images to obtain a fusion image; and inputting the fusion image into a visual detection model running on an embedded AI computing platform to obtain a detection result of the target product output by the visual detection model. The method improves the defect detection precision and provides more reliable guarantee for the discrete manufacturing industry.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of product detection, and in particular to a discrete manufacturing product defect AI visual detection method based on multi-modal fusion. BACKGROUND

[0002] In discrete manufacturing, the appearance quality of products not only concerns the external image of the products, but also directly affects their performance and market competitiveness. A small appearance defect, such as a scratch, a crack, or a flaw, may cause a malfunction in the use of the product, thereby affecting the quality and reliability of the entire product. Consumers have increasingly high requirements for product appearance, and products with excellent appearance quality are more likely to gain market favor. Therefore, ensuring that the appearance quality of products is defect-free has become a key link that discrete manufacturing enterprises must pay attention to.

[0003] For a long time, traditional product appearance quality detection has mainly relied on manual (quality inspectors) detection. Quality inspectors rely on naked eye observation and simple tool measurement to check the appearance of products item by item. However, this manual detection method has many drawbacks. On the one hand, manual detection is relatively inefficient, and in the case of large-scale production, it is difficult to meet the demand for rapid delivery; on the other hand, manual detection is prone to errors, and long-term repetitive work can cause visual fatigue of quality inspectors, greatly reducing the detection accuracy. At the same time, there are differences in subjective judgment standards between different quality inspectors, which increases the uncertainty of the detection results and seriously affects the stability of product quality.

[0004] With the continuous progress of science and technology, machine vision detection technology has emerged, bringing new solutions for appearance detection in discrete manufacturing. Machine vision detection uses industrial cameras, lenses, and image processing algorithms to quickly capture and analyze the appearance images of products. Compared with manual detection, machine vision detection has significantly improved in speed and can achieve automated batch detection.

[0005] However, the current machine vision detection method still has many shortcomings.

[0006] In a complex industrial production environment, single-modal visual detection faces serious challenges. Factors such as changes in light within the factory, vibration of equipment, and interference from the surrounding environment can all affect the accuracy of visual detection. Moreover, single-modal visual detection can only obtain information about one aspect of the appearance of the product, making it difficult to fully capture the defect information of the product. For some surface micro-cracks or internal defects, relying solely on visible light image detection may not be able to accurately identify them, limiting the detection accuracy.

[0007] In the context of increasingly stringent product quality requirements in modern manufacturing, single-modal visual inspection techniques have been unable to meet the needs of enterprises for high-precision detection of product appearance defects, and there is an urgent need for scientific and reasonable multi-modal visual inspection techniques to improve detection accuracy. SUMMARY

[0008] To solve the above technical problems, embodiments of the present application propose a discrete manufacturing product defect AI visual inspection method based on multi-modal fusion, which takes advantage of the complementary nature of multi-modal information, fully leverages the strengths of each modality, enhances the detection capability of micro-defects and internal defects, improves adaptability under complex lighting conditions, and realizes batched and automated detection, providing more reliable quality detection assurance for the discrete manufacturing industry.

[0009] To achieve the above purpose, embodiments of the present application propose a discrete manufacturing product defect AI (Artificial Intelligence) visual inspection method based on multi-modal fusion, which is suitable for appearance defect detection of discrete manufacturing products, comprising: acquiring time-aligned visible light images and thermal infrared images of a target product to be detected, performing photometric distortion correction on the visible light images, performing radiation calibration correction on the thermal infrared images, and then performing spatial registration on the photometric distortion corrected visible light images and the radiation calibration corrected thermal infrared images to obtain registered visible light images and registered thermal infrared images; performing feature level fusion based on multi-level DWT (Discrete Wavelet Transform) and decision level fusion based on DS (Dempster Shafer) evidence theory on the registered visible light images and the registered thermal infrared images to obtain a fusion image; inputting the fusion image into a pre-trained visual inspection model running on an embedded AI computing platform to obtain a detection result of the target product output by the visual inspection model; wherein the visual inspection model is composed of a dynamic attention gate module, a multi-scale feature aggregation module, a global attention fusion module and an output module, the dynamic attention gate module dynamically enhances the defect area of the fusion image based on a dynamic attention gate mechanism, the multi-scale feature aggregation module captures defect features of different scales based on the enhanced fusion image, the global attention fusion module performs global attention fusion on the defect features of different scales to obtain global fusion features, and the output module outputs the detection result of the target product based on the global fusion features.

[0010] To achieve the above object, the embodiment of the present application also proposes a discrete manufacturing product defect AI visual detection system based on multi-modal fusion, which is suitable for appearance defect detection of discrete manufacturing products, and comprises a visible light imaging module, a thermal infrared imaging module, a synchronous control module, a data preprocessing module, a multi-modal fusion module, a model construction module, a model training module, an embedded AI computing platform and a model use module; the synchronous control module is used for synchronously starting and controlling the visible light imaging module and the thermal infrared imaging module based on a timestamp synchronization mechanism to shoot a target product to be detected, so as to obtain time-aligned visible light images and thermal infrared images of the target product; the data preprocessing module is used for performing photometric distortion correction on the visible light images, performing radiation calibration correction on the thermal infrared images, and then performing spatial registration on the visible light images after photometric distortion correction and the thermal infrared images after radiation calibration correction, to obtain registered visible light images and registered thermal infrared images; the multi-modal fusion module is used for performing feature-level fusion based on multi-level DWT and decision-level fusion based on DS evidence theory on the registered visible light images and the registered thermal infrared images, to obtain a fusion image; the model construction module is used for constructing a visual detection model composed of a dynamic attention gate module, a multi-scale feature aggregation module, a global attention fusion module and an output module; the model training module is used for iteratively training the visual detection model based on an industrial defect detection standard dataset until convergence, to obtain a trained visual detection model; the embedded AI computing platform is used for running the trained visual detection model; and the model use module is used for inputting the fusion image into the trained visual detection model running on the embedded AI computing platform, to obtain a detection result of the target product output by the visual detection model; wherein the dynamic attention gate module dynamically enhances a defect area of the fusion image based on a dynamic attention gate mechanism, the multi-scale feature aggregation module captures defect features of different scales based on the enhanced fusion image, the global attention fusion module performs global attention fusion on the defect features of different scales to obtain global fusion features, and the output module outputs the detection result of the target product based on the global fusion features.

[0011] To achieve the above object, the embodiment of the present application also proposes an electronic device, which comprises at least one processor and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a discrete manufacturing product defect AI visual detection method based on multi-modal fusion as described above.

[0012] To achieve the above object, the embodiment of the present application also proposes a computer readable storage medium, which stores a computer program, and the computer program can realize the above-mentioned multi-modal fusion based discrete manufacturing product defect AI visual detection method when executed by a processor.

[0013] The multi-modal fusion based discrete manufacturing product defect AI visual detection method proposed in the present application effectively integrates the complementary information of visible light images and thermal infrared images through multi-modal data fusion technology, significantly improves the detection capability of surface micro-defects and internal hidden defects, has higher comprehensive recognition accuracy compared with traditional single modal detection method, and the introduction of dynamic attention mechanism enhances the focusing capability of the visual detection model on the defect area, and shows more stable detection performance under complex lighting conditions. The multi-modal fusion strategy, i.e. feature level fusion based on multi-level DWT and decision level fusion based on DS evidence theory, effectively improves the quality of multi-modal fusion, effectively overcomes the sensitivity of single modal to light changes and noise interference, and still maintains high reliability in complex environments in industrial field. The visual detection model composed of dynamic attention gate module, multi-scale feature aggregation module, global attention fusion module and output module has very strong anti-interference performance, and has very strong robustness to image noise, mechanical vibration and other interference factors, and reduces the false detection rate. The multi-scale feature fusion mechanism set in the multi-scale feature aggregation module effectively enhances the capture capability of defect morphology diversity, which can cover a wider range of defect types. The embedded AI computing platform provides a hardware acceleration scheme for the running of the visual detection model, greatly reduces the computing resource demand, realizes real-time and efficient inference of edge devices, and improves the deployment flexibility. In summary, the present application realizes batch and automatic detection, and provides more reliable quality detection guarantee for discrete manufacturing industry.

[0014] Optionally, the visible light image is obtained by a visible light imaging module, and the thermal infrared image is obtained by a thermal infrared imaging module.

[0015] The visible light imaging module is equipped with a Sony IMX586 CMOS sensor, the resolution is 4800*3200, and the frame rate is 90fps. The visible light imaging module is also integrated with a ring-shaped LED light supplement system, the wavelength adjustment range of the ring-shaped LED light supplement system is 500nm to 700nm, and the illumination uniformity is greater than 92%;

[0016] The thermal infrared imaging module is equipped with a FLIR Tau2 640 thermal imager, the thermal sensitivity is less than 30mK@30℃, the temperature measurement range is-20℃ to 350℃, and the polarization filter with an extinction ratio greater than 1000:1 is carried.

[0017] The visible light image is subjected to photometric distortion correction, the thermal infrared image is subjected to radiometric calibration correction, and the visible light image subjected to photometric distortion correction and the thermal infrared image subjected to radiometric calibration correction are subjected to spatial registration to obtain a registered visible light image and a registered thermal infrared image, comprising:

[0018] The visible light image is subjected to photometric distortion correction by using a polynomial distortion model with an order greater than 5 to compensate for lens radial distortion, to obtain a visible light image subjected to photometric distortion correction;

[0019] A temperature-gray mapping relationship is established based on a blackbody radiation source, and the thermal infrared image is subjected to radiometric calibration correction by using the temperature-gray mapping relationship to obtain a thermal infrared image subjected to radiometric calibration correction;

[0020] The visible light image subjected to photometric distortion correction and the thermal infrared image subjected to radiometric calibration correction are subjected to sub-pixel level spatial registration based on a SURF feature matching algorithm to obtain a registered visible light image and a registered thermal infrared image with a registration error less than 0.5 pixel.

[0021] Optionally, the registered visible light image and the registered thermal infrared image are subjected to feature level fusion based on multi-level DWT and decision level fusion based on DS evidence theory to obtain a fused image, comprising:

[0022] Basic probability values are respectively assigned to the registered visible light image and the registered thermal infrared image to obtain visible light basic probability values and thermal infrared basic probability values, and a conflict measurement factor is calculated based on the visible light basic probability values and the thermal infrared basic probability values;

[0023] It is judged whether the calculated conflict measurement factor is greater than a preset conflict measurement threshold value;

[0024] If the calculated conflict measurement factor is greater than the conflict measurement threshold value, a decision level fusion weight is calculated based on the visible light basic probability values, the thermal infrared basic probability values and the calculated conflict measurement factor, and the registered visible light image and the registered thermal infrared image are subjected to decision level fusion weighting based on the decision level fusion weight to obtain a visible light image subjected to decision level fusion weighting and a thermal infrared image subjected to decision level fusion weighting;

[0025] The visible light image subjected to decision level fusion weighting and the thermal infrared image subjected to decision level fusion weighting are respectively subjected to multi-level DWT, multi-level low frequency low frequency components, multi-level high frequency low frequency components and multi-level low frequency high frequency components are determined based on multi-level DWT results, and an IDWT is performed on a splicing result of the multi-level low frequency low frequency components, the multi-level high frequency low frequency components and the multi-level low frequency high frequency components to obtain a fused image;

[0026] If the calculated conflict metric factor is less than or equal to the conflict metric threshold, directly performing multi-level DWT on the registered visible light image and the registered thermal infrared image respectively, determining multi-level low-frequency low-frequency components, multi-level high-frequency low-frequency components and multi-level low-frequency high-frequency components based on the multi-level DWT result, and then performing IDWT on the spliced results of the multi-level low-frequency low-frequency components, the multi-level high-frequency low-frequency components and the multi-level low-frequency high-frequency components to obtain the fusion image.

[0027] Optionally, the conflict metric factor is calculated based on the visible light basic probability value and the thermal infrared basic probability value, and is realized by the following formula:

[0028] ;

[0029] , wherein, represents the registered visible light image, represents the registered thermal infrared image, represents the visible light basic probability value, represents the thermal infrared basic probability value,

[0030] The decision-level fusion weight is calculated based on the visible light basic probability value, the thermal infrared basic probability value and the calculated conflict metric factor, and is realized by the following formula:

[0031] ;

[0032] , wherein,

[0033] The decision-level fusion weight is calculated based on the visible light basic probability value, the thermal infrared basic probability value and the calculated conflict metric factor, and is realized by the following formula:

[0034] ;

[0035] ;

[0036] , wherein, represents the decision-level fusion weighted visible light image, represents the decision-level fusion weighted thermal infrared image.

[0037] Optionally, the multi-level low-frequency low-frequency components, the multi-level high-frequency low-frequency components and the multi-level low-frequency high-frequency components are determined based on the multi-level DWT result, and are realized by the following formula:

[0038]

[0039] ;​​​

[0040] ;

[0041] ;

[0042] wherein, denotes performing the DWT of the th stage, is the total number of stages, , , , are the dynamic weight coefficients of the th stage, denotes the ReLU function, denotes the Softmax function, , and denote the low-frequency low-frequency component, the high-frequency low-frequency component and the low-frequency high-frequency component of the th stage, respectively;

[0043] performing the IDWT on the spliced results of the multi-stage low-frequency low-frequency component, the multi-stage high-frequency low-frequency component and the multi-stage low-frequency high-frequency component to obtain the fusion image, which is realized by the following formula:

[0044] ;

[0045] wherein, , and denote the low-frequency low-frequency component, the high-frequency low-frequency component and the low-frequency high-frequency component of the first stage, , and denote the low-frequency low-frequency component, the high-frequency low-frequency component and the low-frequency high-frequency component of the th stage, denotes performing the IDWT, denotes the fusion image.

[0046] Optionally, the visual inspection model is obtained by training through the following steps:

[0047] loading an industrial defect detection standard dataset, the industrial defect detection standard dataset containing a plurality of product appearance images labeled with defect labels, the defect types including three types of scratches, bubbles and stains;

[0048] randomly performing preprocessing including combined application of affine transformation, addition of Gaussian noise and addition of salt and pepper noise on each product appearance image in the industrial defect detection standard dataset, taking the preprocessed product appearance image as a training sample to form a training dataset;

[0049] Based on cross-entropy loss, Dice loss, and structural similarity loss, an overall loss function is designed;

[0050] The training samples in the training dataset are input into the visual detection model to obtain the detection results of the visual detection model on the training samples. The loss value is calculated based on the overall loss function, the defect labels corresponding to the training samples, and the detection results of the visual detection model on the training samples.

[0051] Backpropagation is performed based on the loss value to update the network parameters of the visual detection model until the visual detection model is trained to convergence, thus obtaining the trained visual detection model.

[0052] The overall loss function is expressed by the formula:

[0053] ;

[0054] in, For the overall loss function, For cross-entropy loss, For Dice's loss, For structural similarity loss, , , They are respectively , , The corresponding loss weight coefficients are obtained by searching using the Bayesian optimization algorithm. During the search, the F1 score, minor defect recall rate, and model convergence speed are used as evaluation metrics.

[0055] Optionally, the embedded AI computing platform uses an NVIDIA Jetson AGX Xavier with a computing power of 32 TOPS and a memory size of 32GB. It accelerates the inference of the visual detection model through the TensorRT framework, including performing INT8 quantization on the visual detection model to reduce the inference latency, and adopting a channel pruning and weight sharing mechanism to reduce the size of the visual detection model, thereby achieving the purpose of memory optimization. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0057] Figure 1is a flow chart of a multi-modal fusion-based discrete manufacturing product defect AI visual inspection method provided in an embodiment of the present application;

[0058] Figure 2 is a structural diagram of a visual inspection model provided in an embodiment of the present application;

[0059] Figure 3 is a structural diagram of a multi-modal fusion-based discrete manufacturing product defect AI visual inspection system provided in another embodiment of the present application;

[0060] Figure 4 is a structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. Those skilled in the art can understand that, in the embodiments of the present application, many technical details are proposed in order to make the readers better understand. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The following embodiments are classified for the convenience of description, and should not constitute any limitation on the specific implementation of the present application. The following embodiments can be combined and referenced with each other without contradiction.

[0062] Some common product defect detection methods in the art will be introduced first.

[0063] The machine vision detection technology based on a single visible light image is a relatively common detection means. This technology uses an industrial camera to collect a visible light image of a product under specific lighting conditions, removes noise through pre-processing operations such as gray scale transformation and filtering, then extracts the image edge profile using an edge detection algorithm (such as the Canny algorithm), determines the edge by calculating the image gradient amplitude and direction, using non-maximum suppression and double threshold processing, and finally compares the extracted edge with a standard template to determine whether the product has defects.

[0064] The machine vision detection technology based on a single visible light image has the following shortcomings.

[0065] First, small defects are easily missed. Small defects have weak features on the visible light image, and edge detection methods such as the Canny algorithm cannot accurately extract their edge information, which can lead to missed detection.

[0066] Second, poor light adaptability. Changes in light intensity and angle can cause uneven reflection on the product surface, affecting image quality and thus interfering with edge detection and defect judgment.

[0067] Third, internal defects cannot be detected. Visible light images can only reflect the information on the surface of the product, and cannot detect internal defects such as cracks and bubbles.

[0068] Single-modal image detection technology based on deep learning has been widely applied in recent years. Taking the detection based on convolutional neural network (CNN) as an example, first, a large number of product appearance images are collected as training data, and the images are labeled (defect type, position, etc.), and CNN automatically extracts image features through convolutional layers, pooling layers and fully connected layers. The trained CNN can be used for defect detection on new product images.

[0069] The single-modal image detection technology based on deep learning has the following shortcomings.

[0070] First, the data dependence is serious. The detection accuracy is highly dependent on the size and diversity of the training data. When the data is insufficient, the model has poor generalization ability and it is difficult to accurately detect defects that have not appeared before.

[0071] Second, the single modal is limited. Only a single image modal is used, which cannot fuse other useful information, and the detection effect is not good for complex defects.

[0072] Third, the demand for computing resources is large. The structure of CNN is complex, and strong computing resources are needed for training and reasoning, which limits its application in resource-constrained scenarios.

[0073] To solve the above technical problems, one embodiment of the present application proposes a discrete manufacturing product defect AI visual detection method based on multi-modal fusion, which is suitable for appearance defect detection of discrete manufacturing products and applied to electronic devices. The electronic device can be a terminal or a server. In this embodiment and each of the following embodiments, a server is taken as an example for illustration. The implementation details of the discrete manufacturing product defect AI visual detection method based on multi-modal fusion proposed in this embodiment are described in detail below. The following content only provides related implementation details for easy understanding, and is not necessary for implementing the scheme.

[0074] The specific process of the discrete manufacturing product defect AI visual detection method based on multi-modal fusion proposed in this embodiment can be as shown in Figure 1 , which includes:

[0075] Step 11, obtaining time-aligned visible light images and thermal infrared images of the target product to be detected, performing photometric distortion correction on the visible light images, performing radiation calibration correction on the thermal infrared images, and then performing spatial registration on the photometric distortion corrected visible light images and the radiation calibration corrected thermal infrared images to obtain registered visible light images and registered thermal infrared images.

[0076] In a specific implementation, the server first needs to perform multi-modal data acquisition, that is, to obtain time-aligned visible light images and thermal infrared images of the target product to be detected. Due to the reasons of the acquisition device, the visible light images and the thermal infrared images may have distortion and deviation, and therefore, photometric distortion correction needs to be performed on the visible light images, radiation calibration correction needs to be performed on the thermal infrared images, and then the photometric distortion corrected visible light images and the radiation calibration corrected thermal infrared images need to be spatially registered to obtain the registered visible light images and the registered thermal infrared images.

[0077] In one example, the visible light images are captured by a visible light imaging module, and the thermal infrared images are captured by a thermal infrared imaging module.

[0078] The visible light imaging module is equipped with a Sony IMX586 CMOS sensor, with a resolution of 4800x3200 and a frame rate of 90fps. The visible light imaging module is also integrated with a ring-shaped LED fill light system, with a wavelength adjustment range of 500nm to 700nm and an illumination uniformity of greater than 92%.

[0079] The thermal infrared imaging module is equipped with a FLIR Tau2 640 thermal imager, with a thermal sensitivity of less than 30mK@30℃ and a temperature measurement range of -20℃ to 350℃. It is equipped with a polarization filter with an extinction ratio of greater than 1000 to 1.

[0080] The visible light imaging module and the thermal infrared imaging module are controlled by a synchronization control module. Based on the timestamp synchronization mechanism of FPGA, the synchronization control module synchronously starts the visible light imaging module and the thermal infrared imaging module to capture the target product to be detected, thereby obtaining time-aligned visible light images and thermal infrared images.

[0081] In one example, the server compensates for the radial distortion of the lens using a polynomial distortion model with an order greater than 5 to perform photometric distortion correction on the visible light images, thereby obtaining the photometric distortion corrected visible light images.

[0082] In one example, the server establishes a temperature-gray mapping relationship based on a blackbody radiation source, and uses the temperature-gray mapping relationship to perform radiation calibration correction on the thermal infrared images, thereby obtaining the radiation calibration corrected thermal infrared images.

[0083] In one example, the server performs sub-pixel level spatial registration on the photometric distortion corrected visible light images and the radiation calibration corrected thermal infrared images based on the SURF feature matching algorithm, thereby obtaining the registered visible light images and the registered thermal infrared images with a registration error of less than 0.5 pixel.

[0084] Step 12, performing multi-level DWT-based feature level fusion and DS evidence theory-based decision level fusion on the registered visible light image and the registered thermal infrared image to obtain a fused image.

[0085] In a specific implementation, after the server completes the multi-modal data acquisition and obtains the registered visible light image and the registered thermal infrared image, the server can perform multi-level DWT-based feature level fusion and DS evidence theory-based decision level fusion on the registered visible light image and the registered thermal infrared image to obtain a fused image.

[0086] In one example, during the entire multi-modal fusion process, the server needs to first determine whether to perform DS evidence theory-based decision level fusion. If yes, the server first performs DS evidence theory-based decision level fusion, and then performs multi-level DWT-based feature level fusion. If no, the server directly performs multi-level DWT-based feature level fusion.

[0087] In one example, the server first performs basic probability assignment on the registered visible light image and the registered thermal infrared image respectively to obtain visible light basic probability values and thermal infrared basic probability values, and calculates a conflict measurement factor based on the visible light basic probability values and the thermal infrared basic probability values.

[0088] Then, it is determined whether the calculated conflict measurement factor is greater than a preset conflict measurement threshold.

[0089] If the calculated conflict measurement factor is greater than the conflict measurement threshold, a decision level fusion weight is calculated based on the visible light basic probability values, the thermal infrared basic probability values, and the calculated conflict measurement factor. The registered visible light image and the registered thermal infrared image are weighted by decision level fusion based on the decision level fusion weight to obtain a decision level fusion weighted visible light image and a decision level fusion weighted thermal infrared image.

[0090] Next, multi-level DWT is performed on the decision level fusion weighted visible light image and the decision level fusion weighted thermal infrared image respectively. Multi-level low frequency low frequency components, multi-level high frequency low frequency components, and multi-level low frequency high frequency components are determined based on the multi-level DWT results. IDWT is performed on the spliced results of the multi-level low frequency low frequency components, the multi-level high frequency low frequency components, and the multi-level low frequency high frequency components to obtain a fused image.

[0091] If the calculated conflict measurement factor is less than or equal to the conflict measurement threshold, multi-level DWT is directly performed on the registered visible light image and the registered thermal infrared image respectively. Multi-level low frequency low frequency components, multi-level high frequency low frequency components, and multi-level low frequency high frequency components are determined based on the multi-level DWT results. IDWT is performed on the spliced results of the multi-level low frequency low frequency components, the multi-level high frequency low frequency components, and the multi-level low frequency high frequency components to obtain a fused image.

[0092] In one example, the preset conflict metric threshold is set to 0.7, that is, when the calculated conflict metric factor is greater than 0.7, the decision-level fusion based on the DS evidence theory needs to be performed.

[0093] In one example, the conflict metric factor is calculated based on the visible light basic probability value and the thermal infrared basic probability value, and is realized by the following formula:

[0094] ;

[0095] Wherein, represents the registered visible light image, represents the registered thermal infrared image, represents the visible light basic probability value, represents the thermal infrared basic probability value, represents the conflict metric factor.

[0096] In one example, the decision-level fusion weight is calculated based on the visible light basic probability value, the thermal infrared basic probability value and the calculated conflict metric factor, and is realized by the following formula:

[0097] ;

[0098] Wherein, represents the decision-level fusion weight.

[0099] In one example, the registered visible light image and the registered thermal infrared image are decision-level fusion weighted based on the decision-level fusion weight, to obtain the decision-level fusion weighted visible light image and the decision-level fusion weighted thermal infrared image, and is realized by the following formula:

[0100] ;

[0101] ;

[0102] Wherein, represents the decision-level fusion weighted visible light image, represents the decision-level fusion weighted thermal infrared image.

[0103] In one example, the multi-level low frequency low frequency component, the multi-level high frequency low frequency component and the multi-level low frequency high frequency component are determined based on the multi-level DWT result, and are realized by the following formula:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] wherein, denotes performing the DWT of the th stage, is the total number of stages, , , , are the dynamic weight coefficients of the th stage, denotes the ReLU function, denotes the Softmax function, , and denote the low-frequency low-frequency component, the high-frequency low-frequency component and the low-frequency high-frequency component of the th stage, respectively.

[0109] In one example, the IDWT is performed on the spliced results of the multi-stage low-frequency low-frequency component, the multi-stage high-frequency low-frequency component and the multi-stage low-frequency high-frequency component to obtain the fusion image, which is realized by the following formula:

[0110] ;

[0111] wherein, , and denote the low-frequency low-frequency component, the high-frequency low-frequency component and the low-frequency high-frequency component of the first stage, respectively, , and denote the low-frequency low-frequency component, the high-frequency low-frequency component and the low-frequency high-frequency component of the th stage, respectively, denotes performing the IDWT, denotes the fusion image.

[0112] Step 13, input the fusion image into the pre-trained visual detection model running on the embedded AI computing platform to obtain the detection result of the target product output by the visual detection model, wherein the visual detection model is composed of a dynamic attention gate module, a multi-scale feature aggregation module, a global attention fusion module and an output module, the dynamic attention gate module dynamically enhances the defect area of the fusion image based on the dynamic attention gate mechanism, the multi-scale feature aggregation module captures defect features of different scales based on the enhanced fusion image, the global attention fusion module globally fuses the defect features of different scales to obtain global fusion features, and the output module outputs the detection result of the target product based on the global fusion features.

[0113] In a specific implementation, after obtaining the fused image, the server can input the fused image into a pre-trained visual detection model running on an embedded AI computing platform to obtain a detection result of the target product output by the visual detection model. The visual detection model is composed of a dynamic attention gate module, a multi-scale feature aggregation module, a global attention fusion module, and an output module. The dynamic attention gate module dynamically enhances the defect area of the fused image based on a dynamic attention gate mechanism. The multi-scale feature aggregation module captures defect features of different scales based on the enhanced fused image. The global attention fusion module performs global attention fusion on the defect features of different scales to obtain global fusion features. The output module outputs the detection result of the target product based on the global fusion features.

[0114] In one example, the specific structure of the visual detection model can be as shown in Figure 2 The dynamic attention gate module calculates spatial attention and channel attention through parallel convolution flow to achieve dynamic enhancement of the defect area. The multi-scale feature aggregation module uses an atrous spatial pyramid pooling (ASPP) with dilation rates of 6, 12, and 18 to capture defect features of different scales.

[0115] In one example, the visual detection model is trained by the following steps.

[0116] First, load the industrial defect detection standard dataset (MVTec AD Dataset). The industrial defect detection standard dataset contains a plurality of product appearance images labeled with defect labels. The defect types included in the industrial defect detection standard dataset include scratches, bubbles, and stains.

[0117] Next, randomly perform preprocessing including combined application of affine transformation, addition of Gaussian noise, and addition of salt and pepper noise on each product appearance image in the industrial defect detection standard dataset. The preprocessed product appearance images are used as training samples to form a training dataset. The combined application of affine transformation includes rotation and scaling. The rotation angle ranges from -15° to 15°, and the scaling factor ranges from 0.8 to 1.2.

[0118] Subsequently, based on cross-entropy loss, Dice loss, and structural similarity loss, a total loss function is designed.

[0119] After that, the training samples in the training dataset are input into the visual detection model to obtain a detection result of the training samples output by the visual detection model. Based on the total loss function, the defect labels corresponding to the training samples, and the detection result of the training samples output by the visual detection model, a loss value is calculated.

[0120] Finally, backpropagation is performed based on the loss value to update the network parameters of the visual detection model until the visual detection model is trained to convergence, thus obtaining the trained visual detection model.

[0121] In one example, the overall loss function is expressed by the formula:

[0122] ;

[0123] in, For the overall loss function, For cross-entropy loss, For Dice's loss, For structural similarity loss, , , They are respectively , , The corresponding loss weight coefficients are obtained by searching using the Bayesian optimization algorithm. During the search, the F1 score, minor defect recall rate, and model convergence speed are used as evaluation metrics.

[0124] In one example, the optimal loss weight coefficient , , The value can be determined through simulation experiments, that is, by searching using the Bayesian optimization algorithm. The main evaluation metric during the search process is the F1 score, and the auxiliary evaluation metrics are the recall rate of minor defects and the model convergence speed.

[0125] When performing a search, it is necessary to... , , Set the value range, search step size, and constraints.

[0126] The range of values ​​is The search step size is 0.5, and the constraint condition is... .

[0127] The range of values ​​is The search step size is 0.5, and the constraint is the same as... It exhibits a non-linear relationship.

[0128] The range of values ​​is The search step size is 0.2, and the constraint is to suppress the risk of overfitting due to structural similarity loss.

[0129] In the search, first, initialization is performed, that is, 10 groups of loss weight coefficient combinations are randomly sampled, Latin hypercube sampling can be used, then 5 rounds of search are performed, the FI score needs to be calculated in each round, and the surrogate model (Gaussian process) is updated, and finally the loss weight coefficient combination with the highest F1 score is taken.

[0130] Through 5 independent repeated searches (fixed random seed), the optimal loss weight coefficient combination finally determined is: 、 、 .

[0131] In an example, the embedded AI computing platform selects NVIDIA Jetson AGX Xavier with a computing power of 32 TOPS and a memory size of 32 GB, realizes inference acceleration of the visual detection model through the TensorRT framework, including INT8 quantization of the visual detection model, reduction of inference delay rate, adoption of channel pruning and weight sharing mechanism, and reduction of the volume of the visual detection model, so as to achieve the purpose of memory optimization.

[0132] In an example, in addition to the improvement in the hardware aspect, the embodiment also innovatively improves the software design. An online incremental learning mechanism is set, real-time uploading and labeling of defect samples are supported, a knowledge distillation technology is adopted to realize online updating of the model, which significantly shortens the updating cycle, a dynamic resource allocation mechanism is set, computing power resources are automatically allocated according to the priority of the detection task (for example, the micro defect detection task occupies 60% of the computing power), and the real-time performance of the key task is ensured. A cross-platform deployment mechanism is set, Docker containerized deployment is supported, and Linux / Windows industrial control systems can be compatible.

[0133] The embodiment proposes a discrete manufacturing product defect AI visual detection method based on multi-modal fusion. Through multi-modal data fusion technology, the complementary information of visible light images and thermal infrared images is effectively integrated, and the detection capability for surface micro-defects and internal hidden defects is significantly improved. Compared with traditional single modal detection methods, the method has higher comprehensive recognition accuracy. The introduction of dynamic attention mechanism enhances the focusing ability of the visual detection model on the defect area, and shows more stable detection performance under complex lighting conditions. The multi-modal fusion strategy, i.e. feature-level fusion based on multi-level DWT and decision-level fusion based on DS evidence theory, effectively improves the quality of multi-modal fusion and overcomes the sensitivity of single modal to light changes and noise interference. The visual detection model composed of dynamic attention gate module, multi-scale feature aggregation module, global attention fusion module and output module has very strong anti-interference ability and strong robustness to image noise, mechanical vibration and other interference factors, and reduces the false detection rate. The multi-scale feature fusion mechanism set in the multi-scale feature aggregation module effectively enhances the ability to capture the diversity of defect morphology and can cover a wider range of defect types. The embedded AI computing platform provides a hardware acceleration scheme for the operation of the visual detection model, greatly reduces the computing resource demand, realizes real-time and efficient inference of edge devices, and improves the deployment flexibility. In summary, the embodiment realizes batch and automatic detection and provides more reliable quality detection guarantee for the discrete manufacturing industry.

[0134] The division of steps in the above methods is only for the purpose of clear description, and in implementation, one step can be combined or some steps can be divided into multiple steps, as long as the same logical relationship is included, which is within the protection scope of the present application. Adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present application.

[0135] Another embodiment of the present application proposes a discrete manufacturing product defect AI visual detection system based on multi-modal fusion, which is suitable for appearance defect detection of discrete manufacturing products. The details of the discrete manufacturing product defect AI visual detection system based on multi-modal fusion proposed in the embodiment are described below. The following content only provides implementation details for easy understanding, and is not necessary for implementing the embodiment.

[0136] The specific structure of the discrete manufacturing product defect AI visual detection system based on multi-modal fusion proposed in the embodiment can be as follows Figure 3As shown, it comprises: a visible light imaging module 21, a thermal infrared imaging module 22, a synchronous control module 23, a data preprocessing module 24, a multi-modal fusion module 25, a model construction module 26, a model training module 27, an embedded AI computing platform 28 and a model use module 29.

[0137] The synchronous control module 23 is configured to synchronize the start and control of the visible light imaging module 21 and the thermal infrared imaging module 22 based on a timestamp synchronization mechanism to capture the target product to be detected, thereby obtaining time-aligned visible light images and thermal infrared images of the target product.

[0138] The data preprocessing module 24 is configured to perform photometric distortion correction on the visible light images, radiometric calibration correction on the thermal infrared images, and spatial registration on the photometric distortion corrected visible light images and the radiometric calibration corrected thermal infrared images to obtain registered visible light images and registered thermal infrared images.

[0139] The multi-modal fusion module 25 is configured to perform multi-level DWT-based feature level fusion and DS evidence theory-based decision level fusion on the registered visible light images and the registered thermal infrared images to obtain a fusion image.

[0140] The model construction module 26 is configured to construct a visual detection model composed of a dynamic attention gate module, a multi-scale feature aggregation module, a global attention fusion module and an output module.

[0141] The model training module 27 is configured to iteratively train the visual detection model based on an industrial defect detection standard dataset until convergence, thereby obtaining a trained visual detection model.

[0142] The embedded AI computing platform 28 is configured to run the trained visual detection model.

[0143] The model use module 29 is configured to input the fusion image into the trained visual detection model running on the embedded AI computing platform to obtain a detection result of the target product output by the visual detection model; wherein the dynamic attention gate module dynamically enhances the defect area of the fusion image based on a dynamic attention gate mechanism, the multi-scale feature aggregation module captures defect features of different scales based on the enhanced fusion image, the global attention fusion module performs global attention fusion on the defect features of different scales to obtain global fusion features, and the output module outputs the detection result of the target product based on the global fusion features.

[0144] It can be found that the embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and the embodiment can be implemented in cooperation with the above-mentioned method embodiment. The related technical details and technical effects mentioned in the above-mentioned method embodiment are still valid in the embodiment. In order to reduce repetition, they will not be described here. Accordingly, the related technical details mentioned in the embodiment can also be applied to the above-mentioned method embodiment.

[0145] It is worth mentioning that each module and module involved in the embodiment is a logical module. In actual application, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the embodiment, but this does not mean that there are no other units in the embodiment.

[0146] Another embodiment of the present application provides an electronic device, as shown in Figure 4 The electronic device includes at least one processor 31 and a memory 32 connected with the at least one processor 31. The memory 32 stores instructions executable by the at least one processor 31. The instructions are executed by the at least one processor 31 to enable the at least one processor 31 to perform a method for AI visual detection of discrete manufacturing product defects based on multi-modal fusion as described in the above-mentioned method embodiment.

[0147] The memory and the processor are connected in a bus manner. The bus includes any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers, and power management circuits together, which are well known in the art, and therefore will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on a transmission medium. Data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0148] The processor is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor during operation.

[0149] Another embodiment of the present application provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program, when executed by a processor, enables a method for AI visual detection of discrete manufacturing product defects based on multi-modal fusion to be implemented.

[0150] That is, a person skilled in the art can understand that all or part of the steps in the above method embodiments can be completed by programs instructing relevant hardware, and the programs are stored in a storage medium and include a plurality of instructions for enabling a device (such as a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the method embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0151] A person skilled in the art can understand that each of the above embodiments is a specific embodiment for implementing the present application, and in actual application, various changes can be made in form and details without departing from the spirit and scope of the present application. For those skilled in the art, a number of improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements are also considered to be within the protection scope of the present application.

Claims

1. A multimodal fusion-based AI visual inspection method for discrete manufactured products, applicable to the inspection of appearance defects in discrete manufactured products, characterized in that, The method includes: Acquire time-aligned visible light and thermal infrared images of the target product to be inspected. Perform photometric distortion correction on the visible light image and radiometric calibration correction on the thermal infrared image. Then, perform spatial registration on the photometric distortion-corrected visible light image and the radiometric calibration-corrected thermal infrared image to obtain the registered visible light image and the registered thermal infrared image. The registered visible light image and the registered thermal infrared image are subjected to feature-level fusion based on multi-level DWT and decision-level fusion based on DS evidence theory to obtain a fused image. The fused image is input into a pre-trained visual detection model running on an embedded AI computing platform to obtain the detection result of the target product output by the visual detection model. The visual detection model consists of a dynamic attention gating module, a multi-scale feature aggregation module, a global attention fusion module, and an output module. The dynamic attention gating module dynamically enhances the defect region of the fused image based on the dynamic attention gating mechanism. The multi-scale feature aggregation module captures defect features at different scales based on the enhanced fused image. The global attention fusion module performs global attention fusion on the defect features at different scales to obtain global fused features. The output module outputs the detection result of the target product based on the global fused features. The visible light image undergoes photometric distortion correction, the thermal infrared image undergoes radiometric calibration correction, and then the photometrically distorted visible light image and the radiometrically calibrated thermal infrared image are spatially registered to obtain the registered visible light image and the registered thermal infrared image, including: A polynomial distortion model of order greater than 5 is used to compensate for the radial distortion of the lens to correct the photometric distortion of the visible light image, resulting in a photometrically corrected visible light image. A temperature grayscale mapping relationship is established based on a blackbody radiation source. The thermal infrared image is then calibrated and corrected using the temperature grayscale mapping relationship to obtain the calibrated and corrected thermal infrared image. Based on the SURF feature matching algorithm, sub-pixel spatial registration is performed on the visible light image after photometric distortion correction and the thermal infrared image after radiometric calibration correction, resulting in a registered visible light image and a registered thermal infrared image with a registration error of less than 0.5 pixels. The registered visible light image and the registered thermal infrared image are subjected to feature-level fusion based on multi-level DWT and decision-level fusion based on DS evidence theory to obtain a fused image, including: Basic probability values ​​are assigned to the registered visible light image and the registered thermal infrared image respectively to obtain the basic probability values ​​of visible light and thermal infrared. The conflict metric factor is then calculated based on the basic probability values ​​of visible light and thermal infrared. Determine whether the calculated conflict metric factor is greater than the preset conflict metric threshold; If the calculated conflict metric factor is greater than the conflict metric threshold, then the decision-level fusion weight is calculated based on the basic probability value of visible light, the basic probability value of thermal infrared and the calculated conflict metric factor. The registered visible light image and the registered thermal infrared image are then subjected to decision-level fusion weighting based on the decision-level fusion weight to obtain the decision-level fusion weighted visible light image and the decision-level fusion weighted thermal infrared image. Multi-level DWT was performed on the visible light image and the thermal infrared image after decision-level fusion weighting, respectively. Based on the multi-level DWT results, multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components were determined. Then, IDWT was performed on the stitching results of the multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components to obtain the fused image. If the calculated conflict metric factor is less than or equal to the conflict metric threshold, then multi-level DWT is directly performed on the registered visible light image and the registered thermal infrared image respectively. Based on the multi-level DWT results, multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components are determined. Then, IDWT is performed on the stitching results of the multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components to obtain the fused image.

2. The AI ​​visual inspection method for discrete manufacturing product defects based on multimodal fusion according to claim 1, characterized in that, Visible light images are captured by a visible light imaging module, and thermal infrared images are captured by a thermal infrared imaging module. The visible light imaging module is equipped with a Sony IMX586 CMOS sensor with a resolution of 4800×3200 and a frame rate of 90fps. The visible light imaging module also integrates a ring LED fill light system with a wavelength adjustment range of 500nm to 700nm and an illumination uniformity of more than 92%. The thermal infrared imaging module is equipped with a FLIR Tau2 640 thermal imager with a thermal sensitivity of less than 30mK@30℃ and a temperature measurement range of -20℃ to 350℃. It also features a polarizing filter with an extinction ratio greater than 1000:

1.

3. The AI ​​visual inspection method for discrete manufacturing product defects based on multimodal fusion according to claim 1, characterized in that, The conflict metric factor is calculated based on the fundamental probability values ​​of visible light and thermal infrared light, and is achieved through the following formula: ； in, This represents the registered visible light image. This represents the registered thermal infrared image. This represents the basic probability value of visible light. This represents the basic probability value of thermal infrared radiation. Indicates the conflict measurement factor; Based on the basic probability values ​​of visible light and thermal infrared light, and the calculated conflict metric factor, the decision-level fusion weights are calculated using the following formula: ； in, Indicates the decision-level fusion weight; Based on decision-level fusion weights, the registered visible light image and the registered thermal infrared image are fused and weighted at the decision level to obtain the decision-level fused weighted visible light image and the decision-level fused weighted thermal infrared image, which is achieved by the following formula: ； ； in, This represents the visible light image after decision-level fusion and weighting. This represents the thermal infrared image after decision-level fusion and weighting.

4. The AI ​​visual inspection method for discrete manufacturing product defects based on multimodal fusion according to claim 3, characterized in that, The multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components are determined based on the multi-level DWT results using the following formula: ； ； ； ； in, Indicates proceeding to the first DWT of the grade, For the total series, , , , All are the first The dynamic weighting coefficient of the level, Represents the ReLU function. This represents the Softmax function. , and They represent the first The low-frequency components, high-frequency components, and low-frequency components of the class. The stitching results of multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components are subjected to IDWT to obtain a fused image, which is achieved by the following formula: ； in, , and These represent the low-frequency component, high-frequency component, and low-frequency component of the first stage, respectively. , and They represent the first The low-frequency components, high-frequency components, and low-frequency components of the class. This indicates that IDWT is being performed. This indicates a fused image.

5. The AI ​​visual inspection method for discrete manufacturing product defects based on multimodal fusion according to claim 1, characterized in that, The visual detection model is trained through the following steps: Load the industrial defect detection standard dataset. The industrial defect detection standard dataset contains several product appearance images labeled with defect tags. The defect types include three categories: scratches, bubbles, and stains. Each product appearance image in the industrial defect detection standard dataset is randomly preprocessed, including a combination of affine transformation, Gaussian noise, and salt-and-pepper noise. The preprocessed product appearance images are used as training samples to form a training dataset. Based on cross-entropy loss, Dice loss, and structural similarity loss, an overall loss function is designed; The training samples in the training dataset are input into the visual detection model to obtain the detection results of the visual detection model on the training samples. The loss value is calculated based on the overall loss function, the defect labels corresponding to the training samples, and the detection results of the visual detection model on the training samples. Backpropagation is performed based on the loss value to update the network parameters of the visual detection model until the visual detection model is trained to convergence, thus obtaining the trained visual detection model. The overall loss function is expressed by the formula: ； in, For the overall loss function, For cross-entropy loss, For Dice's loss, For structural similarity loss, , , They are respectively , , The corresponding loss weight coefficients are obtained by searching using the Bayesian optimization algorithm. During the search, the F1 score, minor defect recall rate, and model convergence speed are used as evaluation metrics.

6. A method for AI visual inspection of defects in discrete manufactured products based on multimodal fusion according to any one of claims 1 to 5, characterized in that, The embedded AI computing platform uses an NVIDIA Jetson AGX Xavier with a computing power of 32 TOPS and a memory size of 32GB. It accelerates the inference of the visual inspection model through the TensorRT framework, including performing INT8 quantization on the visual inspection model to reduce the inference latency rate, and adopting channel pruning and weight sharing mechanisms to reduce the size of the visual inspection model, thereby achieving the purpose of memory optimization.

7. A multimodal fusion-based AI visual inspection system for discrete manufactured products, suitable for detecting appearance defects in discrete manufactured products, characterized in that, The system includes: a visible light imaging module, a thermal infrared imaging module, a synchronization control module, a data preprocessing module, a multimodal fusion module, a model building module, a model training module, an embedded AI computing platform, and a model usage module; The synchronization control module is used to synchronously start and control the visible light imaging module and the thermal infrared imaging module based on the timestamp synchronization mechanism to capture images of the target product to be inspected, thereby obtaining time-aligned visible light and thermal infrared images of the target product. The data preprocessing module is used to perform photometric distortion correction on the visible light image, radiometric calibration correction on the thermal infrared image, and then spatial registration on the photometric distortion-corrected visible light image and the radiometric calibration-corrected thermal infrared image to obtain the registered visible light image and the registered thermal infrared image. The multimodal fusion module is used to perform feature-level fusion based on multi-level DWT and decision-level fusion based on DS evidence theory on the registered visible light image and the registered thermal infrared image to obtain a fused image; The model building module is used to build a visual detection model consisting of a dynamic attention gating module, a multi-scale feature aggregation module, a global attention fusion module, and an output module. The model training module is used to iteratively train the visual detection model based on the industrial defect detection standard dataset until convergence, so as to obtain the trained visual detection model. An embedded AI computing platform for running trained visual detection models; The model uses modules to input fused images into a trained visual detection model running on an embedded AI computing platform, and obtain the detection results of the target product output by the visual detection model. Among them, the dynamic attention gating module dynamically enhances the defect regions of the fused image based on the dynamic attention gating mechanism, the multi-scale feature aggregation module captures defect features at different scales based on the enhanced fused image, the global attention fusion module performs global attention fusion on the defect features at different scales to obtain global fused features, and the output module outputs the detection results of the target product based on the global fused features. The visible light image undergoes photometric distortion correction, the thermal infrared image undergoes radiometric calibration correction, and then the photometrically distorted visible light image and the radiometrically calibrated thermal infrared image are spatially registered to obtain the registered visible light image and the registered thermal infrared image, including: A polynomial distortion model of order greater than 5 is used to compensate for the radial distortion of the lens to correct the photometric distortion of the visible light image, resulting in a photometrically corrected visible light image. A temperature grayscale mapping relationship is established based on a blackbody radiation source. The thermal infrared image is then calibrated and corrected using the temperature grayscale mapping relationship to obtain the calibrated and corrected thermal infrared image. Based on the SURF feature matching algorithm, sub-pixel spatial registration is performed on the visible light image after photometric distortion correction and the thermal infrared image after radiometric calibration correction, resulting in a registered visible light image and a registered thermal infrared image with a registration error of less than 0.5 pixels. The registered visible light image and the registered thermal infrared image are subjected to feature-level fusion based on multi-level DWT and decision-level fusion based on DS evidence theory to obtain a fused image, including: Basic probability values ​​are assigned to the registered visible light image and the registered thermal infrared image respectively to obtain the basic probability values ​​of visible light and thermal infrared. The conflict metric factor is then calculated based on the basic probability values ​​of visible light and thermal infrared. Determine whether the calculated conflict metric factor is greater than the preset conflict metric threshold; If the calculated conflict metric factor is greater than the conflict metric threshold, then the decision-level fusion weight is calculated based on the basic probability value of visible light, the basic probability value of thermal infrared and the calculated conflict metric factor. The registered visible light image and the registered thermal infrared image are then subjected to decision-level fusion weighting based on the decision-level fusion weight to obtain the decision-level fusion weighted visible light image and the decision-level fusion weighted thermal infrared image. Multi-level DWT was performed on the visible light image and the thermal infrared image after decision-level fusion weighting, respectively. Based on the multi-level DWT results, multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components were determined. Then, IDWT was performed on the stitching results of the multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components to obtain the fused image. If the calculated conflict metric factor is less than or equal to the conflict metric threshold, then multi-level DWT is directly performed on the registered visible light image and the registered thermal infrared image respectively. Based on the multi-level DWT results, multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components are determined. Then, IDWT is performed on the stitching results of the multi-level low-frequency components, multi-level high-frequency components, and multi-level low-frequency components to obtain the fused image.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform an AI visual inspection method for discrete manufacturing product defects based on multimodal fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement an AI visual detection method for discrete manufacturing product defects based on multimodal fusion as described in any one of claims 1 to 6.

Images