A multi-band defect image multi-task joint intelligent analysis method, system, device and medium

Abstract

The application discloses a kind of multi-band defect image multi-task joint intelligent analysis method, system, equipment and medium, it is related to intelligent defect detection field.Therein method includes: obtaining the visible light image and infrared thermal image sequence of measured object;With heterochronous space double-dimensional registration model, by thermal contrast sorting and combining the matching effect evaluation criterion Transformer matching network will both registration alignment;With the contrast feature extraction algorithm based on heat diffusion model, solve thermal parameters as defect characteristics in combination with three-dimensional heat conduction equation;Finally visible light image and the infrared thermal image after feature extraction are input into different scale feature extraction layer, and self-adaptive essential learning multi-source fusion recognition network with embedded Dropout operator is retained Dropout operator and carried out multiple inference, and the output target detection recognition information and uncertainty.The application solves the problem that multi-modal data is difficult to align, sub-surface defect is difficult to extract and model is not interpretable, realizes the intelligent comprehensive analysis of defect high precision.

Description

Technical Field

[0001] This invention relates to the field of intelligent defect detection, specifically to a multi-band defect image multi-task joint intelligent analysis method, system, device, and medium. Background Technology

[0002] The statements in this section are provided only as background information in connection with this disclosure and may not constitute prior art.

[0003] As modern manufacturing accelerates its evolution towards higher precision, higher speed, and greater intelligence, traditional manual visual inspection and contact measurement methods are no longer sufficient to meet the stringent requirements of large-scale industrial production due to their susceptibility to subjective experience, visual fatigue, and low efficiency. In recent years, machine vision defect detection technology based on deep learning has been widely applied.

[0004] However, existing defect detection technologies still have the following significant shortcomings: Defect detection based on single-band visual images often only captures surface features and cannot obtain subsurface defect information of the object being tested. When defects exist inside or on the reverse side, this type of detection method is prone to failure due to the lack of defect features.

[0005] To obtain rich defect information, the industry has begun to further integrate electromagnetic eddy current thermal imaging technology to acquire infrared thermal image sequence data. However, visible light images and pulsed eddy current infrared thermal images differ in imaging band, time step, and spatial dimension, making conventional image registration methods (such as SIFT and SURF) prone to matching failures. The combination of multiple detection methods significantly increases data complexity, and ineffective data alignment leads to inefficient comprehensive analysis of multi-source defect data.

[0006] Although data processing methods based on deep learning intelligent models have been widely adopted, these models generally suffer from "black box" characteristics, meaning that the model's cognitive mechanism for the task is unexplainable. In industrial-grade scenarios with high reliability requirements, simply outputting detection results without assessing the "uncertainty (reliability)" of their prediction conclusions leads to unknown risks in the detection results. Summary of the Invention

[0007] The purpose of this invention is to address the technical problems of current visual image-based defect detection, which cannot acquire subsurface defect information, the difficulty in registering multi-band image data leading to inefficient comprehensive analysis, and the "black box" characteristics of existing intelligent recognition models resulting in unknown reliability of detection conclusions. This invention provides a multi-task joint intelligent analysis method, system, device, and medium for multi-band defect images. It achieves spatiotemporal alignment of multi-source heterogeneous data based on a heterogeneous spatiotemporal registration model, deeply mines internal defect parameter features using a contrastive feature extraction algorithm based on a thermal diffusion model, and combines it with an adaptive essential learning multi-source fusion recognition network embedding Dropout operators. This achieves high precision and efficiency in the joint interpretation of various surface and subsurface defects, and provides interpretable joint interpretation with cognitive uncertainty assessment.

[0008] The technical solution of the present invention is as follows: A multi-band defect image multi-task joint intelligent analysis method includes: Acquire visible light images of the object under test, as well as infrared thermal image sequences acquired through pulsed eddy current thermal imaging; The visible light image and the infrared thermal image sequence are registered and aligned using a spatiotemporal two-dimensional registration model. Specifically, this includes: calculating the thermal contrast of each frame of the infrared thermal image sequence and sorting them in descending order; inputting the sorted infrared thermal images frame by frame with the visible light image into a Transformer-based matching network for registration; and performing iterative matching based on matching effect evaluation criteria until the matching requirements are met, thereby obtaining the registered visible light image and infrared thermal image. The registered infrared thermal image is used to extract features using a contrast feature extraction algorithm based on a thermal diffusion model. Specifically, the thermal parameters of the internal heat change of the object under test are solved using the surface temperature distribution formula, and the solved thermal parameters are used as defect features to form a new infrared thermal image. The registered visible light image and the new infrared thermal image are input into a pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation, which outputs the target detection and recognition information and uncertainty of the measured object. The adaptive essential learning multi-source fusion recognition network embeds Dropout operators in feature extraction layers at different scales. During the inference phase, the Dropout operators are retained, and multiple inferences are performed on the same input visible light image and the new infrared thermal image to obtain multiple output results. The mean and variance of the multiple output results are calculated to obtain the target detection and recognition information and the uncertainty, respectively.

[0009] Furthermore, the calculation of the thermal contrast of each frame of the infrared thermal image sequence is performed using the following formula:

[0010] in, For thermal contrast, The maximum pixel value in a certain frame of the infrared thermal image. This is the minimum pixel value in the infrared thermal image of that frame.

[0011] Furthermore, the visible light image and the infrared thermal image have pre-marked coil positions. The step of performing cyclic matching based on matching effect evaluation criteria until the matching requirements are met includes: Set the intersection-union ratio threshold; The cross-union ratio (CUNR) function is used to calculate the CUNR of the coil positions in the visible light image and the infrared thermal image currently being registered. If the calculated cross-union ratio is less than the cross-union ratio threshold, the next frame of the sorted infrared thermal image and the visible light image are selected and input into the matching network again for registration calculation. If the calculated cross-union ratio is greater than or equal to the cross-union ratio threshold, the registration effect is evaluated using the matching effect evaluation criterion based on normalized mutual information. If the requirements are met, the registration ends.

[0012] Furthermore, the registration effect is evaluated using a matching effect evaluation criterion based on normalized mutual information, including: A joint grayscale histogram of the registered visible light image and infrared thermal image is established. The grayscale levels of 0-255 are divided into bins, and the proportion of the number of pixels in each bin to the total number is calculated to obtain the probability of the current grayscale value. The entropy of the infrared thermal image is calculated based on the probability of the occurrence of the grayscale value. Entropy of visible light images and the joint entropy of the two types of images. ; Using the normalized mutual information formula Evaluate the differences between the two registered images; The formulas for calculating the image entropy and joint entropy are as follows:

[0013]

[0014] In the formula, Represents grayscale value The probability of occurrence Indicates the grayscale of the infrared thermal image. And the grayscale of the visible light image is The probability of a pixel appearing.

[0015] Furthermore, the thermal parameters of the internal heat change of the object under test are solved using the surface temperature distribution formula, and the solved thermal parameters are used as defect features to form a new infrared thermal image, specifically including: Formula for obtaining the surface temperature distribution of the object under test based on the three-dimensional heat conduction equation; The thermal parameter in the surface temperature distribution formula is defined as... ; in: Thermal conductivity; Mass density; Specific heat capacity; For depth The density of the measured area; For depth The specific heat capacity of the tested area; The shallow surface depth of the object being measured; The thermal parameters in the surface temperature distribution formula were obtained using the Levenberg-Marquardt algorithm. An estimation solution is performed, and the thermal parameters that cause inconsistent material properties due to defects are taken as defect features.

[0016] Furthermore, the adaptive essential learning multi-source fusion recognition network is a dual-stream backbone network; the method also includes a training phase for the adaptive essential learning multi-source fusion recognition network: The acquired labeled pairs of visible light images and infrared thermal images are input to the visible light branch input and infrared branch input of the dual-stream backbone network, respectively. The Dropout operator is added as a regularization term to the feature extraction layers at different scales to constrain the error function of the model. Convolutional connections are randomly dropped during training to obtain the pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation.

[0017] Furthermore, after performing multiple inferences on the same input visible light image and the new infrared thermal image to obtain multiple output results, and calculating the mean and variance of the multiple output results, the method further includes: The output of the adaptive essential learning multi-source fusion recognition network is filtered using a post-processing method of maximum suppression to obtain a comprehensive detection and recognition result; The defect targets in the comprehensive detection and identification results are processed using a contour extraction algorithm to extract the defect contours; The defect contour, the target detection and identification information, and the uncertainty are output together to complete the joint interpretation of the defect information.

[0018] This invention also proposes a multi-band defect image multi-task joint intelligent analysis system, applied to the method described above, the system comprising: The data acquisition module is used to acquire visible light images of the object under test, as well as infrared thermal image sequences acquired through pulsed eddy current thermal imaging. The spatiotemporal two-dimensional registration module is used to register and align the visible light image and the infrared thermal image sequence using a spatiotemporal two-dimensional registration model. Specifically, it includes: calculating the thermal contrast of each frame of the infrared thermal image sequence and sorting them in descending order; inputting the sorted infrared thermal images frame by frame with the visible light image into a Transformer-based matching network for registration; and performing iterative matching in combination with matching effect evaluation criteria until the matching requirements are met, thereby obtaining the registered visible light image and infrared thermal image. The contrast feature extraction module is used to extract features from the registered infrared thermal image using a contrast feature extraction algorithm based on a thermal diffusion model. Specifically, it includes: solving the thermal parameters of the internal heat change of the object under test using the surface temperature distribution formula, and using the solved thermal parameters as defect features to form a new infrared thermal image. An adaptive fusion recognition module is used to input the registered visible light image and the new infrared thermal image into a pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation, and output the target detection and recognition information and uncertainty of the measured object. The adaptive essential learning multi-source fusion recognition network embeds Dropout operators in feature extraction layers at different scales. During the inference phase, the Dropout operators are retained, and multiple inferences are performed on the same input visible light image and the new infrared thermal image to obtain multiple output results. The mean and variance of the multiple output results are calculated to obtain the target detection and recognition information and the uncertainty, respectively.

[0019] The present invention also proposes an electronic device, comprising: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the instructions stored in the memory to perform the method described above.

[0020] The present invention also proposes a computer-readable storage medium for storing instructions that, when executed, cause the method described above to be implemented.

[0021] Compared with existing technologies, the advantages of this invention are: 1. Achieved efficient and stable alignment of data across time and space, and across modalities. This invention utilizes a heterogeneous spatiotemporal two-dimensional registration model to reduce the temporal dimension of infrared thermal sequences using thermal contrast sorting. It also innovatively connects the Transformer matching network with the Intersection over Union (IoU) and Normalized Mutual Information (NMI) evaluation criteria, overcoming the inherent weakness of traditional registration algorithms in bridging visible light and infrared thermal images. This lays a precise data foundation for subsequent multi-source data analysis.

[0022] 2. Breakthrough in Feature Extraction of Subsurface Hidden Defects. This invention introduces a contrastive feature extraction algorithm based on a thermal diffusion model, mapping pixels in infrared thermal images to surface temperature representations, and extracting thermal parameters related to thermal conductivity, mass density, and specific heat capacity through a three-dimensional heat conduction equation. This approach abandons simple extraction of surface texture, fundamentally uncovering the differences in material properties caused by defects, and significantly enhancing the effective defect information in infrared images.

[0023] 3. This invention endows the intelligent detection model with extremely high reliability and interpretability. It embeds the Dropout operator into the feature extraction layers at different scales of the adaptive essential learning multi-source fusion recognition network, breaking the conventional practice of disabling Dropout during the inference phase. By retaining Dropout during the inference phase to perform multiple inference calculations of the mean and variance, it achieves a quantitative assessment of the "uncertainty" of the model's cognition, effectively solving the "black box" trust crisis of deep learning models and greatly improving the reliability of industrial defect detection conclusions. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0025] Figure 1 This is a structural diagram of a joint defect interpretation method provided in an embodiment of the present invention; Figure 2 This is a diagram of a spatiotemporal two-dimensional registration structure provided in an embodiment of the present invention; Figure 3 The essential feature adaptive fusion detection and recognition network structure diagram for uncertainty estimation provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0027] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0028] Example 1 Please see Figure 1 , Figure 1 This is a structural diagram of a joint defect interpretation method provided in an embodiment of the present invention. The present invention provides a multi-band defect image multi-task joint intelligent analysis method, applicable to the comprehensive analysis of defect image data such as pulsed eddy current infrared thermal image sequences and visible light, to solve problems such as incomplete defect features, inefficient comprehensive analysis, and weak model interpretability in defect detection methods such as electromagnetic eddy current and visual imaging. The method specifically includes the following steps: Step S1: Acquire a visible light image of the object under test, as well as an infrared thermal image sequence acquired by pulsed eddy current thermal imaging.

[0029] In this embodiment, a large number of image pairs can be acquired via the Internet or through the collaborative acquisition of visible light and infrared sensors. Visible light images acquired using visual detection methods have the advantages of high resolution and rich color and texture characteristics. Infrared thermal image sequences obtained through eddy current infrared thermal imaging can effectively image subsurface defects that are undetectable by visible light images, thus overcoming the problem of visible light surface defect detection failure caused by environmental conditions such as lighting and dirt. During the model training phase, the image pairs to be aligned are labeled with the target (defect) location and type as the training set. Specifically, the location of defects in the images can be marked with rectangular boxes and labeled with their categories; unaligned image pairs can be used as the test set.

[0030] Step S2: Use a different spatiotemporal two-dimensional registration model to register and align the visible light image and the infrared thermal image sequence.

[0031] Please see Figure 2 , Figure 2 This is a diagram illustrating the spatiotemporal two-dimensional registration structure. Because the data generated by the two detection methods differ in duration and imaging band, conventional registration methods (such as SIFT and SURF) are prone to failure. This step aims to align the original defect data generated by different detection methods, facilitating subsequent algorithms to uniformly process data from different domains and time steps.

[0032] The registration and alignment specifically includes: calculating the thermal contrast of each frame of the infrared thermal image sequence and sorting them in descending order.

[0033] Specifically, the calculation formula for the thermal contrast of each frame of the infrared thermal image sequence is as follows:

[0034] in, For thermal contrast, The maximum pixel value in a certain frame of the infrared thermal image. This represents the minimum pixel value in the infrared thermal image frame. Thermal contrast filtering and sorting utilizes the physical principle of eddy current heating thermal imaging to select the N frames with the highest thermal contrast from different heating stages. This completes the dimensionality reduction of the infrared thermal sequence in the time dimension, allowing it to enter the step of cyclic registration with the visible light image.

[0035] The sorted first frame infrared thermal image and the visible light image are input into a Transformer-based matching network for registration. The matching is then performed iteratively using matching effect evaluation criteria until the matching requirements are met, resulting in the registered visible light image and infrared thermal image.

[0036] In this specific implementation, the visible light image and the infrared thermal image are pre-marked with coil positions (i.e., the physical anchor points of the coils in the pulsed eddy current heating device). The step of performing cyclic matching based on matching effect evaluation criteria until the matching requirements are met includes: (2.1) Set the Intersection over Union (IoU) threshold a; (2.2) Calculate the cross-union ratio (CUNR) of the coil positions in the visible light image and the infrared thermal image currently being registered using the cross-union ratio function; (2.3) If the calculated cross-union ratio is less than the cross-union ratio threshold a, it indicates that the matching degree of the current frame is poor. The next frame of infrared thermal image after sorting is selected and input into the matching network again with the visible light image for registration calculation. (2.4) If the calculated cross-union ratio is greater than or equal to the cross-union ratio threshold a, the registration effect is further evaluated using the matching effect evaluation criterion based on normalized mutual information (NMI). If the requirement is met, the registration ends; if the requirement is not met, the infrared thermal image is replaced and the registration continues.

[0037] The registration effect is evaluated using a matching effect evaluation criterion based on normalized mutual information. This includes: establishing a joint grayscale histogram of the registered visible light image and infrared thermal image; dividing the grayscale levels from 0 to 255 into bins; calculating the proportion of pixels in each bin to the total number of pixels; and obtaining the probability of the current grayscale value appearing based on the probability of the grayscale value appearing. Entropy of visible light images and the joint entropy of the two types of images. Furthermore, the normalized mutual information formula is used. Evaluate the differences between the two images after registration.

[0038] The formulas for calculating the image entropy and joint entropy are as follows:

[0039]

[0040] In the formula, Represents grayscale value The probability of occurrence Indicates the grayscale of the infrared thermal image. And the grayscale of the visible light image is The probability of a pixel appearing.

[0041] As a preferred embodiment, the Transformer-based matching network described above can be a CRFT network dedicated to multimodal image registration. Its principle is as follows: first, the Transformer is used to eliminate modal differences, and then the inner product is used to calculate the global correlation between pixels, thereby greatly improving the success rate of heterogeneous image registration.

[0042] Step S3: Use a contrastive feature extraction algorithm based on a thermal diffusion model to extract features from the registered infrared thermal image.

[0043] This step aims to enhance the defect information in the infrared thermal image, improving the accuracy of subsequent identification networks. Specifically, it includes using the pixel values ​​of the infrared thermal image as a characterization of the surface temperature of the object under test, and applying this characterization based on the three-dimensional heat conduction equation. Establish a formula for the surface temperature distribution of the object being measured.

[0044] The thermal diffusion model is a physical representation of the internal heat changes of an object under test after eddy current heating. The surface temperature distribution of the object under test can be simplified as follows:

[0045] in, For a moment Surface temperature, The initial ambient temperature, The heating temperature, is the time constant.

[0046] Further, the thermal parameters in the surface temperature distribution formula are determined, and these thermal parameters are related to the thermal conductivity, mass density, and specific heat capacity of the object being measured. Specifically, the thermal parameters in the surface temperature distribution formula are defined as follows: ; in: Thermal conductivity; Mass density; Specific heat capacity; For depth The density of the measured area; For depth The specific heat capacity of the tested area; The shallow surface depth of the object being measured; The presence of defects can lead to inconsistent local properties of the material (i.e., localized defects). (Changes occur), thus causing thermal parameters different.

[0047] Therefore, this step utilizes the Levenberg-Marquardt (LM) algorithm to calculate the thermal parameters in the surface temperature distribution formula. The thermal parameters that cause inconsistent material properties due to defects are used as defect features to form a new infrared thermal image.

[0048] Step S4: Input the registered visible light image and the new infrared thermal image into the pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation, and output the target detection and recognition information and uncertainty of the object under test.

[0049] Please see Figure 3 , Figure 3The diagram shows the structure of the adaptive fusion detection and recognition network for uncertainty estimation. This adaptive essential learning multi-source fusion recognition network is a dual-stream backbone network that adaptively learns and fuses multi-scale compatible features from visible light and infrared thermal images to improve the overall detection and recognition accuracy of various defects.

[0050] In practical applications, the network first needs to be trained: prepare the environment required for model building and training, such as using a Docker + VScode + PyTorch coding and runtime environment. The acquired labeled pairs of visible light images and infrared thermal images are input to the visible light branch input and infrared branch input of the dual-stream backbone network, respectively. Because the network is a dual-stream backbone, the Dropout operator is embedded after each feature extraction layer at different scales in each first-level network. The Dropout operator is used as a regularization term to constrain the model's error function, and convolutional connections are randomly dropped during training to fine-tune the pre-trained model, thereby obtaining the pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation.

[0051] Furthermore, the model after training and fine-tuning is saved and tested against test images.

[0052] In the inference (prediction) phase: This invention breaks with the conventional practice of disabling Dropout during inference in conventional networks, instead retaining the Dropout operator during the inference phase. Dropout is enabled, and multiple (e.g., M times) inferences are performed on the same input visible light image and the new infrared thermal image to obtain M output results; the mean and variance of the M output results are then calculated to obtain the target detection and recognition information (including category confidence and location coordinates) and the uncertainty, respectively.

[0053] Its underlying physical and algorithmic logic lies in the fact that retaining multiple forward propagations of Dropout is equivalent to Bayesian approximation of deep networks. The variance of its output can effectively reflect the "uncertainty" of the model's cognition. That is, this probability is used as the model's cognitive uncertainty to indicate the reliability of the model's prediction results, thus solving the problems of traditional intelligent models being uninterpretable and having unknown reliability of conclusions.

[0054] Finally, after obtaining the results of M inferences, the method further includes: filtering the output of the adaptive essential learning multi-source fusion recognition network using the post-processing method of maximum suppression (NMS) to obtain a comprehensive detection and recognition result; processing the defect target in the comprehensive detection and recognition result using a contour extraction algorithm to extract the defect contour; and finally outputting the defect contour, the target detection and recognition information, and the uncertainty together to complete the joint interpretation of the defect information.

[0055] Example 2 Corresponding to the multi-band defect image multi-task joint intelligent analysis method described in Embodiment 1, this embodiment of the invention also provides a multi-band defect image multi-task joint intelligent analysis system, applied to the method in Embodiment 1. Since the system embodiment is basically similar to the method embodiment, the description is relatively simple; relevant details can be found in the descriptions of the method embodiment.

[0056] The multi-band defect image multi-task joint intelligent analysis system provided in this embodiment includes the following virtual functional modules: Data acquisition module: used to acquire visible light images of the object under test, as well as infrared thermal image sequences acquired through pulsed eddy current thermal imaging.

[0057] Specifically, this module can access or connect to a visible light sensor and an infrared sensor to collaboratively acquire image pairs via an internet interface. The acquired visible light images have high resolution and rich color and texture characteristics; the infrared thermal image sequence can effectively image subsurface defects.

[0058] Spatiotemporal dual-dimensional registration module: used to register and align the visible light image and the infrared thermal image sequence using a spatiotemporal dual-dimensional registration model.

[0059] Specifically, the process includes: calculating the thermal contrast of each frame of the infrared thermal image sequence and sorting them in descending order; inputting the sorted infrared thermal images frame by frame with the visible light image into a Transformer-based matching network for registration; and performing iterative matching in conjunction with matching effect evaluation criteria until the matching requirements are met, thereby obtaining the registered visible light image and infrared thermal image.

[0060] In the specific operational logic of this module, coil positions need to be pre-marked in both visible light and infrared thermal images. An initial matching judgment is made based on the Intersection over Union (IoU) threshold. For image frames with an IoU that meets the threshold, the matching effect evaluation criterion based on Normalized Mutual Information (NMI) is further invoked to evaluate the registration effect. If the requirements are met, the registration ends and the aligned image is output; otherwise, the next frame is extracted in a loop for matching.

[0061] The contrast feature extraction module is used to extract features from the registered infrared thermal image using a contrast feature extraction algorithm based on a thermal diffusion model.

[0062] Specifically, this includes: using the pixel values ​​of the infrared thermal image as a characterization of the surface temperature of the object under test, and establishing a surface temperature distribution formula for the object under test based on the three-dimensional heat conduction equation; determining the thermal parameters in the surface temperature distribution formula, which are related to the thermal conductivity, mass density, and specific heat capacity of the object under test; subsequently, using the Levenberg-Marquardt algorithm to estimate and solve the thermal parameters in the surface temperature distribution formula, and using the thermal parameters that cause inconsistencies in material properties due to defects as defect features to form a new infrared thermal image. This module physically avoids the limitations of conventional direct pixel value extraction and deeply mines the anomalies in the physical parameters inside the material.

[0063] Adaptive fusion recognition module: used to input the registered visible light image and the new infrared thermal image into a pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation, and output the target detection and recognition information and uncertainty of the measured object.

[0064] The adaptive essential learning multi-source fusion recognition network is a dual-stream backbone network, with Dropout operators embedded in feature extraction layers at different scales. During the inference phase, this module keeps the Dropout operator enabled, performs multiple inferences on the same input visible light image and the new infrared thermal image to obtain multiple output results, and calculates the mean and variance of the multiple output results to obtain the target detection and recognition information and the uncertainty, respectively.

[0065] Furthermore, the module also includes a post-processing and interpretation subunit, which is used to filter the network output using a maximum value suppression post-processing method after obtaining multiple inference results to obtain a comprehensive detection and recognition result; and to process the defect target in the comprehensive detection and recognition result using a contour extraction algorithm to extract the defect contour; finally, the defect contour, the target detection and recognition information, and the uncertainty are output together to complete the joint interpretation of the defect information.

[0066] The system provided in this invention achieves alignment of multi-source data with inconsistent time and modality through a spatiotemporal dual-dimensional registration module, deeply mines subsurface defect features through a comparative feature extraction module combined with a thermal diffusion model, and innovatively achieves quantitative evaluation of model uncertainty by utilizing the Dropout mechanism embedded in the adaptive fusion recognition module. This system significantly improves the accuracy, timeliness, reliability, and interpretability of multi-band defect image comprehensive analysis.

[0067] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the multi-band defect image multi-task joint intelligent analysis method flow provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 4 As shown, the electronic device may include: At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 4 The example used is the connection between the processor and memory via a bus. The bus... Figure 4 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 4 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0068] In this embodiment of the invention, the memory stores instructions executable by at least one processor. By executing the instructions stored in the memory, the at least one processor can perform the multi-band defect image multi-task joint intelligent analysis method described above. The processor can implement... Figure 4 The functions of each module in the device shown.

[0069] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0070] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0071] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the multi-band defect image multi-task joint intelligent analysis method disclosed in the embodiments of this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0072] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia cards, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), and electrically erasable programmable read-only memory (EPROM). Only memory (EEPROM), magnetic storage, magnetic disks, optical disks, etc. A memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. The memory in embodiments of this invention can also be a circuit or any other device capable of performing storage functions for storing program instructions and / or data.

[0073] By designing and programming the processor, the code corresponding to the multi-band defect image multi-task joint intelligent analysis method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the method described in the foregoing embodiments during operation. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0074] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions, which, when executed on a computer, cause the computer to perform a multi-band defect image multi-task joint intelligent analysis method described above.

[0075] In some alternative embodiments, the present invention also provides a multi-band defect image multi-task joint intelligent analysis method that can also be implemented as a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the multi-band defect image multi-task joint intelligent analysis method according to various exemplary embodiments of the present invention as described above.

[0076] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can be implemented in one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs) containing computer-usable program code. The form of a computer program product implemented on ROM, optical memory, etc.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0080] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

[0084] This background section is provided to generally present the context of the invention. The work of the currently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither expressly nor impliedly acknowledged as prior art to the invention.

Claims

1. A multi-band defect image multi-task joint intelligent analysis method, characterized in that, include: Acquire visible light images of the object under test, as well as infrared thermal image sequences acquired through pulsed eddy current thermal imaging; The visible light image and the infrared thermal image sequence are registered and aligned using a spatiotemporal two-dimensional registration model. Specifically, this includes: calculating the thermal contrast of each frame of the infrared thermal image sequence and sorting them in descending order; inputting the sorted infrared thermal images frame by frame with the visible light image into a Transformer-based matching network for registration; and performing iterative matching based on matching effect evaluation criteria until the matching requirements are met, thereby obtaining the registered visible light image and infrared thermal image. The registered infrared thermal image is used to extract features using a contrast feature extraction algorithm based on a thermal diffusion model. Specifically, the thermal parameters of the internal heat change of the object under test are solved using the surface temperature distribution formula, and the solved thermal parameters are used as defect features to form a new infrared thermal image. The registered visible light image and the new infrared thermal image are input into a pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation, which outputs the target detection and recognition information and uncertainty of the measured object. The adaptive essential learning multi-source fusion recognition network embeds Dropout operators in feature extraction layers at different scales. During the inference phase, the Dropout operators are retained, and multiple inferences are performed on the same input visible light image and the new infrared thermal image to obtain multiple output results. The mean and variance of the multiple output results are calculated to obtain the target detection and recognition information and the uncertainty, respectively.

2. The multi-band defect image multi-task joint intelligent analysis method according to claim 1, characterized in that, The formula for calculating the thermal contrast of each frame of the infrared thermal image sequence is as follows: in, For thermal contrast, The maximum pixel value in a given frame of the infrared thermal image. This is the minimum pixel value in the infrared thermal image of that frame.

3. The multi-band defect image multi-task joint intelligent analysis method according to claim 1, characterized in that, The visible light image and the infrared thermal image have pre-marked coil positions. The step of performing cyclic matching based on matching effect evaluation criteria until the matching requirements are met includes: Set the intersection-union ratio threshold; The cross-union ratio (CUNR) function is used to calculate the CUNR of the coil positions in the visible light image and the infrared thermal image currently being registered. If the calculated cross-union ratio is less than the cross-union ratio threshold, the next frame of the sorted infrared thermal image and the visible light image are selected and input into the matching network again for registration calculation. If the calculated cross-union ratio is greater than or equal to the cross-union ratio threshold, the registration effect is evaluated using the matching effect evaluation criterion based on normalized mutual information. If the requirements are met, the registration ends.

4. The multi-band defect image multi-task joint intelligent analysis method according to claim 3, characterized in that, The registration effect is evaluated using a matching effect evaluation criterion based on normalized mutual information, including: A joint grayscale histogram of the registered visible light image and infrared thermal image is established. The grayscale levels of 0-255 are divided into bins, and the proportion of the number of pixels in each bin to the total number is calculated to obtain the probability of the current grayscale value. The entropy of the infrared thermal image is calculated based on the probability of the occurrence of the grayscale value. Entropy of visible light images and the joint entropy of the two types of images. ; Using the normalized mutual information formula Evaluate the differences between the two registered images; The formulas for calculating the image entropy and joint entropy are as follows: In the formula, Represents grayscale value The probability of occurrence Indicates the grayscale of the infrared thermal image. And the grayscale of the visible light image is The probability of a pixel appearing.

5. The multi-band defect image multi-task joint intelligent analysis method according to claim 1, characterized in that, The thermal parameters of the internal heat change of the object under test are solved using the surface temperature distribution formula, and the solved thermal parameters are used as defect features to form a new infrared thermal image, specifically including: Formula for obtaining the surface temperature distribution of the object under test based on the three-dimensional heat conduction equation; The thermal parameter in the surface temperature distribution formula is defined as... ,in, Thermal conductivity, For mass density, For specific heat capacity, For depth The density of the measured area, For depth The specific heat capacity of the tested area, The shallow surface depth of the object being measured; The thermal parameters in the surface temperature distribution formula were obtained using the Levenberg-Marquardt algorithm. The estimation and solution are performed, and the thermal parameters that cause inconsistent material properties due to defects are taken as defect features.

6. The multi-band defect image multi-task joint intelligent analysis method according to claim 1, characterized in that, The adaptive essential learning multi-source fusion recognition network is a dual-stream backbone network; the method also includes a training phase for the adaptive essential learning multi-source fusion recognition network: The acquired labeled pairs of visible light images and infrared thermal images are input to the visible light branch input and infrared branch input of the dual-stream backbone network, respectively. The Dropout operator is added as a regularization term to the feature extraction layers at different scales to constrain the error function of the model. Convolutional connections are randomly dropped during training to obtain the pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation.

7. The multi-band defect image multi-task joint intelligent analysis method according to claim 1, characterized in that, After performing multiple inferences on the same input visible light image and the new infrared thermal image to obtain multiple output results, and calculating the mean and variance of the multiple output results, the method further includes: The output of the adaptive essential learning multi-source fusion recognition network is filtered using a post-processing method of maximum suppression to obtain a comprehensive detection and recognition result; The defect targets in the comprehensive detection and identification results are processed using a contour extraction algorithm to extract the defect contours; The defect contour, the target detection and identification information, and the uncertainty are output together to complete the joint interpretation of the defect information.

8. A multi-band defect image multi-task joint intelligent analysis system, characterized in that, The system, applicable to the method as described in any one of claims 1 to 7, comprises: The data acquisition module is used to acquire visible light images of the object under test, as well as infrared thermal image sequences acquired through pulsed eddy current thermal imaging. The spatiotemporal two-dimensional registration module is used to register and align the visible light image and the infrared thermal image sequence using a spatiotemporal two-dimensional registration model. Specifically, it includes: calculating the thermal contrast of each frame of the infrared thermal image sequence and sorting them in descending order; inputting the sorted infrared thermal images frame by frame with the visible light image into a Transformer-based matching network for registration; and performing iterative matching in combination with matching effect evaluation criteria until the matching requirements are met, thereby obtaining the registered visible light image and infrared thermal image. The contrast feature extraction module is used to extract features from the registered infrared thermal image using a contrast feature extraction algorithm based on a thermal diffusion model. Specifically, it includes: solving the thermal parameters of the internal heat change of the object under test using the surface temperature distribution formula, and using the solved thermal parameters as defect features to form a new infrared thermal image. An adaptive fusion recognition module is used to input the registered visible light image and the new infrared thermal image into a pre-trained adaptive essential learning multi-source fusion recognition network for uncertainty estimation, and output the target detection and recognition information and uncertainty of the measured object. The adaptive essential learning multi-source fusion recognition network embeds Dropout operators in feature extraction layers at different scales. During the inference phase, the Dropout operators are retained, and multiple inferences are performed on the same input visible light image and the new infrared thermal image to obtain multiple output results. The mean and variance of the multiple output results are calculated to obtain the target detection and recognition information and the uncertainty, respectively.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which executes the instructions stored in the memory to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store instructions that, when executed, cause the method as described in any one of claims 1-7 to be implemented.

Images