Online quality monitoring method and system based on optical multispectral fusion

Abstract

The invention provides an on-line quality monitoring method and system based on optical multispectral fusion, and relates to the technical field of multispectral fusion, and the method comprises the steps: collecting visible light and infrared multispectral images of an object to be measured, carrying out the preprocessing and registration, inputting the images into a variational auto-encoder, and carrying out the probability distribution modeling, and obtaining a spectrum basis vector and an initial abundance graph; and constructing a multi-scale morphological feature pyramid, performing graph segmentation in combination with spatial information to obtain a local consistency region, and fusing the local consistency region with the initial abundance graph to obtain an enhanced abundance feature. And fusing the spectrum basis vector and the enhanced abundance feature by adopting a multi-head cross attention mechanism to obtain a multi-spectrum fusion feature. And inputting the fusion features into a dynamic neural network, analyzing a topological connection relationship between nodes, obtaining a structured representation, performing discriminative enhancement, calculating node significance distribution, determining a quality defect area, analyzing a formation mechanism of the quality defect area, and finally generating a quality monitoring report.

Description

Technical Field

[0001] The present invention relates to multi-spectrum fusion technology, and in particular to an online quality monitoring method and system based on optical multi-spectrum fusion. Background Art

[0002] Optical multispectral detection technology plays an important role in industrial product quality monitoring. Traditional single-spectrum detection methods are difficult to fully reflect the quality status of the product surface and interior, while multispectral fusion detection can simultaneously obtain image information of different bands such as visible light and infrared. However, the current multispectral detection technology has the following problems: First, the registration accuracy of different spectral images is insufficient, which affects the accuracy of subsequent analysis; second, the fusion method of multispectral data is relatively simple and fails to fully utilize the complementary information of different spectra; third, the defect detection method is mainly based on shallow features, which makes it difficult to accurately identify defective areas under complex backgrounds; fourth, there is a lack of analysis capabilities for the defect formation mechanism, and it is impossible to provide effective guidance for quality improvement. Therefore, it is urgent to develop an online quality monitoring method that can achieve high-precision multispectral image registration, deep feature fusion, and defect mechanism analysis. Summary of the invention

[0003] The embodiments of the present invention provide an online quality monitoring method and system based on optical multi-spectrum fusion, which can solve the problems in the prior art.

[0004] According to a first aspect of the embodiments of the present invention, Provided is an online quality monitoring method based on optical multi-spectrum fusion, comprising: Collect multispectral image data of the object to be tested at the detection station, set a visible light camera group and an infrared camera group at the detection station, and the trigger signals of the visible light camera group and the infrared camera group are uniformly dispatched by a synchronous control module, and the collected multispectral image data are pre-processed, and the different spectral images are registered by using a feature point matching algorithm combined with camera calibration parameters to obtain the registered multispectral image data; The registered multi-spectral image data is input into a variational autoencoder for probability distribution modeling and relative entropy constraints are introduced. The spectral basis vector and the initial abundance map are obtained by minimizing the reconstruction error. A multi-scale morphological feature pyramid is constructed for the initial abundance map. The non-local feature similarity matrix is ​​calculated. A regional affinity map is constructed in combination with spatial position information. A graph cut algorithm is used to segment the regional affinity map to obtain local consistency regions. The local consistency regions are adaptively fused with the initial abundance map to obtain enhanced abundance features. A multi-head cross-attention mechanism is used to align and complementarily enhance the spectral basis vector and the enhanced abundance features to obtain multi-spectral fusion features. A dynamic neural network is constructed to map the multi-spectral fusion features into network node features, and the topological connection relationship between nodes is constructed based on the feature distance. The dynamic neural network enables the nodes to obtain key information from neighboring nodes to obtain the post-propagation features, and the post-propagation features are subjected to multi-level dimensionality reduction to obtain a structured representation. A feature discrimination criterion is constructed to enhance the discriminability of the structured feature representation, and the node significance distribution is calculated based on the structured feature representation after the discriminability enhancement. The area in the node significance distribution with a significance higher than a preset threshold is determined as the quality defect area. At the same time, the formation mechanism of the quality defects is analyzed based on the propagation path of the network node features after propagation, and a quality monitoring report is generated according to the spatial distribution of the quality defect area and the formation mechanism of the quality defects.

[0005] In an optional embodiment, The collected multispectral image data is preprocessed, and the feature point matching algorithm is combined with the camera calibration parameters to align the different spectral images. The aligned multispectral image data includes: Calculating a local cumulative distribution function through a sliding window, and using the local cumulative distribution function to perform local adaptive histogram equalization processing on the visible light image and the infrared light image to obtain a contrast enhanced image; Performing bilateral filtering on the contrast enhanced image, combining noise suppression processing with wavelet transform, to obtain a denoised image, and performing geometric distortion correction on the denoised image based on a camera imaging model to obtain a preprocessed image; Constructing a multi-layer image pyramid for the preprocessed image, extracting image feature points based on the scale space in each layer of the multi-layer image pyramid, and screening the image feature points by using corner point response values ​​to obtain screened feature points; Extracting the gradient direction histogram and the local binary pattern of the selected feature points through neighborhood block division to generate a feature description vector, matching the feature description vector based on the nearest neighbor ratio method and random consistency test to obtain a set of feature matching point pairs; Calibrate the cameras corresponding to the visible light image and the infrared light image based on the calibration plate image sequence, obtain the camera parameter matrix and distortion parameters, calculate the homography matrix according to the feature matching point pair set, and optimize the transformation parameters of the homography matrix by using an iterative optimization algorithm to obtain an optimized homography matrix; The pre-processed image is resampled by bilinear interpolation using the optimized homography matrix and the camera parameter matrix to obtain a registered visible light image and infrared light image.

[0006] In an optional embodiment, The gradient direction histogram and the local binary pattern are extracted from the selected feature points through neighborhood block, and a feature description vector is generated. The feature description vector is matched based on the nearest neighbor ratio method and random consistency test to obtain a feature matching point pair set including: Dividing a feature description area with the screened feature point as the center, and dividing the feature description area into a plurality of feature sub-blocks; Calculating the gradient magnitude and the gradient direction in the feature sub-block, constructing a gradient direction histogram to obtain a gradient feature vector, extracting a local binary pattern in the feature sub-block and using a rotationally invariant uniform pattern encoding to obtain a texture feature vector, normalizing and concatenating the gradient feature vector and the texture feature vector to form a feature description vector; A tree index structure is constructed for the feature description vector, and a nearest neighbor search method is used to obtain a nearest neighbor point set for each feature point; the nearest neighbor point set is screened according to a nearest neighbor ratio threshold to obtain an initial matching point pair set; a plurality of groups of matching points are selected from the initial matching point pair set by a random sampling method to calculate a perspective transformation model, and the number of matching points that satisfy the perspective transformation model is counted, and the perspective transformation model with the largest number of matching points is determined as the optimal transformation model, and candidate feature matching point pairs that satisfy the optimal transformation model are obtained; Dividing the image containing the candidate matching point pairs into a plurality of grid areas, counting the number of feature matching points in each grid area to obtain a spatial distribution variance, and calculating the distance ratio and angle difference between the feature matching point pairs to obtain a geometric consistency parameter; A weighted voting function is constructed according to the spatial distribution variance and the geometric consistency parameter, and the feature matching point pairs are screened by voting, and the feature matching point pairs that do not meet the spatial distribution constraints and the geometric consistency constraints are eliminated to obtain a final feature matching point pair set.

[0007] In an optional embodiment, The registered multispectral image data is input into a variational autoencoder for probability distribution modeling and relative entropy constraints are introduced. The spectral basis vector and the initial abundance map are obtained by minimizing the reconstruction error. A multi-scale morphological feature pyramid is constructed for the initial abundance map. The non-local feature similarity matrix is ​​calculated. The regional affinity map is constructed in combination with the spatial position information, including: Reconstructing the registered multispectral image data into a two-dimensional matrix, wherein the number of rows of the two-dimensional matrix is ​​the number of image pixels, and the number of columns is the number of spectral bands, and normalizing the two-dimensional matrix to obtain a normalized matrix; Input the normalized matrix into an encoder for feature extraction to obtain a mean vector and a variance vector, sample the mean vector and the variance vector to obtain a latent variable, input the latent variable into a decoder for reconstruction, and restore the data dimension through deconvolution operation and upsampling to obtain a reconstructed matrix; Constructing a loss function including a reconstruction error term and a relative entropy constraint term, obtaining a spectral basis vector and an initial abundance map by minimizing the loss function, and performing non-negative constraint and normalization processing on the spectral basis vector to obtain a standardized spectral basis vector; Based on the initial abundance map, an opening operation is performed to sequentially extract dark area features to obtain a multi-scale dark area feature map set, and a closing operation is performed to sequentially extract bright area features to obtain a multi-scale bright area feature map set; the response values ​​of each pixel position in the multi-scale dark area feature map set and the multi-scale bright area feature map set are combined to obtain a multi-scale feature vector of the current pixel position; The covariance matrix of the multi-scale feature vector is calculated, and the Mahalanobis distance between pixel pairs is calculated based on the covariance matrix. The Mahalanobis distance is mapped through a Gaussian kernel function to obtain a feature similarity matrix. The gradient variance of the local area is calculated to obtain texture complexity, and the texture complexity is used as a weight factor to perform a weighted combination of the feature similarity matrix and the spatial distance of the pixel points to obtain the edge weight of the regional affinity graph. A threshold is set for the edge weight to retain the maximum connection value to obtain the final regional affinity graph.

[0008] In an optional embodiment, The regional affinity map is segmented by a graph cut algorithm to obtain a local consistency region, the local consistency region is adaptively fused with the initial abundance map to obtain an enhanced abundance feature, and the spectral basis vector and the enhanced abundance feature are feature aligned and complementary enhanced by a multi-head cross attention mechanism to obtain multi-spectral fusion features including: The regional affinity graph is constructed as a weighted undirected graph, in which the vertex set is the spatial position of the image pixel points, the edge set is the connection relationship between adjacent pixel points, and the edge weight value is determined by the association strength of the corresponding pixel pairs in the regional affinity graph to obtain a weight matrix; Constructing a degree matrix based on the weight matrix, wherein the diagonal elements of the degree matrix are the sum of the elements of the corresponding rows in the weight matrix, constructing a normalized Laplace matrix based on the weight matrix and the degree matrix, solving the eigenvalue equation of the normalized Laplace matrix, and obtaining an eigenvector; Divide the vertex set into mutually non-intersecting subgraphs according to the sign value of the eigenvector, calculate the cut cost value and subgraph volume value between each subgraph, and when the cut cost value is less than the cut cost threshold and the subgraph volume value is greater than the volume threshold, recursively divide the subgraph until a set of local consistency regions is obtained; Calculating the arithmetic mean of the initial abundance vectors of the pixels in each area of ​​the local consistency area set to obtain the regional abundance vector, calculating the Euclidean distance between the initial abundance vector of the pixels in the local consistency area and the regional abundance vector, and converting the Euclidean distance into a fusion weight coefficient; The initial abundance vector of the pixel point and the regional abundance vector are linearly weighted combined by using the fusion weight coefficient to obtain an enhanced abundance feature matrix, and the spectral basis vector is subjected to query transformation to obtain a query feature matrix, and the enhanced abundance feature matrix is ​​subjected to key transformation and value transformation to obtain a key feature matrix and a value feature matrix, respectively, and the query feature matrix is ​​multiplied by the key feature matrix and normalized, and an attention weight matrix is ​​obtained by an activation function, and the attention weight matrix is ​​multiplied by the value feature matrix to obtain an interaction feature matrix; A plurality of the interactive feature matrices are concatenated on the feature dimension and then mapped to obtain a fused feature matrix; the fused feature matrix is ​​added to the enhanced abundance feature matrix and normalized to obtain a multi-spectrum fused feature matrix.

[0009] In an optional embodiment, Constructing a feature discrimination criterion to enhance the discriminability of the structured feature representation, calculating the node significance distribution according to the structured feature representation after the discriminability enhancement, determining the area with a significance higher than a preset threshold in the node significance distribution as a quality defect area, and analyzing the formation mechanism of the quality defect based on the propagation path of the propagated network node feature, and generating a quality monitoring report according to the spatial distribution of the quality defect area and the formation mechanism of the quality defect, including: Constructing a feature discrimination criterion, wherein the feature discrimination criterion includes a pairwise feature distance term calculated based on a node feature vector, a local structure preservation term based on a graph Laplacian matrix, a global feature distribution constraint term based on a high-order moment, and a feature regularization term based on a sparse representation, and taking the feature discrimination criterion as an optimization target; The feature discrimination criterion is optimized by adopting an alternating optimization strategy of block coordinate descent, the node feature similarity is calculated by an adaptive kernel function to obtain a node feature weight matrix, a graph Laplacian matrix is ​​constructed based on the node feature weight matrix, and feature decomposition is performed to obtain a feature mapping matrix, and the node features are updated using the feature mapping matrix to obtain discriminative enhancement features; The discriminative enhancement features of different scales are combined by attention weights to obtain multi-scale fusion features, the Mahalanobis distance and cosine distance between the multi-scale fusion features and the standard features are calculated, and the significance score is obtained by weighted fusion; Performing double threshold segmentation on the significance score to obtain a candidate defect region, and optimizing the candidate defect region through morphological processing to obtain a target quality defect region; Constructing a double-layer feature propagation network, calculating edge weights based on the difference of the multi-scale fusion features at the local layer, calculating edge weights based on node importance at the global layer, and combining the edge weights of the double-layer network to extract the propagation path of the target quality defect area; A recurrent neural network is constructed based on the propagation path, the multi-scale fusion feature is encoded into a state sequence along the propagation path, a characteristic state transition probability is obtained by decoding the state sequence, a significant transition mode is extracted from the characteristic state transition probability, and the significant transition mode is matched and analyzed with a preset material knowledge base to obtain a defect formation mechanism; A quality monitoring report is generated based on the spatial distribution characteristics of the target quality defect area, the network topology characteristics of the propagation path, and the evolution characteristics of the defect formation mechanism.

[0010] In an optional embodiment, Extracting a significant transition mode from the characteristic state transition probability, and matching and analyzing the significant transition mode with a preset material knowledge base to obtain a defect formation mechanism includes: Performing eigenvalue decomposition on the characteristic state transition probability matrix to obtain an eigenvalue matrix and an eigenvector matrix, selecting an eigenvector corresponding to the maximum eigenvalue from the eigenvector matrix as a transfer mode, calculating a transfer feature score based on the ratio of the eigenvalue in the eigenvalue matrix to the first-order norm and the second-order norm of the transfer mode, and selecting the transfer mode with the highest transfer feature score to construct a significant transfer mode set; Constructing a hierarchical knowledge base including a material physical property layer and a defect formation mechanism layer, and calculating the defect similarity with the defect formation mechanism in the hierarchical knowledge base based on the state transition sequence characteristics and physical property characteristics in the significant transition pattern set; The significant transfer pattern set and defect formation mechanism are used as two vertex sets to construct a bipartite graph, and the defect similarity is used as the edge weight. The maximum weight matching algorithm is used to obtain the optimal matching pair. A target state sequence is extracted from the optimal matching pair, the target state sequence is mapped to the evolution stage of the corresponding defect formation mechanism, the physical process of the evolution stage is analyzed based on the hierarchical knowledge base, a quantitative model of the component field and temperature field including the key state sequence is constructed, the pattern matching probability and the physical rationality probability of the quantitative model are calculated to obtain the mechanism credibility, and the defect formation mechanism is determined based on the mechanism credibility.

[0011] According to a second aspect of the embodiments of the present invention, Provided is an online quality monitoring system based on optical multi-spectrum fusion, comprising: The first unit is used to collect multispectral image data of the object to be tested at the detection station, and a visible light camera group and an infrared camera group are set at the detection station. The trigger signals of the visible light camera group and the infrared camera group are uniformly dispatched by a synchronous control module, and the collected multispectral image data are pre-processed, and different spectral images are registered by using a feature point matching algorithm combined with camera calibration parameters to obtain the registered multispectral image data; The second unit is used to input the registered multi-spectral image data into a variational autoencoder for probability distribution modeling and introduce relative entropy constraints, obtain spectral basis vectors and an initial abundance map by minimizing reconstruction errors, construct a multi-scale morphological feature pyramid for the initial abundance map, calculate a non-local feature similarity matrix, construct a regional affinity map in combination with spatial position information, segment the regional affinity map using a graph cut algorithm to obtain local consistency regions, adaptively fuse the local consistency regions with the initial abundance map to obtain enhanced abundance features, and use a multi-head cross attention mechanism to align and complementarily enhance the spectral basis vectors and the enhanced abundance features to obtain multi-spectral fusion features; The third unit is used to construct a dynamic neural network, map the multi-spectral fusion features into network node features, construct the topological connection relationship between nodes based on the feature distance, enable the nodes to obtain key information from neighboring nodes through the dynamic neural network to obtain the post-propagation features, perform multi-level dimensionality reduction on the post-propagation features to obtain a structured representation, construct a feature discrimination criterion to enhance the discriminability of the structured feature representation, calculate the node significance distribution according to the structured feature representation after the discriminability enhancement, determine the area with significance higher than a preset threshold in the node significance distribution as the quality defect area, and at the same time analyze the formation mechanism of the quality defect based on the propagation path of the network node features after propagation, and generate a quality monitoring report according to the spatial distribution of the quality defect area and the formation mechanism of the quality defect.

[0012] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0013] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0014] In this embodiment, by combining visible light and infrared information with multi-spectral image fusion technology, the surface feature information of the object to be tested can be obtained more comprehensively. By using technologies such as variational autoencoders, multi-scale morphological features, and graph cut algorithms, noise and background interference are effectively reduced, and the accuracy of feature extraction is improved. At the same time, a multi-head cross-attention mechanism is used to align and complement the spectral basis vectors and enhanced abundance features, further improving the quality of the fusion features, thereby improving the accuracy and reliability of quality defect detection. By constructing a dynamic neural network, mapping the multi-spectral fusion features to network node features, and propagating information based on the topological connection relationship between nodes, the defect area can be more accurately located. At the same time, by analyzing the propagation path, the formation mechanism of quality defects can be deeply understood, providing guidance for subsequent process improvements, which goes beyond simple defect detection and achieves a deeper level of quality analysis. The online multi-spectral image acquisition and processing method can monitor the product quality in the production process in real time. The defect area is judged by the node significance distribution, and the quality monitoring report is automatically generated in combination with the defect formation mechanism, which realizes the automation and intelligence of quality monitoring, improves production efficiency, and reduces labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of an online quality monitoring method based on optical multi-spectrum fusion according to an embodiment of the present invention; Figure 2 It is a structural schematic diagram of an online quality monitoring system based on optical multi-spectrum fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0018] Figure 1 FIG. 1 is a flow chart of an online quality monitoring method based on optical multi-spectrum fusion according to an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Collect multispectral image data of the object to be tested at the detection station, set a visible light camera group and an infrared camera group at the detection station, the visible light camera group includes a plurality of visible light cameras evenly distributed along the circumference of the object to be tested, and the infrared camera group includes a plurality of infrared cameras corresponding to the position of the visible light camera group, and the trigger signals of the visible light camera group and the infrared camera group are uniformly dispatched by a synchronous control module; pre-process the collected multispectral image data, and use a feature point matching algorithm combined with camera calibration parameters to align different spectral images to obtain aligned multispectral image data; S102. Input the registered multi-spectral image data into a variational autoencoder for probability distribution modeling and introduce relative entropy constraints, obtain spectral basis vectors and an initial abundance map by minimizing the reconstruction error, construct a multi-scale morphological feature pyramid for the initial abundance map, calculate a non-local feature similarity matrix, construct a regional affinity map in combination with spatial position information, use a graph cut algorithm to segment the regional affinity map to obtain local consistency regions, adaptively fuse the local consistency regions with the initial abundance map to obtain enhanced abundance features, use a multi-head cross attention mechanism to align and complementarily enhance the spectral basis vectors and the enhanced abundance features to obtain multi-spectral fusion features; S103. Construct a dynamic neural network, map the multi-spectral fusion features into network node features, construct the topological connection relationship between nodes based on the feature distance, enable the node to obtain key information from the neighboring nodes through the dynamic neural network to obtain the post-propagation features, perform multi-level dimensionality reduction on the post-propagation features to obtain a structured representation, construct a feature discrimination criterion to enhance the discriminability of the structured feature representation, calculate the node significance distribution according to the structured feature representation after the discriminability enhancement, determine the area with significance higher than the preset threshold in the node significance distribution as the quality defect area, and analyze the formation mechanism of the quality defect based on the propagation path of the network node features after propagation, and generate a quality monitoring report according to the spatial distribution of the quality defect area and the formation mechanism of the quality defect.

[0019] Among them, a visible light camera group and an infrared camera group are set up at the inspection station, each group contains eight cameras. The visible light camera is evenly installed every 45 degrees along the circumference of the object to be tested, with a field of view of 60 degrees. The infrared camera uses a medium-wave band with a wavelength range of three to five microns, and the installation position corresponds to the visible light camera one by one. The synchronization control module is implemented using a field programmable gate array to generate a trigger signal with a frequency of one hundred hertz to ensure the synchronization of the trigger timing between different cameras.

[0020] In an optional implementation, the collected multispectral image data is preprocessed, and the different spectral images are registered using a feature point matching algorithm combined with camera calibration parameters, and the registered multispectral image data includes: Calculating a local cumulative distribution function through a sliding window, and using the local cumulative distribution function to perform local adaptive histogram equalization processing on the visible light image and the infrared light image to obtain a contrast enhanced image; Performing bilateral filtering on the contrast enhanced image, combining noise suppression processing with wavelet transform, to obtain a denoised image, and performing geometric distortion correction on the denoised image based on a camera imaging model to obtain a preprocessed image; Constructing a multi-layer image pyramid for the preprocessed image, extracting image feature points based on the scale space in each layer of the multi-layer image pyramid, and screening the image feature points by using corner point response values ​​to obtain screened feature points; Extracting the gradient direction histogram and the local binary pattern of the selected feature points through neighborhood block division to generate a feature description vector, matching the feature description vector based on the nearest neighbor ratio method and random consistency test to obtain a set of feature matching point pairs; Calibrate the cameras corresponding to the visible light image and the infrared light image based on the calibration plate image sequence, obtain the camera parameter matrix and distortion parameters, calculate the homography matrix according to the feature matching point pair set, and optimize the transformation parameters of the homography matrix by using an iterative optimization algorithm to obtain an optimized homography matrix; The pre-processed image is resampled by bilinear interpolation using the optimized homography matrix and the camera parameter matrix to obtain a registered visible light image and infrared light image.

[0021] Obtain a set of visible light images and their corresponding infrared images, both of which capture the same scene. The goal is to accurately align the infrared image with the visible light image.

[0022] First, the collected multispectral image data is preprocessed. In order to enhance the image contrast, a local adaptive histogram equalization method is used. Specifically, the image is divided into several small windows, such as 16x16 pixels in size. In each window, the cumulative distribution function of the pixel value is calculated. Then, the pixel values ​​in the window are histogram equalized using the cumulative distribution function. This local equalization processing method can better adapt to the brightness changes in different areas of the image, thereby improving the overall contrast of the image. For example, assuming that in a window, the pixel values ​​are mainly concentrated in the darker areas, the local histogram equalization will stretch these dark details to make them more clearly visible.

[0023] After contrast enhancement, in order to remove image noise, the image is subjected to noise suppression processing combining bilateral filtering and wavelet transform. Bilateral filtering is a nonlinear filtering method that can retain edge information while smoothing the image. Wavelet transform can decompose the image into sub-bands of different scales and directions, thereby better separating noise and image signals. For example, the db4 wavelet basis can be selected for three-layer decomposition, and then the high-frequency sub-bands can be thresholded to remove noise, and then the inverse wavelet transform can be performed to reconstruct the image. This can effectively remove high-frequency noise in the image while retaining the image's detail information.

[0024] Since the camera lens has distortion, the image will be geometrically deformed. Therefore, it is necessary to correct the geometric distortion of the image. This requires calibrating the camera in advance to obtain the camera's intrinsic parameter matrix and distortion coefficient. For example, a checkerboard calibration plate is used to calibrate the camera to obtain the camera's focal length, principal point coordinates, radial distortion coefficient, tangential distortion coefficient and other parameters. Then, these parameters are used to correct the image distortion and restore the image to the ideal imaging plane.

[0025] Next, feature points are extracted and matched on the preprocessed image. In order to improve the robustness of the matching, a multi-layer image pyramid is constructed. On each layer of the image, the scale-invariant feature transform (SIFT) algorithm is used to extract feature points. The SIFT algorithm can extract feature points with scale invariance and rotation invariance, and is also robust to changes in illumination and perspective. For example, on each layer of the pyramid image, the Hessian matrix of the image is calculated, and local extreme points are found as candidate feature points. Then, the candidate feature points are screened by the corner point response value to remove some unstable feature points.

[0026] For the filtered feature points, extract the feature description vector. Here, a method combining the gradient directional histogram (HOG) and the local binary pattern (LBP) is used to generate the feature description vector. The HOG feature describes the gradient direction distribution of the area around the feature point, and the LBP feature describes the texture information of the area around the feature point. Combining these two features can improve the distinguishability of the feature description. For example, the area around the feature point is divided into several small blocks, the gradient direction histogram is calculated in each small block, and these histograms are connected to form the HOG feature vector. At the same time, the LBP feature is calculated in the area around the feature point, and the LBP feature vector is connected with the HOG feature vector to form the final feature description vector.

[0027] After obtaining the feature description vector, the nearest neighbor ratio method and random consistency check (RANSAC) are used for feature matching. The nearest neighbor ratio method can remove some mismatched points. The RANSAC algorithm can further remove mismatched points and estimate the homography matrix. For example, the nearest neighbor ratio threshold is set to 0.8, and matching point pairs with a distance ratio greater than 0.8 are considered mismatched points and removed. Then, the RANSAC algorithm is used to estimate the homography matrix, and matching point pairs that do not conform to the homography matrix transformation are considered mismatched points and removed.

[0028] In order to obtain more accurate registration results, an iterative optimization algorithm is used to optimize the homography matrix. For example, the Levenberg-Marquardt algorithm is used to iteratively optimize the parameters of the homography matrix to minimize the reprojection error.

[0029] Finally, the optimized homography matrix is ​​used to perform bilinear interpolation resampling on the preprocessed image to obtain the registered visible light image and infrared image. For example, the pixel coordinates of the infrared image are transformed to the coordinate system of the visible light image through the homography matrix, and then the bilinear interpolation method is used to calculate the pixel values ​​corresponding to the transformed coordinates.

[0030] In this embodiment, the contrast and signal-to-noise ratio of the image are effectively improved by local adaptive histogram equalization and bilateral filtering combined with wavelet transform noise reduction processing, which lays a good foundation for subsequent feature extraction and matching. The SIFT algorithm is used to extract feature points, and the feature description vector is generated by combining HOG and LBP features. It has strong robustness and discrimination, and can effectively cope with complex situations such as illumination changes, perspective changes, and scale changes. The strategy of combining the nearest neighbor ratio method and the RANSAC algorithm is used for feature matching, and the homography matrix is ​​optimized using an iterative optimization algorithm, which can effectively remove mismatched points and improve the registration accuracy.

[0031] In an optional implementation, the gradient direction histogram and the local binary pattern are extracted from the screened feature points by neighborhood block, a feature description vector is generated, and the feature description vector is matched based on the nearest neighbor ratio method and random consistency test to obtain a feature matching point pair set including: Dividing a feature description area with the screened feature point as the center, and dividing the feature description area into a plurality of feature sub-blocks; Calculating the gradient magnitude and the gradient direction in the feature sub-block, constructing a gradient direction histogram to obtain a gradient feature vector, extracting a local binary pattern in the feature sub-block and using a rotationally invariant uniform pattern encoding to obtain a texture feature vector, normalizing and concatenating the gradient feature vector and the texture feature vector to form a feature description vector; A tree index structure is constructed for the feature description vector, and a nearest neighbor search method is used to obtain a nearest neighbor point set for each feature point; the nearest neighbor point set is screened according to a nearest neighbor ratio threshold to obtain an initial matching point pair set; a plurality of groups of matching points are selected from the initial matching point pair set by a random sampling method to calculate a perspective transformation model, and the number of matching points that satisfy the perspective transformation model is counted, and the perspective transformation model with the largest number of matching points is determined as the optimal transformation model, and candidate feature matching point pairs that satisfy the optimal transformation model are obtained; Dividing the image containing the candidate matching point pairs into a plurality of grid areas, counting the number of feature matching points in each grid area to obtain a spatial distribution variance, and calculating the distance ratio and angle difference between the feature matching point pairs to obtain a geometric consistency parameter; A weighted voting function is constructed according to the spatial distribution variance and the geometric consistency parameter, and the feature matching point pairs are screened by voting, and the feature matching point pairs that do not meet the spatial distribution constraints and the geometric consistency constraints are eliminated to obtain a final feature matching point pair set.

[0032] Exemplarily, two images to be matched are first obtained. The two images are preprocessed, including grayscale and Gaussian filtering, to reduce noise interference and enhance the texture information of the images. For example, a Gaussian kernel with a standard deviation of 1.5 is used for filtering.

[0033] Then, extract feature points from the preprocessed image. Use the SIFT (Scale Invariant Feature Transform) algorithm to detect key points in the image and calculate the scale and direction information of each key point. Filter feature points and remove those with low contrast and unclear edge response. For example, set the contrast threshold to 0.04 and the edge response threshold to 10.

[0034] With the filtered feature points as the center, a feature description area of ​​16x16 pixels is divided. The area is divided into 4x4 sub-blocks, each of which is 4x4 pixels. In each sub-block, the gradient amplitude and gradient direction are calculated. The gradient direction is divided into 8 intervals and a gradient direction histogram is constructed. The gradient direction histograms of each sub-block are connected to form a 128-dimensional gradient feature vector. At the same time, the local binary pattern (LBP) is extracted in each sub-block, and a rotation-invariant uniform pattern encoding is used to obtain a 59-dimensional texture feature vector. The gradient feature vector and texture feature vector are normalized and cascaded to form a 187-dimensional feature description vector. For example, if the gradient feature vector of a feature point in image A is [0.1, 0.2, ..., 0.05] and the texture feature vector is [0.01, 0.02, ..., 0.03], then the feature description vector of the feature point is [0.1, 0.2, ..., 0.05, 0.01, 0.02, ..., 0.03].

[0035] Construct a KD tree index structure for the feature description vector in the image B to be matched. For each feature point in image A, use the nearest neighbor search method to search for its nearest neighbor point set in the KD tree. For example, two nearest neighbor points are searched. Calculate the distance ratio between the nearest neighbor points. If the distance ratio is less than a preset threshold (for example, 0.8), the two points are regarded as the initial matching point pair. Assuming that the distance of the first nearest neighbor point is 2 and the distance of the second nearest neighbor point is 3, the distance ratio is 2 / 3=0.67, which is less than 0.8, so the two points are added to the initial matching point pair set.

[0036] The random sampling consensus (RANSAC) algorithm is used to screen the initial set of matching point pairs. Four groups of matching point pairs are randomly selected to calculate the perspective transformation model. The number of matching point pairs that satisfy the model, i.e., the number of inliers, is counted. Repeat the random sampling multiple times and select the perspective transformation model with the largest number of inliers as the optimal transformation model. Obtain the candidate feature matching point pairs that satisfy the optimal transformation model. For example, if the random sampling is performed 100 times and the number of inliers in the model obtained by one sampling is 80, which is the maximum value, the matching point pairs corresponding to the model are selected as the candidate matching point pairs.

[0037] Divide the image containing candidate matching point pairs into 4x4 grid regions. Count the number of feature matching points in each grid region and calculate the spatial distribution variance. Calculate the distance ratio and angle difference between feature matching point pairs as geometric consistency parameters. Suppose there are two matching point pairs, the distance ratio between them is 0.9, and the angle difference is 5 degrees.

[0038] A weighted voting function is constructed based on the spatial distribution variance and geometric consistency parameters. Candidate feature matching point pairs are screened by voting. Feature matching point pairs that do not meet the spatial distribution constraints and geometric consistency constraints are eliminated. For example, the spatial distribution variance threshold is set to 0.5, and the geometric consistency parameter threshold is set to (0.8, 10 degrees). Finally, a set of feature matching point pairs is obtained.

[0039] In this embodiment, a more discriminative feature description vector is constructed by combining gradient and texture features, which can more accurately describe the local information of feature points, thereby improving the accuracy of feature matching. The RANSAC algorithm and spatial distribution constraints and geometric consistency constraints are used to eliminate mismatched point pairs, effectively reducing the impact of factors such as noise and perspective changes on the matching results and enhancing the robustness of the algorithm. By constructing a KD tree index structure, the nearest neighbor search process is accelerated and the efficiency of feature matching is improved.

[0040] In an optional implementation, the registered multispectral image data is input into a variational autoencoder for probability distribution modeling and relative entropy constraints are introduced, spectral basis vectors and an initial abundance map are obtained by minimizing the reconstruction error, a multi-scale morphological feature pyramid is constructed for the initial abundance map, a non-local feature similarity matrix is ​​calculated, and a regional affinity map is constructed in combination with spatial position information, including: Reconstructing the registered multispectral image data into a two-dimensional matrix, wherein the number of rows of the two-dimensional matrix is ​​the number of image pixels, and the number of columns is the number of spectral bands, and normalizing the two-dimensional matrix to obtain a normalized matrix; Input the normalized matrix into an encoder for feature extraction to obtain a mean vector and a variance vector, sample the mean vector and the variance vector to obtain a latent variable, input the latent variable into a decoder for reconstruction, and restore the data dimension through deconvolution operation and upsampling to obtain a reconstructed matrix; Constructing a loss function including a reconstruction error term and a relative entropy constraint term, obtaining a spectral basis vector and an initial abundance map by minimizing the loss function, and performing non-negative constraint and normalization processing on the spectral basis vector to obtain a standardized spectral basis vector; Based on the initial abundance map, an opening operation is performed to sequentially extract dark area features to obtain a multi-scale dark area feature map set, and a closing operation is performed to sequentially extract bright area features to obtain a multi-scale bright area feature map set; the response values ​​of each pixel position in the multi-scale dark area feature map set and the multi-scale bright area feature map set are combined to obtain a multi-scale feature vector of the current pixel position; The covariance matrix of the multi-scale feature vector is calculated, and the Mahalanobis distance between pixel pairs is calculated based on the covariance matrix. The Mahalanobis distance is mapped through a Gaussian kernel function to obtain a feature similarity matrix. The gradient variance of the local area is calculated to obtain texture complexity, and the texture complexity is used as a weight factor to perform a weighted combination of the feature similarity matrix and the spatial distance of the pixel points to obtain the edge weight of the regional affinity graph. A threshold is set for the edge weight to retain the maximum connection value to obtain the final regional affinity graph.

[0041] Exemplarily, the registered multispectral image data, such as a hyperspectral image or a multispectral remote sensing image, first needs to be preprocessed. Suppose a multispectral image containing 1000x1000 pixels and 200 spectral bands is obtained. The image is reconstructed into a two-dimensional matrix of 1000000 rows and 200 columns, where each row represents a pixel and each column represents the value of a spectral band. Then, the two-dimensional matrix is ​​normalized. The normalization method can use maximum and minimum value normalization, that is, scaling each band value of each pixel to between 0 and 1.

[0042] The normalized matrix is ​​input to the encoder part of the variational autoencoder (VAE). The encoder is usually composed of multiple convolutional layers and fully connected layers to extract the spectral features of the image. Assume that the encoder reduces the input data to a 10-dimensional latent space. The output of the encoder is a 10-dimensional mean vector and a 10-dimensional variance vector. Based on these two vectors, a 10-dimensional latent variable is sampled from a standard normal distribution.

[0043] The sampled 10-dimensional latent variables are input to the decoder part of the VAE. The decoder is usually composed of multiple deconvolution layers and upsampling layers to reconstruct the latent variables back to the original data dimensions. The output of the decoder is a reconstructed matrix with the same dimensions as the input matrix, that is, a matrix of 1,000,000 rows and 200 columns.

[0044] In order to train VAE, a loss function needs to be constructed. The loss function consists of two parts: a reconstruction error term and a relative entropy constraint term. The reconstruction error term measures the difference between the reconstructed matrix and the original input matrix, which can be calculated using the mean square error. The relative entropy constraint term is used to constrain the distribution of the latent variable to make it close to the standard normal distribution. By minimizing the loss function, the spectral basis vector and the initial abundance map can be obtained. The spectral basis vector corresponds to the column vector of the decoder weight matrix and represents the typical spectral characteristics of different objects. The initial abundance map corresponds to the reconstruction matrix, which represents the mixing ratio of different objects at each pixel. The obtained spectral basis vector is subjected to non-negative constraints and normalization, such as L1 normalization, to make it a standardized spectral basis vector.

[0045] Next, a multi-scale morphological feature pyramid is constructed for the initial abundance map. Take one of the abundance maps as an example and assume that the abundance map represents the distribution of a certain specific feature. The abundance map is opened using structural elements of different sizes (e.g., square structural elements of 3x3, 5x5, and 7x7), and the dark area features are extracted in sequence to obtain a set of multi-scale dark area feature maps. At the same time, the abundance map is closed using structural elements of the same size to extract the bright area features in sequence to obtain a set of multi-scale bright area feature maps. The response values ​​of each pixel position in the multi-scale dark area feature map set and the multi-scale bright area feature map set are combined to form a multi-scale feature vector for the pixel position. For example, if three sizes of structural elements are used, the multi-scale feature vector of each pixel contains 6 values.

[0046] Calculate the covariance matrix of the multi-scale feature vectors of all pixels. Then, calculate the Mahalanobis distance between any two pixels based on the covariance matrix. Map the calculated Mahalanobis distance to between 0 and 1 through the Gaussian kernel function to obtain the feature similarity matrix.

[0047] Calculate the gradient variance of the local area (e.g., a 5x5 window centered on each pixel) as the texture complexity. Use the texture complexity as a weight factor to perform a weighted combination of the feature similarity matrix and the spatial distance of the pixel point (e.g., Euclidean distance) to obtain the edge weight of the regional affinity graph. Set a threshold for the edge weight and only retain the connection between each pixel and the pixels with the largest connection weights, and finally obtain the regional affinity graph.

[0048] In this embodiment, by learning the spectral basis vector and the initial abundance map through the variational autoencoder, the spectral characteristics and spatial distribution information of the objects can be better expressed, thereby improving the accuracy of object classification. The construction of the multi-scale morphological feature pyramid can capture the features of objects at different scales and enhance the robustness of the algorithm to noise and scale changes. By constructing a regional affinity map, the computational complexity of the subsequent graph segmentation algorithm can be effectively reduced, and the overall computational efficiency can be improved.

[0049] In an optional implementation, a graph cut algorithm is used to segment the regional affinity map to obtain a local consistency region, the local consistency region is adaptively fused with the initial abundance map to obtain an enhanced abundance feature, and a multi-head cross attention mechanism is used to align and complementarily enhance the spectral basis vector and the enhanced abundance feature to obtain a multi-spectral fusion feature including: The regional affinity graph is constructed as a weighted undirected graph, in which the vertex set is the spatial position of the image pixel points, the edge set is the connection relationship between adjacent pixel points, and the edge weight value is determined by the association strength of the corresponding pixel pairs in the regional affinity graph to obtain a weight matrix; Constructing a degree matrix based on the weight matrix, wherein the diagonal elements of the degree matrix are the sum of the elements of the corresponding rows in the weight matrix, constructing a normalized Laplace matrix based on the weight matrix and the degree matrix, solving the eigenvalue equation of the normalized Laplace matrix, and obtaining an eigenvector; Divide the vertex set into mutually non-intersecting subgraphs according to the sign value of the eigenvector, calculate the cut cost value and subgraph volume value between each subgraph, and when the cut cost value is less than the cut cost threshold and the subgraph volume value is greater than the volume threshold, recursively divide the subgraph until a set of local consistency regions is obtained; Calculating the arithmetic mean of the initial abundance vectors of the pixels in each area of ​​the local consistency area set to obtain the regional abundance vector, calculating the Euclidean distance between the initial abundance vector of the pixels in the local consistency area and the regional abundance vector, and converting the Euclidean distance into a fusion weight coefficient; The initial abundance vector of the pixel point and the regional abundance vector are linearly weighted combined by using the fusion weight coefficient to obtain an enhanced abundance feature matrix, and the spectral basis vector is subjected to query transformation to obtain a query feature matrix, and the enhanced abundance feature matrix is ​​subjected to key transformation and value transformation to obtain a key feature matrix and a value feature matrix, respectively, and the query feature matrix is ​​multiplied by the key feature matrix and normalized, and an attention weight matrix is ​​obtained by an activation function, and the attention weight matrix is ​​multiplied by the value feature matrix to obtain an interaction feature matrix; A plurality of the interactive feature matrices are concatenated on the feature dimension and then mapped to obtain a fused feature matrix; the fused feature matrix is ​​added to the enhanced abundance feature matrix and normalized to obtain a multi-spectrum fused feature matrix.

[0050] Exemplarily, firstly, remote sensing image data containing multiple spectral band information and the corresponding initial abundance map are obtained. For example, a remote sensing image containing three spectral bands of visible light, near infrared, and short-wave infrared is obtained, and an initial abundance map is obtained by methods such as non-negative matrix decomposition, where the dimension of the abundance map is the number of pixels multiplied by the number of end members, for example, for an image of 256*256 pixels, the number of end members is 5, and the size of the abundance map is 256*256*5.

[0051] Calculate the regional affinity map. The regional affinity map reflects the similarity between adjacent pixels in the image. For example, the similarity between pixels can be calculated based on the spectral angle distance, and the larger the value, the higher the similarity. Assuming that the spectral vectors of two pixels are [0.2, 0.3, 0.5] and [0.25, 0.35, 0.4] respectively, their spectral angle distance can be obtained by calculating the cosine value of the vector angle.

[0052] Convert the region affinity graph to a weighted undirected graph. Each pixel in the image is considered a vertex in the graph, and adjacent pixels are connected by edges. The weight of the edge is determined by the similarity of the corresponding pixel pairs in the region affinity graph. For example, if the similarity of two adjacent pixels is 0.9, the weight of the edge between them is 0.9. The final result is a weight matrix whose size is the number of pixels multiplied by the number of pixels.

[0053] Construct the degree matrix and the normalized Laplacian matrix. The degree matrix is ​​a diagonal matrix. Each element on the diagonal represents the degree of the corresponding vertex, that is, the sum of the weights of the edges connected to the vertex. For example, if a vertex is connected to three edges with weights of 0.8, 0.7, and 0.9, the degree of the vertex is 0.8+0.7+0.9=2.4. The normalized Laplacian matrix can be calculated from the degree matrix and the weight matrix.

[0054] Compute the eigenvalues ​​and eigenvectors of the normalized Laplacian matrix. By solving the eigenvalue equation, a series of eigenvalues ​​and corresponding eigenvectors are obtained.

[0055] Graph partitioning based on eigenvectors. Select the first k eigenvectors and partition the vertex set into disjoint subgraphs based on the sign value of each eigenvector. For example, if the value of an element in an eigenvector is positive, the corresponding vertex is assigned to one subgraph; if the value is negative, it is assigned to another subgraph.

[0056] Recursively partition to obtain local consistency regions. Calculate the cut cost between each subgraph, which represents the connection strength between subgraphs. At the same time, calculate the volume of the subgraph, which represents the size of the subgraph. If the cut cost is less than the preset threshold and the volume is greater than the preset threshold, recursively partition the subgraph until the stop condition is met. Finally, a set of local consistency regions is obtained.

[0057] Calculate the regional abundance vector. For each local consistency region, calculate the arithmetic mean of the initial abundance vectors of all pixels in the region as the abundance vector of the region. For example, if a region contains 10 pixels, add the initial abundance vectors of these 10 pixels and divide by 10 to get the abundance vector of the region.

[0058] Calculate the fusion weight coefficient. Calculate the Euclidean distance between the initial abundance vector of each pixel and the abundance vector of the region to which it belongs. Convert the Euclidean distance to a fusion weight coefficient. The smaller the distance, the greater the weight. For example, a Gaussian kernel function can be used to convert the distance to a weight.

[0059] The enhanced abundance feature is obtained by fusion. The initial abundance vector of each pixel is linearly weighted combined with the abundance vector of the region to which it belongs using the fusion weight coefficient to obtain the enhanced abundance feature.

[0060] Perform a multi-head cross attention mechanism. Transform the spectral basis vector through the query to obtain the query feature matrix, and transform the enhanced abundance features through the key transformation and value transformation to obtain the key feature matrix and value feature matrix respectively. Multiply the query feature matrix with the key feature matrix and normalize them, and then pass the activation function to obtain the attention weight matrix. Multiply the attention weight matrix with the value feature matrix to obtain the interaction feature matrix. Repeat multiple times to obtain multiple interaction feature matrices.

[0061] The multi-spectral fusion feature is obtained by fusion. Multiple interactive feature matrices are concatenated and mapped on the feature dimension to obtain a fusion feature matrix. The fusion feature matrix is ​​added to the enhanced abundance feature matrix and normalized to obtain the final multi-spectral fusion feature matrix.

[0062] In this embodiment, the image is segmented into local consistency regions by a graph cut algorithm, and the abundance feature is enhanced by using regional information, which effectively reduces the impact of noise and outliers and improves the accuracy and reliability of the abundance feature. Through the multi-head cross attention mechanism, the spectral basis vector and the enhanced abundance feature are aligned and complementarily enhanced, making full use of the spectral information and abundance information and improving the expressiveness of the feature. The enhanced multi-spectral fusion feature can be used in various remote sensing image applications, such as target detection, classification and recognition, effectively improving the performance of these applications.

[0063] In an optional implementation, a feature discrimination criterion is constructed to enhance the discriminability of the structured feature representation, node significance distribution is calculated according to the structured feature representation after discriminability enhancement, and an area in the node significance distribution with a significance higher than a preset threshold is determined as a quality defect area, and at the same time, a formation mechanism of quality defects is analyzed based on the propagation path of the propagated network node features, and a quality monitoring report is generated according to the spatial distribution of the quality defect area and the formation mechanism of the quality defect, including: Constructing a feature discrimination criterion, wherein the feature discrimination criterion includes a pairwise feature distance term calculated based on a node feature vector, a local structure preservation term based on a graph Laplacian matrix, a global feature distribution constraint term based on a high-order moment, and a feature regularization term based on a sparse representation, and taking the feature discrimination criterion as an optimization target; The feature discrimination criterion is optimized by adopting an alternating optimization strategy of block coordinate descent, the node feature similarity is calculated by an adaptive kernel function to obtain a node feature weight matrix, a graph Laplacian matrix is ​​constructed based on the node feature weight matrix, and feature decomposition is performed to obtain a feature mapping matrix, and the node features are updated using the feature mapping matrix to obtain discriminative enhancement features; The discriminative enhancement features of different scales are combined by attention weights to obtain multi-scale fusion features, the Mahalanobis distance and cosine distance between the multi-scale fusion features and the standard features are calculated, and the significance score is obtained by weighted fusion; Performing double threshold segmentation on the significance score to obtain a candidate defect region, and optimizing the candidate defect region through morphological processing to obtain a target quality defect region; Constructing a double-layer feature propagation network, calculating edge weights based on the difference of the multi-scale fusion features at the local layer, calculating edge weights based on node importance at the global layer, and combining the edge weights of the double-layer network to extract the propagation path of the target quality defect area; A recurrent neural network is constructed based on the propagation path, the multi-scale fusion feature is encoded into a state sequence along the propagation path, a characteristic state transition probability is obtained by decoding the state sequence, a significant transition mode is extracted from the characteristic state transition probability, and the significant transition mode is matched and analyzed with a preset material knowledge base to obtain a defect formation mechanism; A quality monitoring report is generated based on the spatial distribution characteristics of the target quality defect area, the network topology characteristics of the propagation path, and the evolution characteristics of the defect formation mechanism.

[0064] Exemplarily, first, a feature discrimination criterion is constructed. This criterion takes into account multiple factors, including the distance between node feature vectors, the preservation of local structure, the constraints of global feature distribution, and the sparsity of features. For example, if the feature vectors of two nodes are very similar, the distance between them should be small. At the same time, in order to maintain the local structure, the features of adjacent nodes should be as similar as possible. In addition, the global feature distribution should meet certain constraints, for example, the variance of the feature cannot be too large. Finally, in order to avoid overfitting, the features should have a certain sparsity.

[0065] Next, the feature discrimination criterion is optimized by using the alternating optimization strategy of block coordinate descent. Specifically, first, other variables are fixed and the node feature weight matrix is ​​optimized. The node feature similarity is calculated by the adaptive kernel function to obtain the node feature weight matrix. Then, the graph Laplacian matrix is ​​constructed based on the node feature weight matrix, and the feature decomposition is performed to obtain the feature mapping matrix. Finally, the node features are updated using the feature mapping matrix to obtain discriminative enhanced features. For example, suppose there are two nodes, and their feature vectors are [0.1, 0.2, 0.3] and [0.2, 0.3, 0.4] respectively. The node feature weights can be obtained by calculating the cosine similarity between them.

[0066] Then, the discriminative enhancement features of different scales are combined through attention weights to obtain multi-scale fusion features. For example, fine-grained features and coarse-grained features can be fused. The Mahalanobis distance and cosine distance between the multi-scale fusion features and the standard features are calculated, and the significance score is obtained by weighted fusion. For example, if the multi-scale fusion features of a region are very different from the standard features, the significance score of the region is higher. Assuming that the multi-scale fusion features of a region are [0.5, 0.6, 0.7] and the standard features are [0.1, 0.2, 0.3], their Mahalanobis distance is larger, the cosine distance is smaller, and the final significance score is higher.

[0067] Perform double threshold segmentation on the significance score to obtain candidate defect regions. For example, a higher threshold and a lower threshold can be set. Regions with significance scores higher than the higher threshold are directly determined as defect regions, and regions with significance scores lower than the lower threshold are directly excluded. For regions with significance scores between the two thresholds, further analysis is required. The candidate defect regions are optimized through morphological processing to obtain the target quality defect regions. For example, dilation and erosion operations can be performed on the candidate defect regions to remove noise and small isolated regions.

[0068] Construct a two-layer feature propagation network. At the local layer, edge weights are calculated based on the difference of multi-scale fusion features. At the global layer, edge weights are calculated based on node importance. The edge weights of the two-layer network are combined to extract the propagation path of the target quality defect area. For example, if the multi-scale fusion features of two nodes are very different, the edge weight between them is large.

[0069] A recurrent neural network is constructed based on the propagation path, and the multi-scale fusion features are encoded into a state sequence along the propagation path. By decoding the state sequence, the characteristic state transition probability is obtained. The significant transition pattern is extracted from the characteristic state transition probability. The significant transition pattern is matched and analyzed with the preset material knowledge base to obtain the defect formation mechanism. For example, if a state always transfers to another specific state, it can be considered as a significant transition pattern.

[0070] Finally, a quality monitoring report is generated based on the spatial distribution characteristics of the target quality defect area, the network topology characteristics of the propagation path, and the evolution characteristics of the defect formation mechanism. For example, the report can include the location, size, shape of the defect area, as well as the cause and evolution process of the defect formation.

[0071] In this embodiment, by constructing feature discrimination criteria and multi-scale feature fusion, the discriminability of features can be effectively enhanced, thereby improving the accuracy of defect detection. Through feature propagation networks and recurrent neural networks, the propagation path and evolution law of defects can be analyzed, thereby revealing the formation mechanism of defects. Quality monitoring reports can be automatically generated, allowing users to quickly understand product quality status.

[0072] In an optional implementation, extracting a significant transition pattern from the characteristic state transition probability, and matching and analyzing the significant transition pattern with a preset material knowledge base to obtain a defect formation mechanism includes: Performing eigenvalue decomposition on the characteristic state transition probability matrix to obtain an eigenvalue matrix and an eigenvector matrix, selecting an eigenvector corresponding to the maximum eigenvalue from the eigenvector matrix as a transfer mode, calculating a transfer feature score based on the ratio of the eigenvalue in the eigenvalue matrix to the first-order norm and the second-order norm of the transfer mode, and selecting the transfer mode with the highest transfer feature score to construct a significant transfer mode set; Constructing a hierarchical knowledge base including a material physical property layer and a defect formation mechanism layer, and calculating the defect similarity with the defect formation mechanism in the hierarchical knowledge base based on the state transition sequence characteristics and physical property characteristics in the significant transition pattern set; The significant transfer pattern set and defect formation mechanism are used as two vertex sets to construct a bipartite graph, and the defect similarity is used as the edge weight. The maximum weight matching algorithm is used to obtain the optimal matching pair. A target state sequence is extracted from the optimal matching pair, the target state sequence is mapped to the evolution stage of the corresponding defect formation mechanism, the physical process of the evolution stage is analyzed based on the hierarchical knowledge base, a quantitative model of the component field and temperature field including the key state sequence is constructed, the pattern matching probability and the physical rationality probability of the quantitative model are calculated to obtain the mechanism credibility, and the defect formation mechanism is determined based on the mechanism credibility.

[0073] Exemplarily, firstly, the material state transition data to be analyzed is obtained, which contains a series of discrete time steps, each of which records the characteristic state of the material. For example, in the crystal growth process, the characteristic state can be the size, shape, number of defects, etc. of the crystal. Suppose we have obtained 100 time steps of data for a crystal growth process, and each time step records the average size and defect density of the crystal.

[0074] The collected state transition data is preprocessed, including data cleaning, normalization and other operations. For example, outliers are removed and the data of different characteristic states are scaled to the same numerical range. In this example, assuming that the crystal size range is 1-10 microns and the defect density range is 0-100 / square micron, we scale the data of these two characteristic states to between 0 and 1.

[0075] Construct a characteristic state transition probability matrix. This matrix describes the probability of a material transitioning from one characteristic state to another. For example, the element in the i-th row and j-th column of the matrix represents the probability of a material transitioning from state i to state j. In this example, we construct a 10x10 transition probability matrix based on the preprocessed data by counting the frequency of crystal size and defect density transitioning from one interval to another at each time step (assuming that the crystal size and defect density are divided into 10 intervals respectively).

[0076] Perform eigenvalue decomposition on the characteristic state transition probability matrix. The purpose of this step is to extract the main transition modes. Through eigenvalue decomposition, the transition probability matrix can be decomposed into an eigenvalue matrix and an eigenvector matrix. In this example, by performing eigenvalue decomposition on the 10x10 transition probability matrix, 10 eigenvalues ​​and corresponding 10 eigenvectors are obtained.

[0077] The eigenvector corresponding to the largest eigenvalue is selected from the eigenvector matrix as the main transfer mode. The size of the eigenvalue reflects the importance of the corresponding transfer mode. In this example, it is assumed that the eigenvector corresponding to the largest eigenvalue mainly reflects the trend of synchronous increase of crystal size and defect density.

[0078] Calculate the transition feature score. The transition feature score is calculated based on the ratio of the eigenvalue to the first-order norm and second-order norm of the corresponding transition pattern. The higher the score, the more significant the transition pattern. In this example, we calculated the scores of 10 transition patterns and selected the ones with the highest scores.

[0079] Construct a set of significant transition patterns. Combine the transition patterns with the highest scores to form a set of significant transition patterns. In this example, assume that we select the three transition patterns with the highest scores, which represent the trends of synchronous increase in crystal size and defect density, increase in crystal size but unchanged defect density, and unchanged crystal size but increased defect density.

[0080] Construct a hierarchical knowledge base. The knowledge base contains a material physical property layer and a defect formation mechanism layer. The physical property layer stores various physical properties of the material, such as melting point, density, conductivity, etc. The defect formation mechanism layer stores the formation mechanisms of various defects, such as vacancies, interstitial atoms, dislocations, etc. In this example, we constructed a knowledge base in which the physical property layer contains information such as the melting point and density of the crystal, and the defect formation mechanism layer contains the formation mechanisms of defects such as vacancies and interstitial atoms.

[0081] Calculate defect similarity. Based on the state transition sequence characteristics and physical property characteristics in the set of significant transition patterns, calculate the similarity with the defect formation mechanism in the hierarchical knowledge base. In this example, we compare the similarity of significant transition patterns with different defect formation mechanisms. For example, if a transition pattern reflects an increase in crystal size but a constant defect density, it may be more similar to the formation mechanism of interstitial atomic defects.

[0082] Construct a bipartite graph and match it. Take the set of significant transfer patterns and the defect formation mechanism as two vertex sets respectively to construct a bipartite graph. Take the defect similarity as the edge weight and use the maximum weight matching algorithm to get the best matching pair. In this example, three significant transfer patterns and several defect formation mechanisms in the knowledge base are constructed into a bipartite graph, and the optimal matching relationship is found through the maximum weight matching algorithm.

[0083] Extract target state sequences and map them to defect evolution stages. Extract target state sequences from the best matching pairs and map them to the evolution stages of the corresponding defect formation mechanisms. In this case, assuming that a significant transition pattern matches the formation mechanism of a vacancy defect, we map the transition pattern to the formation stage of the vacancy defect.

[0084] Build a quantitative model and calculate the physical credibility. Based on the hierarchical knowledge base, analyze the physical process of the defect evolution stage and build a quantitative model of the composition field and temperature field containing the key state sequence. Calculate the pattern matching probability and physical rationality probability of the model to obtain the mechanism credibility. In this example, we built a quantitative model containing variables such as crystal size, defect density, and temperature based on the formation mechanism of vacancy defects, and calculated the degree of match between the model and the actual observed data and the physical rationality of the model.

[0085] Determine the defect formation mechanism based on the mechanism credibility. Based on the calculated mechanism credibility, determine the most likely defect formation mechanism. In this case, if the quantitative model of the vacancy defect formation mechanism has a high pattern match probability and a high physical plausibility probability, we believe that the vacancy defect is the main cause of the observed state transition phenomenon.

[0086] In this embodiment, by combining the material knowledge base and the state transition probability analysis, the formation mechanism of material defects can be more accurately identified. By building a quantitative model and analyzing the physical process of the defect evolution stage, the cause of defect formation can be explained more clearly. By automatically analyzing the state transition data and matching the defect formation mechanism, the efficiency of defect analysis can be improved.

[0087] Figure 2 FIG. 1 is a schematic diagram of the structure of an online quality monitoring system based on optical multi-spectrum fusion according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to collect multispectral image data of the object to be tested at the detection station, and a visible light camera group and an infrared camera group are set at the detection station. The trigger signals of the visible light camera group and the infrared camera group are uniformly dispatched by a synchronous control module, and the collected multispectral image data are pre-processed, and different spectral images are registered by using a feature point matching algorithm combined with camera calibration parameters to obtain the registered multispectral image data; The second unit is used to input the registered multi-spectral image data into a variational autoencoder for probability distribution modeling and introduce relative entropy constraints, obtain spectral basis vectors and an initial abundance map by minimizing reconstruction errors, construct a multi-scale morphological feature pyramid for the initial abundance map, calculate a non-local feature similarity matrix, construct a regional affinity map in combination with spatial position information, segment the regional affinity map using a graph cut algorithm to obtain local consistency regions, adaptively fuse the local consistency regions with the initial abundance map to obtain enhanced abundance features, and use a multi-head cross attention mechanism to align and complementarily enhance the spectral basis vectors and the enhanced abundance features to obtain multi-spectral fusion features; The third unit is used to construct a dynamic neural network, map the multi-spectral fusion features into network node features, construct the topological connection relationship between nodes based on the feature distance, enable the nodes to obtain key information from neighboring nodes through the dynamic neural network to obtain the post-propagation features, perform multi-level dimensionality reduction on the post-propagation features to obtain a structured representation, construct a feature discrimination criterion to enhance the discriminability of the structured feature representation, calculate the node significance distribution according to the structured feature representation after the discriminability enhancement, determine the area with significance higher than a preset threshold in the node significance distribution as the quality defect area, and at the same time analyze the formation mechanism of the quality defect based on the propagation path of the network node features after propagation, and generate a quality monitoring report according to the spatial distribution of the quality defect area and the formation mechanism of the quality defect.

[0088] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0089] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0090] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An online quality monitoring method based on optical multi-spectrum fusion, characterized in that: include: Collect multispectral image data of the object to be tested at the detection station, set a visible light camera group and an infrared camera group at the detection station, and the trigger signals of the visible light camera group and the infrared camera group are uniformly dispatched by a synchronous control module, and the collected multispectral image data are pre-processed, and the different spectral images are registered by using a feature point matching algorithm combined with camera calibration parameters to obtain the registered multispectral image data; The registered multi-spectral image data is input into a variational autoencoder for probability distribution modeling and relative entropy constraints are introduced. The spectral basis vector and the initial abundance map are obtained by minimizing the reconstruction error. A multi-scale morphological feature pyramid is constructed for the initial abundance map. The non-local feature similarity matrix is ​​calculated. A regional affinity map is constructed in combination with spatial position information. A graph cut algorithm is used to segment the regional affinity map to obtain local consistency regions. The local consistency regions are adaptively fused with the initial abundance map to obtain enhanced abundance features. A multi-head cross-attention mechanism is used to align and complementarily enhance the spectral basis vector and the enhanced abundance features to obtain multi-spectral fusion features. A dynamic neural network is constructed to map the multi-spectral fusion features into network node features, and the topological connection relationship between nodes is constructed based on the feature distance. The dynamic neural network enables the nodes to obtain key information from neighboring nodes to obtain the post-propagation features, and the post-propagation features are subjected to multi-level dimensionality reduction to obtain a structured representation. A feature discrimination criterion is constructed to enhance the discriminability of the structured feature representation, and the node significance distribution is calculated based on the structured feature representation after the discriminability enhancement. The area in the node significance distribution with a significance higher than a preset threshold is determined as the quality defect area. At the same time, the formation mechanism of the quality defects is analyzed based on the propagation path of the network node features after propagation, and a quality monitoring report is generated according to the spatial distribution of the quality defect area and the formation mechanism of the quality defects.

2. The method according to claim 1, characterized in that The collected multispectral image data is preprocessed, and the feature point matching algorithm is combined with the camera calibration parameters to align the different spectral images. The aligned multispectral image data includes: Calculating a local cumulative distribution function through a sliding window, and using the local cumulative distribution function to perform local adaptive histogram equalization processing on the visible light image and the infrared light image to obtain a contrast enhanced image; Performing bilateral filtering on the contrast enhanced image, combining noise suppression processing with wavelet transform, to obtain a denoised image, and performing geometric distortion correction on the denoised image based on a camera imaging model to obtain a preprocessed image; Constructing a multi-layer image pyramid for the preprocessed image, extracting image feature points based on the scale space in each layer of the multi-layer image pyramid, and screening the image feature points by using corner point response values ​​to obtain screened feature points; Extracting the gradient direction histogram and the local binary pattern of the selected feature points through neighborhood block division to generate a feature description vector, matching the feature description vector based on the nearest neighbor ratio method and random consistency test to obtain a set of feature matching point pairs; Calibrate the cameras corresponding to the visible light image and the infrared light image based on the calibration plate image sequence, obtain the camera parameter matrix and distortion parameters, calculate the homography matrix according to the feature matching point pair set, and optimize the transformation parameters of the homography matrix by using an iterative optimization algorithm to obtain an optimized homography matrix; The pre-processed image is resampled by bilinear interpolation using the optimized homography matrix and the camera parameter matrix to obtain a registered visible light image and infrared light image.

3. The method according to claim 2, characterized in that The gradient direction histogram and the local binary pattern are extracted from the selected feature points through neighborhood block, and a feature description vector is generated. The feature description vector is matched based on the nearest neighbor ratio method and random consistency test to obtain a feature matching point pair set including: Dividing a feature description area with the screened feature point as the center, and dividing the feature description area into a plurality of feature sub-blocks; Calculating the gradient magnitude and the gradient direction in the feature sub-block, constructing a gradient direction histogram to obtain a gradient feature vector, extracting a local binary pattern in the feature sub-block and using a rotationally invariant uniform pattern encoding to obtain a texture feature vector, normalizing and concatenating the gradient feature vector and the texture feature vector to form a feature description vector; A tree index structure is constructed for the feature description vector, and a nearest neighbor search method is used to obtain a nearest neighbor point set for each feature point; the nearest neighbor point set is screened according to a nearest neighbor ratio threshold to obtain an initial matching point pair set; a plurality of groups of matching points are selected from the initial matching point pair set by a random sampling method to calculate a perspective transformation model, and the number of matching points that satisfy the perspective transformation model is counted, and the perspective transformation model with the largest number of matching points is determined as the optimal transformation model, and candidate feature matching point pairs that satisfy the optimal transformation model are obtained; Dividing the image containing the candidate matching point pairs into a plurality of grid areas, counting the number of feature matching points in each grid area to obtain a spatial distribution variance, and calculating the distance ratio and angle difference between the feature matching point pairs to obtain a geometric consistency parameter; A weighted voting function is constructed according to the spatial distribution variance and the geometric consistency parameter, and the feature matching point pairs are screened by voting, and the feature matching point pairs that do not meet the spatial distribution constraints and the geometric consistency constraints are eliminated to obtain a final feature matching point pair set.

4. The method according to claim 1, characterized in that: The registered multispectral image data is input into a variational autoencoder for probability distribution modeling and relative entropy constraints are introduced. The spectral basis vector and the initial abundance map are obtained by minimizing the reconstruction error. A multi-scale morphological feature pyramid is constructed for the initial abundance map. The non-local feature similarity matrix is ​​calculated. The regional affinity map is constructed in combination with the spatial position information, including: Reconstructing the registered multispectral image data into a two-dimensional matrix, wherein the number of rows of the two-dimensional matrix is ​​the number of image pixels, and the number of columns is the number of spectral bands, and normalizing the two-dimensional matrix to obtain a normalized matrix; Input the normalized matrix into an encoder for feature extraction to obtain a mean vector and a variance vector, sample the mean vector and the variance vector to obtain a latent variable, input the latent variable into a decoder for reconstruction, and restore the data dimension through deconvolution operation and upsampling to obtain a reconstructed matrix; Constructing a loss function including a reconstruction error term and a relative entropy constraint term, obtaining a spectral basis vector and an initial abundance map by minimizing the loss function, and performing non-negative constraint and normalization processing on the spectral basis vector to obtain a standardized spectral basis vector; Based on the initial abundance map, an opening operation is performed to sequentially extract dark area features to obtain a multi-scale dark area feature map set, and a closing operation is performed to sequentially extract bright area features to obtain a multi-scale bright area feature map set; the response values ​​of each pixel position in the multi-scale dark area feature map set and the multi-scale bright area feature map set are combined to obtain a multi-scale feature vector of the current pixel position; The covariance matrix of the multi-scale feature vector is calculated, and the Mahalanobis distance between pixel pairs is calculated based on the covariance matrix. The Mahalanobis distance is mapped through a Gaussian kernel function to obtain a feature similarity matrix. The gradient variance of the local area is calculated to obtain texture complexity, and the texture complexity is used as a weight factor to perform a weighted combination of the feature similarity matrix and the spatial distance of the pixel points to obtain the edge weight of the regional affinity graph. A threshold is set for the edge weight to retain the maximum connection value to obtain the final regional affinity graph.

5. The method according to claim 1, characterized in that The regional affinity map is segmented by a graph cut algorithm to obtain a local consistency region, the local consistency region is adaptively fused with the initial abundance map to obtain an enhanced abundance feature, and the spectral basis vector and the enhanced abundance feature are feature aligned and complementary enhanced by a multi-head cross attention mechanism to obtain multi-spectral fusion features including: The regional affinity graph is constructed as a weighted undirected graph, in which the vertex set is the spatial position of the image pixel points, the edge set is the connection relationship between adjacent pixel points, and the edge weight value is determined by the association strength of the corresponding pixel pairs in the regional affinity graph to obtain a weight matrix; Constructing a degree matrix based on the weight matrix, wherein the diagonal elements of the degree matrix are the sum of the elements of the corresponding rows in the weight matrix, constructing a normalized Laplace matrix based on the weight matrix and the degree matrix, solving the eigenvalue equation of the normalized Laplace matrix, and obtaining an eigenvector; Divide the vertex set into mutually non-intersecting subgraphs according to the sign value of the eigenvector, calculate the cut cost value and subgraph volume value between each subgraph, and when the cut cost value is less than the cut cost threshold and the subgraph volume value is greater than the volume threshold, recursively divide the subgraph until a set of local consistency regions is obtained; Calculating the arithmetic mean of the initial abundance vectors of the pixels in each area of ​​the local consistency area set to obtain the regional abundance vector, calculating the Euclidean distance between the initial abundance vector of the pixels in the local consistency area and the regional abundance vector, and converting the Euclidean distance into a fusion weight coefficient; The initial abundance vector of the pixel point and the regional abundance vector are linearly weighted combined by using the fusion weight coefficient to obtain an enhanced abundance feature matrix, and the spectral basis vector is subjected to query transformation to obtain a query feature matrix, and the enhanced abundance feature matrix is ​​subjected to key transformation and value transformation to obtain a key feature matrix and a value feature matrix, respectively, and the query feature matrix is ​​multiplied by the key feature matrix and normalized, and an attention weight matrix is ​​obtained by an activation function, and the attention weight matrix is ​​multiplied by the value feature matrix to obtain an interaction feature matrix; A plurality of the interactive feature matrices are concatenated on the feature dimension and then mapped to obtain a fused feature matrix; the fused feature matrix is ​​added to the enhanced abundance feature matrix and normalized to obtain a multi-spectrum fused feature matrix.

6. The method according to claim 1, characterized in that Constructing a feature discrimination criterion to enhance the discriminability of the structured feature representation, calculating the node significance distribution according to the structured feature representation after the discriminability enhancement, determining the area with a significance higher than a preset threshold in the node significance distribution as a quality defect area, and analyzing the formation mechanism of the quality defect based on the propagation path of the propagated network node feature, and generating a quality monitoring report according to the spatial distribution of the quality defect area and the formation mechanism of the quality defect, including: Constructing a feature discrimination criterion, wherein the feature discrimination criterion includes a pairwise feature distance term calculated based on a node feature vector, a local structure preservation term based on a graph Laplacian matrix, a global feature distribution constraint term based on a high-order moment, and a feature regularization term based on a sparse representation, and taking the feature discrimination criterion as an optimization target; The feature discrimination criterion is optimized by adopting an alternating optimization strategy of block coordinate descent, the node feature similarity is calculated by an adaptive kernel function to obtain a node feature weight matrix, a graph Laplacian matrix is ​​constructed based on the node feature weight matrix, and feature decomposition is performed to obtain a feature mapping matrix, and the node features are updated using the feature mapping matrix to obtain discriminative enhancement features; The discriminative enhancement features of different scales are combined by attention weights to obtain multi-scale fusion features, the Mahalanobis distance and cosine distance between the multi-scale fusion features and the standard features are calculated, and the significance score is obtained by weighted fusion; Performing double threshold segmentation on the significance score to obtain a candidate defect region, and optimizing the candidate defect region through morphological processing to obtain a target quality defect region; Constructing a double-layer feature propagation network, calculating edge weights based on the difference of the multi-scale fusion features at the local layer, calculating edge weights based on node importance at the global layer, and combining the edge weights of the double-layer network to extract the propagation path of the target quality defect area; A recurrent neural network is constructed based on the propagation path, the multi-scale fusion feature is encoded into a state sequence along the propagation path, a characteristic state transition probability is obtained by decoding the state sequence, a significant transition mode is extracted from the characteristic state transition probability, and the significant transition mode is matched and analyzed with a preset material knowledge base to obtain a defect formation mechanism; A quality monitoring report is generated based on the spatial distribution characteristics of the target quality defect area, the network topology characteristics of the propagation path, and the evolution characteristics of the defect formation mechanism.

7. The method according to claim 6, characterized in that Extracting a significant transition mode from the characteristic state transition probability, and matching and analyzing the significant transition mode with a preset material knowledge base to obtain a defect formation mechanism includes: Performing eigenvalue decomposition on the characteristic state transition probability matrix to obtain an eigenvalue matrix and an eigenvector matrix, selecting an eigenvector corresponding to the maximum eigenvalue from the eigenvector matrix as a transfer mode, calculating a transfer feature score based on the ratio of the eigenvalue in the eigenvalue matrix to the first-order norm and the second-order norm of the transfer mode, and selecting the transfer mode with the highest transfer feature score to construct a significant transfer mode set; Constructing a hierarchical knowledge base including a material physical property layer and a defect formation mechanism layer, and calculating the defect similarity with the defect formation mechanism in the hierarchical knowledge base based on the state transition sequence characteristics and physical property characteristics in the significant transition pattern set; The significant transfer pattern set and defect formation mechanism are used as two vertex sets to construct a bipartite graph, and the defect similarity is used as the edge weight. The maximum weight matching algorithm is used to obtain the optimal matching pair. A target state sequence is extracted from the optimal matching pair, the target state sequence is mapped to the evolution stage of the corresponding defect formation mechanism, the physical process of the evolution stage is analyzed based on the hierarchical knowledge base, a quantitative model of the component field and temperature field including the key state sequence is constructed, the pattern matching probability and the physical rationality probability of the quantitative model are calculated to obtain the mechanism credibility, and the defect formation mechanism is determined based on the mechanism credibility.

8. An online quality monitoring system based on optical multi-spectral fusion, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect multispectral image data of the object to be tested at the detection station, and a visible light camera group and an infrared camera group are set at the detection station. The trigger signals of the visible light camera group and the infrared camera group are uniformly dispatched by a synchronous control module, and the collected multispectral image data are pre-processed, and different spectral images are registered by using a feature point matching algorithm combined with camera calibration parameters to obtain the registered multispectral image data; The second unit is used to input the registered multi-spectral image data into a variational autoencoder for probability distribution modeling and introduce relative entropy constraints, obtain spectral basis vectors and an initial abundance map by minimizing reconstruction errors, construct a multi-scale morphological feature pyramid for the initial abundance map, calculate a non-local feature similarity matrix, construct a regional affinity map in combination with spatial position information, segment the regional affinity map using a graph cut algorithm to obtain local consistency regions, adaptively fuse the local consistency regions with the initial abundance map to obtain enhanced abundance features, and use a multi-head cross attention mechanism to align and complementarily enhance the spectral basis vectors and the enhanced abundance features to obtain multi-spectral fusion features; The third unit is used to construct a dynamic neural network, map the multi-spectral fusion features into network node features, construct the topological connection relationship between nodes based on the feature distance, enable the nodes to obtain key information from neighboring nodes through the dynamic neural network to obtain the post-propagation features, perform multi-level dimensionality reduction on the post-propagation features to obtain a structured representation, construct a feature discrimination criterion to enhance the discriminability of the structured feature representation, calculate the node significance distribution according to the structured feature representation after the discriminability enhancement, determine the area with significance higher than a preset threshold in the node significance distribution as the quality defect area, and at the same time analyze the formation mechanism of the quality defect based on the propagation path of the network node features after propagation, and generate a quality monitoring report according to the spatial distribution of the quality defect area and the formation mechanism of the quality defect.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images