Multimodal cnn / transformer image defect diagnosis tracking decision method

Abstract

The application discloses a multimodal CNN / Transformer image defect diagnosis tracking decision method, and belongs to the field of power electronic industry detection. The method collects multimodal data of defect positions of different batches, process stages and equipment, and extracts features by using position coding, a CNN shallow network and ResNet-18 after time and space alignment, image enhancement preprocessing and alignment. The features are combined and fused into a multimodal feature vector by using weights, a detection model is obtained through CNN / Transformer model training, defect diagnosis and tracking decision are realized, and the problems of accurate detection, positioning, diagnosis tracking and decision of small component changes and structural defects in electronic products, and single local and overall multimodal defects of power are solved. The detection accuracy is improved by 20%, the diagnosis accuracy is more than 98.5%, the tracking defect error is less than 5%, the decision efficiency is improved by 40%, and the defect prediction capability is improved by 22%.

Description

Technical Field

[0001] This invention relates to the field of machine vision and visualization technology for power electronics industrial products, specifically to the fusion of single-process and local defects with multi-process and overall defects and CNN / Transformer for detection, measurement, diagnosis, tracking and decision-making. It relates to a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method. Background Technology

[0002] The rough surfaces of some power electronics products (such as gas-insulated switchgear, inverters, power cables, circuit breakers, etc.) severely interfere with the perception and detection of internal and external defects by optical and non-destructive testing equipment. For example, the trend towards miniaturization in chips, IGBTs, and crystal oscillators, coupled with the complexity of their backgrounds and the subtlety of their defects, makes effective detection of internal defects increasingly difficult. Existing technologies, particularly image processing, cannot clearly (or are blurry) reflect the finer details caused by variations in the internal composition and structure of the product (low brightness and contrast), have low pixel correlation, cannot analyze and reflect finer texture information, and provide limited and disorganized information, lacking depth and failing to reflect information in complex areas. Many finer structural or component color variations are hidden, and low spatial resolution makes it impossible to detect minute component changes and structural defects within the product. Existing technologies can only provide a general location and condition of the defect. Image blurring and poor visualization hinder computer-based detection, fault location, and diagnosis.

[0003] The zero-order moments of existing X-ray images are generated from two dimensions: objects of different sizes and depths. They are related to or related to the area and volume of the object, product, and defects. Independent zero-order moments are difficult to directly use for area and volume measurement; they cannot distinguish shape changes, and shape cues are blurred. First-order moments of X-ray images can only provide a preliminary guess and judgment of the intensity distribution, centroid location, and current state of the product and defects. A single first-order moment is difficult to measure the length, width, and height of the product and defects. Second-order moments of X-ray images can only roughly determine the shape and orientation of the product and defects, and assist in understanding their approximate area and volume, but cannot provide precise measurement. Even combining zero-order, first-, and second-order moments from X-ray images, infrared cameras, etc., can only construct a rough model of the defect, providing only a basis for further analysis and evaluation, and cannot serve as a basis for product improvement, defect repair, process design, production, or prediction. Existing depth imaging methods such as X-ray, ultrasound, infrared, and laser suffer from attenuation at different depths, and single-modal imaging has limitations in depth imaging.

[0004] With the continuous advancement of industrialization, welding technology is increasingly widely used in manufacturing. However, quality problems in welded structures, such as cracks, porosity, lack of fusion, voids, delamination, deformation, separation, poor solder joint contact, partial missing solder joints, and incomplete soldering, seriously affect the safety, reliability, and performance of welded structures and products. To ensure the quality, performance, stability, reliability, and safety of products and welds, defect detection in welded structures and products has become an indispensable part. PCBs (Printed Circuit Boards), components, and PCBAs (Printed Circuit Board Assemblies) can all exhibit defects such as cracks, voids, delamination, deformation, separation, bridging, open circuits in metal layers (wires, cores), fractures, dust and impurities, metal particles, loosening, misalignment, and damage to the insulating layer. These defects severely affect the safety, reliability, stability, performance, and potential failure of welded structures, PCBs (PCBAs), components, and processes. To ensure the quality, safety, reliability, stability, performance, and lifespan of PCBs (PCBAs), components, and processes, defect detection of PCBs (PCBAs), components, and processes has become an indispensable part.

[0005] Traditional methods for defect detection include non-destructive testing techniques such as ultrasonic testing, magnetic particle testing, penetrant testing, and X-ray testing. Among these, X-ray testing offers intuitive imaging, sensitivity to volumetric defects, accurate dimensional measurement, and no strict requirements on the surface roughness of the tested object. X-ray non-destructive testing technology has advantages such as strong penetrating power and high image resolution, and it can also penetrate deep into the internal structure of materials, making it widely used for internal defect detection and quality inspection of metallic materials. However, existing welding structures and semiconductor packages such as SOP (Standard Operating Procedure), QFP (Quad Flat Package), BGA (Ball Grid Array), CSP (Chip Scale Package), and IGBT, as well as industrial devices and batteries, have discontinuities (microcracks, porosity, etc.) that cause defects. These defects result in differences in the density, thickness, shape, and size of the welding structure and product components. In particular, it is difficult to detect discontinuities on tiny micro-surfaces or sub-surfaces, which are the root cause of progressive degradation and failure in product lifespan, reliability, performance, stability, and metrological accuracy. Existing X-ray non-destructive testing images have low contrast and clarity, are insensitive to microcracks, and do not penetrate, making it difficult to detect these defects.

[0006] Currently, X-ray crack detection primarily relies on manual visual inspection, which is not only inefficient and labor-intensive, but also prone to inaccurate results due to eye damage from strong light and subjective factors. In the power industry, meters and metering automation terminals, transformers, batteries, transformers, and conductors are crucial equipment in the power system; their quality and reliability directly affect the accuracy of power metering and the normal operation of the power system. Defects such as cracks, voids, and delamination in welding, PCB (PCBA) boards, and components not only affect the lifespan and stability of meters and metering automation terminals, batteries, transformers, and conductors, leading to power metering errors, but can even cause safety hazards. Traditional methods for detecting cracks, voids, bubbles, and delamination in welding, PCB (PCBA) boards and processes, transformers, and components (e.g., eight types of chips) mainly rely on manual visual inspection, which also suffers from low efficiency, high workload, and poor accuracy. With the increasing demands for efficiency and accuracy in industrial production, traditional manual inspection methods can no longer fully meet the requirements.

[0007] Current integrated circuits and semiconductors (13-layer metal interconnect chips), whether for circuit testing or device (multilayer chips and transistors) electrical performance testing, require the removal of the capping metal and dielectric layers layer by layer. This involves using FIB (Focused Ion Beam) to cut the sample, and TEM (Transmission Electron Microscope) to inspect the oxide layer thickness and defects such as voids and cracks in the metal interconnects after the capping layer is removed. EMM (Electron Emission Microscope) and OBIRCH (Optical Beam Induced Resistance Change) optical current emission microscopy also require the removal of the capping layer before testing can detect defects in circuits and devices. This process suffers from high efficiency and cost, significant destructiveness, and low image detection accuracy (88% at best experimental levels, less than 80% at industrial levels).

[0008] In recent years, with the development of image processing and machine learning technologies, automated X-ray crack detection has gradually become a new detection approach. Although deep learning-based automated detection methods have made significant progress in image recognition and classification, these methods typically require large amounts of labeled datasets for training and validation. Furthermore, the model training process is often time-consuming and highly dependent on computational resources. This poses a significant challenge to the practical application of deep learning methods, especially in industrial scenarios where data is scarce and real-time processing is required.

[0009] The current research indicates that product testing can only detect and diagnose defects in a single process or a localized area. Any defect is caused by multi-parameter, multi-process, and multi-modal overall factors. Therefore, the improvement effect of any single or localized defect is limited. Solving product defects requires an automated fusion method that integrates single-process and localized defects with multi-modal and multi-process overall factors for detection, diagnosis, tracking, and decision-making. Currently, there is a lack of a method that can automatically integrate single-process and localized defects with multi-process and overall defect detection, diagnosis, tracking, and decision-making in a single process. Summary of the Invention

[0010] In view of this, the present invention provides a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method to solve the problem of being unable to accurately detect, locate, diagnose, track and make decisions on minute component changes and structural defects inside power electronic products, as well as single process and local defects, and multi-process and overall defects.

[0011] In a first aspect, the present invention provides a multimodal CNN / Transformer image defect diagnosis and tracking decision method, the method comprising:

[0012] The defect location images of the product to be diagnosed are obtained from multiple multimodal data. These multiple defect location images are defect location images containing the above multimodal data, taken from different production batches, different process stages, and different equipment. The multimodal data includes image data, electrical data, and operating condition data.

[0013] Data preprocessing and alignment are performed on defect location images from multiple multimodal datasets;

[0014] Based on the defect location images of each image type, feature extraction is performed on the defect location images of multiple multimodal data using a position encoding method to obtain the position encoded image features of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. The image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and location texture parameter features.

[0015] Based on a preset feature weight allocation scheme, feature weight fusion is performed on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and location texture parameter feature as image features. The image features, electrical features and working condition features are then concatenated into a unified vector to obtain multiple multimodal fusion feature vectors.

[0016] A single-process image defect detection model is obtained by sequentially training and evaluating independent sub-models of a convolutional neural network model based on multiple multimodal fusion feature vectors, and a multi-process image defect detection model is obtained by sequentially training and evaluating a Transformer network model based on multiple multimodal fusion feature vectors.

[0017] Multiple images of the product to be diagnosed are acquired and input into the single-process image defect detection model and / or multi-process image defect detection model to perform image defect diagnosis, and the detection result of whether there is a defect or no defect in the corresponding area of ​​the product image is obtained.

[0018] The defective product image output by the defect detection model is subjected to hybrid modeling and analysis to obtain a hybrid modeled image defect. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0019] The defect detection results are obtained by using the improved U-Net model to obtain the defects in the hybrid modeled image, thereby realizing defect region segmentation and localization. Then, the location of the defect region in the image is tracked, and the decision is made based on the defect tracking and fault tracing results to optimize the production process parameters.

[0020] This invention provides a multimodal CNN / Transformer image defect diagnosis, tracking, and decision-making method. It collects multimodal data (including image, electrical, and operational condition data) on defect locations from different batches, process stages, and equipment. Based on multiple multimodal fusion feature vectors combined with a convolutional neural network model, an image defect detection model is obtained. This model is applied to real-world scenarios to obtain detection results indicating whether defects exist or not in the corresponding image regions. This enables precise detection and location of minute component changes and structural defects within products, improving detection accuracy. The method also performs shape measurement calculations on defective product images to diagnose the severity of image defects and make tracking decisions regarding the location of image defect regions. After spatiotemporal alignment, image enhancement preprocessing, and alignment, features are extracted using positional encoding, shallow CNN networks, and ResNet-18, and fused with weights to form multimodal feature vectors. These vectors are then trained using CNN and Transformer models to obtain a detection model, enabling defect diagnosis and tracking decisions. The innovation lies in the complementary cross-modal features, the hybrid modeling of discrete and continuous modes, and the multi-stage fusion strategy. This solves the problems of accurate detection, localization, diagnosis, tracking, and decision-making for minute component variations and structural defects within electronic products, as well as single local and overall multimodal electrical defects. Detection accuracy is improved by 20%, diagnostic accuracy exceeds 98.5%, defect tracking error is less than 5%, decision-making efficiency is improved by 40%, and defect prediction capability is improved by 22%.

[0021] In one optional implementation, data preprocessing and data alignment are performed on defect location images from multiple multimodal data sets, including:

[0022] The defect location images are preprocessed sequentially by data cleaning, data annotation, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization.

[0023] Among them, data cleaning is used to remove noise data, blurred defect location images, and incomplete defect location images from defect location images; data annotation uses a combination of manual annotation and semi-automatic annotation to annotate defect locations and defect types; image usage classification divides multimodal data into image data, electrical data, and operating condition data according to the production batch, process stage, equipment type, and multimodal data type of the image; image pixel normalization normalizes the pixel values ​​of image data to a preset range to unify the scale of image data;

[0024] Image data, electrical data, and operating condition data are aligned, merged, and standardized according to timestamps to construct a spatiotemporally consistent dataset.

[0025] This invention provides a multimodal CNN / Transformer image defect diagnosis and tracking decision-making method. It aligns data from different sources, such as 3D structural information, thermal radiation information, and internal structure images, by timestamp to construct a spatiotemporally consistent multimodal dataset. Preprocessing of the 3D / microscopic structure images, including noise reduction and contrast enhancement, improves feature extraction quality. Electrical parameters (current / voltage / resistance) are normalized to eliminate dimensional differences. Defect location images undergo sequential data cleaning, data annotation, and image usage classification, improving data reliability and image quality. Feature normalization before fusion prevents certain features from dominating the fusion process due to excessively large numerical ranges, ensuring that each feature participates equally in the fusion process, thus improving the fusion effect and model stability.

[0026] In one alternative implementation, the operating condition data includes load fluctuations and ambient temperature;

[0027] Based on the defect location images of each image type, feature extraction is performed on the defect location images of the multiple multimodal data using a location encoding method to obtain the location-encoded image features of each defect location image. Simultaneously, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features, including:

[0028] A shallow CNN network is used to extract basic edge and texture features from images of defect locations;

[0029] The defect center position of each defect location image is determined based on edge and texture features; the two-dimensional coordinate values ​​of each defect center position are obtained and each two-dimensional coordinate value is discretized; each discretized coordinate value is converted into a feature vector through one-hot encoding, and the feature vector is used as the image feature after position encoding of each defect location image;

[0030] Electrical data is converted into time series data, and frequency domain features are extracted as electrical features using Fourier transform. Real-time data on load fluctuations and ambient temperature in the operating condition data are sampled using a 5-minute sliding window to extract operating condition features.

[0031] The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided in this invention considers location information and incorporates defect location features into multimodal fusion through location encoding, enabling the model to better utilize location information to detect defects. For devices like smart meters, defects at different locations may have different impacts, allowing for more accurate handling of location-related issues.

[0032] In one alternative implementation, the feature weight allocation scheme is set in the following manner:

[0033] The weights for contrast features are allocated as follows: 20%–30% for contrast features, 10%–25% for correlation features, 10%–25% for energy features, 30%–40% for entropy features, 10%–20% for defect entropy features, and 10%–30% for positional texture parameter features. The sum of the weights for contrast features, correlation features, energy features, entropy features, defect entropy features, and positional texture parameter features is equal to 1.

[0034] In one optional implementation, feature weight fusion is performed on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and location texture parameter feature based on a preset feature weight allocation scheme, and the image features, electrical features, and operating condition features are concatenated into a unified vector, including:

[0035] Calculate the first product of each contrast feature and its weight, the second product of each correlation feature and its weight, the third product of each energy feature and its weight, the fourth product of each entropy feature and its weight, the fifth product of each defect entropy value feature and its weight, and the sixth product of each positional texture parameter feature and its weight.

[0036] By adding the corresponding first, second, third, fourth, fifth, and sixth products, multiple multimodal fusion feature vectors are obtained.

[0037] Early fusion involves concatenating multiple multimodal fusion feature vectors with electrical and operating condition features into a unified vector.

[0038] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided in this invention weights and fuses multiple features such as contrast, correlation, energy, entropy, and defect entropy value, thereby fully leveraging the role of each feature in defect detection. By assigning weights based on the effectiveness of different features in distinguishing different types of defects, the fused feature vector becomes more discriminative and meets diagnostic requirements. It considers the characteristics of different image types and the performance of different features under different defect types, making the fusion scheme more targeted and better adaptable to specific detection tasks, thus meeting the need for high adaptability.

[0039] In one optional implementation, the convolutional neural network model includes an input layer, multiple convolutional and pooling layers, multiple fully connected layers, and an output layer. The single-process image defect detection model, obtained by sequentially training independent sub-models and evaluating the convolutional neural network model based on multiple multimodal fusion feature vectors, further includes:

[0040] Multiple multimodal fusion feature vectors are fed into multiple convolutional and pooling layers through the input layer for deep learning. The multiple convolutional layers use convolution kernels to perform convolution operations on the multimodal fusion feature vectors to extract local features, and the pooling layers obtain multiple key features through downsampling operations.

[0041] Multiple fully connected layers are used to integrate multiple key features, transforming the extracted features from local information into global information, and mapping them to the output probability distribution through an activation function;

[0042] The output layer is used to transform the probability distribution mapped to the output into multiple fused probability distribution images and output them. The value of each pixel in the fused probability distribution image represents the probability that there is a defect at that location.

[0043] The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by this invention extracts key features through multiple convolutional layers, pooling layers, and multiple fully connected layers, making image details more prominent and providing conditions for subsequent model training.

[0044] In one optional implementation, a single-process image defect detection model is obtained by sequentially training and evaluating independent sub-models based on multiple multimodal fusion feature vectors on a convolutional neural network model, including:

[0045] Multiple fused probability distribution images are divided into training set, validation set and test set according to a preset ratio;

[0046] The training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weighted and fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and the evaluation result is output. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model.

[0047] During training, the loss is calculated based on a preset loss function, and the Adam optimizer is used for backpropagation to adjust the weights and biases of the convolutional neural network model. Based on the validation set, the parameters of the multiple convolutional layers, pooling layers, and multiple fully connected layers are adjusted to obtain the convolutional neural network model to be evaluated. The Adam optimizer combines the momentum method and the Adagrad optimization algorithm to dynamically adjust the learning rate of each parameter in the model based on the first-order moment estimate and the second-order moment estimate of the gradient, and corrects the first-order moment estimate and the second-order moment estimate through bias.

[0048] In one optional implementation, the training set is input into a convolutional neural network model to train independent sub-models sequentially, and the outputs of each independent sub-model are weighted and fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on a test set, and an evaluation result is output. When the evaluation result meets a preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as a single-process image defect detection model, including:

[0049] The unified vector of defect features from the early-stage multimodal fusion image is input into a convolutional neural network for later-stage fusion.

[0050] Training convolutional neural network models specifically for image data to extract defect features from images;

[0051] Train a long short-term memory network model to process time-series electrical data and capture the patterns of change in electrical data over time;

[0052] Train a multilayer perceptron model to process operating condition data and analyze the impact of operating conditions on defects.

[0053] The outputs of the independent sub-models obtained from the above training are weighted and fused to obtain the detection model to be evaluated. The test set is input into the detection model to be evaluated. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model, and the evaluation result is used as the final defect diagnosis result.

[0054] In one optional implementation, a multi-step image defect detection model is obtained by sequentially training and evaluating a Transformer network model based on multiple multimodal fusion feature vectors, including:

[0055] The early fusion path of the multimodal fusion Transformer for image defects utilizes a self-attention mechanism to effectively capture the semantic relationships between features of different modalities, and directly performs cross-modal early fusion of multimodal features;

[0056] A complex defect feature detection model is formed by fusing local features extracted from multiple convolutional neural networks from different sources using a cross-attention mechanism in parallel. The complex defect feature detection model is combined with a detection model that processes temporal correlations using Transformer. The image to be diagnosed is input into this combined detection model, and the final defect diagnosis result is output.

[0057] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided by this invention selects early or late fusion based on data characteristics, balances computational efficiency and information preservation, and the technical characteristics of combining CNN / Transformer models can intuitively present the core logic of defect diagnosis and tracking decision-making.

[0058] In an optional implementation, the method further includes: visualizing the loss changes, evaluation result changes, key features learned by multiple convolutional and pooling layers, and the probability distribution of feature integration and mapping to the output of multiple fully connected layers during the training process of the convolutional neural network model using visualization tools.

[0059] The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by this invention uses visualization tools to visualize the loss changes, evaluation result changes, key features learned by multiple convolutional and pooling layers, and the probability distribution of feature integration and mapping of multiple fully connected layers to the output during the training process of the convolutional neural network model. This enables real-time monitoring of the training process and improves training accuracy.

[0060] In one optional implementation, multiple images of the product to be diagnosed are acquired and input into an image defect detection model for image defect diagnosis, resulting in a detection result indicating the presence or absence of defects, including:

[0061] Image defect diagnosis is performed on multiple product images to be diagnosed that are input into the image defect detection model using a decision function, which is expressed by the following formula:

[0062]

[0063] Where G(F) = 1 indicates that a defect is detected in the corresponding region of the image, G(F) = 0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that the preset defect condition is met;

[0064] The multimodal fusion feature vector of the input product image to be diagnosed is matched with the defect feature threshold set during model training. That is, when the feature parameters such as contrast, energy, and entropy in the feature vector exceed the preset threshold, it is determined that there is a defect; otherwise, it is determined that there is no defect.

[0065] In one optional implementation, hybrid modeling and analysis are performed on the defective product image output by the defect detection model from the input image to be diagnosed, including:

[0066] Calculate the geometric features of the image defect region contour and determine the preliminary shape of the image defect region; the geometric features of the preliminary shape of the image defect region include the contour perimeter, the contour convex hull perimeter, and the shape factor;

[0067] Based on the preliminary shape of the defect region in the image, a hybrid shape model is obtained by using the second moments of discrete data and the second moments of continuous functions to perform shape hybrid modeling, thus obtaining a mathematical model for measuring the shape of the image defect region.

[0068] The degree of defect in an image defect region is measured using a mathematical model for measuring the shape of the defect region, and the final shape, width, length, area, and volume of the defect region are obtained.

[0069] The severity of image defects is diagnosed and graded based on the final shape, width, length, area, and volume of the defective region.

[0070] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided by this invention obtains accurate internal three-dimensional structural information through multi-parameter data fusion and reconstruction technology. This method combines discrete second moments with continuous function second moments, addressing the challenges in combining discrete second moments with continuous function second moments and weight allocation in image accuracy, which hinders online analysis and visualization modeling of high-precision images. It also needs to address the limitations of depth imaging technology attenuation at different depths and single-modality limitations, overcoming the difficulties in establishing product defect models and weight allocation by combining discrete second moments with continuous function second moments, thus achieving online analysis and visualization modeling of high-precision images.

[0071] In one optional implementation, a shape hybrid model is obtained by using the second moments of discrete data and the second moments of continuous functions based on the preliminary shape of the defect region in the image, to obtain a mathematical model for measuring the shape of the image defect region, including:

[0072] Based on the preliminary shape of the defect region in the image, determine the discrete second moment and the continuous function second moment of the defect region;

[0073] By fusing the second moments of discrete data and the second moments of continuous functions in the image defect region, a mathematical model for measuring the shape of the image defect region is obtained.

[0074] In one optional implementation, the discrete second moments and the continuous second moments of the image defect region are fused and modeled to obtain a mathematical model for measuring the shape of the image defect region, including:

[0075] The first moment of inertia is calculated based on the second moment of discrete data, and the first equivalent ellipse parameters are calculated based on the first moment of inertia.

[0076] The second moment of inertia is calculated based on the second moment of the continuous function, and the parameters of the second equivalent ellipse are calculated based on the second moment of inertia.

[0077] The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent elliptic parameter and the second equivalent elliptic parameter, to obtain a mathematical model for measuring the shape of the image defect region.

[0078] In one optional implementation, a weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent elliptic parameters and the second equivalent elliptic parameters, to obtain a mathematical model for measuring the shape of the image defect region, including:

[0079] Define the weights of the second moments of continuous functions, and calculate the weights of the second moments of discrete functions based on the weights of the second moments of continuous functions;

[0080] The first moment of inertia and the second moment of inertia are fused using a weighted average method based on the weights of the second moments of continuous functions and discrete functions to obtain the fused second moment; the first equivalent elliptic parameter and the second equivalent elliptic parameter are fused to obtain the fused equivalent elliptic parameter.

[0081] A mathematical model for measuring the shape of the image defect region is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters.

[0082] The formula for the fused second moment is as follows:

[0083] M fusion =aM c +(1-a)M d ,

[0084] Among them, M c M is the second moment of a continuous function. dis the second moment of discrete data; a (0≤a≤1) is a weighting coefficient used to measure the proportion of the second moment of continuous function in the fused second moment; when a=0, it means that the contribution of the second moment of continuous function is not considered at all when calculating the fused second moment, and only the second moment of discrete data is used; when a=1, it depends entirely on the second moment of continuous function and does not consider the second moment of discrete data.

[0085] The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by this invention uses a weighted average method to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent elliptic parameter and the second equivalent elliptic parameter, to obtain a mathematical model for measuring the shape of the image defect region. This realizes the construction of the shape of the image defect region and provides conditions for tracking decision of the image defect region.

[0086] In one optional implementation, the defects in the hybrid modeled image are processed by an improved U-Net model to obtain defect detection results, thereby achieving defect region segmentation and localization, and then tracking the location of the image defect region, including:

[0087] The defects in the hybrid modeled image are processed by the improved U-Net model to obtain defect detection results, and the defect regions are segmented and located based on the defect detection results;

[0088] Based on the segmented and localized defect regions, the location of image defects is tracked.

[0089] In one alternative implementation, the semantic segmentation model of the improved U-Net model includes an encoder, a decoder, and skip connections, wherein the encoder includes residual connections and the decoder includes an attention mechanism.

[0090] The defects in the hybrid modeled image are processed using an improved U-Net model to obtain defect detection results, achieving defect region segmentation and localization, including:

[0091] After convolution processing of the image defect region in the encoder, the output feature map and the input feature map are obtained. The output feature map and the input feature map are then added together through residual connection to obtain a new output feature map.

[0092] The new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, which is then convolved to obtain a TFT ray map.

[0093] The TFT ray image is input into the improved U-Net model to perform image region segmentation and obtain a binary image of the image defect region.

[0094] In one optional implementation, after convolution processing of the image defect region in the encoder to obtain an output feature map and an input feature map, the output feature map and the input feature map are added together through residual connection to obtain a new output feature map, including:

[0095] Residual connections are introduced into the encoder, and the weight matrix of the residual connections is: The output feature map after two convolutional layers Input features Figure X i (x, y, z), and add the output feature map and the input feature map through a residual connection to obtain a new output feature map. The formula is as follows:

[0096]

[0097] Where (x, y, z) are the image coordinates.

[0098] In one optional implementation, the new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, which is then subjected to a convolution operation to obtain a TFT ray map, including:

[0099] An attention mechanism is introduced into the decoder, with the attention weight matrix as follows: Feature maps after encoder upsampling and stitching Weighted features are obtained through an attention mechanism. That is, the TFT beam pattern, the formula is as follows:

[0100]

[0101] Where (x, y, z) are image coordinates, a represents the relative position offset involved in the weighted operation when the feature map is subjected to the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[0102] In one alternative implementation, decisions are made based on defect tracking and fault tracing results to optimize production process parameters, including:

[0103] A dynamic model of defect evolution is constructed based on discrete data points of defect area and perimeter and a second-order moment model of continuous function.

[0104] Based on defect location images, a preset tracking function is used to analyze the morphological changes of crack propagation angle and wear depth increment of defects in the defect area at different times, and to visualize and quantify the defect evolution process.

[0105] The physical defect locations in the three-dimensional structural information are obtained, and a spatial mapping algorithm is used to match the physical defect locations in the three-dimensional structural information with electrical anomaly point data and microstructure images, respectively, to establish a correlation map between physical defects and electrical faults.

[0106] Based on the correlation map of physical defects and electrical faults and the morphological change characteristics, and combined with the abnormal current distribution analysis of the wear area of ​​the resistor surface, the fault source of the image defect area is traced.

[0107] Based on defect tracking and fault tracing, improvements are made to the welding process, material replacement process, and online monitoring process of the product to be decided.

[0108] In one alternative implementation, the welding process of the product to be decided is improved, including:

[0109] Finite element method (FEM) simulation was used to analyze the effects of different welding parameters (laser power, welding speed, and spot diameter) on the residual stress, fatigue life, and resistance of the weld joint of the product under consideration. The simulation effect was verified by combining multimodal data from ultrasonic scanning results to determine the optimal process parameters for laser welding and resistance brazing. The influence of multiple power sources (photovoltaics, green electricity, energy storage, mains power, combined power, and nuclear power) on the welding parameters was also considered. The defect location images of welding cracks and incomplete welds detected by convolutional neural networks were combined with the multimodal data from ultrasonic scanning to quantify the welding quality.

[0110] Improvements to the material replacement process for the product being decided upon, including:

[0111] A material performance database was established to record the resistance change rate and capacitance stability of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments. Through accelerated aging tests simulating extreme working conditions, convolutional neural networks were used to detect defects in images of aged materials, and the durability of the materials was evaluated in combination with performance data.

[0112] Improvements to the online monitoring process for decision-making products include:

[0113] Deploy a deep learning-based real-time image recognition system to perform millisecond-level analysis on X-ray or OBIRCH inspection images to automatically label and classify defects;

[0114] A defect recognition model is established by training X-ray or OBIRCH inspection images using deep learning algorithms.

[0115] In a second aspect, the present invention provides a multimodal CNN / Transformer image defect diagnosis and tracking decision-making device, the device comprising:

[0116] The multimodal data acquisition module is used to acquire defect location images of multiple multimodal data of the product to be diagnosed. These multiple defect location images are defect location images containing the aforementioned multimodal data, taken from different production batches, different process stages, and different equipment. The multimodal data includes image data, electrical data, and operating condition data.

[0117] The data preprocessing and data alignment module is used to perform data preprocessing and data alignment on defect location images of multiple multimodal data.

[0118] The feature extraction module is used to extract features from defect location images of multiple multimodal data based on the defect location images of each image type data using a location encoding method, to obtain the image features after location encoding of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. The image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and location texture parameter features.

[0119] The multimodal feature fusion module is used to perform feature weight fusion on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and location texture parameter feature based on a preset feature weight allocation scheme as image features, and to concatenate the image features, electrical features and operating condition features into a unified vector to obtain multiple multimodal fusion feature vectors;

[0120] The multimodal detection model construction and detection module is used to obtain a single-process image defect detection model by sequentially training and evaluating independent sub-models of a convolutional neural network model based on multiple multimodal fusion feature vectors, and to obtain a multi-process image defect detection model by sequentially training and evaluating a Transformer network model based on multiple multimodal fusion feature vectors; it acquires multiple actual captured product images to be diagnosed and inputs them into the single-process image defect detection model and / or the multi-process image defect detection model for image defect diagnosis, and obtains the detection results of whether there are defects or no defects in the corresponding areas of the product images;

[0121] The hybrid modeling and analysis module is used to perform hybrid modeling and analysis on the product image with defects output by the defect detection model of the input image to be diagnosed, so as to obtain hybrid modeled image defects. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0122] The tracking and decision module is used to obtain defect detection results from hybrid modeled image defects through the improved U-Net model, realize defect region segmentation and localization, and then track the location of image defect regions. Based on the defect tracking and fault tracing results, decisions are made to optimize production process parameters.

[0123] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the multimodal CNN / Transformer image defect diagnosis and tracking decision method described in the first aspect or any corresponding embodiment thereof.

[0124] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the multimodal CNN / Transformer image defect diagnosis and tracking decision method described in the first aspect or any corresponding embodiment thereof.

[0125] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the multimodal CNN / Transformer image defect diagnosis and tracking decision method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0126] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0127] Figure 1 This is a schematic diagram of the multimodal CNN / Transformer image defect diagnosis and tracking decision-making process according to an embodiment of the present invention;

[0128] Figure 2 This is a schematic diagram of another multimodal CNN / Transformer image defect diagnosis and tracking decision-making process according to an embodiment of the present invention;

[0129] Figure 3 yes Figure 1 A flowchart illustrating another multimodal CNN / Transformer image defect diagnosis and tracking decision method in an embodiment of tracking the location of defects in an image of a product to be diagnosed;

[0130] Figure 4 yes Figure 1 A flowchart illustrating another multimodal CNN / Transformer image defect diagnosis and tracking decision-making method for tracking the location of defects in product images;

[0131] Figure 5This is a flowchart illustrating another multimodal CNN / Transformer image defect diagnosis and tracking decision-making method according to an embodiment of the present invention;

[0132] Figure 6 This is a schematic diagram of the structure of a multimodal CNN / Transformer image defect diagnosis and tracking decision system according to an embodiment of the present invention;

[0133] Figure 7 This is a schematic diagram illustrating how the TVS diode (SMBJ10CA) used to identify and detect smart meters according to an embodiment of the present invention detects the location of welding voids and measures their defect shape, size, and area using X-rays.

[0134] Figure 8(a) is a front X-ray schematic diagram of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0135] Figure 8(b) is a magnified X-ray schematic diagram of the front part of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0136] Figure 8(c) is a side X-ray schematic diagram of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0137] Figure 8(d) is a magnified X-ray schematic diagram of a partial side view of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0138] Figure 9 This is a schematic diagram illustrating how the rectifier diode (M7) of a smart meter, according to an embodiment of the present invention, detects the location of welding voids and measures their defect shape, size, and area using X-rays.

[0139] Figure 10 This is a schematic diagram illustrating the location tracking of welding cracks in a current transformer of a smart meter according to an embodiment of the present invention.

[0140] Figure 11(a) is a schematic diagram of the overall measurement dimensions of the chip, leads, lead-out terminals, substrate soldering holes, detection, and tracking of a smart meter according to an embodiment of the present invention.

[0141] Figure 11(b) is a partial schematic diagram of the location, detection, and tracking of the chip, leads, lead-out terminals, and substrate soldering voids of a smart meter according to an embodiment of the present invention.

[0142] Figure 11(c) is a schematic diagram of the chip, leads, lead-out terminals, substrate welding voids, and packaged body of a smart meter for identification, detection, and measurement according to an embodiment of the present invention.

[0143] Figure 12This is a schematic diagram illustrating the detection of protrusions, voids, cracks, and insufficient soldering in the PCB, crystal oscillator, and seven major chips of a smart meter according to an embodiment of the present invention.

[0144] Figure 13 This is a structural block diagram of a multimodal CNN / Transformer image defect diagnosis and tracking decision device according to an embodiment of the present invention;

[0145] Figure 14 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0146] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the multimodal CNN / Transformer image defect diagnosis and tracking decision method.

[0147] The relationship between images and defects is complex. When an image has a complex background and many interfering factors, the low contrast (C) and low contrast weight make it impossible to locate subtle texture defects (defects are not clearly visible). High-noise textures are chaotic and uncorrelated between pixels, and weld defects are negatively correlated with cold-state pixels, making it difficult or impossible to confirm the edges, extent, and shape of defects. Pixel gradients and image changes have a high positive correlation, making it easy to identify edges, extent, and shape. The correlation R is influenced by the complex relationship between defects, location texture, and weighting factors. The relationship and weighting of energy (E) with defects and location texture primarily reflect the concentration of image grayscale distribution. Image grayscale changes are mainly caused by defects, so the weight of energy features will be higher, and vice versa. The relationship and weighting of entropy (H) with defects and location texture reflect the amount of information in the image. Defect areas have complex structures or grayscale changes, increasing the entropy value. Normal textures have relatively stable entropy values, while defect entropy values ​​change significantly. Therefore, entropy features have a higher weight when detecting defect locations and texture changes. Existing image cleaning and resolution are not high. The error is caused by the complex relationship between the combined error of image parameters (contrast C, correlation R, energy E, entropy H, etc.) and defects. How to reduce the combined error and integrate the low contrast C, correlation R, energy E, entropy H, etc. with defect features is a difficult problem in the industry.

[0148] Existing depth imaging technologies, such as X-ray, ultrasound, infrared, and laser, suffer from attenuation at different depths and limitations of single-modal imaging. Current depth imaging techniques lack multi-parameter data fusion and reconstruction techniques to obtain more accurate 3D structural information (visualization) of the internal structure of power computing equipment. The challenge in improving image accuracy by combining zero-order moments, first-order moments, and second-order moments with the image's own multi-parameter and multi-modal fusion lies in establishing a model of the product's performance defects (including position and shape models) by combining the second-order moments of discrete data and continuous functions, and accurately allocating the weights of the second-order moments of discrete data and continuous functions. This severely restricts the realization of high-precision image online analysis and visualization modeling.

[0149] By combining non-destructive testing such as X-rays with visual image pattern recognition, reliable quality can be ensured for smart meters, automated metering terminals, power electronic components, transformers, PCB boards, and component welding connections. This enables new demands such as high-frequency data acquisition, dynamic adjustment of time-of-use electricity prices, proactive real-time perception of power outages and restorations, and low-voltage monitoring. It also addresses the shortcomings of existing technical equipment in terms of comprehensive status perception and flexible expansion and upgrade capabilities, establishes a complete technical standard system for electrical measurement enterprises, and achieves full coverage and collection of metering and billing data across the entire process of power generation, transmission, transformation, distribution, consumption, and computing power integration. This provides strong technical support for the marketing, production, operation, planning, and construction of power computing power and related products, fully leveraging the fundamental support and technological leadership role of metering and billing in modern power supply service systems, the integration of electricity, carbon, computing, and grids, digital grids, and new power systems.

[0150] This research addresses the challenges of surface roughness, subtle internal defects, and complex backgrounds in power electronics products, which hinder effective internal defect detection. It also addresses issues such as interference with optical and non-destructive testing equipment, resulting in poor image clarity, insufficient information, limited and chaotic information, and low spatial resolution. These limitations mean that equipment can only provide approximate defect location and characteristics, which is detrimental to computer-aided inspection, fault location, and diagnosis. Furthermore, it fails to accurately detect and locate minute changes in internal components and structural defects within products. Figure 1 The more detailed embodiment shown provides a multimodal CNN / Transformer image defect diagnosis and tracking decision method. By combining multimodal imaging of power electronics industrial products with deep learning of CNN / Transformer through internal structure visualization, the method achieves the effects of image defect detection, measurement, diagnosis and tracking.

[0151] This embodiment provides a multimodal CNN / Transformer image defect diagnosis and tracking decision-making method, which can be used in computer devices. Figure 2This is a flowchart of a multimodal CNN / Transformer image defect diagnosis and tracking decision method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0152] Step S101: Obtain defect location images of multiple multimodal data of the product to be diagnosed. The multiple defect location images are defect location images containing the above multimodal data taken by different production batches, different process stages and different equipment. The multimodal data includes image data, electrical data and operating condition data.

[0153] Specifically, the defect location image acquisition range includes: acquiring multiple defect location images of the product to be diagnosed; these multiple defect location images are defect location images taken from different production batches, different process stages, and different equipment.

[0154] Furthermore, the acquired defect location image area includes multimodal data of various data types; the definition of this multimodal data and the method steps for multimodal data fusion processing are as follows:

[0155] Multimodal data definition: In this embodiment of the invention, multimodal data refers to different types of data about the product to be diagnosed, collected from multiple dimensions and using various technical means, specifically covering the following categories:

[0156] (1) Image data:

[0157] Three-dimensional structural information: The three-dimensional structural information of the product to be decided is obtained through X-ray tomography, which can present the overall spatial structure of the product and its internal structure.

[0158] Thermal radiation information: Thermal radiation information of the product to be decided is obtained by thermal infrared imaging, reflecting the heat distribution on the surface and inside of the product.

[0159] Internal structural information: Information on the internal structure of the product to be decided can be obtained by using ultrasonic imaging, which can detect the physical structural features inside the product.

[0160] Microstructure images: Microstructure images acquired using the FIB-SEM dual-beam system, showcasing the microstructure of the product material at the micro level.

[0161] (2) Electrical data:

[0162] Electrical anomaly point data: Electrical anomaly point data located using OBIRCH technology, used to determine the specific location of anomalies in the product's electrical system.

[0163] Electrical parameter data: Electrical parameter data such as current, voltage, and resistance collected through nanoprobes reflect the electrical performance indicators of the product.

[0164] (3) Operating condition data: Real-time operating condition data of the product to be decided, including load fluctuation and ambient temperature data, reflecting the external conditions and load changes during product operation.

[0165] These data from different modalities describe product characteristics from multiple perspectives and scales, providing a rich information foundation for comprehensive analysis of product status, defect diagnosis, and decision-making.

[0166] Step S102: Perform data preprocessing and data alignment on the defect location images of multiple multimodal data.

[0167] Specifically, such as Figure 1 As shown, data preprocessing and alignment of defect location images from multiple multimodal datasets include:

[0168] 1. Timestamp synchronization: Align data from different sources, such as 3D structural information, thermal radiation information, and internal structural images, according to timestamps to construct a spatiotemporally consistent multimodal dataset.

[0169] 2. Image Enhancement: Preprocessing such as noise reduction and contrast enhancement is performed on 3D structural information and microstructure images to improve the quality of feature extraction.

[0170] 3. Feature standardization: Normalize electrical data (current / voltage / resistance) to eliminate dimensional differences.

[0171] Step S103: Based on the defect location image of each image type data, feature extraction is performed on the defect location images of multiple multimodal data using a position encoding method to obtain the image features after position encoding of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. The image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features.

[0172] Specifically, positional encoding adds positional information to the input sequence, transforming each defect location image into a form that can be effectively processed by a neural network model. Contrast feature, denoted as C, refers to the brightness difference between the brightest and darkest areas in each defect location image. Correlation feature, denoted as R, refers to the similarity or correlation between different images or different parts within the same image. Energy feature, denoted as E, is a concept based on neuropsychology or brain science, referring to the psychological impact of an image. It is the result of the combined effects of factors such as color, position, shape, and relationships within the image, which determine the image's psychological and physiological influence. Entropy feature, denoted as H, is an estimate of the image's "busyness," expressed as the bit average of the image's grayscale set, in bits per pixel, and also describes the average information content of the image source. Defect entropy value feature, denoted as D, is a measure of the "busyness" or information content of the defective portion of the image. Positional texture parameter features, denoted as P, include texture size, texture coordinates, and texture pixel data.

[0173] like Figure 1 As shown, image feature extraction:

[0174] (1) Location encoding extraction: contrast features, correlation features, energy features, entropy features, defect entropy value features and location texture parameter features.

[0175] (2) Use CNN shallow network to extract basic image features, including: extract edge / texture features: use ResNet-18 to extract edge features and texture features.

[0176] Step S104: Based on the preset feature weight allocation scheme, feature weight fusion is performed on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature as image features, and the image features, electrical features and working condition features are concatenated into a unified vector to obtain multiple multimodal fusion feature vectors.

[0177] Specifically, such as Figure 1 As shown, time-series feature transformation: convert electrical data into a time series and extract frequency domain features through Fourier transform.

[0178] Operating condition feature processing: Real-time data such as load fluctuations and ambient temperature are sampled using a sliding window to extract statistical features. A 5-minute sliding window is used for statistical analysis.

[0179] Based on a preset feature weight allocation scheme, feature weight fusion is performed on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and location texture parameter feature. The image features, electrical features and working condition features are then stitched together into a unified vector to obtain multiple multimodal fusion feature vectors.

[0180] Step S105: Based on multiple multimodal fusion feature vectors, the convolutional neural network model is trained and evaluated sequentially to obtain a single-process image defect detection model. Based on multiple multimodal fusion feature vectors, the Transformer network model is trained and evaluated sequentially to obtain a multi-process image defect detection model.

[0181] Specifically, based on multiple multimodal fusion feature vectors, the single-process convolutional neural network model and the multi-process Transformer network model are trained and evaluated sequentially to obtain an image defect detection model; multiple actual captured product images to be diagnosed are acquired and input into the image defect detection model for image defect diagnosis, and the detection results of whether there are defects or no defects in the corresponding areas of the image are obtained;

[0182] The specific path for the above two unified vectors using multimodal fusion is as follows: The multimodal fusion method of this invention is divided into two types: multimodal fusion of CNN image defects and multimodal fusion of Transformer image defects. Multimodal fusion of CNN image defects, together with other solutions of this invention, constitutes Invention Scheme 1; multimodal fusion of Transformer image defects, together with other solutions of this invention, constitutes Invention Scheme 2. The difference between the two is:

[0183] In early-stage fusion (feature level), image defects in multimodal fusion Transformers can be directly fused across modalities, whereas image defects in multimodal fusion CNNs need to be stitched together first.

[0184] In the later fusion stage (decision level), the image defects of the multimodal fusion Transformer can be directly fused to achieve the fusion of complex image defects from multiple CNNs.

[0185] The fusion path of image defects in CNN (simple image defects): early fusion (feature level): concatenating image features, electrical features, and operating condition features into a unified vector.

[0186] Late-stage integration (decision level):

[0187] (1) Train sub-models independently and merge the outputs.

[0188] (2) Train a CNN to process image data.

[0189] (3) Train LSTM to process time-series electrical data.

[0190] (4) Train the MLP to process working condition data.

[0191] (5) Output of weighted fusion model.

[0192] The fusion path of complex image defects in multiple processes using Transformer;

[0193] Early fusion (feature level) uses a self-attention mechanism to directly fuse across modalities, which has the advantages of not requiring feature concatenation and being able to capture semantic associations.

[0194] Decision-level late-stage integration:

[0195] Parallel processing of multi-channel CNN features and cross-attention fusion of complex defect features; outputting actual detection defect results based on multimodal fusion strategy and path; and performing hybrid modeling and shape analysis.

[0196] Step S106: Acquire multiple actual captured images of the product to be diagnosed and input them into a single-process image defect detection model and / or a multi-process image defect detection model for image defect diagnosis, and obtain the detection result that the corresponding area of ​​the product image has defects or no defects.

[0197] Step S107: Perform hybrid modeling and analysis on the product image with defects output by the image defect detection model to be diagnosed, and obtain hybrid modeled image defects. Hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0198] Specifically, shape model construction, i.e., second-order rectangular modeling, includes:

[0199] (1) Construct a shape model based on the second moment of discrete data and the second moment of continuous function.

[0200] (2) Calculate the second-order moment matrix of the discrete points.

[0201] (3) Fit the continuous function and calculate the second moment characteristics.

[0202] (4) Integrate the two moment features.

[0203] Defect quantification by fusing two rectangular feature shape models: Calculation of defect quantification parameters such as perimeter, area, and volume of the defect region based on geometric features and shape model. These parameters, based on shape measurement results, are then used for severity grading assessment.

[0204] Severity diagnosis is performed based on shape measurement results, followed by decision-level fusion and application.

[0205] Step S108: The defects in the hybrid modeled image are processed by the improved U-Net model to obtain the defect detection results, thereby achieving defect region segmentation and localization. Then, the location of the defect region in the image is tracked, and decisions are made based on the defect tracking and fault tracing results to optimize the production process parameters.

[0206] Specifically, such as Figure 1As shown, the defect detection results are obtained by applying the improved U-Net model to the hybrid modeled image defects, realizing defect region segmentation and localization. Then, the location of the image defect region is tracked, and decisions are made based on the defect tracking and fault tracing results to optimize production process parameters. This constitutes the decision-level fusion and application steps, including:

[0207] 1. Defect Diagnosis Network: Train an improved U-Net defect detection model, and obtain defect detection results by processing the fused feature image through the improved U-Net model, thereby achieving defect region segmentation and localization.

[0208] 2. Defect location tracking and fault tracing;

[0209] Process improvement decisions: Optimize and adjust production processes (such as water flow rate, air flow rate, temperature, pressure, current, and time parameters) based on defect tracking and fault tracing results.

[0210] The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by this invention also includes the following key technical features:

[0211] Multimodal data fusion: Spatiotemporal alignment ensures data consistency.

[0212] Feature extraction combination: CNN extracts local features + Transformer processes temporal correlations.

[0213] Decision-level applications: Closed-loop feedback from defect detection to process optimization.

[0214] Figure 1 It clearly demonstrates the entire process from data acquisition to decision optimization, highlighting the technical features of multimodal data fusion combined with CNN / Transformer models, and intuitively presenting the core logic of defect diagnosis and tracking decisions. Figure 1 The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by this invention will be described in detail below:

[0215] I. Multimodal Data Acquisition:

[0216] Acquire multiple defect location images of the product to be diagnosed; these multiple defect location images are taken from different production batches, different process stages, and different equipment; the multimodal data is defined as follows:

[0217] The multimodal data in this invention refers to different types of data about the product to be decided, collected from multiple dimensions and using various technical means, specifically covering the following categories:

[0218] 1. Image data

[0219] (1) Three-dimensional structural information: The three-dimensional structural information of the product to be decided is obtained through X-ray tomography, which can present the overall spatial structure of the product and its internal structure.

[0220] (2) Thermal radiation information: Thermal radiation information of the product to be decided is obtained by thermal infrared imaging, which reflects the heat distribution on the surface and inside of the product.

[0221] (3) Internal structure information: The internal structure information of the product to be decided can be obtained by using ultrasonic imaging, which can detect the physical structure characteristics of the product.

[0222] (4) Microstructure images: Microstructure images acquired using the FIB-SEM dual-beam system to show the microstructure of the product material.

[0223] 2. Electrical data:

[0224] (1) Electrical anomaly point data: Electrical anomaly point data located using OBIRCH technology, used to determine the specific location of anomalies in the product's electrical system.

[0225] (2) Electrical parameter data: Electrical parameter data such as current, voltage, and resistance collected by nanoprobes reflect the electrical performance indicators of the product.

[0226] 3. Operating condition data: Real-time operating condition data of the product to be decided, including load fluctuations and ambient temperature data, reflecting the external conditions and load changes during product operation.

[0227] These data from different modalities describe product characteristics from multiple perspectives and scales, providing a rich information foundation for comprehensive analysis of product status, defect diagnosis, and decision-making.

[0228] The multimodal data fusion framework of this invention includes a phased integrated fusion framework and a multi-strategy multimodal data fusion processing integration method, divided into five core stages from data acquisition to decision application: data preprocessing and alignment → feature extraction and transformation → multimodal fusion strategy → hybrid modeling and shape analysis → decision-level fusion and application. This framework achieves accurate detection and tracking of minute internal defects in electronic products through temporal alignment technology, cross-modal feature fusion algorithms, and dynamic weight allocation mechanisms.

[0229] II. Data Preprocessing and Alignment: Establishing a Unified Data Benchmark

[0230] 1. Timestamp Synchronization: Modal data from different sources, such as X-ray 3D structural information, thermal infrared radiation information, and ultrasonic internal structure images, are spatiotemporally aligned according to the acquisition timestamp to ensure consistency of multi-source data in the time dimension. For example, 3D CT images and thermal imaging data of the same product within 30 minutes after the welding process are aligned to avoid defect location errors caused by time misalignment.

[0231] Technical advantages: By building a spatiotemporal consistency system, the problem of temporal disorder in asynchronous data collection from multiple devices is solved, providing a unified time benchmark for subsequent feature fusion.

[0232] 2. Image enhancement preprocessing:

[0233] Preprocessing techniques such as median filtering for noise reduction and histogram equalization for contrast enhancement are applied to 3D CT images and FIB-SEM microstructure images to improve the clarity of feature defect edges. For example, in chip solder joint defect detection, the enhancement processing can improve the grayscale contrast of the poor solder joint area by more than 40%.

[0234] Technical advantages: Eliminates noise interference during image acquisition and improves the accuracy of subsequent feature extraction.

[0235] 3. Feature standardization: The electrical parameters such as current (mA level), voltage (mV level), and resistance (kΩ level) collected by the nanoprobe are Z-score normalized to map the data to the [-1,1] interval.

[0236] Technical advantages: Eliminates the influence of dimensional differences on the fusion model and avoids high-value-range features (such as resistance) from dominating the fusion process.

[0237] III. Feature Extraction and Transformation:

[0238] Image feature extraction: Based on the defect location of each defect location image, feature extraction is performed on the defect location image using a location encoding method to obtain the feature set of each defect location image after location encoding. The feature set includes contrast features, correlation features, energy features, entropy features, defect entropy value features, and location texture parameter features.

[0239] Load the pre-trained ResNet-18 model for CNN feature extraction, including: truncating the first 4 layers as a shallow feature extractor (extracting basic features such as edges and textures).

[0240] Initialize feature weight configuration (the importance of each feature can be customized).

[0241] 1. Feature extraction:

[0242] (1) Location coding feature extraction, including:

[0243] Region of interest extraction based on defect mask.

[0244] Calculate the features of the gray-level co-occurrence matrix.

[0245] Calculate image entropy and defect entropy (the difference in entropy between defective and normal regions).

[0246] Extract shallow features from CNN (using the first 4 layers of ResNet-18).

[0247] Extract edge features (Canny operator) and texture features (LBP operator).

[0248] Returns the integrated multimodal feature set.

[0249] (2) Basic feature extraction methods, including:

[0250] Gray-level co-occurrence matrix features: Convert to grayscale image and calculate GLCM, extract three key features: contrast, correlation and energy.

[0251] CNN shallow features: Image preprocessing and using ResNet-18 shallow convolutional layers to extract basic features such as image edges and textures. For example, in chip packaging defect detection, 3×3 convolutional kernels are used to capture crack edge features on the surface of solder joints.

[0252] Edge and texture features: By combining the Canny operator and the LBP texture operator, the geometric features of the defect area are extracted, such as the direction of cracks and the texture complexity of the wear area.

[0253] Among them, LBP (Local Binary Pattern) is used to extract texture features: the LBP histogram is calculated as the texture feature vector.

[0254] 2. Feature Fusion: Based on a preset feature weight allocation scheme, feature weights are fused for each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and location texture parameter feature to obtain multiple multimodal fused feature vectors; specifically including:

[0255] Multimodal features can be fused based on preset weights, various types of features can be fused according to weights, dictionary-type features (such as edge features containing multiple sub-features) can be processed, and scalar or vector features can be processed.

[0256] 3. Technical features and advantages:

[0257] (1) Multimodal fusion: integrates image geometric features, CNN features, texture features and statistical features to comprehensively describe defect characteristics.

[0258] (2) Location encoding: Based on defect masking, the region of interest is accurately located, improving the targeting of feature extraction.

[0259] (3) Configurable weights: Supports custom weights for each feature to adapt to different industrial scenarios.

[0260] (4) Hybrid feature processing: Unify the processing of scalar features (such as contrast) and vector features (such as CNN features).

[0261] (5) Industrial-grade precision: Combining traditional image processing with deep learning, it is suitable for detecting defects in precision manufacturing such as chips.

[0262] 4. Typical application scenarios: solder joint defect detection in semiconductor chip manufacturing, circuit breakage detection on PCB circuit boards, surface wear analysis of precision mechanical parts, internal structural defect diagnosis in battery production, and real-time monitoring of equipment under high temperature and high pressure environments.

[0263] 5. Time-series feature conversion: Convert real-time current and voltage data into time series, and extract frequency domain features (such as the proportion of harmonic components) through Fourier transform. For example, in the defect detection of electrolytic capacitors, frequency domain features can reflect abnormal changes in internal dielectric loss.

[0264] 6. Operating condition feature processing: Statistical features such as mean and variance are extracted from data such as load fluctuation and ambient temperature using a sliding window (window size 5 minutes). For example, in the monitoring of smart meter operation status, the variance of temperature fluctuation can be used as an early warning indicator of thermal defects.

[0265] IV. Multimodal Fusion Strategy: Technical Breakdown of CNN and Transformer:

[0266] This invention's multimodal fusion is divided into two types: image defects in single-process (single) defect multimodal fusion CNNs and image defects in multi-process multimodal fusion Transformer models. Image defects in multimodal fusion CNNs, together with other solutions of this invention, constitute Invention Solution 1; image defects in multimodal fusion Transformer models, together with other solutions of this invention, constitute Invention Solution 2. Detailed technical breakdowns of both are shown in Table 1 below:

[0267] Table 1

[0268]

[0269]

[0270]

[0271]

[0272]

[0273]

[0274]

[0275] Typical application scenarios of multimodal fusion Transformer: a case study of its implementation in smart meter manufacturing.

[0276] Taking multimodal detection of smart meter chip flip-chip bonding as an example, in the flip-chip bonding process of metering chips using 0.13μm technology, a multimodal fusion Transformer can be used to achieve this, specifically including:

[0277] Image modality: AOI detection of solder ball offset (accuracy ±0.01mm).

[0278] Electrical mode: Micro probe test of solder joint contact resistance (resolution 1mΩ).

[0279] Thermal mode: The infrared thermal imager captures the temperature field at the moment of welding (accuracy ±1℃).

[0280] After fusion by Transformer, composite defects such as "solder ball misalignment + critical contact resistance + local overheating" that are missed by traditional methods can be identified. For example, a combination defect of 0.03mm solder ball misalignment, 15mΩ contact resistance (critical value 10mΩ), and 230℃ welding temperature (standard 250℃) can be found. The miss rate of such defects in traditional single-mode inspection is as high as 30%, while the Transformer inspection solution can reduce the miss rate to less than 1%.

[0281] The Transformer model can be integrated to manage the entire lifecycle quality control of sampling component welding, specifically for the relay copper plate welding of the sampling circuit of smart meters.

[0282] Production stage: X-ray inspection of solder joint porosity (threshold <5%).

[0283] Testing phase: Contact resistance stability during 100A high-current on / off test.

[0284] Operational phase: Temperature, humidity, and load data collected on-site.

[0285] By using long-distance dependency modeling, Transformer can identify solder joints with a porosity of 4.5% (close to the threshold) during the production stage. After one year of operation, these joints experience early failure due to porosity expansion caused by temperature cycling, ultimately leading to a sharp increase in contact resistance. After applying this technology, a meter factory reduced the field failure rate related to welding in the sampling circuit from 0.8% to 0.15%, saving 2 million yuan in after-sales maintenance costs annually.

[0286] V. Hybrid modeling and shape analysis, i.e., precise characterization of defect geometry:

[0287] The shape of the defective area in the product image is measured and calculated to diagnose the severity level of the image defect and to track and decide the location of the defective area.

[0288] 1. Modeling a second-order rectangle:

[0289] Discrete data second moment: The second moment matrix of the profile is calculated based on the pixel coordinates of the binary image of the defect region. The principal direction and eccentricity of the defect can be obtained. For example, the discrete second moment of the crack defect can reflect its extension direction.

[0290] Second moment of continuous function: By fitting the defect profile with a Gaussian function, the second moment in continuous space is calculated, which improves the accuracy of describing irregular defect shapes. For example, continuous modeling of wear areas can reduce shape distortion caused by discrete sampling.

[0291] 2. Fusion Strategy: A weighted average method is used to fuse the two moment features. The formula is as follows:

[0292] M fusion =0.6M C +0.4M d ;

[0293] Among them, M C M is the second moment of a continuous function. d The second moment of discrete data can reduce the error in defect shape description by 30%.

[0294] 3. Geometric feature calculation: Based on the fused second-order moment model, the perimeter, area, volume and other parameters of the defect region are calculated. For example, in multilayer chip defects, the severity of void defects can be quantified by three-dimensional volume calculation.

[0295] VI. Decision-level integration and application:

[0296] Defect Diagnosis Network: The improved U-Net model (which integrates residual connections and attention mechanisms) inputs fused features to achieve pixel-level segmentation of defect areas. For example, in the detection of defects in lithium battery electrodes, this model can accurately segment micron-level coating peeling areas.

[0297] 1. Process improvement decisions:

[0298] Welding process optimization: By combining multimodal data (such as the three-dimensional morphology of X-ray weld joints and the stress distribution of ultrasonic welding), the optimal laser power (error ±5W) and welding speed (error ±0.1mm / s) are determined through finite element simulation, so that the failure rate is reduced to below 0.3%.

[0299] Material durability assessment: By combining accelerated aging tests (high temperature 85℃ / high humidity 85% RH) with CNN image detection, a correlation model between the aging degree of metal film resistors and resistance value drift is established to provide data support for material selection.

[0300] 2. Summary of Technological Innovations:

[0301] Breakthrough in cross-modal fusion efficiency: Compared with traditional CNN, the Transformer strategy reduces feature splicing loss by 40% in multimodal feature association modeling, making it suitable for deep semantic understanding of complex defects.

[0302] Improved accuracy through hybrid modeling: The discrete-continuous second-moment fusion model overcomes the limitations of traditional single modeling methods in describing irregular defects, making the measurement error of 3D defect volume ≤5%.

[0303] Decision-level application closed loop: The entire process from data acquisition to process optimization is integrated to achieve a complete closed loop of "detection-tracking-source tracing-improvement" for electronic product defects. Compared with traditional single-modal detection solutions, the efficiency of process improvement is increased by more than 2 times.

[0304] According to an embodiment of the present invention, a method for diagnosing and tracking image defects using a multimodal CNN / Transformer is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0305] This embodiment provides a multimodal CNN / Transformer image defect diagnosis and tracking decision method, which can be used in computer devices, such as... Figure 3 This is a flowchart of a multimodal CNN / Transformer image defect diagnosis and tracking decision method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0306] Step S201: Obtain defect location images of multiple multimodal data of the product to be diagnosed. The multiple defect location images are defect location images containing the above multimodal data taken from different production batches, different process stages and different equipment. The multimodal data includes image data, electrical data and operating condition data.

[0307] Specifically, the products to be diagnosed include, but are not limited to, smart meters, metering automation terminals, instrument transformers, batteries, transformers and wires, power electronic components, multilayer chips, transistors, instrument transformers and PCBs, component welding connections and processes, existing welding structures and semiconductor packages such as SOP, QFP, BGA, CSP, and IGBT, industrial devices, and defects such as internal discontinuities in batteries (microcracks, porosity, etc.), cracks, voids, delamination, deformation, separation, bridging, open circuits in metal layers (wires, cores), fractures, dust, foreign matter and impurities, metal particles, loosening, misalignment, and damaged insulation layers. Images of the defect locations of the products to be diagnosed, taken from different production batches, different process stages, and different equipment, are obtained. Simultaneously, label data corresponding to each image is collected, such as whether the product has a defect and the type of defect.

[0308] Step S202: Perform data preprocessing and data alignment on the defect location images of multiple multimodal data.

[0309] Specifically, step S202 includes:

[0310] Step S2021 involves performing data preprocessing on the defect location image in sequence, including data cleaning, data annotation, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization.

[0311] Among them, data cleaning is used to remove noise data, blurred defect location images, and incomplete defect location images from defect location images; data annotation uses a combination of manual annotation and semi-automatic annotation to annotate defect locations and defect types; image usage classification divides multimodal data into image data, electrical data, and operating condition data according to the production batch, process stage, equipment type, and multimodal data type to which the image belongs; image pixel normalization normalizes the pixel values ​​of image data to a preset range to unify the scale of image data.

[0312] Image pixel normalization refers to normalizing image pixel values ​​to a specific range, such as [0,1] or [-1,1]. This helps to speed up the training of subsequent convolutional neural network models and improve their stability. For a certain pixel value x, normalization can be performed using the following formula:

[0313]

[0314] Where, x min and x max These are the minimum and maximum values ​​of the pixel among all pixels, respectively.

[0315] Step S2022: Align, merge, and standardize image data, electrical data, and operating condition data according to timestamps to construct a spatiotemporally consistent dataset.

[0316] Specifically, modal data from different sources, such as X-ray three-dimensional structural information, thermal infrared radiation information, and ultrasonic internal structure images, are spatiotemporally aligned according to the acquisition timestamp to ensure consistency of multi-source data in the time dimension. For example, three-dimensional CT images and thermal imaging data of the same product within 30 minutes after the welding process are aligned to avoid defect location deviations caused by time misalignment.

[0317] Step S203: Based on the defect location image of each image type data, feature extraction is performed on the defect location images of multiple multimodal data using a position encoding method to obtain the image features after position encoding of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. The image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features.

[0318] Specifically, the operating condition data includes load fluctuations and ambient temperature, and step S203 above includes:

[0319] Step S2031: Use a shallow CNN network to extract basic edge features and texture features from the defect location image.

[0320] Specifically, a shallow CNN network is used to extract basic image features, including: extracting edge / texture features: using ResNet-18 to extract edge and texture features.

[0321] Step S2032: Determine the defect center position of each defect location image based on edge features and texture features; obtain the two-dimensional coordinate value of each defect center position and discretize each two-dimensional coordinate value; convert each discretized coordinate value into a feature vector through one-hot encoding, and use the feature vector as the image feature after position encoding of each defect location image.

[0322] Specifically, drawing tools, such as CAD software, are used to determine the center position of each defect in the image and to obtain the two-dimensional coordinates of each defect center position. The two-dimensional coordinates are represented by (x... d y d ) is represented, and then the discrete coordinate method is used to represent the two-dimensional coordinate values ​​(x). d y d Discretize it.

[0323] One-hot encoding transforms each coordinate value after discretization into a vector form. For example, dividing the defect location image into grid regions, one-hot encoding is performed on the grid region where the defect center is located, resulting in a vector of length m. 2The vector is used as the feature set after position encoding.

[0324] Step S2033: Convert electrical data into time series and extract frequency domain features as electrical features through Fourier transform; perform 5-minute sliding window sampling on real-time data of load fluctuation and ambient temperature in the operating condition data to extract operating condition features.

[0325] Step S204: Based on the preset feature weight allocation scheme, feature weight fusion is performed on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and location texture parameter feature as image features, and the image features, electrical features and working condition features are concatenated into a unified vector to obtain multiple multimodal fusion feature vectors.

[0326] Specifically, step S204 includes:

[0327] In step S2041, the feature weight allocation scheme is set as follows: the weight allocation for contrast features is 20% to 30%, the weight allocation for correlation features is 10% to 25%, the weight allocation for energy features is 10% to 25%, the weight allocation for entropy features is 30% to 40%, the weight allocation for defect entropy value features is 10% to 20%, and the weight allocation for position texture parameter features is 10% to 30%. The sum of the weights for contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features is equal to 1.

[0328] Furthermore, let the input image of the product to be diagnosed be I. d The smart meter image is I B The contrast characteristic is denoted as C(I). d The correlation feature is denoted as R(I). d The energy characteristic is denoted as E(I). d The entropy feature is denoted as H(I). d The defect entropy value characteristic is denoted as D(I). d The positional texture parameter feature is denoted as P(I). d ).

[0329] Contrast characteristics C(I) B If contrast C(I) is found B It has a strong ability to distinguish between normal and defective areas. For example, when the meter display shows defects such as blurriness, the contrast characteristic C(I) is strong. B If the difference between the defective area and the surrounding normal area is clearly visible, then a contrast feature C(I) is assigned. B A relatively high initial weight range, such as 20% to 30%. For some minor defects (such as fine scratches), if the contrast feature C(I) BIf the change is not significant, then take the lower value within the weight range of 20% to 30%.

[0330] For the correlation feature R(I) B ): When the correlation feature R(I) B When pixel correlation is significantly reduced around the defect area (e.g., the defect disrupts texture continuity), a weight range of 15% to 25% can be assigned. If the correlation R(I) B If the change is not significant for most defect types, the weight range should be appropriately reduced to 10% to 20%.

[0331] For energy characteristic E(I) B If the energy E(I) B When a defect occurs, it exhibits a specific pattern of change (e.g., a certain defect always leads to an increase or decrease in the energy of a specific frequency component), and is assigned a weight of 15% to 25%. If the energy characteristic E(I) B The changes are complex and have no obvious pattern; the weight can be set to 10% to 20%.

[0332] For entropy feature H(I) B ): Since the entropy characteristic is denoted as H(I) B This is related to the uncertainty of image information. For complex textures or mixed defects, if the entropy feature is denoted as H(I)... B This can well reflect these situations and can be assigned a weight of 30% to 40%. If the entropy feature is denoted as H(I... B If the performance is unstable in some simple defect scenarios, the weight should be adjusted appropriately within the range of 30% to 40%.

[0333] For the defect entropy feature D(I) B If the defect entropy value is a key factor in distinguishing defects of different severity (e.g., a high entropy value defect may indicate a more serious internal circuit fault in the smart meter), then a separate weight should be assigned to the defect entropy value. In addition to the total weight, 10% to 20% of the weight can be specifically allocated to adjust accuracy based on the defect entropy value. Defects with high entropy values ​​and significant impact on meter function should be given higher weight.

[0334] For the positional texture parameter feature P(I) B When a defect is located in a critical part of a smart meter (such as the display screen, around the metering chip, etc.), its weight should be higher than that of defects in non-critical parts. Depending on the importance of the location, the weight of the location factor can be allocated between 10% and 30%. Meanwhile, for areas with obvious texture features (such as specific patterns on the meter casing), if texture changes are significant for defect detection, the weight of texture-related parameters (such as correlation features R(I)) can be increased. B ), entropy feature H(I) BThe weights of texture variations (such as texture) should be adjusted appropriately to highlight their impact on accuracy.

[0335] Different types of defects (such as scratches, wear, display abnormalities, etc.) are detected, and then a preliminary weight range is determined based on the different types of defects.

[0336] The preset feature weight allocation scheme is set according to the actual situation. For example, if the input image of the defect location of the product to be diagnosed is a high-precision chip defect location image, denoted as I... d Then its weight allocation scheme is: contrast feature C(I d The weight is 0.25, and the correlation feature R(I) d The weight is 0.15, and the energy feature E(I) d The weight is 0.12, and the entropy feature H(I) d The weight is 0.20, and the defect entropy feature D(I) d The weight is 0.25, and the positional texture parameter feature P(I) is... d The weight is 0.03. During the feature fusion stage, feature weights are fused according to the given weight allocation scheme, resulting in a feature vector F(I) after feature weight fusion. d ).

[0337] Step S2042: Calculate the first product corresponding to each contrast feature and contrast feature weight, the second product corresponding to each correlation feature and correlation feature weight, the third product corresponding to each energy feature and energy feature weight, the fourth product corresponding to each entropy feature and entropy feature weight, the fifth product corresponding to each defect entropy value feature and defect entropy value feature weight, and the sixth product corresponding to each positional texture parameter feature and positional texture parameter feature weight; add the first, second, third, fourth, fifth, and sixth products of each corresponding feature to obtain multiple multimodal fusion feature vectors.

[0338] For smart meter images, during the feature fusion stage, fusion is performed according to a given weight allocation scheme. Let the fused feature vector be F(I). B ),but:

[0339] F(I B )=(20% to 30%)×C(I B )+(10% to 25%)×R(I B )+E(I d )×(15% to 25)+

[0340] H(I B )×(30% to 40%)+D(I B )×(10% to 20%)+P(I B)×(10% to 30%))(2);

[0341] The principle of weight allocation is that the sum of the weights of each feature must be equal to 1.

[0342] For example, when the defect image of the product to be diagnosed is a high-precision chip, the final weight allocation scheme is: contrast feature C(I d The weight is 0.25, and the correlation feature R(I) d The weight is 0.15, and the energy feature E(I) d The weight is 0.12, and the entropy feature H(I) d The weight is 0.20, and the defect entropy feature D(I) d The weight is 0.25, and the positional texture parameter feature P(I) is... d The weight is 0.03. During the feature fusion stage, feature weights are fused according to the given weight allocation scheme, resulting in a feature vector F(I) after feature weight fusion. d The formula is as follows: F(I) d )=0.25×C(I d )+0.15×R(I d )+E(I d )×0.12+H(I d )×0.2+D(I d )×0.25+P(I d )×0.03(3).

[0343] Step S2043 involves concatenating multiple multimodal fusion feature vectors with electrical features and operating condition features to form an early fusion vector.

[0344] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided in this embodiment weights and fuses multiple features such as contrast, correlation, energy, entropy, and defect entropy value, which can fully leverage the role of each feature in defect detection. By assigning weights based on the effectiveness of different features in distinguishing different types of defects, the fused feature vector becomes more discriminative and meets diagnostic requirements. It considers the characteristics of different image types and the performance of different features under different defect types, making the fusion scheme more targeted and better adaptable to specific detection tasks, thus meeting the need for high adaptability.

[0345] Step S205: Based on multiple multimodal fusion feature vectors, the convolutional neural network model is trained and evaluated sequentially to obtain a single-process image defect detection model. Based on multiple multimodal fusion feature vectors, the Transformer network model is trained and evaluated sequentially to obtain a multi-process image defect detection model.

[0346] Specifically, a convolutional neural network model includes an input layer, multiple convolutional and pooling layers, multiple fully connected layers, and an output layer. Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computation and has a deep structure. CNNs mainly analyze visual images by extracting and recognizing image features. By sequentially training and evaluating the CNN model using multiple multimodal fusion feature vectors, the CNN model gains the ability to detect image defects, thus obtaining an image defect detection model.

[0347] Step S205 above includes:

[0348] Step S2051: Multiple multimodal fused feature vectors are fed into multiple convolutional and pooling layers through the input layer for deep learning. The convolutional layers use convolutional kernels to perform convolution operations on the multimodal fused feature vectors to extract local features. The pooling layers obtain multiple key features through downsampling. Multiple fully connected layers integrate these key features, converting the extracted features from local information to global information, and mapping them to the output probability distribution through an activation function. The output layer transforms the mapped probability distributions into multiple fused probability distribution images and outputs them. The value of each pixel in the fused probability distribution image represents the probability of a defect existing at that location.

[0349] Specifically, such as Figure 5 As shown, the input layer receives multimodal fused feature vectors. Its number of nodes is equal to The number of dimensions.

[0350] Convolutional and Pooling Layers: Multiple convolutional and pooling layers are used to automatically extract high-level information, i.e., key features, from the features. This embodiment uses three convolutional layers with kernel sizes of 3×3, 3×3, and 5×5, each with a stride of 1. Each convolutional layer is followed by a pooling layer with a 2×2 kernel and a stride of 2. The activation function used for the convolutional layers is the ReLU (Rectified Linear Unit) function, defined as: f(x) = max(0, x). In this formula, if x is less than or equal to 0, the function value f(x) is 0; if x is greater than 0, the function value f(x) is x. This activation function increases the model's non-linear expressive power, where x represents the feature vector. Each feature.

[0351] Fully connected layers: After key feature extraction via convolutional and pooling layers, several fully connected layers are used to further integrate information and map it to the output probability distribution. This embodiment uses two fully connected layers. The first fully connected layer has k nodes (a suitable value can be determined experimentally, such as 128), and the second fully connected layer has n nodes, corresponding to the number of elements P = [p0, p1…p...] in the probability distribution of the image type output by the neural network model. n ], where n is the number of image types, P i This represents the probability that an image belongs to the i-th type, and the type with the highest probability is the predicted image type. To alleviate gradient vanishing and improve computational efficiency, neuron death, and model overfitting issues in multimodal and high-resolution images, and to filter unfavorable negative signals, the activation function of the fully connected layer can be the RReLU function, f(x) = max:(ax,x), where a is a learnable, adaptively adjusted slope parameter averaged after training, and x represents the feature vector. Each feature.

[0352] The output layer uses the Softmax function (a common activation function in multi-class classification problems) as the activation function, transforming the output of the fully connected layer into a probability representation. For the output Z of the i-th neuron in the output layer... i The output after processing by the Softmax function is: This ensures that the sum of all probability values ​​in the output is 1, which meets the requirements of the probability distribution.

[0353] Where: P i This represents the probability value output by the i-th neuron after processing by the Softmax function. This probability value represents the likelihood that the input sample belongs to the i-th class. For example, in an image classification task with 10 classes, when i=3, it means that the image belongs to the 3rd class.

[0354] z j This represents the raw output value of the j-th neuron in the fully connected layer before it is input into the Softmax function. When calculating the probability of the j-th output, all values ​​(j from 1 to n) need to be calculated. These raw output values ​​contain information about the model's judgment for each class, but are not yet in probabilistic form. n represents the total number of classes. In classification tasks, the total number of classes is fixed, representing the number of all possible classes. For example, for a five-class classification problem, n = 5, meaning that when calculating the probability of each class, the raw output values ​​of all five classes (j = 1, 2, 3, 4, 5) need to be considered.

[0355] Initialize parameters: Initialize parameters such as convolution kernels and biases in the network. Common initialization methods include random initialization (such as Xavier initialization (for neural network weights) and He initialization (for ReLU activation functions)). These initialization methods can help the model converge faster.

[0356] Step S2052: Divide multiple fused probability distribution images into training set, validation set and test set according to a preset ratio; input the training set into the convolutional neural network model to train independent sub-models in sequence, and perform weighted fusion of the outputs of each independent sub-model to obtain the detection model to be evaluated; evaluate the detection model to be evaluated based on the test set and output the evaluation result. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model.

[0357] In an optional implementation, step S2052 includes:

[0358] The training set, validation set, and test set are divided into a ratio of 7:2:1 or 8:1:1.

[0359] A unified vector of early-stage multimodal fusion image defect features is input into a convolutional neural network for later-stage fusion. A convolutional neural network model is trained specifically to process image data to extract defect features from images. A long short-term memory network model is trained to process time-series electrical data to capture the patterns of electrical data changes over time. A multilayer perceptron model is trained to process operating condition data to analyze the impact of operating conditions on defects. The outputs of the independent sub-models obtained above are weighted and fused to obtain the detection model to be evaluated. The test set is input into the detection model to be evaluated. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single-process image defect detection model, and the evaluation result is used as the final defect diagnosis result.

[0360] Step S2053: During training, the loss is calculated based on a preset loss function, and the Adam optimizer is used for backpropagation to adjust the weights and biases of the convolutional neural network model. Based on the validation set, the parameters of multiple convolutional layers, pooling layers, and multiple fully connected layers are adjusted to obtain the convolutional neural network model to be evaluated. The Adam optimizer combines the momentum method and the Adagrad optimization algorithm to dynamically adjust the learning rate of each parameter in the model based on the first-order moment estimate and the second-order moment estimate of the gradient, and corrects the first-order moment estimate and the second-order moment estimate through the bias.

[0361] Specifically, the preset loss function uses the cross-entropy loss function to measure the difference between the probability distribution predicted by the model and the actual probability distribution, and its expression is:

[0362]

[0363] Among them, y i y represents the elements in the actual probability distribution p, i.e., the actual labels. In classification problems, if it is a one-hot encoding, such as for a three-class classification problem, where the class labels are [0,0,1] indicating belonging to the third class, then y i It is 1 when i=3, and 0 in other positions. It represents the probability of the sample's true class (true probability distribution), and i is the sum of numbers from 1 to j, used to calculate the cross-entropy loss.

[0364] Where, p i `predicted` represents the true probability value and the probability value predicted by the model, respectively. For example, if the model predicts that a sample belongs to the first class with a probability of 0.2, to the second class with a probability of 0.3, and to the third class with a probability of 0.5, these probability values ​​are p. i , predicted(i=1,2,3), is the output after passing through the convolutional neural network and the Softmax function. It is used to compare with the probability corresponding to the actual label to measure the quality of the model's prediction.

[0365] By using the Softmax function and minimizing the cross-entropy loss, the convolutional neural network model can effectively learn the correct classification boundary.

[0366] The training process is as follows: Training set data is input into the constructed single-process image defect detection model. The output is calculated via forward propagation, and the loss is calculated based on the loss function. Then, the network parameters are updated via backpropagation using an optimizer. During training, model parameters are periodically saved, and model performance is evaluated on the validation set. Training parameters (such as learning rate and number of training epochs) are adjusted based on the validation results. In this embodiment, training set data is used to train the convolutional neural network model. The backpropagation algorithm combined with optimization algorithms and the Adam optimizer (improved stochastic gradient descent) is used to adjust the model's weights and biases, continuously reducing the value of the loss function, i.e., continuously reducing the difference between the model's predicted probability distribution and the actual probability distribution. During training, the parameters (such as kernel size, stride, and number of nodes) of the convolutional, pooling, and fully connected layers can be adjusted based on the training results (such as accuracy and loss values ​​on the validation set) to improve model performance.

[0367] A detailed description of the Adam optimizer (an improved stochastic gradient descent method):

[0368] The Adam optimizer maintains two moving average estimators: the first moment estimator m and the second moment estimator v.

[0369] First, at each iteration t, for a weight of the convolutional layer with parameter θ... The formula for calculating the gradient is as follows:

[0370]

[0371] Among them, g t Let θ be the gradient, and θ be a parameter. The gradient is typically represented as the gradient of the loss function L with respect to the parameter θ (here, it's an averaged gradient, or the average of gradients over a mini-batch of data, or other forms of gradient processing). The loss function L measures the difference between the model's predictions and the true labels; when training machine learning models such as neural networks, our goal is to minimize this loss function. The gradient... This represents the rate of change of the loss function in the parameter space, indicating how the parameters should be adjusted to reduce the loss function.

[0372] To calculate the gradient, first calculate the gradient g of the loss function L with respect to the parameters θ (specifically, a weight w in the convolutional layer). t This gradient calculation, based on the current batch of data, indicates the direction in which the loss function grows fastest at the current position of the parameter.

[0373] Update the first moment estimator (momentum term):

[0374] m t =β t m t-1 +(1-β1)g t (6);

[0375] Where β1 is a hyperparameter, set to 0.9, β t It is a function used to control the previous momentum information m t-1 The parameter that assigns weight when updating the current momentum. When β t When it is close to 1, it indicates that m is being updated. t At that time, it relies more on the previous momentum m t-1 When β t When the gradient is close to 0, the current gradient information (1-β1)g will be given more importance. t .

[0376] In the given formula, the hyperparameter is only defined as β1 = 0.9, where β t It is a parameter similar to β1, and in practical applications (such as optimization algorithms like Adam), β t Typically, β is also a number within the (0,1) interval, used to balance the contributions of historical and current information to the momentum term update. If this is during a specific algorithm iteration, β...t It may also change with the number of iterations; the specific rules for this change need to be determined based on the detailed settings of the algorithm. Formula (6)g t Used to update the first moment estimate m t m t It can be viewed as the gradient g t This is a weighted moving average that accumulates information from past gradients, analogous to momentum with friction. This momentum term helps accelerate convergence, especially when dealing with parameter updates with similar gradient directions. Initially, m0 is typically set to 0.

[0377] Update the second moment estimator:

[0378]

[0379] Here, β1 is a hyperparameter, which is set to 0.999.

[0380] The bias corrections for the first-order moment estimators and second-order moment estimators are as follows:

[0381]

[0382] Finally, update the parameters using the following formula:

[0383]

[0384] Where η is the learning rate, and ∈ is a very small number to prevent the denominator from being 0.

[0385] The improved Adam optimizer combines the momentum method (using the first moment estimate m) and the idea of ​​Adagrad (using the second moment estimate v). It can dynamically adjust the learning rate of each parameter in the model based on the first and second moments of the gradient, and achieve good learning results in the early stages of training through bias correction.

[0386] Step S2054: Early fusion of the image defect fusion path of the multimodal fusion Transformer. The self-attention mechanism is used to effectively capture the semantic association between features of different modalities and directly perform cross-modal early fusion of multimodal features. The cross-attention mechanism is used to process the local features extracted by multi-path convolutional neural networks from different sources in parallel to fuse the complex defect feature detection model. The complex defect feature detection model and the detection model that processes temporal associations by Transformer are combined to form a combined detection model. The image to be diagnosed is input into the combined detection model and the final defect diagnosis result is output.

[0387] For example, the obtained image defect detection model is deployed to the production and application environment, and the image of the product to be diagnosed captured by the detection equipment is input into the image defect detection model to achieve real-time detection. Taking binary classification to determine whether there is a defect, the detection result of whether there is a defect or not in the corresponding area of ​​the image is obtained.

[0388] Evaluation using a test set involves using a test set to evaluate the trained model after training is complete. Evaluation metrics such as accuracy, precision, recall, mean intersection-over-union ratio, and F1-score are calculated to assess the model's performance on unseen data.

[0389] Taking the performance evaluation indicators of image accuracy for products to be diagnosed as an example, the calculation formulas for evaluation indicators such as accuracy, precision, recall, and F1-score are as follows:

[0390] Accuracy:

[0391]

[0392] Wherein, TP (True Positive) represents the number of samples correctly detected as positive (e.g., correctly detecting a defective chip image); TN (True Negative) represents the number of samples correctly detected as negative (e.g., correctly detecting a defect-free chip image); FP (False Positive) represents the number of samples incorrectly detected as positive (actually defect-free but detected as defective); and FN (False Negative) represents the number of samples incorrectly detected as negative (actually defective but detected as defect-free).

[0393] Precision:

[0394]

[0395] Precision measures the proportion of truly positive samples out of those that are detected as positive, reflecting the reliability of the detection results.

[0396] Recall:

[0397]

[0398] Recall rate represents the proportion of actual positive samples that are correctly detected, reflecting the model's ability to capture positive samples.

[0399] Mean Intersection over Union (mIoU):

[0400]

[0401] Suppose that k categories (e.g., different types of defects or normal areas are considered different categories) are detected in the product image to be diagnosed. intersection(i) represents the area of ​​intersection between the predicted result and the true label in the i-th category, and Union(i) represents the area of ​​union between the predicted result and the true label in the i-th category.

[0402] Step S206: Acquire multiple actual captured images of the product to be diagnosed and input them into the single-process image defect detection model and / or multi-process image defect detection model for image defect diagnosis, obtaining detection results indicating whether defects exist or not in the corresponding areas of the product images. For details, please refer to... Figure 1 Step S106 of the illustrated embodiment will not be described again here.

[0403] Step S207 involves performing hybrid modeling and analysis on the product image with defects output by the defect detection model from the input image to be diagnosed, resulting in a hybrid modeled image defect. Hybrid modeling and analysis includes shape model construction, defect quantification, and severity diagnosis. For details, please refer to [link to relevant documentation]. Figure 1 Step S107 of the illustrated embodiment will not be described again here.

[0404] Step S208 involves using an improved U-Net model to obtain defect detection results from the hybrid modeled image defects, achieving defect region segmentation and localization. This allows for the tracking of defect locations within the image, and decision-making based on defect tracking and fault tracing results to optimize production process parameters. For details, please refer to [link to details]. Figure 1 Step S108 of the illustrated embodiment will not be described again here.

[0405] The multimodal CNN / Transformer image defect diagnosis, tracking, and decision-making method provided in this embodiment determines the encoded feature set through positional encoding and uses a preset feature weight allocation scheme to fuse the feature weights of each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and positional texture parameter feature in the feature set, resulting in multiple multimodal fused feature vectors. Based on these multiple multimodal fused feature vectors and a convolutional neural network model, an image defect detection model is obtained. Applying the image defect detection model to a real-world scenario yields detection results indicating whether a defect exists or not in the corresponding region of the image. This enables accurate detection and localization of minute component changes and structural defects within the product, improving detection accuracy. The method also performs shape measurement calculations on the image of the product to be diagnosed, which contains defects, to diagnose the severity level of the image defect and track the location of the image defect region. This achieves the diagnosis and tracking of image defect locations, solving the problem of the inability to accurately detect, locate, diagnose, track, and make decisions regarding minute component changes and structural defects within electronic products.

[0406] This embodiment provides a multimodal CNN / Transformer image defect diagnosis and tracking decision-making method, which can be used in computer terminals, such as central processing units, servers, etc. Figure 4 This is a flowchart of a multimodal CNN / Transformer image defect diagnosis and tracking decision-making method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0407] Step S301: Acquire defect location images from multiple multimodal data of the product to be diagnosed. These multiple defect location images are taken from different production batches, different process stages, and different equipment, containing the aforementioned multimodal data. The multimodal data includes image data, electrical data, and operating condition data. For details, please refer to... Figure 3 Step S201 of the illustrated embodiment will not be described again here.

[0408] Step S302 involves preprocessing and aligning the defect location images from multiple multimodal datasets. For details, please refer to [link to relevant documentation]. Figure 3 Step S202 of the illustrated embodiment will not be described again here.

[0409] Step S303: Based on the defect location image of each image type, feature extraction is performed on the defect location images of multiple multimodal data using a location encoding method to obtain the location-encoded image features of each defect location image. Simultaneously, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. Image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and location texture parameter features. For details, please refer to [link to details]. Figure 3 Step S203 of the illustrated embodiment will not be described again here.

[0410] Step S304: Based on a preset feature weight allocation scheme, feature weights are fused for each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and location texture parameter feature to obtain image features. The image features, electrical features, and operating condition features are then concatenated into a unified vector to obtain multiple multimodal fused feature vectors. For details, please refer to [link to relevant documentation]. Figure 3 Step S204 of the illustrated embodiment will not be described again here.

[0411] Step S305: Based on multiple multimodal fusion feature vectors, the convolutional neural network model is sequentially trained as an independent sub-model and then evaluated to obtain a single-process image defect detection model. Similarly, based on multiple multimodal fusion feature vectors, the Transformer network model is sequentially trained and evaluated to obtain a multi-process image defect detection model. For details, please refer to [link to details]. Figure 3 Step S205 of the illustrated embodiment will not be described again here.

[0412] Step S306: Visualize the changes in loss and evaluation results during the training process of the convolutional neural network model, the key features learned by multiple convolutional and pooling layers, and the probability distribution of feature integration and mapping to the output of multiple fully connected layers using visualization tools.

[0413] Specifically, visualization tools (such as TensorBoard) can be used to analyze curves showing changes in loss and accuracy during model training, providing a direct understanding of the model's training progress. Simultaneously, the feature maps learned by the convolutional layers can be visualized to see if the model has truly learned valuable features from the chip (smart meter) image.

[0414] Step S307: Acquire multiple actual captured images of the product to be diagnosed and input them into a single-process image defect detection model and / or a multi-process image defect detection model for image defect diagnosis, and obtain the detection result that the corresponding area of ​​the product image has defects or no defects.

[0415] Specifically, deployment to production and application environments: The trained and evaluated model is deployed to the actual product testing production line and application. The actual product images to be diagnosed captured by the testing equipment are input into the tested image defect detection model to achieve real-time detection, measurement, diagnosis, and tracking decisions.

[0416] The above step S307 includes:

[0417] Step a: Perform image defect diagnosis on multiple product images to be diagnosed input into the image defect detection model using a decision function. The decision function is expressed by the following formula:

[0418]

[0419] Wherein, G(F) = 1 indicates that a defect is detected in the corresponding region of the image, G(F) = 0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that the preset defect condition is met.

[0420] Specifically, G(F) is a binary classification judgment value. The preset defect condition can be a judgment rule such as the value of F being greater than a certain set threshold. The threshold can be determined through training or other methods. That is, the preset defect condition can be a judgment rule such as the value of F being greater than a certain set threshold. It is a variable related to the features and parameters of the image of the product to be diagnosed. Its meaning needs to be further clarified by combining more specific background information about the image defect detection model and related settings.

[0421] The multimodal fusion feature vector of the input product image to be diagnosed is matched with the defect feature threshold set during model training. That is, when the feature parameters such as contrast, energy, and entropy in the feature vector exceed the preset threshold, it is determined that there is a defect; otherwise, it is determined that there is no defect.

[0422] Step S308: Perform hybrid modeling and analysis on the product image with defects output by the image defect detection model to be diagnosed, and obtain hybrid modeled image defects. Hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0423] Specifically, a semantic segmentation model based on an improved U-Net model using CNN deep learning is employed to segment the image defect region, obtaining a binary image of the defect region. Image processing library functions are then used to extract the contour of the binary image of the defect region, yielding the image defect region contour. The geometric features of the image defect region contour are calculated to determine the preliminary shape of the image defect region; these geometric features include the contour perimeter, the contour convex hull perimeter, and the shape factor.

[0424] The above step S308 includes:

[0425] Step S3081: Calculate the geometric features of the image defect region contour and determine the preliminary shape of the image defect region; the geometric features of the preliminary shape of the image defect region include the contour perimeter, the contour convex hull perimeter, and the shape factor.

[0426] Specifically, the first step is to accurately segment the defect region from the image. A semantic segmentation model based on an improved U-Net model using CNN deep learning can be employed. The improved U-Net model has excellent segmentation performance, especially suitable for multi-task, multi-modal segmentation of complex images such as welds, crystal oscillators, chips, IGBTs, GBA (Game Boy Advance, core processor chip), and subtle details within artificial intelligence. Establishing a coordinate system is the first step. Assuming the imaging plane is used as a reference, let the point within the defect region of the image be (x, y), and its corresponding grayscale value be I(x, y).

[0427] Step S3082: Based on the preliminary shape of the defect region in the image, shape hybrid modeling is performed using the second moments of discrete data and the second moments of continuous functions to obtain a mathematical model for measuring the shape of the image defect region.

[0428] In some alternative implementations, the semantic segmentation model based on CNN deep learning improves the U-Net model by introducing residual connections in the encoder (downsampling path) and an attention mechanism in the decoder (upsampling path). That is, the semantic segmentation model based on CNN deep learning improves the U-Net model by including an encoder, a decoder and skip connections. The encoder includes residual connections and the decoder includes an attention mechanism.

[0429] The above step S3082 includes:

[0430] Based on the preliminary shape of the defect region in the image, determine the discrete second moment and the continuous function second moment of the defect region;

[0431] By fusing the second moments of discrete data and the second moments of continuous functions in the image defect region, a mathematical model for measuring the shape of the image defect region is obtained.

[0432] In one optional implementation, the discrete second moments and the continuous second moments of the image defect region are fused and modeled to obtain a mathematical model for measuring the shape of the image defect region, including:

[0433] The first moment of inertia is calculated based on the second-order moments of discrete data, and the first equivalent ellipse parameter is calculated based on the first moment of inertia. The second moment of inertia is calculated based on the second-order moments of continuous functions, and the second equivalent ellipse parameter is calculated based on the second moment of inertia. The first moment of inertia and the second moment of inertia are fused using a weighted average method, and the first equivalent ellipse parameter is fused with the second equivalent ellipse parameter, respectively, to obtain a mathematical model for measuring the shape of the image defect region.

[0434] Furthermore, a weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent elliptic parameters and the second equivalent elliptic parameters, to obtain a mathematical model for measuring the shape of the image defect region, including:

[0435] Define the weights of the second moments of continuous functions, and calculate the weights of the second moments of discrete functions based on the weights of the second moments of continuous functions;

[0436] The first moment of inertia and the second moment of inertia are fused using a weighted average method based on the weights of the second moments of continuous functions and discrete functions to obtain the fused second moment; the first equivalent elliptic parameter and the second equivalent elliptic parameter are fused to obtain the fused equivalent elliptic parameter.

[0437] A mathematical model for measuring the shape of the image defect region is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters.

[0438] The formula for the fused second moment is as follows:

[0439] M fusion =aM c +(1-a)M d ;

[0440] Among them, M c M is the second moment of a continuous function. dis the second moment of discrete data; a (0≤a≤1) is a weighting coefficient used to measure the proportion of the second moment of continuous function in the fused second moment; when a=0, it means that the contribution of the second moment of continuous function is not considered at all when calculating the fused second moment, and only the second moment of discrete data is used; when a=1, it depends entirely on the second moment of continuous function and does not consider the second moment of discrete data.

[0441] Step S3083: Measure the degree of defect in the image defect region based on the mathematical model for measuring the shape of the image defect region, and obtain the final shape, width, length, area and volume of the image defect region.

[0442] The final shape, width, length, area, and volume of the image defect region are calculated based on the major axis, minor axis, and orientation angle.

[0443] The following is an example of a common method for calculating image defect region-related parameters by fusing the second moments of continuous functions and discrete data using the weighted average method to obtain the major axis, minor axis, and orientation angle (major axis a, minor axis b, orientation angle Э). d )as follows:

[0444] Final shape: The approximate elliptical shape of the defect area can be determined based on the ratio of the major axis to the minor axis and the orientation angle (if the defect shape is similar to an ellipse). For example, when the major axis is significantly longer than the minor axis, it presents a relatively long and narrow ellipse. The orientation angle determines its tilt angle in the plane, which together outlines its final shape characteristics on the two-dimensional plane.

[0445] Width: The minor axis length b is usually regarded as the width of the defect area, which represents the size occupied by the defect in the direction perpendicular to the major axis.

[0446] Length: The length of the major axis 'a' is defined as the length of the defect region, which is the distance the defect spans in its main extension direction.

[0447] Area: For a defect region that is approximately elliptical in shape, the area S is calculated by the formula S = πaab. The area occupied by the defect on the two-dimensional plane is obtained by multiplying the values ​​of the major axis and minor axis by pi.

[0448] Volume: If we want to consider the three-dimensional case (assuming that the defect also extends in a certain thickness direction, and the thickness is set as h), the volume V can be simply approximated by the volume of an elliptical cylinder (when the defect shape conforms to such characteristics). The calculation formula is V = πabh, that is, the volume is obtained by multiplying the two-dimensional area calculated above by the dimension in the thickness direction.

[0449] In practical applications, these calculation methods can be appropriately adjusted or more realistic professional models can be used to measure relevant parameters based on the specific physical characteristics of the defect and the degree of shape conformity.

[0450] Step S3084: Diagnose and classify the severity level of image defects based on the final shape, width, length, area, and volume of the image defect region.

[0451] Calculate the geometric features of the image defect region contour to determine the initial shape of the image defect region; the geometric features of the initial shape of the image defect region include the contour perimeter, the contour convex hull perimeter, and the shape factor.

[0452] Specifically, after obtaining the contour of the defect region, the shape of the defect can be described by calculating some geometric features of the contour. For defects in a two-dimensional image, the perimeter C of the contour can be calculated. p convex hull perimeter C hc Indicators such as shape factor are used to describe the shape of image defects from different perspectives.

[0453] Among them, the perimeter C p calculate:

[0454] Contour extraction is performed on the binary image of the defect region. Let the extracted contour point set be... N is the number of contour points, then the contour perimeter C p It can be obtained by calculating the sum of the Euclidean distances between adjacent contour points, and the calculation formula is as follows:

[0455]

[0456] Where, x i+1 With x i For adjacent contour points, y i+1 With y i These are adjacent contour points.

[0457] convex hull perimeter C hc calculate:

[0458] First, calculate the convex hull of the defect region's contour. Let the set of convex hull points be... M represents the number of points on the convex hull. The perimeter of the convex hull is C. hc Similarly, it is determined by calculating the sum of the Euclidean distances between adjacent convex hull points, using the following formula:

[0459]

[0460] Shape factors, such as roundness, are calculated using the following formula:

[0461]

[0462] Where A is the area of ​​the image defect region. The closer the circularity value is to 1, the closer the defect shape is to a circle. The smaller the circularity value is, the more irregular the shape is.

[0463] Calculate the moment characteristics of the defect region, such as the zeroth moment M0 (whose value equals the area of ​​the defect region), the first moment M1 (which can be used to determine the centroid of the defect region), and the second moment M2 (related to the shape of the defect region). These moment characteristics can also describe the shape of the defect to some extent. Calculate the second moment of the defect region, including the central moment.

[0464] Mean Squared Error (MSE) is used to evaluate measurement tasks, and its calculation formula is as follows:

[0465]

[0466] When measuring certain features (such as component dimensions, line width, etc.) in a product image, It is the actual value. Here, n is the predicted value, and n is the number of measurement samples. The mean square error (MSE) can be used to assess the deviation between the measured value and the true value; the smaller the value, the higher the measurement accuracy.

[0467] Measurement process (taking the measurement of a target dimension as an example, let the measurement function be M:)

[0468] M(F)=size value calue calculated on F(20);

[0469] In other words, the dimensional values ​​of the target to be measured (such as the width, length, area, volume, etc. of chip circuits (smart meter defects, faults)) are calculated based on the fused feature vector F.

[0470] Step S309: The defects in the hybrid modeled image are processed by the improved U-Net model to obtain the defect detection results, thereby achieving defect region segmentation and localization. Then, the location of the defect region in the image is tracked, and decisions are made based on the defect tracking and fault tracing results to optimize production process parameters.

[0471] Specifically, step S309 includes:

[0472] Step S3091: The defects in the hybrid modeled image are processed by the improved U-Net model to obtain the defect detection results, and the defect region is segmented and located based on the defect detection results.

[0473] In an optional implementation, step S3091 includes:

[0474] After convolution processing of the image defect region in the encoder, output feature map and input feature map are obtained. The output feature map and input feature map are added together through residual connection to obtain a new output feature map. The new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, which is then convolved to obtain a TFT ray map. The TFT ray map is input into the improved U-Net model to perform image region segmentation to obtain a binary image of the image defect region.

[0475] The semantic segmentation model based on CNN deep learning improves the U-Net model by introducing residual connections in the encoder (downsampling path) and an attention mechanism in the decoder (upsampling path). That is, the semantic segmentation model based on CNN deep learning improves the U-Net model by introducing residual connections in the encoder and an attention mechanism in the decoder.

[0476] Residual connections are introduced into the encoder, and the weight matrix of the residual connections is given by... The output feature map after two convolutional layers Input features Figure X i (x, y, z) are summed through residual connections to obtain a new output feature map. The formula is as follows:

[0477]

[0478] The new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, which is then convolved to obtain a TFT ray map.

[0479] Specifically, an attention mechanism is introduced into the decoder, where the attention weight matrix is ​​denoted as follows. Feature maps after encoder upsampling and stitching Weighted features are obtained through an attention mechanism. That is, the TFT beam pattern, the formula is as follows:

[0480]

[0481] Where (x, y, z) are image coordinates, a represents the relative position offset involved in the weighted operation when the feature map is subjected to the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[0482] Then replace Then proceed with subsequent convolutional layer operations. When And (C) i-1 =C i+1 When the number of channels changes after passing through the convolutional layer, the number of channels may change.

[0483] The TFT ray image is input into the trained improved U-Net model to obtain a binary image of the defect region, where the pixel value of the defect region is 1 and the pixel value of the background region is 0.

[0484] Specifically, the calculation process for the discrete second moment is as follows:

[0485] Relationship between zeroth moment and area: Let the binary image function of the defect region be f(x, y), where the pixel value of the defect region is 1 and the pixel value of the background region is 0. The formula for calculating the zeroth moment M0 is:

[0486] M0=∑∑ f f(x, y) (23);

[0487] As can be seen from the above formula, the value of the zeroth moment is equal to the area A of the defect region, that is, A = M0.

[0488] The first moment is denoted by M1, and the first moment and the position of the centroid are calculated as follows:

[0489] The formulas for calculating the first moments M1x (about the x-axis) and M1y (about the y-axis) are as follows:

[0490] M1x=∑ x ∑ y xf(x,y) (24);

[0491] M1y=∑ x ∑ y yf(x, y) (25);

[0492] The centroid position of the defect area (x) g -y g It can be calculated using the following formula:

[0493]

[0494] Relationship between second moment and defect shape:

[0495] The formulas for calculating the second moments M2xx (the second moment about the x-axis), M2yy (the second moment about the y-axis), and M2xy (the combined second moment about the x and y axes) are as follows:

[0496] M2xy=∑ x ∑ y xyf(x,y) (27);

[0497] M2xx=∑ x ∑ y x 2 f(x, y) (28);

[0498] M2yy=∑ x∑ y y 2 f(x, y) (29);

[0499] The second moment of a continuous function is calculated as follows:

[0500] M 00 =∫∫ D xf(x,y)dx dy (30);

[0501] For a continuous function z = f(x, y) defined on the defect location region D.

[0502] Zeroth moment M 00 calculate:

[0503] M 00 =∫∫ D f(x,y)dx dy (31);

[0504] The above equation is a double integral, and the integration region is the domain D of the function f(x, y).

[0505] First-order moment calculation:

[0506] M 10 =∫∫ D xf(x,y)dx dy (32);

[0507] M 01 =∫∫ D yf(x,y)dx dy (33);

[0508] The centroid coordinates (x, y) are given by and Sure.

[0509] The calculations for the second-order central moments are as follows:

[0510] M 2xy =∫∫ D xf(x,y)dx dy (34);

[0511] M 2xx =∫∫ D x 2 f(x,y)dx dy (35);

[0512] M 2yy =∫∫ D y 2 f(x,y)dx dy (36).

[0513] The calculation of these second-order central moments requires first determining the coordinates of the centroid, and then performing a double integral.

[0514] A hybrid second-order model combining the second moments of continuous functions and discrete data is constructed, integrating discrete data with continuous functions. For the defect shape, it is preferable that the main body of the defect can be approximated by a continuous function; in this embodiment, a combination of polynomial functions or trigonometric functions is preferred. Simultaneously, at some key boundary locations or local irregularities, discrete data points are used to supplement details.

[0515] In some optional implementations, this embodiment also uses a weighted average method to fuse the second moments of continuous functions and discrete data, and the second moments are used to calculate a mathematical model of the major axis, minor axis, and orientation angle of the defect shape:

[0516] The desired defect shape is represented by a continuous function z = f(x, y) in a two-dimensional plane, with its domain being Ω. Calculate the moment of inertia. The formulas are as follows:

[0517]

[0518] M 2xx =∫∫ D x 2 f(x,y)dx dy (38);

[0519] M 2yy =∫∫ D y 2 f(x,y)dx dy (39);

[0520] In one alternative implementation, the above includes:

[0521] The first moment of inertia is calculated based on the second moment of discrete data, and the first equivalent ellipse parameters are calculated based on the first moment of inertia.

[0522] Specifically, there are n discrete data points (x i -y i ), with a weight of w i (If the weights are the same, then w) i =1).

[0523] Calculate the moment of inertia and The formula is as follows:

[0524]

[0525] Let k be the major axis corresponding to the discrete data portion. d short axis b d and direction angle Э d First, calculate:

[0526]

[0527] Then the major axis short axis Direction angle

[0528] Where, j d It is an intermediate variable, obtained through the moment of inertia. and It was calculated. From the subsequent calculation of the major axis k... d short axis b d and direction angle Э d From the formula, j d With k d The values ​​are first obtained through a fixed calculation method (formulas (43) and (44)), and then used to calculate the lengths of the major and minor axes of the equivalent ellipse.

[0529] j d With k d The calculation method is similar to combining the second moments of discrete data to calculate the parameters (major axis, minor axis, and orientation angle) of an equivalent ellipse that can more easily represent the shape of the discrete data distribution.

[0530] The second moment of inertia is calculated based on the second moment of the continuous function, and the second equivalent ellipse parameters are calculated based on the second moment of inertia.

[0531] Let the continuous function be y = f(x), its domain be [a, b], and the discrete data points be (x, b). i y i )i = 1, 2, ..., n. Calculate the second moment of the continuous function part:

[0532]

[0533] in,

[0534] For the discrete data portion, its second moment is:

[0535]

[0536] in, The description of the defect shape is optimized by minimizing M by adjusting the parameters of the continuous function and the positions of the discrete data points.

[0537] Let k be the major axis corresponding to the continuous function part. c short axis b c and direction angle Э c .

[0538] First, calculate:

[0539]

[0540] Long axis short axis Direction angle

[0541] The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent elliptic parameter and the second equivalent elliptic parameter, to obtain a mathematical model for measuring the shape of the image defect region.

[0542] Let the weight of the second moment of the continuous function be a (0 ≤ a ≤ 1), then the weight of the second moment of the discrete data is 1 - a. The fused moment of inertia:

[0543]

[0544] Calculate the equivalent ellipse parameters after fusion (major axis a, minor axis b, direction angle θ). d First, calculate:

[0545]

[0546] Long axis short axis Direction angle

[0547] Weighted average fused second moment: M fusion＝ =aM c +(1-a)M d , of which M c M is the second moment of a continuous function. d It is the second moment of discrete data;

[0548]

[0549] In this model, 'a' (0 ≤ a ≤ 1) is a weighting coefficient. It measures the proportion of the continuous function's second moment in the fused second moment. When a = 0, the contribution of the continuous function's second moment is completely disregarded when calculating the fused second moment; only the discrete data's second moment is used. When a = 1, the model relies entirely on the continuous function's second moment, ignoring the discrete data's second moment. By adjusting the value of 'a', the influence of the continuous function's second moment and the discrete data's second moment on the final result can be flexibly balanced according to the characteristics of the data and accuracy requirements, thus better adapting to the description needs of different image defect region shapes. 'b' mainly appears in the continuous function's second moment M. c In the calculation, when the continuous function is y = f(x) and its domain is [a, b], where b is the upper limit of the domain of this function. In calculating M... c During the process, the integration interval is from a to b, and this interval determines the range of values ​​to be considered when calculating the second moment of a continuous function.

[0550] For example, in calculation (This is used to calculate M) c When considering intermediate quantities in the process, ba is used as the denominator to normalize the integral result, while the value of b directly affects the integration interval and the final calculation result. Together with a, it defines the range involved in the calculation of the second moment of a continuous function.

[0551] Using the above model, the value of weight 'a' is adjusted according to the characteristics and accuracy of the data, and the major axis, minor axis, and orientation angle of the defect shape are determined based on the fusion result of the second moment of the continuous function and the second moment of the discrete data.

[0552] For example, taking a real electricity meter sampling component as an example, the weighted average method is used to fuse the second moment of the continuous function and the second moment of the discrete data to express the long axis, rectangular axis, and direction angle of the defect shape to reproduce and construct the defect shape.

[0553] The sampling components of an electricity meter are crucial for accurate electricity measurement. During production and use, defects may occur due to material issues, welding cracks, voids, poor soldering, or wear and tear from long-term operation. These defects affect the sampling accuracy, leading to inaccurate meter readings. For example, the sampling resistor may exhibit localized wear or uneven conductive layer, the sampling hard connection may have welding cracks, and the capacitor may have dielectric defects.

[0554] Calculation of the second moment of a continuous function:

[0555] For a typical defect in the sampling components of an electricity meter, such as material loss on the surface of the sampling resistor or capacitor due to wear, its shape on a two-dimensional plane can be approximated by a parabolic function based on the principles of materials physics and electricity.

[0556] Let the function f(x, y) = AB[(x - x0)] 2 +(y-y0) 2 (Within the defective area);

[0557] Where (x0, y0) are parameters related to the resistive material and the degree of wear, and represent the location of the wear center.

[0558] Based on the second-order moment model of a continuous function, the moment of inertia is obtained through integration (the integration region is the wear, crack, and circular wear radius of the resistive surface, with radius r).

[0559]

[0560] Among them, the functions x0 and y0 are symmetric and have the same direction angle of 0.

[0561] Calculation of the second moment of discrete data:

[0562] Data acquisition: such as Figure 6As shown, the sampled components were inspected using a high-precision optical microscope and an electron microscope, and discrete data points (x, y, z) of the suspected defect area were obtained. i y i Weights w are assigned based on the clarity and reliability of discrete data points. i (For example, data points that are clear and confirmed through multiple tests have higher weights). Weights are determined based on the accuracy assessment of the ideal model and the test data. The weight of the second moment of the continuous function is determined as a = 0.3, then the weight of the second moment of the discrete data is 1 - a = 0.7. A weighted average fusion calculation is then performed based on the calculated moments of inertia and equivalent ellipse parameters to obtain the fused moments of inertia and equivalent ellipse parameters (major axis a, minor axis b, direction angle θ). d By using this weighted average fusion method, the defect shape of the meter sampling components was reproduced, providing a basis for evaluating component quality and improving production processes.

[0563] Taking multilayer chip manufacturing and inspection as another example, various defects may arise during the production of multilayer chips due to complex process steps such as photolithography, etching, and deposition. These defects may exist between or within different layers, severely impacting chip performance. To detect these defects, various techniques are typically used, including electron microscopy imaging and X-ray inspection. These methods can provide theoretical models of defects in the form of continuous functions, as well as discrete data points.

[0564] For a certain type of short-circuit defect in the conductive layer of a multilayer chip, based on electromagnetic field theory and circuit models, its shape in a two-dimensional plane can be approximated by a function combining a log-normal distribution and trigonometric functions. For example, Where A is a coefficient related to the defect current density, (x0, y) is the estimated position of the defect center on the x-axis, and a and b are parameters related to the defect range.

[0565] Weight determination: Based on the accuracy assessment of the theoretical model (based on previous studies and verifications of similar defects) and the reliability assessment of electron microscope data, the weight of the second moment of the continuous function is determined to be 0.4, and the weight of the second moment of the discrete data is 0.6.

[0566] By using the steps of this invention, the second moments of continuous functions and discrete data are fused using a weighted average method to obtain the major axis, minor axis, and orientation angle parameters that can reproduce the shape of chip defects. This allows for a more accurate description of defects in the chip and provides a basis for subsequent chip repair or quality improvement.

[0567] Step S3092: Track the image defect location based on the segmented and located defect region.

[0568] Specifically, in the diagnostic phase (taking the assessment of defect severity as an example, let the diagnostic function be J):

[0569] J(F)= severity level determined by F (58);

[0570] The severity level of the defect is determined based on the fusion feature F. For example, different F ranges can be set to correspond to different severity levels (such as mild, moderate, severe, etc.).

[0571] Tracking process: Taking the change in the position of the target at different times as an example, let the tracking function be T, and let the different times be denoted as t1, t2, ..., the tracking function is expressed as:

[0572] T(F,t,)= position of target at time t: calculated based on F (59);

[0573] By fusing feature F at different times t j It calculates the position of the target (such as the movement of a defect or the displacement of a component) and then tracks the target.

[0574] Step S310: The defect detection result is obtained by using the improved U-Net model to obtain the defect in the hybrid modeled image, thereby realizing defect region segmentation and localization, and then tracking the location of the defect region in the image. Decisions are made based on the defect tracking and fault tracing results to optimize production process parameters.

[0575] Decision-making process: Discrete data second-order moment calculation. Data acquisition: The sampled original was inspected using high-precision optical microscopes, electron microscopes, and optical current emission microscopes to obtain discrete data points (x) of suspected defect areas. i y i Weights w are assigned based on the clarity and reliability of the data points. i (For example, data points that are clear and have been repeatedly verified have higher weights). Weights are determined based on the accuracy assessment of the ideal model and the test data. The weight of the second moment of the continuous function is determined to be a = 0.3, and the weight of the second moment of the discrete data is 1 - a = 0.7. The equivalent ellipse parameters of the moment of inertia are calculated using the previous formula, and the weighted average fusion is used to calculate the equivalent ellipse parameters of the fused moment of inertia (major axis k, minor axis b, direction angle Эd). Through this weighted average fusion method, the defect shape of the meter sampling components is reproduced, providing a basis for evaluating component quality and improving production processes.

[0576] Specifically, step S310 includes:

[0577] A dynamic model of defect evolution is constructed based on discrete data points of defect area and perimeter and a second-order moment model of continuous function.

[0578] Based on defect location images, a preset tracking function is used to analyze the morphological changes of crack propagation angle and wear depth increment of defects in the defect area at different times, and to visualize and quantify the defect evolution process.

[0579] The physical defect locations in the three-dimensional structural information are obtained, and a spatial mapping algorithm is used to match the physical defect locations in the three-dimensional structural information with electrical anomaly point data and microstructure images, respectively, to establish a correlation map between physical defects and electrical faults.

[0580] Based on the correlation map of physical defects and electrical faults and the morphological change characteristics, and combined with the abnormal current distribution analysis of the wear area on the resistor surface, the fault source of the defect area in the image is traced.

[0581] Based on defect tracking and fault tracing, improvements are made to the welding process, material replacement process, and online monitoring process of the product to be decided.

[0582] Among these improvements are the welding processes for the products requiring decision-making, including:

[0583] Finite element method (FEM) simulation was used to analyze the effects of different welding parameters (laser power, welding speed, and spot diameter) on the residual stress, fatigue life, and resistance of the weld joint of the product under consideration. The simulation effect was verified by combining multimodal data from ultrasonic scanning results to determine the optimal process parameters for laser welding and resistance brazing. The influence of multiple power sources (photovoltaics, green electricity, energy storage, mains power, combined power, and nuclear power) on the welding parameters was also considered. The defect location images of welding cracks and incomplete welds detected by convolutional neural networks were combined with the multimodal data from ultrasonic scanning to quantify the welding quality.

[0584] Improvements to the material replacement process for the product being decided upon, including:

[0585] A material performance database was established to record the resistance change rate and capacitance stability of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments. Through accelerated aging tests simulating extreme working conditions, convolutional neural networks were used to detect defects in images of aged materials, and the durability of the materials was evaluated in combination with performance data.

[0586] Improvements to the online monitoring process for decision-making products include:

[0587] Deploy a real-time image recognition system based on deep learning to perform millisecond-level analysis on X-ray or OBIRCH inspection images to automatically label and classify defects; use deep learning algorithms to train X-ray or OBIRCH inspection images to establish a defect recognition model.

[0588] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided in this embodiment obtains accurate internal three-dimensional structural information through multi-parameter data fusion and reconstruction technology. It combines discrete second moments with continuous function second moments, addressing the difficulty in combining discrete second moments with continuous function second moments and weight allocation in image accuracy, which restricts online analysis and visualization modeling of high-precision images. It is necessary to solve the problems of attenuation at different depths and limitations of single-modality depth imaging technology, and overcome the difficulties in establishing product defect models and weight allocation by combining discrete second moments with continuous function second moments, thus realizing online analysis and visualization modeling of high-precision images. A weighted average method is used to fuse the first and second moments of inertia, as well as the first and second equivalent elliptic parameters, to obtain a mathematical model for measuring the shape of the image defect region. This enables the construction of the image defect region shape and provides the conditions for tracking the image defect region.

[0589] Combination Figure 5 The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by this invention will be further described in detail below:

[0590] like Figure 5 As shown, the overall process of the multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by this invention is as follows:

[0591] Step 1, Multimodal Data Collection: Obtain multi-layer chip (smart meter and component) images (data types include image type, electrical type and operating condition type as described above) from multiple channels, including chip (smart meter and component) images taken by different production batches, different process stages and different equipment, and collect corresponding tag data, such as whether the chip (smart meter and component) has defects and the type of defects.

[0592] Step 2, Data Cleaning: Remove blurry, incorrectly labeled, or damaged image data to ensure data quality.

[0593] Step 3, Data annotation: For images that are not annotated or are incompletely annotated, annotate them with the help of professionals or by using annotation tools. The annotation content can include the different layers of the chip, the location and type of defects, etc.

[0594] Step 4, clarify the intended use and categorize the operations:

[0595] A. Image data of different modalities have significant differences in content, scene, target object, etc.; appearance inspection and internal structure inspection, key or critical part location defects or product inspection and measurement are classified; classification is used to facilitate annotation and targeted model training.

[0596] B. Image data of different modalities are similar in content and purpose, and have high data homogeneity; when the plan is to use a unified tool and method architecture and feature extraction strategy, and to emphasize the correlation and complementarity between multimodalities, there is no need to classify them.

[0597] Data preprocessing and alignment (including the previously described fusion processing, image preprocessing, and electrical parameter processing).

[0598] Step 5, Feature Normalization: Normalize the image pixel values ​​to a specific range, such as [0,1] or [-1,1]. This helps to speed up model training and improve stability. For a certain feature value x, the formula can be used:

[0599]

[0600] Where, x min and x max These are the minimum and maximum values ​​of the feature across all samples, respectively.

[0601] Step 6, Location Encoding: For the texture parameters of the defect location (in two-dimensional coordinates (x...)... d y d The coordinates (representing the center location of the defect) are transformed into a form that can be effectively processed by a neural network using positional encoding. A simple method is to discretize the coordinates and then convert them into an m x m vector using one-hot encoding. For example, the image can be divided into grid regions, and one-hot encoding can be performed on the grid region where the defect center is located to obtain a vector of length m. 2 The vector is used as the feature after position encoding.

[0602] Step 7, Weight Allocation and Feature Fusion: When x min and x max This is used for defining a simple range; all data points have the same function at this stage and no weighting is required. When x min and x max When used for further comparison and classification, as a threshold judgment, each feature has different importance or defects in subsequent operations, and the significance and severity of feature changes are weighted according to contrast C, correlation R, energy E, entropy H, and defect entropy value.

[0603] Different sensors, different defects, or different equipment for product visualization inspection have different detection conditions and rules. The image information of the same defect or product varies greatly and is difficult to integrate. The entropy weight method is used to calculate the minimum and maximum values ​​in the sample, and the minimum entropy weight of each value is taken and spliced ​​together. The average value is used for splicing the gap.

[0604] The relationship between various parameters and the accuracy of multilayer chip images is analyzed as follows: Let the input multilayer chip image be I. d The smart meter image is I B The contrast characteristic is denoted as C(I). d The correlation feature is denoted as R(I). d The energy characteristic is denoted as E(I). d The entropy feature is denoted as H(I). d The defect entropy value characteristic is denoted as D(I). d (Here, D represents the defect entropy value for ease of subsequent description), and the positional texture parameter feature is denoted as P(I d ).

[0605] The contrast characteristic is denoted as C(I) B If contrast C(I) is found B It has a strong ability to distinguish between normal and defective areas. For example, when the meter display shows defects such as blurriness, the contrast ratio C(I) is high. B If the difference between the displayed area and the surrounding normal area is clearly visible, then a contrast ratio C(I) is given. B A relatively high initial weighting range, such as 20% to 30%. For minor defects (such as fine scratches), if the contrast ratio C(I) is relatively high... B If the change is not significant, then a lower value should be taken within this weight range.

[0606] The correlation feature is denoted as R(I) B ): When the correlation R(I) B When pixel correlation is significantly reduced around the defect area (e.g., the defect disrupts texture continuity), a weight range of 15% to 25% can be assigned. If the correlation R(I) B If the change is not significant for most defect types, the weight range should be appropriately reduced to 10% to 20%.

[0607] The energy characteristic is denoted as E(I) B If the energy E(I) B When a defect exhibits a specific pattern of change (e.g., a certain defect always leads to an increase or decrease in the energy of a specific frequency component), it is given a weight of 15% to 25%. If the energy E(I) B The changes are complex and have no obvious pattern; the weight can be set to 10% to 20%.

[0608] Entropy features are denoted as H(I)B ): Since the entropy characteristic is denoted as H(I) B This is related to the uncertainty of image information. For complex textures or mixed defects, if the entropy feature is denoted as H(I)... B This can well reflect these situations and can be assigned a weight of 30% to 40%. If the entropy feature is denoted as H(I... B If the performance is unstable in some simple defect scenarios, then make appropriate adjustments within this range.

[0609] The defect entropy characteristic is denoted as D(I) B (Here, D represents the defect entropy value for ease of subsequent description): If the defect entropy value is a key factor in distinguishing defects of different severity (for example, a high entropy value defect may indicate a more serious internal circuit fault in the meter), then a certain weight should be assigned to the defect entropy value separately. Based on the total weight, 10% to 20% of the weight can be specifically allocated to adjust the accuracy according to the defect entropy value. Defects with high entropy values ​​and significant impact on meter function should be given higher weight.

[0610] The positional texture parameter feature is denoted as P(I B When a defect is located in a critical part of a smart meter (such as the display screen, around the metering chip, etc.), its weight should be higher than that of defects in non-critical parts. Depending on the importance of the location, the weight of the location factor can be allocated between 10% and 30%. Meanwhile, for areas with obvious texture features (such as specific patterns on the meter casing), if texture changes are significant for defect detection, the weight of texture-related parameters (such as correlation features denoted as R(I)) can be increased. B The entropy feature is denoted as H(I). B The weights of texture variations (such as texture) should be adjusted appropriately to highlight their impact on accuracy.

[0611] The principle of the weight allocation scheme is that the sum of the weights must equal 1.

[0612] Different types of defects (such as scratches, wear, display anomalies, etc.) are detected, and a preliminary weight range is determined based on the different types of defects. In the feature fusion stage, fusion is performed according to a given weight allocation scheme. Let the fused feature vector be F(I...). B ),but:

[0613] F(I B ) = (20% to 30%) x C(I) B )+(10% to 25%)x R(I B )+E(I d )x(15% to 25)+H(I B )x(30%)

[0614] Up to 40%) + D(I B)x(10% to 20%)+P(I B (10% to 30%)

[0615] The advantage of performing multimodal fusion after location encoding is that encoding location information before fusion allows for better alignment of location features with other modal features in spatial and other conceptual ways. This enables the fused feature vector to comprehensively consider the correlation between location and other physical attribute features, which may be more advantageous for tasks such as defect identification with location dependencies. In some industrial inspection scenarios, if there is a pattern-based correlation between the defect location and its own features such as contrast and energy, this method can better capture this relationship. In the power electronics industry, the primary requirement is to accurately locate the defect and implement rapid countermeasures; shape and size are secondary requirements. This improvement meets these needs.

[0616] For example, X-ray images undergo multimodal fusion after location encoding, including: normalizing the contrast features C(I) d ), correlation feature R(I) d Energy characteristics E(I) d ), entropy feature H(I) d ), defect entropy feature D(I) d (Here, D represents the defect entropy value for ease of subsequent description), and the positional texture parameter feature is denoted as P(I d The feature vector is formed by concatenating and fusing the defect location features after location encoding p, and the defect location features after location encoding p. or After the above processing, the features are C_{normalized}, R_{normalized}, E_{normalized}, H_{normalization}, H_d_{normalized}, and the position encoding vector p, respectively. Then:

[0617] \mathbf{F}=[C_{normalized},R_{normalized},E_{normalized},H_{normalization},H_d_{normalized},\mathbf{P}].

[0618] The advantages and effects of the above innovations include: Considering location information: By incorporating defect location features into multimodal fusion through location encoding, the model can better utilize location information to detect defects. For devices like smart meters, defects in different locations may have different impacts; this approach can more accurately handle location-related issues.

[0619] Feature integration: By weighting and fusing multiple features such as contrast, correlation, energy, entropy, and defect entropy value, the role of each feature in defect detection can be fully utilized. Weighting is assigned based on the effectiveness of different features in distinguishing different types of defects, making the fused feature vector more discriminative and meeting diagnostic requirements.

[0620] Advantages of normalization: Normalizing features before fusion can prevent certain features from dominating the fusion process due to their large numerical range, ensuring that each feature can participate in the fusion equally, thus improving the fusion effect and the stability of the model.

[0621] Targeted weight allocation: Weight allocation schemes are determined for multi-layer chip images and smart meter images respectively, taking into account the characteristics of different image types and the performance of different features under different defect types, making the fusion scheme more targeted and better able to adapt to specific detection tasks, meeting the need for high adaptability.

[0622] Step 8, Explanation of the model architecture based on CNN (Convolutional Neural Networks):

[0623] The basic CNN part involves choosing a suitable convolutional neural network architecture as the foundation, such as ResNet (Residual Network) or VGG (Visual Geometry Group, a deep convolutional neural network architecture). The depth and width of the network are determined based on the complexity of the chip image and the task requirements. For example, if the chip image is rich in detail and requires more convolutional layers to extract features, ResNet-50 or a more complex network can be chosen. The CNN model includes an input layer, multiple convolutional and pooling layers, multiple fully connected layers, and an output layer.

[0624] The input layer receives multimodal fused feature vectors. Its number of nodes is equal to Dimensions.

[0625] Convolutional and Pooling Layers: Multiple convolutional and pooling layers are used to automatically extract high-level information from features. This embodiment uses three convolutional layers with kernel sizes of 3x3, 3x3, and 5x5, each with a stride of 1. Each convolutional layer is followed by a pooling layer with a 2x2 kernel and a stride of 2. The activation function used for the convolutional layers is the RReLU (Randomized ReLU) activation function, defined as: f(x) = max(0.x).

[0626] Fully connected layers: After feature extraction through convolutional and pooling layers, several fully connected layers are set to further integrate information and map it to the output probability distribution. This embodiment sets two fully connected layers. The first fully connected layer has k nodes (a suitable value can be determined experimentally, such as 128), and the second fully connected layer has n nodes, corresponding to the number of elements P = [p0, o1…p] in the probability distribution of the model's output image type. n ], where n is the number of image types, P i This represents the probability that an image belongs to the i-th type, and the type with the highest probability is the predicted image type. To alleviate gradient vanishing and improve computational efficiency, neuron death, and model overfitting problems in multimodal and high-resolution images, and to filter out unfavorable negative signals, the activation function of the fully connected layer can be the RReLU function (f(x) = max:(ax,x)), where a is a learnable, adaptively adjusted slope parameter that is averaged after training.

[0627] The output layer uses the Softmax function (an activation function used in multi-class classification problems) as the activation function, transforming the output of the fully connected layer into a probability representation. For the output Z of the i-th neuron in the output layer... i The output after processing by the Softmax function is: This ensures that the sum of all probability values ​​in the output is 1, which meets the requirements of the probability distribution.

[0628] Step 9, Initialize parameters: Initialize parameters such as convolution kernels and biases in the network. Common initialization methods include random initialization (such as Xavier initialization and He initialization), which can help the model converge faster.

[0629] Step 10, Divide the model training dataset: Divide the collected image data into training set, validation set and test set, which can be divided in a ratio of 7:2:1 or 8:1:1.

[0630] Step 11, Loss Function Selection: The Cross-Entropy Loss function is used to measure the difference between the probability distribution predicted by the model and the actual probability distribution. Its expression is:

[0631]

[0632] Among them, y i y represents the elements in the actual probability distribution p, i.e., the actual labels. In classification problems, if it is a one-hot encoding, such as for a three-class classification problem, where the class labels are [0,0,1] indicating belonging to the third class, then y iIt is 1 when i=3, and 0 in other positions. It represents the probability of the sample's true class (true probability distribution), where i is the sum of values ​​from 1 to j, used to calculate the cross-entropy loss. i `predicted` represents the true probability value and the probability value predicted by the model, respectively. For example, if the model predicts that a sample belongs to the first class with a probability of 0.2, to the second class with a probability of 0.3, and to the third class with a probability of 0.5, these probability values ​​are p. i `predicted(i=1,2,3)` is the output after passing through the convolutional neural network and the softmax function. It is used to compare the probability with the actual label to measure the quality of the model's prediction. Through the softmax function and minimizing the cross-entropy loss, the convolutional neural network model can effectively learn the correct classification boundary.

[0633] Step 12, Multimodal Fusion CNN Training: Input the training set data into the constructed multimodal fusion CNN model, calculate the output through forward propagation, calculate the loss according to the loss function, and then use the optimizer for backpropagation to update the network parameters. During training, save the model parameters periodically and evaluate the model performance on the validation set. Adjust the training parameters (such as learning rate, number of training epochs, etc.) based on the validation results. In this embodiment, the training set data is used to train the convolutional neural network model. The backpropagation algorithm combined with the optimization algorithm and the Adam optimizer (improved stochastic gradient descent) is used to adjust the model's weights and biases, so that the value of the loss function continuously decreases, that is, the difference between the probability distribution predicted by the model and the actual probability distribution continuously decreases. During training, the parameters (such as kernel size, stride, number of nodes, etc.) of the convolutional layer, pooling layer, and fully connected layer of the model can be adjusted according to the training effect (such as the accuracy and loss value on the validation set) to improve the model's performance.

[0634] Step 13 provides a detailed description of the Adam optimizer (an improved stochastic gradient descent method):

[0635] The Adam optimizer maintains two moving average estimators: the first-order moment estimator *m* and the second-order moment estimator *v*. First, at each iteration *t*, for a weight of the convolutional layer with parameter θ... The formula for calculating the gradient is as follows:

[0636]

[0637] Among them, g t Let θ be the gradient, and θ be a parameter. The gradient is typically represented as the gradient of the loss function L with respect to the parameter θ (here, it's an averaged gradient, or the average of gradients over a mini-batch of data, or other forms of gradient processing). The loss function L measures the difference between the model's predictions and the true labels; when training machine learning models such as neural networks, our goal is to minimize this loss function. The gradient... This represents the rate of change of the loss function in the parameter space, indicating how the parameters should be adjusted to reduce the loss function.

[0638] To calculate the gradient, first calculate the gradient g of the loss function L with respect to the parameters θ (specifically, a weight w in the convolutional layer). t This gradient calculation, based on the current batch of data, indicates the direction in which the loss function grows fastest at the current position of the parameter.

[0639] Update the first moment estimator (momentum term):

[0640] m t =β t m t-1 +(1-β1)g t (6);

[0641] Where β1 is a hyperparameter, set to 0.9, β t It is a function used to control the previous momentum information m t-1 The parameter that assigns weight when updating the current momentum. When β t When it is close to 1, it indicates that m is being updated. t At that time, it relies more on the previous momentum m t-1 When β t When the gradient is close to 0, the current gradient information (1-β1)g will be given more importance. t .

[0642] In the given formula, the hyperparameter is only defined as β1 = 0.9, where β t It is a parameter similar to β1, and in practical applications (such as optimization algorithms like Adam), β t Typically, β is also a number within the (0,1) interval, used to balance the contributions of historical and current information to the momentum term update. If this is during a specific algorithm iteration, β... t It may also change with the number of iterations; the specific rules for this change need to be determined based on the detailed settings of the algorithm. Formula (6)g t Used to update the first moment estimate m t m t It can be viewed as the gradient g tThis is a weighted moving average that accumulates information from past gradients, analogous to momentum with friction. This momentum term helps accelerate convergence, especially when dealing with parameter updates with similar gradient directions. Initially, m0 is typically set to 0.

[0643] Update the second moment estimator:

[0644]

[0645] Here, β1 is a hyperparameter, which is set to 0.999.

[0646] The bias corrections for the first-order moment estimators and second-order moment estimators are as follows:

[0647]

[0648] Finally, update the parameters using the following formula:

[0649]

[0650] Where η is the learning rate, and ∈ is a very small number to prevent the denominator from being 0.

[0651] The improved Adam optimizer combines the momentum method (using the first moment estimate m) and the idea of ​​Adagrad (using the second moment estimate v). It can dynamically adjust the learning rate of each parameter in the model based on the first and second moments of the gradient, and achieve good learning results in the early stages of training through bias correction.

[0652] Step 14, Image Performance Evaluation: Evaluation using a test set. After training, the trained model is evaluated using a test set. Evaluation metrics such as accuracy, precision, recall, mean intersection-over-union ratio, and F1-score are calculated to assess the model's performance on unseen data.

[0653] Taking the performance evaluation indicators of multi-layer chip image accuracy as an example, the calculation formulas for evaluation indicators such as accuracy, precision, recall, and F1-score are as follows:

[0654] Accuracy:

[0655]

[0656] Wherein, TP (True Positive) represents the number of samples correctly detected as positive (e.g., correctly detecting a defective chip image); TN (True Negative) represents the number of samples correctly detected as negative (e.g., correctly detecting a defect-free chip image); FP (False Positive) represents the number of samples incorrectly detected as positive (actually defect-free but detected as defective); and FN (False Negative) represents the number of samples incorrectly detected as negative (actually defective but detected as defect-free).

[0657] Precision:

[0658]

[0659] Precision measures the proportion of truly positive samples out of those that are detected as positive, reflecting the reliability of the detection results.

[0660] Recall:

[0661]

[0662] Recall rate represents the proportion of actual positive samples that are correctly detected, reflecting the model's ability to capture positive samples.

[0663] Mean Intersection over Union (mIoU):

[0664]

[0665] Suppose we are detecting k categories in a multilayer chip image (e.g., different types of defects or normal regions are considered different categories). intersection(i) represents the area of ​​intersection between the predicted result and the ground truth label in the i-th category, and Union(i) represents the area of ​​union between the predicted result and the ground truth label in the i-th category.

[0666] Step 15, Model Visualization Analysis: Analyze the curves of loss and accuracy changes during model training using visualization tools (such as TensorBoard) to intuitively understand the model's training progress. Simultaneously, visualize the feature maps learned by the convolutional layers to see if the model has truly learned valuable features from the chip (smart meter) image.

[0667] Step 16, Determination of defects based on fused feature images:

[0668] Deployment to Production and Application Environments: The trained and evaluated model is deployed to actual product testing production lines and applications. Actual multi-layer chip images captured by the testing equipment are input into the tested image defect detection model to achieve real-time detection, measurement, diagnosis, and tracking. The detection process (taking binary classification to determine the presence of defects as an example, let the decision function be G, expressed by the following formula):

[0669]

[0670] Wherein, G(F) = 1 indicates that a defect is detected in the corresponding region of the image, G(F) = 0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that the preset defect condition is met.

[0671] The value of G(F) is a binary classification decision value. The preset defect condition can be a judgment rule such as the value of F being greater than a certain set threshold. The threshold can be determined through training or other methods. In other words, the preset defect condition, such as the value of F being greater than a certain set threshold, is a variable related to the features and parameters of the multi-layer chip image. Its meaning needs to be further clarified by combining more specific background information about the image defect detection model and related settings. For example, G(F) = 1 indicates that a defect is detected in the corresponding region of the image, and G(F) = 0 indicates that there is no defect.

[0672] Step 17, Image Defect Segmentation: First, it is necessary to accurately segment the defect region from the image. A semantic segmentation model based on the improved U-Net model of CNN deep learning can be used. The improved U-Net model has good segmentation performance and is especially suitable for multi-task and multi-modal segmentation of complex images such as weld seams, crystal oscillators, chips, IGBTs, GBAs, and the fine details inside multi-layer chips of artificial intelligence.

[0673] The semantic segmentation model based on CNN deep learning improves the U-Net model by introducing residual connections in the encoder (downsampling path) and an attention mechanism in the decoder (upsampling path).

[0674] Residual Connections (Improved) (x, y, z): Residual connections are introduced into the encoder. Let the weight matrix of the residual connections be... The output feature map after two convolutional layers Input features Figure X i (x, y, z) are summed through residual connections to obtain a new output feature map. The formula is as follows:

[0675]

[0676] Attention Mechanism (Improved): An attention mechanism is introduced into the decoder, where the attention weight matrix is ​​set as follows. Feature maps after encoder upsampling and stitching Weighted features are obtained through an attention mechanism. That is, the TFT beam pattern, the formula is as follows:

[0677]

[0678] Where (x, y, z) are image coordinates, a represents the relative position offset involved in the weighted operation when the feature map is subjected to the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[0679] Then replace Then proceed with subsequent convolutional layer operations. Once And (C) i-1 =C i+1 When the TFT ray image is input into the trained improved U-Net model, a binary image of the defect region is obtained, where the pixel value of the defect region is 1 and the pixel value of the background region is 0.

[0680] Step 18, Defect Shape Contour Analysis: Contour extraction is performed on the binary image of the defect region, which can be achieved using functions provided by image processing libraries such as OpenCV. After obtaining the contour of the defect region, the defect shape can be described by calculating some geometric features of the contour. For defects in a two-dimensional image, the perimeter C of the contour can be calculated. p convex hull perimeter C hc Shape factors (such as roundness) A represents the area of ​​the image defect region. The closer the circularity value is to 1, the closer the defect shape is to a circle. The smaller the circularity value is, the more irregular the shape is. These indicators are used to describe the shape of the image defect from different perspectives.

[0681] Perimeter C p Calculation: Extract contours from the binary image of the defect region. Let the extracted contour point set be... N is the number of contour points, then the contour perimeter C p It can be obtained by calculating the sum of the Euclidean distances between adjacent contour points, and the calculation formula is as follows:

[0682]

[0683] Where, x i+1 With x i For adjacent contour points, y i+1 With y i These are adjacent contour points.

[0684] convex hull perimeter C hc Calculation: First, calculate the convex hull of the defect region's contour. Let the set of points on the convex hull be... M represents the number of points on the convex hull. The perimeter of the convex hull is C. hc Similarly, it is determined by calculating the sum of the Euclidean distances between adjacent convex hull points, using the following formula:

[0685]

[0686] Step 19, high-precision defect feature shape hybrid detection modeling:

[0687] Calculate the moment characteristics of the defect region, such as the zeroth moment M0 (whose value equals the area of ​​the defect region), the first moment M1 (which can be used to determine the centroid of the defect region), and the second moment M2 (related to the shape of the defect region). These moment characteristics can also describe the shape of the defect to some extent. Calculate the second moment of the defect region, including the central moment.

[0688] Mean Squared Error (MSE) is used to evaluate measurement tasks, and its calculation formula is as follows:

[0689]

[0690] When measuring certain features (such as component dimensions, line width, etc.) in a product image, It is the actual value. Here, n is the predicted value, and n is the number of measurement samples. The mean square error (MSE) can be used to assess the deviation between the measured value and the true value; the smaller the value, the higher the measurement accuracy.

[0691] Step 20: Calculate the target measurement based on the fusion feature F:

[0692] Measurement process (taking the measurement of a target dimension as an example, let the measurement function be M:)

[0693] M(F)=size value calue calculated on F (22);

[0694] In other words, the dimensional values ​​of the target to be measured (such as the width, length, area, volume, etc. of chip circuits (smart meter defects, faults)) are calculated based on the fused feature vector F.

[0695] Step 21, the specific process of outputting the width, length, area, volume, etc. of defects and faults:

[0696] 1. Second moment based on discrete data: Defined as the second moment of discrete data in the defect location region.

[0697] Relationship between zeroth moment and area: Let the binary image function of the defect region be f(x, y), where the pixel value of the defect region is 1 and the pixel value of the background region is 0. The formula for calculating the zeroth moment M0 is:

[0698] M0=∑∑ f f(x, y) (23);

[0699] As can be seen from the above formula, the value of the zeroth moment is equal to the area A of the defect region, that is, A = M0.

[0700] The first moment is denoted by M1, and the first moment and the position of the centroid are calculated as follows:

[0701] The formulas for calculating the first moments M1x (about the x-axis) and M1y (about the y-axis) are as follows:

[0702] M1x=∑ x ∑ y xf(x,y) (24);

[0703] M1y=∑ x ∑ y yf(x, y) (25);

[0704] The centroid position of the defect area (x) g -y g It can be calculated using the following formula:

[0705]

[0706] Relationship between second moment and defect shape:

[0707] The formulas for calculating the second moments M2xx (the second moment about the x-axis), M2yy (the second moment about the y-axis), and M2xy (the combined second moment about the x and y axes) are as follows:

[0708] M2xy=∑ x ∑ y xyf(x,y) (27);

[0709] M2xx=∑ x ∑ y x 2 f(x, y) (28);

[0710] M2yy=∑ x ∑ y y 2 f(x, y) (29).

[0711] 2. The calculation of the second moment of a continuous function (taking two dimensions as an example) is as follows:

[0712] M 00 =∫∫D xf(x,y)dx dy (30);

[0713] For a continuous function z = f(x, y) defined on the defect location region D.

[0714] Zeroth moment M 00 calculate:

[0715] M 00 =∫∫ D f(x,y)dx dy (31);

[0716] The above equation is a double integral, and the integration region is the domain D of the function f(x, y).

[0717] First-order moment calculation:

[0718] M 10 =∫∫ D xf(x,y)dx dy (32);

[0719] M 01 =∫∫ D yf(x,y)dx dy (33);

[0720] The centroid coordinates (x, y) are given by and Sure.

[0721] The calculations for the second-order central moments are as follows:

[0722] M 2xy =∫∫ D xf(x,y)dx dy (34);

[0723] M 2xx =∫∫ D x 2 f(x,y)dx dy (35);

[0724] M 2yy =∫∫ D y 2 f(x, y)dx dy (36). The calculation of these second-order central moments requires first finding the coordinates of the centroid, and then completing the calculation through double integrals.

[0725] 3. Hybrid Second-Order Model: A hybrid second-order model is constructed, combining the second moments of continuous functions and discrete data. For the defect shape, it is preferable that the main part of the defect can be approximated by a continuous function; in this embodiment, a combination of polynomial functions or trigonometric functions is preferred. Meanwhile, at some key boundary locations or local irregularities, discrete data points are used to supplement details.

[0726] Let the continuous function be y = f(x), its domain be [a, b], and the discrete data points be (x, b). i y i )i = 1, 2, ..., n. Calculate the second moment of the continuous function part:

[0727]

[0728] in,

[0729] For the discrete data portion, its second moment is:

[0730]

[0731] in, The description of the defect shape is optimized by minimizing M by adjusting the parameters of the continuous function and the positions of the discrete data points.

[0732] 4. The following is a mathematical model for calculating the major axis, minor axis, and orientation angle of a defect shape by using a weighted average method to fuse the second moments of continuous functions and discrete data:

[0733] The desired defect shape is represented by a continuous function z = f(x, y) on a two-dimensional plane, and its domain is Ω.

[0734] Calculate the moment of inertia The formulas are as follows:

[0735]

[0736] M 2xx =∫∫ D x 2 f(x,y)dx dy (38);

[0737] M 2yy =∫∫ D y 2 f(x,y)dx dy (39);

[0738] 4.1 Calculate the equivalent ellipse parameters (continuous function part):

[0739] Let k be the major axis corresponding to the continuous function part. c short axis b c and direction angle Э c .

[0740] First, calculate:

[0741]

[0742]

[0743] Long axis short axis Direction angle

[0744] 4.2 Second Moments for Discrete Data:

[0745] There are n discrete data points (x i -y i ), with a weight of w i (If the weights are the same, then w) i =1).

[0746] Calculate the moment of inertia and The formula is as follows:

[0747]

[0748] 4.3 Calculate the equivalent ellipse parameters (discrete data part):

[0749] Let k be the major axis corresponding to the discrete data portion. d short axis b d and direction angle Э d First, calculate:

[0750]

[0751] Then the major axis short axis Direction angle

[0752] 4.4 Weighted average fusion calculation:

[0753] Let the weight of the second moment of the continuous function be a (0 ≤ a ≤ 1), then the weight of the second moment of the discrete data is 1-a. The fused moment of inertia:

[0754]

[0755] Calculate the equivalent ellipse parameters after fusion (major axis k, minor axis b, direction angle Э) d First, calculate:

[0756]

[0757] Long axis short axis Direction angle

[0758] Weighted average fused second moment: M fusion＝ =aM c +(1-a)M d , of which M c M is the second moment of a continuous function.d It is the second moment of discrete data.

[0759] Using the above model, the value of weight 'a' is adjusted according to the characteristics and accuracy of the data. The major axis, minor axis, and orientation angle of the defect shape are determined based on the fusion result of the second moment of the continuous function and discrete data. Shape visualization is then available online.

[0760] Final shape: The approximate elliptical shape of the defect area can be determined based on the ratio of the major axis to the minor axis and the orientation angle (if the defect shape is similar to an ellipse). For example, when the major axis is significantly longer than the minor axis, it presents a relatively long and narrow ellipse. The orientation angle determines its tilt angle in the plane, which together outlines its final shape characteristics on the two-dimensional plane.

[0761] Width: The minor axis length b is usually regarded as the width of the defect area, which represents the size occupied by the defect in the direction perpendicular to the major axis.

[0762] Length: The length of the major axis 'a' is defined as the length of the defect region, which is the distance the defect spans in its main extension direction.

[0763] Area: For a defect region that is approximately elliptical in shape, the area S is calculated by the formula S = πaab. The area occupied by the defect on the two-dimensional plane is obtained by multiplying the values ​​of the major axis and minor axis by pi.

[0764] Volume: If we want to consider the three-dimensional case (assuming that the defect also extends in a certain thickness direction, and the thickness is set as h), the volume V can be simply approximated by the volume of an elliptical cylinder (when the defect shape conforms to such characteristics). The calculation formula is V = πabh, that is, the volume is obtained by multiplying the two-dimensional area calculated above by the dimension in the thickness direction.

[0765] In practical applications, these calculation methods can be appropriately adjusted or more realistic professional models can be used to measure relevant parameters based on the specific physical characteristics of the defect and the degree of shape conformity.

[0766] Step 22, Defect and Fault Diagnosis:

[0767] Diagnostic phase (taking the assessment of defect severity as an example, let the diagnostic function be J:)

[0768] J(F)=severity level determined by F (58);

[0769] The severity level of the defect is determined based on the fusion feature F. For example, different F ranges can be set to correspond to different severity levels (such as mild, moderate, severe, etc.).

[0770] Step 23, Tracking Phase: Taking the change in the position of the target at different times as an example, let the tracking function be T, and let the different times be denoted as t1, t2, ... The tracking function is expressed as:

[0771] T(F,t,)=position of target at time t:calculated based on F (59);

[0772] By fusing feature F at different times t i It calculates the position of the target (such as the movement of a defect or the displacement of a component) and then tracks the target.

[0773] Step 24, Decision-making stage:

[0774] like Figure 6As shown, this is a flexible configuration detection system that integrates X-ray fusion with nanoprobe-based FIB (an algorithm for calculating the Fibonacci sequence), TEM, EMM, and OBIRCH (Optical Beam Induced Resistance Change) detection. This system achieves non-destructive testing of physical and electrical defects and faults in circuits and devices by fusing detection, measurement, diagnosis, tracking, and decision-making. The detection system includes an X-ray source flexibly integrated with FIB, TEM, EMM, and OBIRCH. It uses X-rays, electrons, photocurrents, and ion beams to detect the object under test. When a defect is detected, the nanoprobe simultaneously detects current, voltage, and resistance in the circuit to determine electrical performance faults and defects. For minor defects in the object under test, FIB, TEM, EMM, and OBIRCH are used for further detection. The transmitted X-rays, electrons, photocurrents, and ion beams hit the detector, and the information from the detector is imaged and transmitted to the image acquisition and transmission circuit. The image acquisition and transmission circuit transmits the signal unidirectionally, while the automatic control unit transmits it bidirectionally to the computer's management data correction unit. The data corrected by the correction unit is then processed by a high-precision image reconstruction unit based on the multimodal CNN / Transformer image defect diagnosis and tracking decision-making method (which consists of a multi-dimensional prediction model for image types, detection, measurement images, and evaluation based on the multimodal CNN / Transformer image defect diagnosis and tracking decision-making method). Part of the process involves layer-by-layer detection of electrical faults and performance assessments; image storage, image display, and image output; defect image visualization; fault diagnosis; fault tracking; and, based on the detection, measurement, and diagnostic results, providing fault and defect countermeasures, and improvement decisions or remaining lifespan performance for processes, materials, structures, electrical systems, quality, production, use, and the entire circuit, system, product, and device. The multimodal fusion CNN high-precision image reconstruction unit interacts with the automatic control unit. One control unit controls the positioning of different inspected items; the nanoprobe, based on defects, integrates with FIB, TEM, EMM, and OBIRCH for detection; movement in different directions achieves layered and surface-level detection of the inspected items; and another control unit controls the high-voltage generator switch. One channel provides detection energy for X-rays while simultaneously controlling the automatic detection and safety protection of FIB, TEM, EMM, and OBIRCH.

[0775] Discrete second-order moment calculation data acquisition: The sampled components were inspected using high-precision optical microscopes, electron microscopes, and optical current emission microscopes to obtain discrete data points (x, y, x) of the suspected defect area. i y i Weights w are assigned based on the clarity and reliability of the data points. i(For example, data points that are clear and have been repeatedly verified have higher weights). The weights are determined based on the accuracy assessment of the ideal model and the test data. The weight of the second moment of the continuous function is determined to be a = 0.3, and the weight of the second moment of the discrete data is 1 - a = 0.7. The equivalent ellipse parameters of the moment of inertia are calculated using the previous formula, and the weighted average fusion is used to calculate the equivalent ellipse parameters of the fused moment of inertia (major axis k, minor axis b, direction angle Эd). Through this weighted average fusion method, the defect shape of the meter sampling components is reproduced, providing a basis for evaluating component quality and improving production processes.

[0776] Based on the above description, the invention details of the improved decision-making scheme for tracking defects and faults in smart meter sampling components are further described in detail: a systematic improvement and predictive decision-making scheme for faults and defects in meter sampling components based on multimodal fusion and second-order moment analysis.

[0777] I. Deepening the Fault and Defect Analysis System:

[0778] 1. Defect type and failure mechanism mapping:

[0779] Material defects: Wear on the resistive surface leads to increased contact resistance, and defects in the capacitor dielectric cause fluctuations in the dielectric constant. Both directly affect the voltage / current conversion accuracy of the sampling circuit. Specifically, when the resistive material wears, its conductive layer thins, according to the law of resistance... A decrease in cross-sectional area S leads to an increase in resistance; defects in the capacitor dielectric alter the internal electric field distribution, causing a change in the dielectric constant ε, thus affecting the capacitance. The stability of the sampling circuit ultimately affects its voltage / current conversion accuracy.

[0780] Welding defects: Cracks and poor soldering in hard connections lead to unstable contact resistance, causing intermittent loss or distortion of the sampling signal. Cracks and poor soldering at the weld joint create an unstable contact interface, causing the contact resistance to fluctuate under different operating conditions, thus resulting in intermittent abnormalities in the sampling signal.

[0781] Structural defects: Voids disrupt the internal electric field distribution of components, causing localized overheating or electrical parameter drift. The presence of voids alters the dielectric distribution inside the component, causing the electric field to concentrate at the void's edge, resulting in excessively high local electric field strength, leading to localized overheating, and consequently affecting the component's electrical parameters.

[0782] By establishing a defect-fault causal matrix, and using the fusion analysis of the second moments of continuous functions and discrete data to analyze defect shape parameters (major axis, minor axis, and orientation angle), the contribution of different defect types to measurement error is quantified. For example, for resistance wear defects, by calculating the major and minor axis parameters of its equivalent ellipse, and combining historical data to establish a correlation model between wear degree and measurement error, the influence weight of this defect on measurement error is determined.

[0783] 2. Dynamic assessment model for defect severity:

[0784] J(F) function extension: The fused feature F is decomposed into three types of sub-features: geometric features (defect area, perimeter), physical features (resistance change rate, dielectric loss angle), and spatiotemporal features (defect propagation speed). The weights are determined using the Analytic Hierarchy Process (AHP), and a weighted diagnostic function is constructed. Among them, geometric features are obtained by analyzing discrete data points (x) i ,y i The data is processed to obtain the moment of inertia and equivalent ellipse parameters calculated by combining the second moments of continuous functions and discrete data using the weighted average method, which further determines the area and perimeter of the defect; physical characteristics are calculated by parameters such as current, voltage, and resistance in the online detection circuit; and spatiotemporal characteristics are obtained by analyzing the changes in the position and shape of the defect at different times using the tracking function T(F,t).

[0785] Dynamic threshold adjustment: By combining the meter's operating conditions (such as load fluctuations and ambient temperature), a threshold correction model is trained using historical data to avoid misjudgments. Defect image information obtained from a multimodal fusion CNN high-precision image reconstruction unit is used, combined with real-time operating data, to dynamically adjust the defect severity threshold under different operating conditions. For example, in high-temperature environments, the threshold for judging defect severity is appropriately lowered to detect potential risks in advance.

[0786] 3. Fault tracing and location optimization:

[0787] Enhanced T(F,t) function: A defect evolution dynamics model is introduced, expanding the tracking target from a single location to changes in defect morphology (such as crack propagation angle and wear depth increment). Based on discrete data points ((xi,yi)) and a second-moment model of continuous functions, the morphological change characteristics of defects at different times are analyzed, and dynamic tracking of defect morphological changes is achieved through the tracking function \(T(F,t)\).

[0788] Multimodal data collaboration: A 3D model of the defect is constructed based on X-ray computed tomography (CT). Microstructural information is acquired using a FIB-SEM dual-beam system. OBIRCH is used to locate electrical anomalies, achieving spatial mapping between physical defects and electrical faults. When a defect is detected, parameters such as current, voltage, and resistance in the circuit are simultaneously detected using nanoprobes. Combined with multimodal detection data, the fault location is precisely pinpointed. For example, the macroscopic location of the defect is determined by X-ray CT scanning, then the microstructure of the defect area is analyzed using FIB-SEM, and finally, electrical performance is detected using OBIRCH, thus correlating and locating the physical defect with the electrical fault.

[0789] II. Improvement and Upgrading of Decision-Making Schemes:

[0790] 1. Quantitative verification of process improvement:

[0791] Welding process: Finite element method (FEM) simulation was used to optimize laser welding parameters, with residual stress distribution and fatigue life of the weld joint as evaluation indicators. Ultrasonic C-scan technology was introduced to achieve non-contact quantitative detection of weld quality. Through finite element simulation, the effects of different laser welding parameters (such as power, welding speed, and spot diameter) on the residual stress distribution and fatigue life of the weld joint were analyzed to determine the optimal welding parameters.

[0792] Welding Process: Finite element method (FEM) simulation was used to optimize resistance brazing parameters, with residual stress distribution and fatigue life of the weld joint as evaluation indicators. Ultrasonic C-scan technology was introduced to achieve non-contact quantitative detection of weld quality. Through FEM simulation, the effects of different multi-source power supply welding parameters (such as power, welding speed, and spot diameter) (photovoltaic, green electricity, energy storage, mains power, combined power, and nuclear power) on residual stress distribution, fatigue life, welded joint area, and resistance of the weld joint were analyzed to determine the optimal welding parameters.

[0793] Ultrasonic C-scan technology is used to perform non-destructive testing on welded joints, obtain information on internal defects (such as cracks and porosity), and quantify the welding quality.

[0794] Material Replacement: A material performance database was established to compare the long-term stability of metal film resistors and carbon film resistors under different temperature and humidity environments. Accelerated aging tests were used to verify the wear resistance of ceramic dielectric capacitors. The performance parameters (such as the rate of change of resistance and capacitance) of different materials under various environmental conditions were recorded in the material performance database. Accelerated aging tests simulated the aging process of materials during long-term use, evaluating the wear resistance of ceramic dielectric capacitors and providing a basis for material replacement.

[0795] Online Inspection: Deploy a deep learning-based real-time image recognition system to perform millisecond-level analysis of X-ray and OBIRCH inspection data, enabling automatic defect labeling and classification. A defect recognition model is established by training X-ray and OBIRCH inspection images using deep learning algorithms. This model can quickly identify defects in images and automatically label and classify defects based on their shape parameters (obtained through second-order moment fusion analysis).

[0796] 2. Intelligent quality inspection system:

[0797] Multimodal inspection process automation: Develop collaborative control software for inspection equipment to automatically switch inspection modes according to preset rules; X-ray coarse positioning of defect areas: Utilize the penetrating power of X-rays to perform an overall scan of the sampled components and quickly locate the approximate position of defects.

[0798] FIB for micro-area sample preparation and compositional analysis: Based on X-ray localization, FIB is used to prepare micro-area samples of the defect area, and the compositional analysis of the sample is performed by electron microscopy to obtain material information of the defect area.

[0799] OBIRCH scanning to locate electrical anomalies: Using OBIRCH technology, the electrical performance of components is scanned to locate electrical anomalies and determine the impact of defects on electrical performance. TEM acquisition of crystal structure information: High-resolution imaging of the defect area is performed using TEM to acquire crystal structure information and conduct in-depth analysis of the defect's microstructure.

[0800] Data fusion decision-making: The weighted average fused second-order moment parameters (major axis k, minor axis b, orientation angle θ) are input into a support vector machine (SVM) classifier to jointly determine the defect shape and severity, and output improvement suggestions (such as rework or scrap). The fused second-order moment parameters are trained and classified by the SVM classifier to establish a mapping relationship between defect shape, severity, and improvement suggestions. Based on the classification results, corresponding improvement suggestions are automatically generated.

[0801] 3. Improved reliability due to design improvements:

[0802] Redundancy Design Optimization: A dynamic redundancy strategy is adopted, automatically switching backup components based on real-time monitored performance parameters of the main component to avoid metering interruptions during the switching process. By setting redundant backup components in the circuit and monitoring the performance parameters of the main component (such as resistance value and capacitance) in real time, the circuit automatically switches to the backup component when the performance of the main component drops to a certain threshold, ensuring the normal operation of the sampling circuit.

[0803] Environmentally Adaptive Design: A nano-level moisture-proof coating is applied to the surface of the components, and a miniature temperature sensor is integrated internally. An adaptive heat dissipation structure regulates the operating temperature. The nano-level moisture-proof coating effectively prevents moisture from penetrating the components, protecting their electrical performance. The miniature temperature sensor monitors the component's operating temperature in real time; when the temperature becomes too high, the adaptive heat dissipation structure automatically activates to adjust the operating temperature, improving the component's environmental adaptability.

[0804] III. Enhancement of Predictive Decision-Making Models:

[0805] 1. Hybrid Modeling for Remaining Life Prediction: This approach combines a physical model (such as the Arrhenius equation for wear rate) with a data-driven model (LSTM neural network). It inputs multi-source data (defect size, operating temperature, voltage fluctuations) and outputs a probability distribution of remaining life. The Arrhenius equation describes the relationship between material wear and temperature, establishing a physical model. Simultaneously, an LSTM neural network is used to learn from historical inspection data, capturing the nonlinear patterns of defect development. The multi-source data is then input into the hybrid model, and through the synergistic effect of the physical and data-driven models, the probability distribution of remaining life is output.

[0806] Monte Carlo simulation is used to quantify prediction uncertainty and generate risk warning intervals. Through Monte Carlo simulation, the model input parameters are randomly sampled to simulate the remaining life prediction results under different conditions. The distribution of the prediction results is statistically analyzed to quantify prediction uncertainty and generate risk warning intervals, providing a reference for maintenance decisions.

[0807] 2. Intelligent Maintenance Decision Support System:

[0808] Tiered maintenance strategy: Minor defects: Activate adaptive filtering algorithm to compensate for measurement errors and extend the detection cycle. Based on the defect severity assessment results, when the defect is minor, the sampled data is processed using an adaptive filtering algorithm to compensate for measurement errors and appropriately extend the detection cycle, thereby reducing maintenance costs.

[0809] Moderate defects: This triggers a reinforcement learning algorithm to optimize operating parameters and schedule planned maintenance. For moderate defects, the operating parameters of the sampling circuit are optimized using a reinforcement learning algorithm to improve circuit performance. Based on the remaining lifetime prediction results, planned maintenance is scheduled to prevent failures.

[0810] Severe defects: The system is activated in conjunction with the power grid dispatching system to achieve "zero-power-outage" replacement. When a severe defect is detected, the system immediately activates the power grid dispatching system to adjust the power grid operation mode, enabling "zero-power-outage" replacement of the sampling components without affecting user power consumption.

[0811] Dynamic Inventory Management: Based on remaining life prediction results, an optimized spare parts inventory strategy is adopted using image defect diagnosis and tracking based on multimodal fusion CNN and Markov Decision Process (MDP) to balance inventory costs and downtime risks. Through the MDP model, the inventory costs and downtime risks under different spare parts inventory strategies are analyzed. Based on the remaining life prediction results, the spare parts inventory level is dynamically adjusted to achieve the optimal balance between inventory costs and downtime risks.

[0812] 3. Full Lifecycle Digital Twin Driven: Constructing a digital twin of the meter sampling components to synchronize detection data, operating status, and maintenance records in real time. Utilizing multimodal detection data and sensor-collected data, a digital twin model of the meter sampling components is built. This model can reflect the actual status of the components in real time, including defect information, operating parameters, and maintenance records.

[0813] By using simulation optimization to evaluate the impact of different improvement schemes (such as material replacement and process adjustment) on the product lifecycle cost, a quantitative basis for long-term decision-making is provided. In a digital twin model, different improvement schemes are simulated to assess their impact on product performance, reliability, and lifecycle cost. Through comparative analysis, the optimal improvement scheme is selected, providing a scientific basis for long-term decision-making.

[0814] IV. Improved Implementation Support System:

[0815] 1. Technology Verification and Standardization: Establish a verification test platform to simulate extreme operating conditions (high temperature, high humidity, strong electromagnetic interference) and verify the effectiveness of the improvement scheme. On the verification test platform, conduct various extreme condition tests on the improved meter sampling components, monitor component performance changes, and evaluate the actual effect of the improvement scheme. Develop the "Specification for Defect Detection and Handling of Meter Sampling Components" to standardize data acquisition, analysis, and decision-making processes. The specification clearly defines the data acquisition methods for multimodal detection, the calculation process for second-order moment analysis, the evaluation criteria for defect severity, and corresponding decision-making measures to ensure consistency and standardization in the detection and handling process.

[0816] 2. Intelligent Inspection System Integration: Develop a multimodal data management platform to achieve seamless integration of inspection equipment, analysis algorithms, and decision-making systems. This platform can integrate data from inspection equipment such as X-ray, FIB, TEM, and OBIRCH, transmit the data to the analysis algorithm module for processing, and then feed the analysis results back to the decision-making system, achieving efficient data flow and collaborative system operation.

[0817] Deploying edge computing nodes allows for local data preprocessing and preliminary diagnosis, reducing the pressure on cloud transmission. Edge computing nodes can preprocess detection data in real time, extract key feature information, and perform preliminary diagnosis. The diagnostic results and necessary data are then transmitted to the cloud, reducing data transmission volume and improving system response speed.

[0818] 3. Talent and Organizational Support: An interdisciplinary team (materials science, electronic engineering, data analysis) will be formed to be responsible for tackling technical challenges and implementing solutions. Team members possess expertise in different fields, enabling them to research and resolve faults and defects in meter sampling components from multiple perspectives, ensuring the feasibility and effectiveness of the improvement plan.

[0819] Conduct AI-assisted detection training to improve operators' ability to interpret multimodal data and handle anomalies. Through training, operators will master the use of multimodal detection equipment, the principles and operating procedures of AI analysis algorithms, accurately interpret multimodal data, promptly detect and handle anomalies, and improve the efficiency and quality of detection and maintenance work.

[0820] The above solution enables closed-loop management across the entire chain, from defect detection and fault diagnosis to predictive maintenance, significantly improving the reliability and intelligence level of meter sampling components.

[0821] In actual electricity meter sampling component examples, the weighted average method is used to fuse the second moments of continuous functions and discrete data to express the defect shape, and the long axis, rectangular axis, and direction angle are used to reproduce and construct the defect shape.

[0822] Background of Testing Electricity Meter Sampling Components: Electricity meter sampling components are crucial for accurate electricity measurement. During production and use, defects may occur due to material issues, welding cracks, voids, poor soldering, or wear and tear from long-term operation. These defects affect sampling accuracy, leading to inaccurate meter readings.

[0823] For example, the sampling resistor may have localized wear or uneven conductive layer, the sampling hard connection may have welding cracks, and the capacitor may have dielectric defects.

[0824] Construction of a theoretical model for calculating the second moment of a continuous function: For a typical defect in the sampling components of an electricity meter, such as material loss on the surface of the sampling resistor or capacitor due to wear, its shape on a two-dimensional plane can be approximated by a parabolic function based on the principles of materials physics and electricity.

[0825] Let the function f(x, y) = AB[(x - x0)] 2 +(y-y0) 2 (Within the defect region), where (x0, y0) are parameters related to the resistive material and the degree of wear, and y0 is the location of the wear center. Based on the second-order moment model of a continuous function, the moment of inertia is obtained through integration (the integration region is the wear area, crack region, and circular wear region with a wear radius of r on the resistive surface).

[0826] (Function x0 and function y0 are symmetric, and their direction angles are both 0) Equivalent ellipse parameter calculation (continuous function part) Refer to step 21.

[0827] Example of multilayer chip defect analysis:

[0828] Background of Multilayer Chip Manufacturing and Inspection: During the manufacturing process of multilayer chips, various defects may arise due to complex process steps such as photolithography, etching, and deposition. These defects may exist between or within different layers, severely impacting chip performance. To detect these defects, various techniques are typically used, including electron microscopy imaging and X-ray inspection. These methods can provide theoretical models of defects in continuous functional form as well as discrete data points. Model Establishment: For a certain type of short-circuit defect in a conductive layer of a multilayer chip, based on electromagnetic field theory and circuit models, its shape in a two-dimensional plane can be approximated by a function combining a log-normal distribution and trigonometric functions. For example, Where A is a coefficient related to the defect current density, (x0, y) is the estimated position of the defect center on the x-axis, and k and b are parameters related to the defect range.

[0829] Weight determination: Based on the accuracy assessment of the theoretical model (based on previous studies and verifications of similar defects) and the reliability assessment of electron microscope data, the weight of the second moment of the continuous function is determined to be 0.4, and the weight of the second moment of the discrete data is 0.6.

[0830] By using the steps of this invention, the second moments of continuous functions and discrete data are fused using a weighted average method to obtain the major axis, minor axis, and orientation angle parameters that can reproduce the shape of chip defects. This allows for a more accurate description of defects in the chip and provides a basis for subsequent chip repair or quality improvement.

[0831] To better understand Figure 5 and Figure 6 The above-mentioned features of the invention are used for defect tracking and decision-making in smart meter sampling components. The overall picture and technical effects of the invention are highlighted as follows:

[0832] I. Deepening the Multimodal Data Acquisition System:

[0833] Defect location image acquisition range: Acquire defect location images of smart meter sampling components (such as resistors, capacitors, solder joints) under different production batches, process stages (photolithography, welding, packaging) and testing equipment (X-ray, electron microscope, OBIRCH).

[0834] Data collection scope coverage:

[0835] Image-based data: 3D structure from X-ray computed tomography, thermal radiation distribution from thermal infrared imaging, internal structure from ultrasound imaging, and microscopic crack images from FIB-SEM.

[0836] Electrical data: Current anomalies located by OBIRCH, resistance / voltage fluctuation parameters collected by nanoprobes;

[0837] Operating condition data: real-time load fluctuations, ambient temperature (-20℃~70℃) and humidity data.

[0838] Multimodal data definition and fusion fundamentals.

[0839] Three-dimensional structural information: The spatial distribution of defects such as voids and interlayer short circuits inside components is obtained through X-ray CT scanning;

[0840] Thermal radiation information: Abnormal heating areas in thermal infrared images (such as a sudden temperature rise when a resistor is overloaded) can map electrical faults;

[0841] Electrical parameter data: Sudden changes in contact resistance detected by nanoprobes (such as resistance fluctuations >10% caused by poor soldering) are directly related to welding defects;

[0842] Operating condition data: Load fluctuations (such as sudden increases or decreases in current) and changes in ambient temperature can accelerate defect evolution and need to be collected simultaneously.

[0843] II. Data Preprocessing and Feature Fusion Process:

[0844] Timestamp synchronization and standardization:

[0845] Align X-ray images (acquisition interval 10ms), electrical parameters (sampling frequency 1kHz), and operating condition data (5-minute sliding window) by timestamps to construct a spatiotemporally consistent dataset;

[0846] Electrical parameters such as resistance, current, water flow rate, pressure, air flow rate, and temperature are normalized using the formula x_normalized=(x-x_min) / (x_max-x_min) to eliminate the influence of dimensions.

[0847] Feature extraction and location encoding:

[0848] Defect location coding:

[0849] For the defect center coordinates (x_d, y_d), the image is divided into an m×m grid, and converted into a length m using one-hot encoding. 2 For example, when m=16, the vector dimension is 256;

[0850] Image feature extraction:

[0851] ResNet-18 was used to extract edge / texture features, and the contrast C (difference between gray levels of defects and background), correlation R (texture continuity), energy E (uniformity of gray level distribution), and entropy H (information uncertainty) were calculated.

[0852] The defect entropy value D is calculated using a grayscale histogram, reflecting the severity of the defect (e.g., the entropy value of a deep crack is higher than that of surface wear).

[0853] Electrical and timing characteristic conversion:

[0854] Perform Fourier transform on the current / voltage data to extract frequency domain features (e.g., abnormal 100Hz harmonic components indicate poor contact).

[0855] Operating condition data (temperature, load) are used to calculate statistical characteristics such as mean and variance through a sliding window.

[0856] Weighting and Multimodal Fusion:

[0857] Dynamically adjust feature weights based on defect type:

[0858] Welding cracks: Contrast C (25%–30%) + Correlation R (20%–25%) + Location texture P (25%–30%);

[0859] Resistance wear: Entropy H (35%–40%) + Defect entropy D (15%–20%) + Electrical parameters (20%–25%);

[0860] The fusion formula is: F = w1·C + w2·R + w3·E + w4·H + w5·D + w6·P, where the sum of the weights is 1.

[0861] III. Comparison of CNN and Transformer Fusion Strategies:

[0862] Option 1: Multimodal fusion CNN (suitable for simple defects).

[0863] Early fusion (feature level): The stitched image features (256-dimensional), electrical features (16-dimensional), and operating condition features (8-dimensional) are combined into a unified 280-dimensional vector;

[0864] Network architecture:

[0865] Input layer → 3 convolutional layers (kernel size 3×3, 3×3, 5×5) → pooling layer → 2 fully connected layers (number of nodes 128 → n) → Softmax output;

[0866] The activation function uses RReLU to alleviate the gradient vanishing problem;

[0867] Loss function: Cross-entropy loss L = -∑yi·log(pi), trained using the Adam optimizer (β1 = 0.9, β2 = 0.999).

[0868] Option 2: Multimodal Fusion Transformer (suitable for complex defects).

[0869] Early fusion (feature level): Direct cross-modal fusion through self-attention mechanism, without feature splicing, capturing semantic associations (such as "high temperature + abnormal resistance" pointing to material aging);

[0870] Decision-level integration:

[0871] Parallel processing of multi-channel CNN features (such as X-ray images + thermal infrared images);

[0872] Cross-attention mechanisms fuse features from different modalities, such as the spatial mapping between OBIRCH electrical anomalies and FIB microstructure images;

[0873] Advantages: It has a stronger ability to model long dependencies in time-series data (such as defect propagation processes).

[0874] IV. Hybrid modeling and defect shape analysis, including:

[0875] Second-order rectangular modeling process:

[0876] Discrete data second moment: Extract n defect edge points (xi, yi) from an electron microscope image, and calculate the contour perimeter Cp = ∑√[(xi+1-xi)]. 2 +(yi+1-yi) 2 ], Circularity Rc=4πA / Cp 2 (A represents the defect area);

[0877] Second moment of a continuous function:

[0878] For resistive wear defects, the fitted parabolic function is f(x,y)=AB[(x-x0)]. 2 +(y-y0) 2 The moment of inertia, Ixx^c, is calculated by integration as πA / 4·r^4 - πB / 6·r^6.

[0879] Weighted fusion:

[0880] Weight a = 0.3 (continuous function) + 0.7 (discrete data), calculate the fused moment of inertia: Ixx = a·Ixx^c + (1-a)·Ixx^d, and then obtain the equivalent ellipse major axis a, minor axis b, and direction angle θ;

[0881] Defect quantification parameters:

[0882] Length = major axis a, width = minor axis b, area = π·a·b, volume = π·a·b·h (h is the defect depth).

[0883] V. Decision-level Applications and Improvement Plans:

[0884] Defect diagnosis and tracking:

[0885] Improved U-Net model: The encoder introduces residual connections, the decoder adds an attention mechanism, and segments defect regions (such as binarized images of weld cracks);

[0886] The tracking function T(F,t) calculates the defect location offset (e.g., crack propagation rate of 0.1 mm / month) and morphological changes (long axis growth rate) using the fusion feature F at different times.

[0887] Process improvement decisions:

[0888] Welding process optimization:

[0889] Based on finite element simulation, the laser welding power was adjusted from 20W to 25W, reducing the residual stress in the joint (from 300MPa to 150MPa), and the welding area was increased by 20% by ultrasonic C-scan.

[0890] Material replacement strategy:

[0891] Replacing carbon film resistors with metal film resistors reduces resistance fluctuation from ±5% to ±1% at high temperatures (70℃), and extends lifespan by 3 times as verified by accelerated aging tests (85℃ / 85% RH, 1000 hours).

[0892] Intelligent maintenance strategy:

[0893] Minor defects (such as surface wear): Adaptive filtering algorithm is activated to compensate for measurement errors, and the detection cycle is extended from 1 month to 3 months;

[0894] Severe defects (such as inter-floor short circuits): Link the power grid dispatch system to achieve "zero power outage" replacement, and measure the interruption time of the switching process through digital twin simulation (<50ms).

[0895] VI. Key Innovations and Implementation Safeguards:

[0896] Technological Innovation:

[0897] Cross-modal complementarity: Combining image geometric features (such as crack length) with electrical features (resistance abrupt changes) improves diagnostic accuracy to 98.5%;

[0898] Hybrid modeling: Fusing discrete point clouds (edge ​​detection) with continuous functions (physical model), defect shape description error <5%;

[0899] Multi-stage fusion: Simple defects are fused in the early stage using CNN, while complex defects are fused at the decision level using Transformer, improving computational efficiency by 40%.

[0900] Implementation support system:

[0901] Establish an extreme working condition test platform (-40℃~125℃, humidity 0~100%) to verify the reliability of the improvement scheme;

[0902] The "Specification for Defect Detection of Smart Meters" was formulated to unify the multimodal data acquisition frequency (10fps for images and 1kHz for electrical parameters) and analysis process.

[0903] An interdisciplinary team (materials science + AI algorithm + electrical engineering) collaborates to visualize the model training process using TensorBoard and optimize parameters in real time.

[0904] The present invention also has the following prominent features:

[0905] A multimodal CNN / Transformer image defect diagnosis and tracking decision method, the method comprising:

[0906] Multimodal data acquisition: Acquires various types of data, including three-dimensional structural information (X-ray CT), thermal radiation information (infrared imaging), and electrical anomaly point data (OBIRCH) of smart meter sampling components, covering different production batches and process stages;

[0907] Data preprocessing and alignment: Construct a spatiotemporally consistent multimodal dataset through timestamp synchronization, image enhancement, and feature normalization;

[0908] Feature extraction and transformation: The defect location is encoded, and image features (contrast, correlation, etc.) are extracted by combining CNN, and the electrical parameters are converted into frequency domain features;

[0909] Multimodal fusion strategy:

[0910] CNN approach: Early fusion and splicing of feature vectors, suitable for simple defects;

[0911] Transformer approach: Self-attention mechanism directly fuses across modalities, suitable for temporal correlation analysis of complex defects;

[0912] Hybrid modeling and shape analysis: Based on the weighted fusion of second moments of discrete data and second moments of continuous functions, parameters such as the major axis and minor axis of defects are calculated to quantify shape characteristics;

[0913] Decision-level applications: Defect segmentation is achieved by improving U-Net, and a closed loop of "detection-diagnosis-decision" is formed by combining the tracking function T(F,t) with process optimization strategies.

[0914] As one or more specific application embodiments of the present invention, combined with Figures 6 to 1 1. The image defect diagnosis and tracking decision method based on multimodal CNN / Transformer provided by the present invention will be further described in detail below:

[0915] Example 1: such as Figure 7As shown, this invention identifies and detects TVS diodes (SMBJ10CA) in smart meters using X-ray detection to pinpoint the location of weld voids, measure their defect shape, size, and area, and illustrates the defect criterion that a total weld void area exceeding 5% exceeds the standard. This invention, through the identification, diagnosis, and tracking of non-conforming products detected by X-ray imaging, provides specific basis for weld repair adjustments in the welding process; thus, during the welding process, non-conforming products are repaired online and converted into conforming products.

[0916] As shown in Figures 8(a), 8(b), 8(c), and 8(d), for... Figure 7 The diagram shows the continuous tracking of non-conforming products with a void ratio of 5.1%. The devices in Figures 8(a), 8(b), 8(c), and 8(d) are all TVS diodes (SMBJ10CA), referred to as sample 107#. Figure 8(a) is a front X-ray image of sample 107#. Figure 8(b) is a magnified X-ray image of a portion of Figure 8(a), showing voids in the contact area (total void ratio of 5.1%). Figure 8(c) is a side X-ray image of sample 107#. Figure 8(d) is a magnified X-ray image of a portion of Figure 8(c). For TVS diodes (SMBJ10CA), the shape, size, and area of ​​voids are continuously tracked over different time periods. The process capability of TVS diodes (SMBJ10CA) is calculated simultaneously (the mean, upper and lower limits, CP, CPL, and CPK of the process capability are calculated). Based on the void development process capability, the consistency of TVS diodes (SMBJ10CA) is predicted and judged. Insufficient process capability of VBR minimum breakdown voltage and IR reverse working voltage may lead to many non-conforming specifications. This provides a basis for improving the performance of products in process manufacturing and the consistency of VBR minimum breakdown voltage and IR reverse working parameters.

[0917] The process capability index and void ratio assessment standards for TVS diode SMBJ10CA are as follows: Process capability assessment of VBR(+). The passing standard is typically Cp ≥ 1.33 and Cpk ≥ 1.0. Cp measures the potential production capacity of the process; Cp ≥ 1.33 indicates that the natural fluctuations of the process are small, and it can better meet the specifications. Cpk considers the process center offset; Cpk ≥ 1.0 indicates that even considering offset, the process still has a high capacity to meet the specifications, and the product consistency is good. The failing standard is when Cp < 1.33 or Cpk < 1.0, the process capability of VBR(+) is considered unqualified, meaning that there is a large variation or center offset in the process, resulting in unstable breakdown voltage characteristics of the product, which may not effectively protect the circuit.

[0918] The process capability assessment for IR(+) includes the following acceptable standards: Generally, CP ≥ 1.0, CPL ≥ 0.8, and CPK ≥ 0.8 are expected. CP reflects the overall process capability; CP ≥ 1.0 indicates a certain degree of stability and reliability. CPL focuses on the lower limit capability; CPL ≥ 0.8 means the process has good control over the lower limit, ensuring the reverse leakage current does not exceed the specified lower limit. CPK comprehensively considers offset; CPK ≥ 0.8 indicates that the overall process capability is still acceptable when considering center offset, and the reverse leakage current performance of the product is relatively stable. The unacceptable standard is: if CP < 1.0, CPL < 0.8, or CPK < 0.8, the process capability of IR(+) is deemed unacceptable. This means the process may have significant fluctuations or center offset, increasing the risk of reverse leakage current exceeding the specification limit and affecting the normal operation of the TVS diode.

[0919] The void ratio is assessed in several ways. The acceptable standard in the electronic components industry is generally required to be below 5%, and this applies to the TVS diode SMBJ10CA as well. A lower void ratio helps ensure the integrity of the diode's internal structure and performance stability, reducing problems such as decreased electrical performance and reliability caused by voids. The unacceptable standard is a void ratio exceeding 5%, which is generally considered unacceptable. An excessively high void ratio may make the diode more prone to localized overheating and electric field concentration during operation, thus affecting key performance parameters such as breakdown voltage and reverse leakage current, reducing product lifespan and reliability.

[0920] Example 2: such as Figure 9 As shown in the diagram, this invention identifies and measures the location of weld voids in the rectifier diode (M7) of a smart meter, and tracks the development of defects from qualified products. The total area of ​​weld voids is less than 2.4%, which does not exceed the standard defect criterion. The arrows indicate the void locations. Based on the identification and location detection of weld voids in the rectifier diode (M7), and the measurement of the void shape, size, and area, it is judged as a qualified product according to the standard, and no resoldering is recommended. Simultaneously, by continuously tracking the shape, size, and area of ​​voids at different time periods, the process capability of the rectifier diode (M7) is calculated (calculating the mean, upper and lower limits, CP, CPL, and CPK). Based on the void development process capability, the consistency of the rectifier diode (M7) is predicted (sufficient VBR minimum breakdown voltage and IR reverse working voltage process capability, ensuring continuous production of qualified products without specification defects; if a qualified CPK exceeds 1.67, it predicts in advance that the product performance will be unqualified, providing a basis for improving the consistency of VBR minimum breakdown voltage and IR reverse working parameters).

[0921] Example 3: such as Figure 10The diagram illustrates the process of identifying, detecting, and measuring the location of welding cracks in the current transformer of a smart meter using this invention, tracking the development of cracks in defective products, and determining the total area of ​​welding cracks to be less than 2.8%, exceeding the standard defect criterion. The left image is an X-ray image of the side of the current transformer, and the right image is a magnified view of a portion of the left image. Based on the identification and location detection of the welding cracks in this current transformer, and the measurement of the crack shape, size, and area, it is judged as a defective product according to the standard, and isolation is recommended. Simultaneously, by continuously tracking the shape, size, and area of ​​cracks at different time periods, the welding crack process capability of the current transformer is calculated (calculating the mean, upper and lower limits, CP, CPL, and CPK of the process capability). Based on the prediction and judgment of the crack development process capability, if the current transformer parameter consistency process capability is sufficient, even if there are non-compliant products, qualified products can be continuously produced. Once the qualified CPK is less than 1.67, it is predicted in advance that the performance of the product produced by the process will be unqualified. The cracks in this signal line and other adverse effects will have poor conductivity and safety reliability. It provides a basis for improving the consistency of its working parameters, equipment maintenance and upgrading, equipment parameter adjustment, and raw material and process optimization.

[0922] Example 4: As shown in Figures 11(a), 11(b), and 11(c), this invention is used to identify, detect, and measure the location of solder holes in the chip, leads, leads, and substrate of a smart meter, and to detect and track the development of defects in the shape, size, and area of ​​qualified products. The total area of ​​the solder holes is less than 2.4%, which does not exceed the standard defect criterion. After identifying and detecting the location of the solder holes in the chip, leads, leads, and substrate, and measuring the shape, size, and area of ​​the holes, it is judged as a qualified product according to the standard, and it is recommended that no resoldering is required. Simultaneously, by continuously tracking the shape, size, and area of ​​voids at different time periods, the bonding void capability between the chip and leads, terminals, and substrate is calculated (calculating the mean, upper and lower limits, CP, CPL, and CPK of the bonding process). Based on the void development capability prediction, it is determined that the parameter consistency of the chip-lead-terminal-substrate bonding assembly is sufficient, preventing specification defects and ensuring continuous production of qualified products. If a qualified CPK exceeds 1.33, it is predicted in advance that the product performance will be substandard, providing a basis for improving the consistency of operating parameters. Generally, in the chip bonding field, a void rate exceeding 5%-10% may attract attention. When the void rate reaches approximately 20%-30%, it is likely to have a serious negative impact on the chip's electrical, mechanical, and heat dissipation performance, and is usually considered substandard. When a chip is found to be defective, X-rays are used to detect defects. TEM is used to display the chip's materials, the gate layer thickness of transistors, and cracks and voids in metal interconnects. FIB enables rapid and accurate defect localization. Nanoprobes are used to detect defects in the source, drain, and gate of transistors, as well as defects below 10 nanometers, or key parameters such as current, voltage, and resistance in metal interconnects. Atomic-level failures such as voids and cracks are also detected. Saturation current failures, leakage, open circuits, and short circuits are detected. This provides a basis for flexibly providing process improvement measures for defects in different layers and surfaces, achieving non-destructive testing and diagnosis of the circuit devices in the chip.

[0923] Example 6, such as Figure 12The diagram illustrates the detection of protrusions, voids, cracks, and insufficient soldering in the PCB, crystal oscillator, and seven major chips of a smart meter. Faults and defects such as protrusions, voids, cracks, and insufficient soldering in the seven major chips of a smart meter—metering chip, main control chip (MCU chip), communication chip, security encryption chip (ESAM chip), clock chip, memory chip, RF front-end chip, and crystal oscillator—can affect metering performance, accuracy, reliability, stability, and lifespan. The real-time example of this invention is not limited to using X-rays (thermal infrared and ultrasonic detection can also be used), combined with electron microscopy, FIB, TEM, EMM, OBIRCH, and a multimodal CNN fusion scheme with X-rays (thermal infrared and ultrasonic detection can also be used) to detect internal circuit defects in the chips. First, X-rays (thermal infrared and ultrasonic detection can also be used) penetrate the chip to obtain an image of its internal structure, which can detect macroscopic defect information such as the location of open circuits and short circuits. Multimodal CNN performs high-precision analysis of images of the chip surface and subsurface, detecting, measuring, and tracking minute fault features such as micro-cracks, scratches, protrusions, voids, and insufficient soldering. When fusing these two images, X-ray images (and possibly thermal infrared or ultrasonic detection) and chip surface images are input into the multimodal CNN model of this invention. After training with a large number of samples, the model can comprehensively judge the type and severity of defects and faults. For example, it can accurately distinguish between circuit defects caused by manufacturing processes and physical damage during use, and make subsequent repair or replacement decisions based on the results, achieving comprehensive and accurate detection and diagnosis of chip defects and faults. In the detection and measurement of voids and cracks in cases with sufficient online testing samples of up to 6,000 pieces and fine training, the accuracy rate reached over 99.9%; however, for cases with fewer samples of protrusions and insufficient soldering, the accuracy rate dropped to around 98.5%.

[0924] Key innovations in this invention include:

[0925] Cross-modal feature complementarity: Combining image geometric features with electrical timing features improves the accuracy of defect diagnosis.

[0926] Hybrid modeling method: Combining discrete point cloud with continuous function modeling to accurately describe the shape of defects.

[0927] Multi-stage fusion strategy: Select early or late fusion based on data characteristics to balance computational efficiency and information retention.

[0928] This integrated solution, in the context of diversified power supply in new power systems (direct connection to green electricity, grid power, energy storage, photovoltaic (wind power), and combined power), and reliable welding and fixing technology to address grid voltage imbalance fluctuations and intermittent fluctuations in new energy sources, achieves a real-time synchronization rate of 93.5% and a conversion efficiency of 99.9% in the practical application of photovoltaic and green electricity control, significantly improving the fault diagnosis capability of photovoltaic systems. The intelligent welding energy (weld heat and temperature) balance regulation through multi-modal and multi-parameter fusion is also significantly enhanced.

[0929] The multimodal data fusion processing described in this invention has significant advantages in the fields of photovoltaic system fault diagnosis and industrial quality inspection:

[0930] Improving diagnostic accuracy and robustness:

[0931] Complementary Information: Combining image-based data (such as X-ray computed tomography and thermal infrared imaging) with electrical parameters (current / voltage) enables multi-dimensional defect detection. For example, temperature anomalies detected by thermal imaging can corroborate electrical parameter anomalies, reducing false alarm rates.

[0932] Anti-interference capability: Multimodal data fusion can maintain diagnostic performance by using data from other modalities even when a single data source is disturbed (e.g., image blurring, electrical noise). Experiments show that the fusion method can reduce harmonic detection error from 20% to below 2.23%.

[0933] Fusion algorithm performance: Data fusion accuracy ≥ 98%.

[0934] Model prediction accuracy: Prediction error of welding parameters (temperature, energy) ≤ ±3%, and accuracy of abnormal state identification ≥ 99%.

[0935] Enhance fault location and tracing capabilities:

[0936] Spatial-temporal correlation: By using timestamp alignment technology, three-dimensional structural information is correlated with real-time operating condition data (load fluctuations, ambient temperature) to achieve spatiotemporal fault localization. For example, by combining internal structural information from ultrasonic imaging with temperature data, the physical location of hot spots can be accurately located.

[0937] Causal relationship mining: By integrating electrical parameters with microstructure images, the causal relationship between process defects (such as weak solder joints) and the degradation of electrical performance can be revealed, supporting root cause analysis.

[0938] Optimize system real-time performance and efficiency:

[0939] Hybrid modeling acceleration: A hybrid modeling method using the second moments of discrete data and the second moments of continuous functions is adopted, which improves computational efficiency by more than 50% while ensuring the accuracy of shape description.

[0940] Intelligent decision support: By dynamically selecting the optimal feature combination through early / late fusion strategies, redundant calculations are reduced, and the system's real-time performance reaches 93.5%, meeting the rapid response requirements of the smart grid.

[0941] Supports standardization and compatibility:

[0942] Multi-protocol adaptation: Integrates distributed power access unit data and equipment operation data conforming to the T / CIMA 0142 standard to achieve collaborative diagnosis of cross-vendor equipment.

[0943] Data interoperability: A unified feature vector representation enables data from different modalities to be processed in the same model, reducing system integration costs and improving compatibility.

[0944] Expanding application scenarios and value:

[0945] Full lifecycle management: Multi-scale data fusion from microstructure to macro performance supports quality monitoring of products throughout their entire lifecycle, from production to retirement.

[0946] Predictive maintenance: By integrating historical fault data with real-time operational data, potential faults can be predicted in advance, reducing downtime, such as reducing unexplained switching losses by 90%.

[0947] In practical applications, in photovoltaic systems, multi-mode fusion technology improves the inverter's low-voltage ride-through capability by 30%, harmonic suppression capability by 85%, and system conversion efficiency to 99.9%, which is significantly better than traditional single-mode methods. This technology has become a core support for smart grid construction and industrial intelligent upgrading.

[0948] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided in this embodiment solves the problem of relying on large labeled datasets, long training time and high computing resources for automated defect detection methods based on deep learning, such as X-ray (and thermal infrared, ultrasonic detection) smart meters, metering automation terminals, transformers, batteries, transformers and wires, power electronic components, multilayer chips, transistors, transformers and PCBs, component welding connections and processes, existing welding structures and semiconductor packaging such as SOP, QFP, BGA, CSP, IGBT, industrial devices, and internal discontinuities (microcracks, porosity, etc.) in batteries, cracks, voids, delamination, deformation, separation, bridging, open circuits in metal layers (wires, cores), fractures, dust and foreign matter and impurities, metal particles, loosening, misalignment, and damage to insulation layers. This makes it suitable for industrial scenarios where data is scarce and real-time processing is required.

[0949] This embodiment also provides a multimodal CNN / Transformer image defect diagnosis and tracking decision-making device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated for details already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0950] This embodiment provides a multimodal CNN / Transformer image defect diagnosis and tracking decision-making device, such as... Figure 13 As shown, it includes:

[0951] The multimodal data acquisition module 1301 is used to acquire defect location images of multiple multimodal data of the product to be diagnosed. The multiple defect location images are defect location images containing the above-mentioned multimodal data taken by different production batches, different process stages and different equipment. The multimodal data includes image data, electrical data and operating condition data.

[0952] The data preprocessing and data alignment module 1302 is used to perform data preprocessing and data alignment on defect location images of multiple multimodal data.

[0953] The feature extraction module 1303 is used to extract features from the defect location images of multiple multimodal data based on the defect location images of each image type data using a position encoding method, to obtain the image features after position encoding of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. The image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features.

[0954] The multimodal feature fusion module 1304 is used to perform feature weight fusion on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature based on a preset feature weight allocation scheme as image features, and to concatenate the image features, electrical features and operating condition features into a unified vector to obtain multiple multimodal fusion feature vectors.

[0955] The multimodal detection model construction and detection module 1305 is used to obtain a single-process image defect detection model by sequentially training and evaluating independent sub-models of a convolutional neural network model based on multiple multimodal fusion feature vectors, and to obtain a multi-process image defect detection model by sequentially training and evaluating a Transformer network model based on multiple multimodal fusion feature vectors; it acquires multiple actual captured product images to be diagnosed and inputs them into the single-process image defect detection model and / or the multi-process image defect detection model for image defect diagnosis, and obtains the detection result of whether there is a defect or no defect in the corresponding area of ​​the product image.

[0956] The hybrid modeling and analysis module 1306 is used to perform hybrid modeling and analysis on the product image with defects output by the defect detection model of the input image to be diagnosed, so as to obtain a hybrid modeled image defect. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0957] The tracking decision module 1307 is used to obtain defect detection results from the hybrid modeled image defects through the improved U-Net model, realize defect region segmentation and localization, and then track the location of the image defect region. Based on the defect tracking and fault tracing results, decisions are made to optimize production process parameters.

[0958] In some alternative implementations, the data preprocessing and data alignment module 1302 includes:

[0959] The preprocessing unit is used to perform data preprocessing on the defect location image in sequence, including data cleaning, data annotation, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization.

[0960] Among them, data cleaning is used to remove noise data, blurred defect location images, and incomplete defect location images from defect location images; data annotation uses a combination of manual annotation and semi-automatic annotation to annotate defect locations and defect types; image usage classification divides multimodal data into image data, electrical data, and operating condition data according to the production batch, process stage, equipment type, and multimodal data type to which the image belongs; image pixel normalization normalizes the pixel values ​​of image data to a preset range to unify the scale of image data.

[0961] The data alignment and standardization unit is used to align, merge, and standardize image data, electrical data, and operating condition data according to timestamps to build a spatiotemporally consistent dataset.

[0962] In some optional implementations, the operating condition data includes load fluctuations and ambient temperature; the feature extraction module 1303 includes:

[0963] Edge and texture feature extraction unit is used to extract basic edge and texture features of defect location images using a shallow CNN network.

[0964] The image feature extraction unit is used to determine the defect center position of each defect location image based on edge features and texture features; obtain the two-dimensional coordinate value of each defect center position, and discretize each two-dimensional coordinate value; convert each discretized coordinate value into a feature vector through one-hot encoding, and use the feature vector as the image feature after position encoding of each defect location image.

[0965] The electrical feature and operating condition feature extraction unit is used to convert electrical data into time series and extract frequency domain features as electrical features through Fourier transform; and to extract operating condition features by sampling real-time data of load fluctuation and ambient temperature in the operating condition data through a 5-minute sliding window.

[0966] In some alternative implementations, the multimodal feature fusion module 1304 includes:

[0967] The feature weight allocation scheme setting unit is used to allocate the weight of contrast features to 20% to 30%, the weight of correlation features to 10% to 25%, the weight of energy features to 10% to 25%, the weight of entropy features to 30% to 40%, the weight of defect entropy value features to 10% to 20%, and the weight of position texture parameter features to 10% to 30%. The sum of the weights of contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features is equal to 1.

[0968] The multimodal fusion feature vector calculation unit is used to calculate the first product corresponding to each contrast feature and contrast feature weight, the second product corresponding to each correlation feature and correlation feature weight, the third product corresponding to each energy feature and energy feature weight, the fourth product corresponding to each entropy feature and entropy feature weight, the fifth product corresponding to each defect entropy value feature and defect entropy value feature weight, and the sixth product corresponding to each positional texture parameter feature and positional texture parameter feature weight. The first, second, third, fourth, fifth, and sixth products are added together to obtain multiple multimodal fusion feature vectors.

[0969] In some optional implementations, the convolutional neural network model includes an input layer, multiple convolutional and pooling layers, multiple fully connected layers, and an output layer. The multimodal detection model construction and detection module 1305 includes:

[0970] The convolutional neural network model initialization unit feeds multiple multimodal fused feature vectors through the input layer to multiple convolutional and pooling layers for deep learning. The convolutional layers use kernels to perform convolution operations on the multimodal fused feature vectors to extract local features, while the pooling layers obtain multiple key features through downsampling. Multiple fully connected layers integrate these key features, transforming the extracted features from local information into global information, and then mapping them to an output probability distribution through an activation function. Finally, the output layer transforms the mapped probability distributions into multiple fused probability distribution images and outputs them. The value of each pixel in the fused probability distribution image represents the probability of a defect existing at that location.

[0971] The single-process image defect detection model construction unit is used to divide multiple fused probability distribution images into training set, validation set and test set according to a preset ratio; the training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weighted and fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and the evaluation result is output. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is taken as the single-process image defect detection model.

[0972] The training constraint unit is used to calculate the loss based on a preset loss function during training, and to use the Adam optimizer for backpropagation to adjust the weights and biases of the convolutional neural network model. Based on the validation set, it adjusts the parameters of the multiple convolutional layers, pooling layers, and multiple fully connected layers to obtain the convolutional neural network model to be evaluated. The Adam optimizer combines the momentum method and the Adagrad optimization algorithm to dynamically adjust the learning rate of each parameter in the model based on the first-order moment estimate and the second-order moment estimate of the gradient, and corrects the first-order moment estimate and the second-order moment estimate through bias.

[0973] A multi-process image defect detection model construction unit is used for early fusion of the image defect fusion path of the Transformer multimodal fusion. It utilizes a self-attention mechanism to effectively capture the semantic association between features of different modalities and directly perform cross-modal early fusion of multimodal features. A cross-attention mechanism is used to process local features extracted by multi-path convolutional neural networks from different sources in parallel to fuse complex defect feature detection models. The complex defect feature detection model and the detection model that processes temporal associations in the Transformer are combined to form a combined detection model. The image to be diagnosed is input into the combined detection model, and the final defect diagnosis result is output.

[0974] The defect diagnosis unit is used to perform image defect diagnosis on multiple multi-layer chip images input to the image defect detection model through a decision function, which is expressed by the following formula:

[0975]

[0976] Wherein, G(F) = 1 indicates that a defect is detected in the corresponding region of the image, G(F) = 0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that the preset defect condition is met.

[0977] In some optional implementations, the single-process image defect detection model building unit includes:

[0978] Independent sub-model training units are used to input the unified vector of defect features from the early fusion of multimodal fusion images into the convolutional neural network for later fusion; training convolutional neural network models specifically for image data is used to extract defect features from images; training long short-term memory network models to process time-series electrical data is used to capture the patterns of electrical data changes over time; training multilayer perceptron models to process operating condition data is used to analyze the impact of operating conditions on defects.

[0979] The defect diagnosis subunit is used to perform weighted fusion of the outputs of the independent sub-models obtained from the above training to obtain the detection model to be evaluated. The test set is input into the detection model to be evaluated. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model, and the evaluation result is used as the final defect diagnosis result.

[0980] In some alternative implementations, the multimodal CNN / Transformer image defect diagnosis and tracking decision apparatus further includes:

[0981] The visualization module is used to visualize the changes in loss and evaluation results during the training process of a convolutional neural network model, the key features learned by deep learning from multiple convolutional and pooling layers, and the probability distribution of feature integration and mapping to the output from multiple fully connected layers.

[0982] In some alternative implementations, the hybrid modeling and analysis module 1306 includes:

[0983] The geometric feature calculation unit is used to calculate the geometric features of the image defect region contour and determine the preliminary shape of the image defect region; the geometric features of the preliminary shape of the image defect region include the contour perimeter, the contour convex hull perimeter, and the shape factor.

[0984] The shape hybrid modeling unit is used to perform shape hybrid modeling based on the initial shape of the defect region in the image by using the second moments of discrete data and the second moments of continuous functions, so as to obtain a mathematical model for measuring the shape of the defect region in the image.

[0985] The defect measurement unit is used to measure the degree of defect in an image defect region based on a mathematical model for measuring the shape of the image defect region, and to obtain the final shape, width, length, area and volume of the image defect region.

[0986] The diagnosis and grading unit is used to diagnose and grade the severity of image defects based on the final shape, width, length, area, and volume of the defective region.

[0987] In some alternative implementations, the shape blending modeling unit includes:

[0988] The second-order moment calculation subunit is used to determine the discrete data second-order moments and continuous function second-order moments of the image defect region based on the shape of the preliminary defect region of the image.

[0989] The fusion modeling unit is used to fuse the second moments of discrete data and the second moments of continuous functions in the image defect region to obtain a mathematical model for measuring the shape of the image defect region. Specifically, it calculates the first moment of inertia based on the second moments of discrete data, and then calculates the first equivalent ellipse parameters based on the first moment of inertia.

[0990] The second moment of inertia is calculated based on the second moment of the continuous function, and the second equivalent ellipse parameter is calculated based on the second moment of inertia. The first moment of inertia and the second moment of inertia are fused by the weighted average method, and the first equivalent ellipse parameter and the second equivalent ellipse parameter are fused by the weighted average method to obtain the mathematical model for measuring the shape of the image defect region.

[0991] Specifically, a weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent elliptic parameters and the second equivalent elliptic parameters, to obtain a mathematical model for measuring the shape of the image defect region, including:

[0992] Define the weights of the second moments of continuous functions, and calculate the weights of the second moments of discrete functions based on the weights of the second moments of continuous functions;

[0993] The first moment of inertia and the second moment of inertia are fused using a weighted average method based on the weights of the second moments of continuous functions and discrete functions to obtain the fused second moment; the first equivalent elliptic parameter and the second equivalent elliptic parameter are fused to obtain the fused equivalent elliptic parameter.

[0994] A mathematical model for measuring the shape of the image defect region is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters.

[0995] The formula for the fused second moment is as follows:

[0996] M fusion =aM c +(1-a)M d ,

[0997] Among them, M c M is the second moment of a continuous function. d is the second moment of discrete data; a (0≤a≤1) is a weighting coefficient used to measure the proportion of the second moment of continuous function in the fused second moment; when a=0, it means that the contribution of the second moment of continuous function is not considered at all when calculating the fused second moment, and only the second moment of discrete data is used; when a=1, it depends entirely on the second moment of continuous function and does not consider the second moment of discrete data.

[0998] In some alternative implementations, the tracking decision module 1307 includes:

[0999] The defect region segmentation and localization unit is used to obtain defect detection results from the hybrid modeled image using the improved U-Net model, and to segment and localize the defect region based on the defect detection results.

[1000] The defect location tracking unit is used to track the location of image defects based on the segmented and localized defect regions.

[1001] The fault tracing unit is used to construct a defect evolution dynamics model based on discrete data points of defect area and perimeter and a second-order moment model of a continuous function; based on defect location images, it uses a preset tracking function to analyze the morphological changes of crack propagation angle and wear depth increment of defects in the defect area at different times, and performs visualization and quantitative analysis of the defect evolution process; it acquires the physical defect location in the three-dimensional structural information, and uses a spatial mapping algorithm to match the physical defect location in the three-dimensional structural information with electrical anomaly point data and microstructure images respectively, to establish a correlation map between physical defects and electrical faults; based on the correlation map between physical defects and electrical faults and the morphological change characteristics, and combined with the analysis of abnormal current distribution in the wear area of ​​the resistive surface, it performs fault tracing of the defect location in the image.

[1002] The process improvement unit is used to improve the welding process, material replacement process, and online monitoring process of the product to be decided based on defect tracking and fault tracing.

[1003] In some optional implementations, the semantic segmentation model based on the CNN deep learning-improved U-Net model includes an encoder, a decoder, and skip connections. The encoder includes residual connections, and the decoder includes an attention mechanism. The defect region segmentation and localization unit includes:

[1004] The downsampling subunit is used to perform convolution processing on the image defect region in the encoder to obtain an output feature map and an input feature map, and then add the output feature map and the input feature map through residual concatenation to obtain a new output feature map. The process of performing convolution processing on the image defect region in the encoder to obtain an output feature map and an input feature map, and then adding the output feature map and the input feature map through residual concatenation to obtain a new output feature map includes:

[1005] Residual connections are introduced into the encoder, and the weight matrix of the residual connections is: The output feature map after two convolutional layers Input features Figure X i (x, y, z), and add the output feature map and the input feature map through a residual connection to obtain a new output feature map. The formula is as follows:

[1006]

[1007] Where (x, y, z) are the image coordinates.

[1008] The upsampling subunit is used to input the new output feature map into the attention mechanism of the decoder to obtain a weighted feature map, which is then subjected to a convolution operation to obtain a TFT ray map. The process of inputting the new output feature map into the attention mechanism of the decoder to obtain a weighted feature map, followed by a convolution operation to obtain a TFT ray map, includes:

[1009] An attention mechanism is introduced into the decoder, with the attention weight matrix as follows: Feature maps after encoder upsampling and stitching Weighted features are obtained through an attention mechanism. That is, the TFT beam pattern, the formula is as follows:

[1010]

[1011] Where (x, y, z) are image coordinates, a represents the relative position offset involved in the weighted operation when the feature map is subjected to the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[1012] In some alternative implementations, the process improvement unit includes:

[1013] The first process improvement subunit is used to improve the welding process of the product to be decided, including:

[1014] Finite element method (FEM) simulation was used to analyze the effects of different welding parameters (laser power, welding speed, and spot diameter) on the residual stress, fatigue life, and resistance of the weld joint of the product under consideration. The simulation effect was verified by combining multimodal data from ultrasonic scanning results to determine the optimal process parameters for laser welding and resistance brazing. The influence of multiple power sources (photovoltaics, green electricity, energy storage, mains power, combined power, and nuclear power) on the welding parameters was also considered. The images of welding cracks and incomplete welds detected by convolutional neural networks were combined with the multimodal data from ultrasonic scanning to quantify the welding quality.

[1015] The second process improvement subunit is used to improve the material replacement process of the product to be decided. This includes: establishing a material performance database to record the resistance change rate and capacitance stability performance parameters of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments; and using convolutional neural networks to detect defects in images of aged materials through accelerated aging tests simulating extreme working conditions, and combining performance data to evaluate material durability.

[1016] The third process improvement subunit is used to improve the online monitoring process of the product to be decided, including: deploying a real-time image recognition system based on deep learning to perform millisecond-level analysis on X-ray or OBIRCH inspection images to automatically label and classify defects; and using deep learning algorithms to train X-ray or OBIRCH inspection images to establish a defect recognition model.

[1017] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[1018] In this embodiment, the multimodal CNN / Transformer image defect diagnosis and tracking decision device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[1019] This invention also provides a computer device having the above-described features. Figure 12 The image defect diagnosis, tracking, and decision-making device shown is based on multimodal fusion CNN.

[1020] Please see Figure 14 , Figure 14 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 14As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 14 Take a processor 10 as an example.

[1021] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[1022] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[1023] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[1024] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[1025] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 13 Taking the example of a connection between China and Israel via a bus.

[1026] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[1027] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[1028] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[1029] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multimodal CNN / Transformer image defect diagnosis and tracking decision-making method, characterized in that, The method includes: The defect location images of the product to be diagnosed are obtained from multiple multimodal data. These defect location images are taken from different production batches, different process stages, and different equipment, and contain the aforementioned multimodal data. The multimodal data includes image data, electrical data, and operating condition data. Data preprocessing and data alignment are performed on the defect location images of the multiple multimodal data. Based on the defect location image of each of the image types, feature extraction is performed on the defect location images of the multiple multimodal data using a position encoding method to obtain the position-encoded image features of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. The image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features. Based on a preset feature weight allocation scheme, feature weight fusion is performed on each of the contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature as image features, and the image features, electrical features and working condition features are concatenated into a unified vector to obtain multiple multimodal fusion feature vectors; A single-process image defect detection model is obtained by sequentially training and evaluating independent sub-models of a convolutional neural network model based on multiple multimodal fusion feature vectors, and a multi-process image defect detection model is obtained by sequentially training and evaluating a Transformer network model based on the same multiple multimodal fusion feature vectors. Multiple images of the product to be diagnosed are acquired and input into the single-process image defect detection model and / or multi-process image defect detection model to perform image defect diagnosis, and the detection result of whether there is a defect or no defect in the corresponding area of ​​the product image is obtained. The defective product image output by the image defect detection model is subjected to hybrid modeling and analysis to obtain a hybrid modeled image defect. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis. The improved U-Net model is used to obtain defect detection results for hybrid modeled image defects, thereby achieving defect region segmentation and localization. The location of the defect region in the image is then tracked, and decision-making is made based on the defect tracking and fault tracing results to optimize production process parameters. The improved U-Net model integrates residual connectivity and attention mechanisms. Hybrid modeling and analysis are performed on the defective product image output by the defect detection model of the input image to be diagnosed, including: Calculate the geometric features of the image defect region contour and determine the preliminary shape of the image defect region; the geometric features of the preliminary shape of the image defect region include the contour perimeter, the contour convex hull perimeter, and the shape factor; Based on the preliminary shape of the defect region in the image, a hybrid shape model is obtained by using the second moments of discrete data and the second moments of continuous functions to measure the shape of the image defect region. This model includes: Based on the preliminary shape of the defect region in the image, determine the discrete second moment and the continuous function second moment of the defect region; The discrete second moments and continuous function second moments of the image defect region are fused and modeled to obtain a mathematical model for measuring the shape of the image defect region, including: The first moment of inertia is calculated based on the second-order moments of the discrete data, and the first equivalent ellipse parameters are calculated based on the first moment of inertia. The second moment of inertia is calculated based on the second moment of the continuous function, and the second equivalent ellipse parameters are calculated based on the second moment of inertia. A weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent elliptic parameters and the second equivalent elliptic parameters, to obtain a mathematical model for measuring the shape of the image defect region, including: The weights of the second moments of the continuous function are set, and the weights of the second moments of the discrete function are calculated based on the weights of the second moments of the continuous function. Based on the weights of the second moments of the continuous function and the weights of the second moments of the discrete function, the first moment of inertia and the second moment of inertia are fused using a weighted average method to obtain the fused second moment; the first equivalent elliptic parameter and the second equivalent elliptic parameter are fused to obtain the fused equivalent elliptic parameter. A mathematical model for measuring the shape of the image defect region is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters. The formula for the fused second moment is as follows: ； in, It is the second moment of a continuous function. It is the second moment of discrete data; It is a weighting coefficient, 0 ≤ ≤1, used to measure the proportion of the second moment of a continuous function in the fused second moment; when When = 0, it means that the contribution of the second moment of continuous function is not considered at all when calculating the fused second moment; only the second moment of discrete data is used. When =1, it depends entirely on the second moment of the continuous function and does not consider the second moment of the discrete data. The degree of defect in an image defect region is measured using a mathematical model for measuring the shape of the defect region, and the final shape, width, length, area, and volume of the defect region are obtained. The severity level of image defects is diagnosed and graded based on the final shape, width, length, area, and volume of the defective region.

2. The method according to claim 1, characterized in that, Data preprocessing and alignment are performed on the defect location images of the multiple multimodal data, including: The defect location image is subjected to data preprocessing in sequence, including data cleaning, data annotation, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization. The data cleaning process removes noise, blurred images, and incomplete images from defect location images. Data annotation combines manual and semi-automatic methods to label defect locations and types. Image usage classification categorizes multimodal data into image-based, electrical, and operational condition-based data according to the production batch, process stage, equipment type, and multimodal data type. Image pixel normalization normalizes the pixel values ​​of image-based data to a preset range to unify the scale of the image-based data. Image data, electrical data, and operating condition data are aligned, merged, and standardized according to timestamps to construct a spatiotemporally consistent dataset.

3. The method according to claim 1, characterized in that, The operating condition data includes load fluctuations and ambient temperature; Based on the defect location image of each of the aforementioned image types, feature extraction is performed on the defect location images of the multiple multimodal data using a location encoding method to obtain the location-encoded image features of each defect location image. Simultaneously, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features, including: A shallow CNN network is used to extract basic edge and texture features from images of defect locations; The defect center position of each defect location image is determined based on the edge features and texture features; the two-dimensional coordinate values ​​of each defect center position are obtained, and each two-dimensional coordinate value is discretized; each discretized coordinate value is converted into a feature vector through one-hot encoding, and the feature vector is used as the image feature after position encoding of each defect location image; Electrical data is converted into time series data, and frequency domain features are extracted as electrical features using Fourier transform. Real-time data on load fluctuations and ambient temperature in the operating condition data are sampled using a 5-minute sliding window to extract operating condition features.

4. The method according to claim 1, characterized in that, The feature weight allocation scheme is set in the following manner: The weights for contrast features are allocated as follows: 20%–30% for contrast features, 10%–25% for correlation features, 10%–25% for energy features, 30%–40% for entropy features, 10%–20% for defect entropy features, and 10%–30% for positional texture parameter features. The sum of the weights for contrast features, correlation features, energy features, entropy features, defect entropy features, and positional texture parameter features is equal to 1.

5. The method according to claim 4, characterized in that, The feature weight fusion method, based on a preset feature weight allocation scheme, performs feature weight fusion on each of the contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and location texture parameter feature, concatenating image features, electrical features, and operating condition features into a unified vector, including: Calculate the first product of each contrast feature and its weight, the second product of each correlation feature and its weight, the third product of each energy feature and its weight, the fourth product of each entropy feature and its weight, the fifth product of each defect entropy value feature and its weight, and the sixth product of each positional texture parameter feature and its weight. Adding the corresponding first, second, third, fourth, fifth, and sixth products together yields multiple multimodal fusion feature vectors. Early fusion involves concatenating multiple multimodal fusion feature vectors with electrical and operating condition features into a unified vector.

6. The method according to claim 5, characterized in that, The convolutional neural network model includes an input layer, multiple convolutional and pooling layers, multiple fully connected layers, and an output layer. Based on multiple multimodal fusion feature vectors, the convolutional neural network model is sequentially trained with independent sub-models and then evaluated to obtain a single-process image defect detection model, including: The multiple multimodal fusion feature vectors are passed through the input layer to multiple convolutional layers and pooling layers for deep learning. The multiple convolutional layers use convolution kernels to perform convolution operations on the multimodal fusion feature vectors to extract local features, and the pooling layers obtain multiple key features through downsampling operations. Multiple fully connected layers are used to integrate the multiple key features, transforming the extracted features from local information into global information, and mapping them to the output probability distribution through an activation function; An output layer is used to transform the probability distribution mapped to the output into multiple fused probability distribution images and output them. The value of each pixel in the fused probability distribution image represents the probability that a defect exists at that location.

7. The method according to claim 6, characterized in that, The single-process image defect detection model, obtained by sequentially training and evaluating independent sub-models of the convolutional neural network model based on multiple multimodal fusion feature vectors, further includes: The multiple fused probability distribution images are divided into a training set, a validation set, and a test set according to a preset ratio; The training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weighted and fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and the evaluation result is output. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model. During training, the loss is calculated based on a preset loss function, and the Adam optimizer is used for backpropagation to adjust the weights and biases of the convolutional neural network model. Based on the validation set, the parameters of the multiple convolutional layers, pooling layers, and fully connected layers are adjusted to obtain the convolutional neural network model to be evaluated. The Adam optimizer combines the momentum method and the Adagrad optimization algorithm to dynamically adjust the learning rate of each parameter in the model based on the first-order moment estimate and the second-order moment estimate of the gradient, and corrects the first-order moment estimate and the second-order moment estimate through bias.

8. The method according to claim 7, characterized in that, The training set is input into a convolutional neural network model to train independent sub-models sequentially, and the outputs of each independent sub-model are weighted and fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set, and the evaluation result is output. When the evaluation result meets a preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as a single-process image defect detection model, including: The unified vector of defect features from the early-stage multimodal fusion image is input into a convolutional neural network for later-stage fusion. Training convolutional neural network models specifically for image data to extract defect features from images; Train a long short-term memory network model to process time-series electrical data and capture the patterns of change in electrical data over time; Train a multilayer perceptron model to process operating condition data and analyze the impact of operating conditions on defects. The outputs of the independent sub-models obtained from the above training are weighted and fused to obtain the detection model to be evaluated. The test set is input into the detection model to be evaluated. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model, and the evaluation result is used as the final defect diagnosis result.

9. The method according to claim 7, characterized in that, Based on the multiple multimodal fusion feature vectors, the Transformer network model is sequentially trained and evaluated to obtain a multi-process image defect detection model, including: The early fusion path of the multimodal fusion Transformer image defect fusion path utilizes a self-attention mechanism to effectively capture the semantic associations between features of different modalities, and directly performs cross-modal early fusion of multimodal features; A complex defect feature detection model is formed by fusing local features extracted by multi-path convolutional neural networks from different sources using a cross-attention mechanism in parallel. The complex defect feature detection model is combined with a detection model that processes temporal correlations using Transformer. The image to be diagnosed is input into this combined detection model, and the final defect diagnosis result is output.

10. The method according to claim 7, characterized in that, The method further includes: using visualization tools to visualize the loss changes, evaluation result changes, key features learned by multiple convolutional and pooling layers during the training process of the convolutional neural network model, and the probability distribution of feature integration and mapping of multiple fully connected layers to the output.

11. The method according to claim 1, characterized in that, The process of acquiring multiple actual captured images of the product to be diagnosed and inputting them into the image defect detection model for image defect diagnosis, to obtain a detection result indicating the presence or absence of defects, includes: Image defect diagnosis is performed on multiple product images to be diagnosed input into the image defect detection model using a decision function, which is expressed by the following formula: ； Where G(F) = 1 indicates that a defect is detected in the corresponding region of the image, G(F) = 0 indicates that there is no defect, and if satisfiescertain defect condition indicates that the preset defect condition is met; The multimodal fusion feature vector of the input product image to be diagnosed is matched with the defect feature threshold set during model training. That is, when the contrast, energy, and entropy feature parameters in the feature vector exceed the preset threshold, it is determined that there is a defect; otherwise, it is determined that there is no defect.

12. The method according to claim 1, characterized in that, Hybrid modeling and analysis are performed on the defective product image output by the defect detection model of the input image to be diagnosed, including: Calculate the geometric features of the image defect region contour and determine the preliminary shape of the image defect region; the geometric features of the preliminary shape of the image defect region include the contour perimeter, the contour convex hull perimeter, and the shape factor; Based on the preliminary shape of the defect region in the image, a hybrid shape model is obtained by using the second moments of discrete data and the second moments of continuous functions to model the shape of the image defect region. The degree of defect in an image defect region is measured using a mathematical model for measuring the shape of the defect region, and the final shape, width, length, area, and volume of the defect region are obtained. The severity level of image defects is diagnosed and graded based on the final shape, width, length, area, and volume of the defective region.

13. The method according to claim 1, characterized in that, The improved U-Net model is used to obtain defect detection results for hybrid modeled image defects, achieving defect region segmentation and localization, and then tracking the location of image defect regions, including: The defects in the hybrid modeled image are processed by the improved U-Net model to obtain defect detection results, and the defect regions are segmented and located based on the defect detection results; Based on the segmented and located defect regions, the image defect location is tracked.

14. The method according to claim 1, characterized in that, The semantic segmentation model of the improved U-Net model includes an encoder, a decoder, and skip connections. The encoder includes residual connections, and the decoder includes an attention mechanism. The defects in the hybrid modeled image are processed using an improved U-Net model to obtain defect detection results, achieving defect region segmentation and localization, including: After performing convolution processing on the image defect region in the encoder, an output feature map and an input feature map are obtained. The output feature map and the input feature map are then added together through the residual connection to obtain a new output feature map. The new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, which is then convolved to obtain a TFT ray map. The TFT ray image is input into the improved U-Net model for image region segmentation to obtain a binary image of the image defect region.

15. The method according to claim 14, characterized in that, The process of performing convolution processing on the image defect region in the encoder to obtain an output feature map and an input feature map, and then adding the output feature map and the input feature map through the residual connection to obtain a new output feature map, includes: A residual connection is introduced into the encoder, and the weight matrix of the residual connection is: The output feature map after two convolutional layers With input feature map The output feature map and the input feature map are then added together via residual connection to obtain a new output feature map. .

16. The method according to claim 1, characterized in that, The decision-making process based on defect tracking and fault tracing results to optimize production process parameters includes: A dynamic model of defect evolution is constructed based on discrete data points of defect area and perimeter and a second-order moment model of continuous function. Based on the defect location image, a preset tracking function is used to analyze the morphological changes of the crack propagation angle and wear depth increment of the defect in the defect area at different times, and to visualize and quantify the defect evolution process. The physical defect locations in the three-dimensional structural information are obtained, and a spatial mapping algorithm is used to match the physical defect locations in the three-dimensional structural information with electrical anomaly point data and microstructure images, respectively, to establish a correlation map between physical defects and electrical faults. Based on the correlation map of physical defects and electrical faults and the morphological change characteristics, and combined with the abnormal current distribution analysis of the wear area on the resistor surface, the fault source of the defect area in the image is traced. Based on defect tracking and fault tracing, improvements are made to the welding process, material replacement process, and online monitoring process of the product to be decided.

17. The method according to claim 16, characterized in that, Improvements to the welding process of the product to be decided upon, including: Finite element method (FEM) simulation was used to analyze the effects of different welding parameters (laser power, welding speed, and spot diameter) on the residual stress, fatigue life, and resistance of the weld joint of the product under consideration. The simulation effect was verified by combining multimodal data from ultrasonic scanning results to determine the optimal process parameters for laser welding and resistance brazing. The influence of multiple power sources (photovoltaics, green electricity, energy storage, mains power, combined power, and nuclear power) on the welding parameters was also considered. The defect location images of welding cracks and incomplete welds detected by convolutional neural networks were combined with the multimodal data from ultrasonic scanning to quantify the welding quality. Improvements to the material replacement process for the product being decided upon, including: A material performance database was established to record the resistance change rate and capacitance stability of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments. Through accelerated aging tests simulating extreme working conditions, convolutional neural networks were used to detect defects in images of aged materials, and the durability of the materials was evaluated in combination with performance data. Improvements to the online monitoring process for decision-making products include: Deploy a deep learning-based real-time image recognition system to perform millisecond-level analysis on X-ray or OBIRCH inspection images to automatically label and classify defects; A defect recognition model is established by training X-ray or OBIRCH inspection images using deep learning algorithms.

18. A multimodal CNN / Transformer image defect diagnosis and tracking decision-making device, characterized in that, The device includes: The multimodal data acquisition module is used to acquire defect location images of multiple multimodal data of the product to be diagnosed. The defect location images are defect location images containing the above-mentioned multimodal data taken from different production batches, different process stages, and different equipment. The multimodal data includes image data, electrical data, and operating condition data. The data preprocessing and data alignment module is used to perform data preprocessing and data alignment on the defect location images of the multiple multimodal data. The feature extraction module is used to extract features from the defect location images of the multiple multimodal data based on the defect location images of each of the image types using a location encoding method, to obtain the location-encoded image features of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features. The image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and location texture parameter features. The multimodal feature fusion module is used to perform feature weight fusion on each of the contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature based on a preset feature weight allocation scheme as image features, and to concatenate the image features, electrical features and operating condition features into a unified vector to obtain multiple multimodal fusion feature vectors; The multimodal detection model construction and detection module is used to obtain a single-process image defect detection model by sequentially training and evaluating independent sub-models of a convolutional neural network model based on multiple multimodal fusion feature vectors, and to obtain a multi-process image defect detection model by sequentially training and evaluating a Transformer network model based on the same multiple multimodal fusion feature vectors; it also acquires multiple actual captured product images to be diagnosed and inputs them into the single-process image defect detection model and / or the multi-process image defect detection model for image defect diagnosis, obtaining detection results indicating whether defects exist or not in the corresponding areas of the product images; The hybrid modeling and analysis module is used to perform hybrid modeling and analysis on the product image with defects output by the defect detection model of the input image to be diagnosed, so as to obtain hybrid modeled image defects. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis. The tracking decision module is used to obtain defect detection results from hybrid modeled image defects through the improved U-Net model, realize defect region segmentation and localization, and then track the location of image defect regions. Based on the defect tracking and fault tracing results, decisions are made to optimize production process parameters; the improved U-Net model integrates residual connections and attention mechanisms. Hybrid modeling and analysis are performed on the defective product image output by the defect detection model of the input image to be diagnosed, including: Calculate the geometric features of the image defect region contour and determine the preliminary shape of the image defect region; the geometric features of the preliminary shape of the image defect region include the contour perimeter, the contour convex hull perimeter, and the shape factor; Based on the preliminary shape of the defect region in the image, a hybrid shape model is obtained by using the second moments of discrete data and the second moments of continuous functions to measure the shape of the image defect region. This model includes: Based on the preliminary shape of the defect region in the image, determine the discrete second moment and the continuous function second moment of the defect region; The discrete second moments and continuous function second moments of the image defect region are fused and modeled to obtain a mathematical model for measuring the shape of the image defect region, including: The first moment of inertia is calculated based on the second-order moments of the discrete data, and the first equivalent ellipse parameters are calculated based on the first moment of inertia. The second moment of inertia is calculated based on the second moment of the continuous function, and the second equivalent ellipse parameters are calculated based on the second moment of inertia. A weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent elliptic parameters and the second equivalent elliptic parameters, to obtain a mathematical model for measuring the shape of the image defect region, including: The weights of the second moments of the continuous function are set, and the weights of the second moments of the discrete function are calculated based on the weights of the second moments of the continuous function. Based on the weights of the second moments of the continuous function and the weights of the second moments of the discrete function, the first moment of inertia and the second moment of inertia are fused using a weighted average method to obtain the fused second moment; the first equivalent elliptic parameter and the second equivalent elliptic parameter are fused to obtain the fused equivalent elliptic parameter. A mathematical model for measuring the shape of the image defect region is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters. The formula for the fused second moment is as follows: ； in, It is the second moment of a continuous function. It is the second moment of discrete data; It is a weighting coefficient, 0 ≤ ≤1, used to measure the proportion of the second moment of a continuous function in the fused second moment; when When = 0, it means that the contribution of the second moment of continuous function is not considered at all when calculating the fused second moment; only the second moment of discrete data is used. When =1, it depends entirely on the second moment of the continuous function and does not consider the second moment of the discrete data. The degree of defect in an image defect region is measured using a mathematical model for measuring the shape of the defect region, and the final shape, width, length, area, and volume of the defect region are obtained. The severity level of image defects is diagnosed and graded based on the final shape, width, length, area, and volume of the defective region.

Images