Multi-mode CNN / Transform image defect diagnosis tracking decision-making method

Abstract

The invention discloses a multi-mode CNN / Transform image defect diagnosis tracking decision-making method, and belongs to the field of power electronic industry detection. According to the method, defect position multi-modal data of different batches, process stages and equipment are collected, after space-time alignment and image enhancement preprocessing and alignment are conducted, features are extracted through position coding, a CNN shallow network and ResNet-18, the features are fused into multi-modal feature vectors in combination with weights, a detection model is obtained through CNN / Transform model training, defect diagnosis and tracking decision making are achieved, and the defect diagnosis and tracking decision making efficiency is improved. The problems of precise detection, positioning, diagnosis tracking and decision-making of internal tiny component changes and structural defects of electronic products and local and overall multi-mode defects of single electric power are solved. The detection accuracy is improved by 20%, the diagnosis accuracy is more than 98.5%, the defect tracking error is less than 5%, the decision-making efficiency is improved by 40%, and the defect prediction capability is improved by 22%.

Description

Technical Field

[0001] The present invention relates to the field of detection, measurement, diagnosis, tracking and decision-making technology for power electronics industry products through machine vision and visualization of single processes and local defects, fusion of multiple processes and overall defects with CNN / Transformer, and more specifically to a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method. Background Art

[0002] Some existing power electronics products (such as gas-insulated switchgear, inverters, power cables, and circuit breakers) suffer from rough surfaces, which severely interferes with the perception and detection of internal and external defects using optical and nondestructive testing equipment. For example, the trend toward miniaturization of chips, IGBTs, and crystal oscillators, coupled with the complexity of the background and subtlety of defects, makes effective internal defect detection increasingly difficult. Existing technologies, in particular, suffer from image blurriness (low brightness and dark contrast) that fails to clearly reflect finer details caused by varying components and structures within the product. Low pixel correlation prevents analysis and reflection of finer texture information, resulting in low and chaotic information and a lack of depth. Complex regions are also obscured, and many subtle structural or component color variations are inherently hidden. Low spatial resolution makes it impossible to detect subtle compositional variations and structural defects within the product, forcing existing technologies to provide only an approximate location and general description of a defect. These image blur and poor visualization hinder computer-based detection, fault location, and diagnosis.

[0003] Existing zero-order moments of X-ray images are generated by objects of varying sizes and depths, and are related to the area and volume of the product and defect. Independent zero-order moments are difficult to directly use for area and volume measurement, as they cannot distinguish changes in shape, resulting in ambiguous shape cues. X-ray image first-order moments can only provide a preliminary guess and preliminary assessment of the center of mass position and current state of the intensity distribution of the product and defect. First-order moments alone are difficult to measure the length, width, and height of the product and defect. X-ray image second-order moments can only roughly determine the shape and orientation of the product and defect, providing a rough understanding of their area and volume, but cannot accurately measure them. Even when combined with zero-order moments from X-ray images, infrared cameras, and other methods, these combined zero-order moments can only construct a rough model of the defect, providing a basis for further analysis and assessment, and cannot serve as a basis for product improvement, defect repair, process improvement, production, design, or forecasting. Existing depth imaging methods such as X-ray, ultrasonic, infrared, and laser imaging suffer from attenuation at different depths and the limitations of single-modality imaging.

[0004] With the continuous advancement of industrialization, the application of welding technology in the manufacturing industry is becoming more and more extensive. However, quality problems in welding structures, such as cracks, pores, lack of fusion, voids, delamination, deformation, separation, poor contact of solder joints, partial solder joint leakage, cold solder joints and other defects, seriously affect the safety and reliability of welding structures and product performance. In order to ensure the quality, performance, stability, reliability and safety of products and welding, defect detection in welding structures and products has become an indispensable part. PCB (Printed Circuit Board, printed circuit board) boards and components and process PCBA (Printed Circuit Board Assembly) will have cracks, voids, delamination, deformation, separation, bridging, open circuits in metal layers (wires, cores), fractures, dust, foreign matter and impurities, metal particles, looseness, dislocation, insulation layer damage and other defects, which seriously affect the safety, reliability, stability, inherent performance and failure of welding structures and PCB (PCBA) boards and components and processes. In order to ensure the quality, safety, reliability, stability, performance and life of PCB (PCBA) boards, components and processes, defect detection of PCB (PCBA) boards, components and processes has become an indispensable part.

[0005] Traditional defect detection methods include non-destructive testing technologies such as ultrasonic testing, magnetic particle testing, penetrant testing, and X-ray testing. X-ray testing offers intuitive imaging, is sensitive to volumetric defects, accurately measures dimensions, and has no strict requirements for surface roughness. X-ray non-destructive testing technology has the advantages of strong penetration and high image resolution. It can also penetrate deep into the internal structure of materials, making it widely used for internal defect detection and quality inspection of metal materials. However, existing soldering structures and semiconductor packages, industrial devices, and batteries, such as SOP (Standard Operation Procedure), QFP (Quad Flat Package), BGA (Ball Grid Array), CSP (Chip Scale Package), and IGBT, have internal discontinuities (microcracks, porosity, etc.). Differences exist between soldering structures and product component density, thickness, shape, and size. In particular, it is difficult to detect small micro-surface or sub-surface discontinuities, which are the root cause of progressive degradation and failure in product life, reliability, performance, stability, and metrological accuracy. Existing X-ray nondestructive testing images have low contrast and clarity, are insensitive to microcracks, and lack penetration, making it difficult to detect these defects.

[0006] Currently, X-ray crack detection and determination generally rely on manual visual inspection, which is not only inefficient and labor-intensive, but also often results inaccurate due to the eye's susceptibility to damage from strong light and subjective factors. In the power industry, electricity meters and automated metering terminals, instrument transformers, batteries, transformers, and conductors are critical components of the power system. Their quality and reliability are directly related to the accuracy of electricity metering and the normal operation of the power system. Defects such as cracks, voids, and delamination in solder joints, printed circuit boards (PCBAs), and components not only affect the service life and stability of electricity meters and automated metering terminals, batteries, instrument transformers, transformers, and conductors, leading to metering errors and even posing safety hazards. Traditional methods for detecting cracks, voids, bubbles, and delamination in solder joints, printed circuit boards (PCBAs), processes, instrument transformers, and components (such as eight types of chips) rely primarily on manual visual inspection, which also suffers from low efficiency, labor-intensiveness, and poor accuracy. With the continuous improvement of detection efficiency and accuracy in industrial production, traditional manual inspection methods can no longer fully meet the requirements.

[0007] Existing integrated circuits and semiconductors (13-layer metal interconnect chips) require peeling off the metal and dielectric layers layer by layer, whether testing circuits or devices (multi-layer chips and transistors) to test their electrical performance. Samples are then cut using a FIB (Focused Ion Beam) microscope. A TEM (Transmission Electron Microscope) is used to inspect oxide thickness and defects such as voids and cracks in the metal interconnects after peeling off the covering layers. Both EMM (Emission Microscope) and OBIRCH (Optical Beam Induced Resistance Change) optical current emission microscopes require testing after peeling off the covering layers to detect circuit and device defects. This process suffers from high efficiency and low cost, high destructiveness, and low image detection accuracy (the best experimental level is 88%, while the industrial level is less than 80%).

[0008] In recent years, with the advancement of image processing and machine learning technologies, automated X-ray crack detection has gradually become a new inspection approach. While automated inspection methods based on deep learning have made significant progress in image recognition and classification, these methods typically require large, annotated datasets for training and validation. Furthermore, model training is often time-consuming and highly dependent on computing resources. This makes deep learning methods challenging to implement in practice, especially in industrial scenarios where data is scarce and real-time processing is required.

[0009] The product under inspection can only detect and diagnose single process or local defects. Any defect is caused by the interaction of multi-parameter, multi-process, and multi-modal overall factors. Any single or local defect has limited effect on defect improvement. Solving product defects requires integrating single process and local factors with multi-modal and multi-process overall factors, using an automatic fusion method for detection, diagnosis, tracking, and decision-making to effectively solve and improve defects. Currently, there is a lack of a method to automatically integrate multi-process and overall defect detection, diagnosis, tracking, and decision-making for single process and local defects at once. Summary of the Invention

[0010] In view of this, the present invention provides a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method to solve the problem of being unable to accurately detect, locate, diagnose, track and make decisions on tiny component changes and structural defects inside power electronic products, single process and local defects, and the fusion of multiple processes and overall defects.

[0011] In a first aspect, the present invention provides a multimodal CNN / Transformer image defect diagnosis and tracking decision method, the method comprising:

[0012] Acquire defect location images of multiple multimodal data of the product to be diagnosed, where the multiple defect location images are defect location images containing the above multimodal data and are taken from different production batches, different process stages, and different equipment; the multimodal data includes image data, electrical data, and operating condition data;

[0013] Perform data preprocessing and data alignment on defect location images of multiple multimodal data;

[0014] Based on the defect location image of each image data, the defect location images of multiple multimodal data are subjected to feature extraction using a position encoding method to obtain the image features of each defect location image after position encoding. At the same time, feature extraction is performed on the electrical data and the working condition data respectively to obtain electrical features and working condition features. Image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features.

[0015] Based on a preset feature weight distribution scheme, each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and position texture parameter feature is fused with feature weights as image features, and the image features, electrical features, and operating condition features are spliced ​​into a unified vector to obtain multiple multimodal fusion feature vectors;

[0016] Based on multiple multimodal fusion feature vectors, the convolutional neural network model is trained and evaluated on independent sub-models to obtain a single-process image defect detection model. Based on multiple multimodal fusion feature vectors, the Transformer network model is trained and evaluated to obtain a multi-process image defect detection model.

[0017] Acquire multiple images of the product to be diagnosed that are actually taken and input them into the single-process image defect detection model and / or the multi-process image defect detection model to perform image defect diagnosis, and obtain a detection result indicating whether the corresponding area of ​​the product image has defects or not;

[0018] Perform hybrid modeling and analysis on the defective product image output by the image defect detection model input to be diagnosed to obtain hybrid modeled image defects. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis;

[0019] The hybrid modeling image defects are used to obtain defect detection results through the improved U-Net model, which realizes the segmentation and positioning of the defect area, and then tracks the position of the image defect area. Decisions are made based on the defect tracking and fault tracing results to optimize the production process parameters.

[0020] The multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method provided by the present invention collects multimodal data (including image, electrical and working condition data) of defect locations from different batches, process stages and equipment, obtains an image defect detection model based on multiple multimodal fusion feature vectors combined with a convolutional neural network model, and applies the image defect detection model to actual scenarios to obtain detection results of whether the corresponding area of ​​the image has defects or no defects, thereby achieving accurate detection and positioning of minor component changes and structural defects inside the product, improving detection accuracy, and performing image defect area shape measurement calculation on images of defective products to be diagnosed to diagnose the severity level of the image defects and track the location of the image defect area, thereby achieving diagnosis and tracking decisions on the image defect location. After spatiotemporal alignment, image enhancement preprocessing and alignment, position encoding, CNN shallow network and ResNet-18 are used to extract features, which are combined with weight fusion to form a multimodal feature vector. The detection model is obtained through CNN and Transformer model training to achieve defect diagnosis and tracking decisions. The innovation lies in cross-modal feature complementarity, discrete and continuous hybrid modeling, and a multi-stage fusion strategy. This solves the problem of accurately detecting, locating, diagnosing, tracking, and making decisions about subtle component changes and structural defects within electronic products, as well as single local and global multimodal defects in electrical systems. Detection accuracy has increased by 20%, diagnostic accuracy has exceeded 98.5%, defect tracking error has decreased to less than 5%, decision-making efficiency has increased by 40%, and defect prediction capability has improved by 22%.

[0021] In an optional embodiment, performing data preprocessing and data alignment on defect location images of a plurality of multimodal data includes:

[0022] The defect location images are pre-processed in sequence, including data cleaning, data labeling, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization;

[0023] Data cleaning is used to remove noise data, blurred defect location images, and incomplete defect location images from defect location images; data labeling uses a combination of manual labeling and semi-automatic labeling to label defect locations and defect types; image usage classification divides multimodal data into image data, electrical data, and working condition data based on the production batch, process stage, equipment type, and multimodal data type to which the image belongs; image pixel normalization normalizes the pixel values ​​of image data to a preset range to unify the scale of image data;

[0024] Image data, electrical data, and operating condition data are aligned, fused, and standardized by timestamp to construct a temporally and spatially consistent dataset.

[0025] The multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method provided by the present invention aligns data from different sources such as three-dimensional structural information, thermal radiation information, and internal structure images by timestamp to construct a multimodal data set that is consistent in time and space. The three-dimensional / microstructure images are pre-processed by noise reduction, contrast enhancement, etc. to improve the quality of feature extraction. The electrical parameters (current / voltage / resistance) are normalized to eliminate dimensional differences. The defect location images are sequentially subjected to data cleaning, data labeling, and image usage classification, which improves the data reliability and image quality of the defect location images. Normalizing the features before fusion can avoid certain features from dominating the fusion process due to their large numerical range, ensuring that each feature can participate equally in the fusion, and improving the fusion effect and model stability.

[0026] In an optional embodiment, the operating condition data includes load fluctuation and ambient temperature;

[0027] Based on the defect location image of each image data, feature extraction is performed on the defect location images of the multiple multimodal data using a position encoding method to obtain image features after position encoding of each defect location image. At the same time, feature extraction is performed on the electrical data and the operating condition data respectively to obtain electrical features and operating condition features, including:

[0028] Use CNN shallow network to extract basic edge features and texture features of defect location images;

[0029] Determine the defect center position of each defect location image based on edge features and texture features; obtain the two-dimensional coordinate value of each defect center position and discretize each two-dimensional coordinate value; convert each discretized coordinate value into a feature vector through one-hot encoding, and use the feature vector as the image feature after the position encoding of each defect location image;

[0030] The electrical data is converted into a time series, and the frequency domain features are extracted as electrical features through Fourier transform. The real-time data of load fluctuations and ambient temperature in the operating condition data are sampled in a 5-minute sliding window to extract the operating condition features.

[0031] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method proposed in this paper considers location information. Through position encoding, it integrates defect location features into multimodal fusion, enabling the model to better utilize location information to detect defects. For devices such as smart meters, defects in different locations may have different impacts, allowing for more precise handling of location-related issues.

[0032] In an optional embodiment, the feature weight allocation scheme is set in the following manner:

[0033] The weight distribution of contrast features is 20% to 30%, the weight distribution of correlation features is 10% to 25%, the weight distribution of energy features is 10% to 25%, the weight distribution of entropy features is 30% to 40%, the weight distribution of defect entropy value features is 10% to 20%, and the weight distribution of position texture parameter features is 10% to 30%, and the sum of the contrast feature weight, correlation feature weight, energy feature weight, entropy feature weight, defect entropy value feature weight and position texture parameter feature weight is equal to 1.

[0034] In an optional embodiment, feature weight fusion is performed on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and position texture parameter feature based on a preset feature weight distribution scheme, and the image features, electrical features, and operating condition features are spliced ​​into a unified vector, including:

[0035] Calculate respectively the first product corresponding to each contrast feature and the contrast feature weight, the second product corresponding to each correlation feature and the correlation feature weight, the third product corresponding to each energy feature and the energy feature weight, the fourth product corresponding to each entropy feature and the entropy feature weight, the fifth product corresponding to each defect entropy value feature and the defect entropy value feature weight, and the sixth product corresponding to each position texture parameter feature and the position texture parameter feature weight;

[0036] A plurality of multimodal fusion feature vectors are obtained by adding each corresponding first product, second product, third product, fourth product, fifth product and sixth product.

[0037] Early fusion of multiple multimodal fusion feature vectors by splicing electrical features and operating condition features into a unified vector.

[0038] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided by this invention assigns weights to multiple features, such as contrast, correlation, energy, entropy, and defect entropy value, before fusing them together, effectively leveraging each feature in defect detection. Weights are assigned based on the effectiveness of different features in distinguishing different types of defects, making the fused feature vector more discriminative and meeting diagnostic requirements. This method also considers the characteristics of different image types and the performance of different features for different defect types, making the fusion solution more targeted and better suited to specific detection tasks, meeting the demand for high adaptability.

[0039] In an optional embodiment, the convolutional neural network model includes an input layer, multiple convolutional layers and pooling layers, multiple fully connected layers, and an output layer. The convolutional neural network model is sequentially trained with independent sub-models and evaluated based on multiple multimodal fusion feature vectors to obtain a single process image defect detection model, which also includes:

[0040] Multiple multimodal fusion feature vectors are transmitted through the input layer to multiple convolutional layers and pooling layers for deep learning. Multiple convolutional layers use convolution kernels to perform convolution operations on the multimodal fusion feature vectors to extract local features, and the pooling layer obtains multiple key features through downsampling operations.

[0041] Multiple fully connected layers are used to integrate multiple key features, convert the extracted features from local information into global information, and map them to the output probability distribution through activation functions;

[0042] The output layer is used to convert the probability distribution mapped to the output into multiple fused probability distribution images and output them. The value of each pixel in the fused probability distribution image represents the probability of a defect at that location.

[0043] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided by the present invention extracts key features through multiple convolutional layers, pooling layers, and multiple fully connected layers, making image details more prominent and providing conditions for subsequent model training.

[0044] In an optional embodiment, a single process image defect detection model is obtained by sequentially performing independent sub-model training and model evaluation on a convolutional neural network model based on multiple multimodal fusion feature vectors, including:

[0045] Divide multiple fused probability distribution images into training set, validation set and test set according to preset ratios;

[0046] The training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weighted and fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and the evaluation result is output. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model;

[0047] During the training process, the loss is calculated based on the preset loss function, and the Adam optimizer is used for backpropagation to adjust the weights and biases of the convolutional neural network model. The parameters of the multiple convolutional layers, pooling layers and multiple fully connected layers are adjusted based on the validation set to obtain the convolutional neural network model to be evaluated. The Adam optimizer combines the ideas of the momentum method and the Adagrad optimization algorithm, and dynamically adjusts the learning rate of each parameter in the model based on the first-order moment estimator and the second-order moment estimator of the gradient, and corrects the first-order moment estimator and the second-order moment estimator through the deviation.

[0048] In an optional embodiment, the training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weightedly fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and an evaluation result is output. When the evaluation result meets a preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as a single process image defect detection model, including:

[0049] The unified vector of defect features of multimodal fusion early fusion images is input into the convolutional neural network for late fusion;

[0050] The convolutional neural network model is trained to process image data specifically to extract defect features in images;

[0051] Train the long short-term memory network model to process time-series electrical data and capture the patterns of electrical data changes over time;

[0052] Train a multi-layer perceptron model to process operating condition data and analyze the impact of operating conditions on defects;

[0053] The outputs of the independent sub-models obtained from the above training are weightedly fused to obtain the detection model to be evaluated. The test set is input into the detection model to be evaluated. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model, and the evaluation result is used as the final defect diagnosis result.

[0054] In an optional embodiment, a multi-process image defect detection model is obtained by sequentially training and evaluating the Transformer network model based on multiple multimodal fusion feature vectors, including:

[0055] The early fusion of the image defect fusion path of the multimodal fusion Transformer uses the self-attention mechanism to effectively capture the semantic associations between different modal features and directly perform early cross-modal fusion of multimodal features;

[0056] A cross-attention mechanism is used to parallelly process local features extracted by multi-channel convolutional neural networks from different sources to fuse the complex defect feature detection model. The complex defect feature detection model and the Transformer processing time-series associated detection model are combined to form a combined detection model. The image to be diagnosed is input into the combined detection model to output the final defect diagnosis result.

[0057] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided by the present invention selects early or late fusion according to data characteristics, balances computational efficiency and information retention, and the technical characteristics of the combined CNN / Transformer model can intuitively present the core logic of defect diagnosis and tracking decision-making.

[0058] In an optional embodiment, the method further includes: using a visualization tool to visualize the loss changes during the training process of the convolutional neural network model, the changes in evaluation results, the key features deeply learned by multiple convolutional layers and pooling layers, and the feature integration of multiple fully connected layers and mapped to the probability distribution of the output.

[0059] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided by the present invention uses visualization tools to visualize the loss changes and evaluation result changes during the training process of the convolutional neural network model, the key features deeply learned by multiple convolutional layers and pooling layers, and the feature integration and mapping of multiple fully connected layers to the probability distribution of the output, thereby realizing real-time monitoring of the training process and improving training accuracy.

[0060] In an optional embodiment, a plurality of images of the product to be diagnosed are obtained and input into an image defect detection model for image defect diagnosis, and a detection result of whether a defect exists or not is obtained, including:

[0061] Image defect diagnosis is performed on multiple product images to be diagnosed that are input into the image defect detection model through a judgment function. The judgment function is expressed by the following formula:

[0062]

[0063] Wherein, G(F)=1 indicates that a defect is detected in the corresponding area of ​​the image, G(F)=0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that the preset defect condition is met;

[0064] The multimodal fusion feature vector of the input product image to be diagnosed is matched with the defect feature threshold set during model training. That is, when the feature parameters such as contrast, energy, and entropy in the feature vector exceed the preset threshold, it is judged that there is a defect, otherwise it is judged as no defect.

[0065] In an optional embodiment, hybrid modeling and analysis is performed on the defective product image output by the image defect detection model input to the image to be diagnosed, including:

[0066] Calculate the geometric features of the image defect area contour and determine the preliminary image defect area shape; the geometric features of the preliminary image defect area shape include contour perimeter, contour convex hull perimeter and shape factor;

[0067] Based on the shape of the initial defect area in the image, the shape hybrid modeling is performed using the second-order moment of discrete data and the second-order moment of continuous function to obtain the mathematical model for measuring the shape of the defect area in the image;

[0068] The defect degree of the image defect area is measured based on the image defect area shape measurement mathematical model, and the final shape, width, length, area and volume of the image defect area are obtained;

[0069] Diagnose and grade the severity of image defects based on the final shape, width, length, area, and volume of the image defect area.

[0070] The multimodal CNN / Transformer image defect diagnosis, tracking, and decision-making method provided by the present invention uses multi-parameter data fusion and reconstruction technology to obtain accurate internal three-dimensional structural information, combining the second-order moments of discrete data with the second-order moments of continuous functions for fusion. This solves the image accuracy problem of combining the second-order moments of discrete data with the second-order moments of continuous functions and the difficulty of weight distribution, which restricts high-precision image online analysis and visualization modeling. It is necessary to solve the problems of depth imaging technology attenuation at different depths and the limitations of a single modality, and to break through the difficulties of establishing a product defect model combining the second-order moments of discrete data with the second-order moments of continuous functions and weight distribution, so as to achieve high-precision image online analysis and visualization modeling.

[0071] In an optional embodiment, a shape hybrid model is performed based on the shape of the preliminary defect area of ​​the image using the second-order moment of discrete data and the second-order moment of continuous function to obtain a mathematical model for measuring the shape of the image defect area, including:

[0072] Determine the discrete data second-order moment and continuous function second-order moment of the image defect area based on the shape of the preliminary defect area of ​​the image;

[0073] The discrete data second-order moment and the continuous function second-order moment of the image defect area are fused and modeled to obtain the mathematical model for shape measurement of the image defect area.

[0074] In an optional embodiment, the discrete data second-order moment of the image defect area and the continuous function second-order moment are fused and modeled to obtain a mathematical model for shape measurement of the image defect area, including:

[0075] Calculating a first moment of inertia based on the second-order moment of discrete data, and calculating a first equivalent ellipse parameter based on the first moment of inertia;

[0076] Calculating a second moment of inertia based on the second-order moment of the continuous function, and calculating a second equivalent ellipse parameter based on the second moment of inertia;

[0077] The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent ellipse parameters and the second equivalent ellipse parameters, to obtain the mathematical model for shape measurement of the image defect area.

[0078] In an optional embodiment, a weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent ellipse parameters and the second equivalent ellipse parameters, respectively, to obtain a mathematical model for measuring the shape of the image defect area, including:

[0079] Set the weight of the second-order moment of the continuous function, and calculate the weight of the second-order moment of the discrete function based on the weight of the second-order moment of the continuous function;

[0080] Based on the second-order moment weight of the continuous function and the second-order moment weight of the discrete function, the first moment of inertia and the second moment of inertia are fused by the weighted average method to obtain the fused second-order moment; the first equivalent ellipse parameters and the second equivalent ellipse parameters are fused to obtain the fused equivalent ellipse parameters;

[0081] The mathematical model for measuring the shape of the image defect area is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters;

[0082] The fused second-order moment formula is as follows:

[0083] M fusion =aM c +(1-a)M d ,

[0084] Among them, M c is the second-order moment of a continuous function, M dis the second-order moment of discrete data; a (0≤a≤1) is a weight coefficient used to measure the proportion of the second-order moment of the continuous function in the fused second-order moment; when a=0, it means that when calculating the fused second-order moment, the contribution of the second-order moment of the continuous function is completely ignored, and only the second-order moment of the discrete data is used; when a=1, it completely relies on the second-order moment of the continuous function, and does not consider the second-order moment of the discrete data.

[0085] The multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method provided by the present invention adopts the weighted averaging method to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent ellipse parameters and the second equivalent ellipse parameters, respectively, to obtain a mathematical model for measuring the shape of the image defect area, thereby realizing the construction of the shape of the image defect area and providing conditions for tracking decisions on the image defect area.

[0086] In an optional embodiment, the hybrid modeling image defects are subjected to an improved U-Net model to obtain defect detection results, implement defect area segmentation and positioning, and then track the position of the image defect area, including:

[0087] The hybrid modeling image defects are applied to the improved U-Net model to obtain defect detection results, and the defect area is segmented and located based on the defect detection results;

[0088] The image defect position is tracked based on the segmented and located defect area.

[0089] In an optional embodiment, the semantic segmentation model of the improved U-Net model includes an encoder, a decoder and a skip connection, the encoder includes a residual connection, and the decoder includes an attention mechanism;

[0090] The hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, achieving defect area segmentation and positioning, including:

[0091] In the encoder, the image defect area is convolved to obtain the output feature map and the input feature map, and the output feature map and the input feature map are added through the residual connection to obtain a new output feature map;

[0092] The new output feature map is input into the attention mechanism of the decoder to obtain the weighted feature map, and then the convolution operation is performed to obtain the TFT ray map;

[0093] The TFT ray map is input into the improved U-Net model for image region segmentation to obtain a binary image of the image defect area.

[0094] In an optional embodiment, after performing convolution processing on the image defect area in the encoder, an output feature map and an input feature map are obtained, and the output feature map and the input feature map are added through a residual connection to obtain a new output feature map, including:

[0095] Residual connection is introduced in the encoder, and the weight matrix of the residual connection is The output feature map after two convolutional layers is With input features Figure X i (x, y, z), and add the output feature map and the input feature map through the residual connection to obtain a new output feature map The formula is as follows:

[0096]

[0097] Where (x, y, z) is the image coordinate.

[0098] In an optional embodiment, the new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, which is then convolved to obtain a TFT ray map, including:

[0099] The attention mechanism is introduced in the decoder, and the attention weight matrix is Feature map after upsampling and concatenation by the encoder The weighted features are obtained through the attention mechanism That is the TFT ray diagram, the formula is as follows:

[0100]

[0101] Among them, (x, y, z) are image coordinates, a represents the relative position offset involved when the feature map performs a weighted operation using the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[0102] In an optional embodiment, defect tracking and fault tracing results are used to make decisions and optimize production process parameters, including:

[0103] A defect evolution dynamics model is constructed based on discrete data points of defect area and perimeter and a continuous function second-order moment model;

[0104] Based on the defect location image, a preset tracking function is used to analyze the morphological change characteristics of the crack propagation angle and wear depth increment of the defect in the defect area of ​​the image at different times, and the defect evolution process is visualized and quantitatively analyzed;

[0105] Obtain the physical defect locations in the 3D structural information and use a spatial mapping algorithm to match the physical defect locations in the 3D structural information with the electrical anomaly data and microstructure images, respectively, to establish a correlation map between physical defects and electrical faults.

[0106] Based on the correlation map between physical defects and electrical faults and the morphological change characteristics, combined with the abnormal current distribution analysis in the wear area of ​​the resistor surface, the fault source is traced to the location of the image defect area;

[0107] Based on defect tracking and fault tracing, the welding process, material replacement process and online monitoring process of the decision-making products are improved.

[0108] In an optional embodiment, improving the welding process of the product to be decided includes:

[0109] Finite element simulation is used to analyze the effects of different welding parameters such as laser power, welding speed, and spot diameter on the residual stress, fatigue life, and resistance value of the welded joint for the product to be determined. The simulation results are verified by combining multimodal data from ultrasonic scanning results to determine the optimal process parameters for laser welding and resistance brazing. The effects of multiple power sources such as photovoltaic, green electricity, energy storage, mains electricity, combined electricity, and nuclear power on the welding parameters are also considered. The defect location images of weld cracks and cold welds detected by convolutional neural networks are combined with multimodal data from ultrasonic scanning to quantify the welding quality.

[0110] Improve the material replacement process for the products to be decided, including:

[0111] Establish a material performance database to record the resistance change rate and capacitance stability performance parameters of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments. Through accelerated aging tests simulating extreme working conditions, use convolutional neural networks to detect defects in images of aged materials, and evaluate material durability based on performance data.

[0112] Improve the online monitoring process of decision-making products, including:

[0113] Deploy a real-time image recognition system based on deep learning to analyze X-ray or OBIRCH inspection images in milliseconds to automatically label and classify defects;

[0114] Use deep learning algorithms to train X-ray or OBIRCH inspection images to establish a defect recognition model.

[0115] In a second aspect, the present invention provides a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making device, which includes:

[0116] A multimodal data acquisition module is used to acquire defect location images of multiple multimodal data of the product to be diagnosed. The multiple defect location images are defect location images containing the above multimodal data and are taken from different production batches, different process stages, and different equipment. The multimodal data includes image data, electrical data, and operating condition data.

[0117] A data preprocessing and data alignment module is used to perform data preprocessing and data alignment on defect location images of multiple multimodal data;

[0118] A feature extraction module is used to extract features from defect location images of multiple multimodal data using a position encoding method based on the defect location image of each image data, thereby obtaining the image features of each defect location image after position encoding. Simultaneously, feature extraction is performed on the electrical data and the operating condition data respectively to obtain electrical features and operating condition features. Image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features.

[0119] A multimodal feature fusion module is used to perform feature weight fusion on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and position texture parameter feature based on a preset feature weight distribution scheme as an image feature, and to concatenate the image features, electrical features, and operating condition features into a unified vector to obtain multiple multimodal fusion feature vectors;

[0120] A multimodal detection model construction and detection module is used to sequentially train and evaluate the convolutional neural network model based on multiple multimodal fusion feature vectors to obtain a single-process image defect detection model, and sequentially train and evaluate the Transformer network model based on multiple multimodal fusion feature vectors to obtain a multi-process image defect detection model; obtain multiple actual images of products to be diagnosed and input them into the single-process image defect detection model and / or the multi-process image defect detection model for image defect diagnosis, and obtain a detection result indicating whether the corresponding area of ​​the product image has defects or not;

[0121] A hybrid modeling and analysis module is used to perform hybrid modeling and analysis on the defective product image output by the image defect detection model input to be diagnosed, thereby obtaining hybrid modeled image defects. The hybrid modeling and analysis includes shape model construction, defect quantification, and severity diagnosis.

[0122] The tracking decision module is used to obtain defect detection results from the hybrid modeling image defects through the improved U-Net model, realize defect area segmentation and positioning, and then track the position of the image defect area. Decisions are made based on the defect tracking and fault tracing results to optimize production process parameters.

[0123] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to thereby execute the multimodal CNN / Transformer image defect diagnosis and tracking decision method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0124] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimodal CNN / Transformer image defect diagnosis and tracking decision method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0125] In a fifth aspect, the present invention provides a computer program product comprising computer instructions, which are used to enable a computer to execute the multimodal CNN / Transformer image defect diagnosis and tracking decision method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0127] Figure 1 1 is a flow chart of a multimodal CNN / Transformer image defect diagnosis and tracking decision-making process according to an embodiment of the present invention;

[0128] Figure 2 2 is a flowchart of another multimodal CNN / Transformer image defect diagnosis and tracking decision-making process according to an embodiment of the present invention;

[0129] Figure 3 yes Figure 1 A flowchart of another multimodal CNN / Transformer image defect diagnosis and tracking decision method according to an embodiment of tracking the regional location of a product image defect to be diagnosed;

[0130] Figure 4 yes Figure 1 A flowchart of another multimodal CNN / Transformer image defect diagnosis and tracking decision method for tracking the regional position of a product image defect to be diagnosed in an embodiment;

[0131] Figure 52 is a flowchart of another multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method according to an embodiment of the present invention;

[0132] Figure 6 2 is a schematic diagram of the structure of a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making system according to an embodiment of the present invention;

[0133] Figure 7 This is a schematic diagram of detecting the position of a welding void and measuring the shape, size, and area of ​​the defect by X-ray detection of a TVS diode (SMBJ10CA) of a smart meter according to an embodiment of the present invention;

[0134] FIG8( a ) is a front X-ray schematic diagram of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0135] FIG8( b ) is a partially enlarged X-ray schematic diagram of the front side of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0136] FIG8( c ) is a side X-ray schematic diagram of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0137] FIG8( d ) is a partially enlarged X-ray diagram of the side surface of a TVS diode (SMBJ10CA) according to an embodiment of the present invention;

[0138] Figure 9 Schematic diagram of detecting the position of a welding void and measuring the shape, size, and area of ​​the defect by X-ray detection of a rectifier diode (M7) of a smart meter according to an embodiment of the present invention;

[0139] Figure 10 2. It is a schematic diagram of position tracking for identifying, detecting and measuring welding cracks in a current transformer of a smart meter according to an embodiment of the present invention;

[0140] FIG11( a ) is a schematic diagram of the overall measurement dimensions of identifying, detecting, and measuring the position, detection, and tracking of the chip, leads, lead terminals, and solder voids of a smart meter according to an embodiment of the present invention;

[0141] FIG11( b ) is a partial schematic diagram of identifying, detecting, and measuring the position, detection, and tracking of solder voids between a chip and leads, lead terminals, and a substrate of a smart meter according to an embodiment of the present invention;

[0142] FIG11( c ) is a schematic diagram illustrating the positions of the chip, leads, lead terminals, and substrate soldering voids of a smart meter according to an embodiment of the present invention;

[0143] Figure 12Schematic diagram of detecting protrusions, voids, cracks, and insufficient soldering in a PCB, crystal oscillator, and seven chips of a smart meter according to an embodiment of the present invention;

[0144] Figure 13 1 is a structural block diagram of a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making device according to an embodiment of the present invention;

[0145] Figure 14 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0146] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method.

[0147] The relationship between images and defects is complex. When the image background is complex and contains numerous interfering factors, low contrast (C) and low contrast weight make it difficult to locate subtle texture defects within the image (making the defect unclear). Highly noisy textures are chaotic and uncorrelated with pixels, and weld defects have a negative correlation with cold pixels, making it difficult or impossible to identify the defect's edge, extent, and shape. Pixel gradients have a high positive correlation with image changes, making it easier to identify the defect's edge, extent, and shape. The correlation R is complexly related to the defect, location, texture, and weight factors. Energy E, its relationship with defect, location, and texture, and its weighting influence energy. Energy primarily reflects the concentration of the image's grayscale distribution. Grayscale variations in the image are primarily caused by defects, so the energy feature will be weighted higher; otherwise, it will be weighted lower. Entropy H, its relationship with defect, location, and texture, and its weighting influence entropy. Entropy reflects the amount of information in the image. Defect regions with complex structures or grayscale variations will have increased entropy. Normal textures have relatively stable entropy, while defects have significant entropy variations. Therefore, entropy features will have a higher weight when detecting defect location and texture changes. Existing images lack high cleanliness and resolution. It is caused by the complex relationship between the comprehensive error of image parameters (contrast C, correlation R, energy E, entropy H, etc.) and defects. How to reduce the comprehensive error and integrate contrast C, correlation R, energy E, entropy H, etc. with defect characteristics is a difficult problem in the industry.

[0148] Existing X-ray, ultrasonic, infrared, and laser depth imaging have limitations in attenuation at different depths and single-modal imaging in depth imaging. Existing depth imaging technology lacks multi-parameter data fusion and reconstruction technology to obtain more accurate three-dimensional structural information (visualization) inside power computing equipment. The difficulty in improving image accuracy lies in the establishment of defect models (including position and shape models) such as the combination of discrete data second-order moments and continuous function second-order moments with the product's own performance, as well as the difficulty in accurately allocating the weights of discrete data second-order moments and continuous function second-order moments, which seriously restricts the realization of high-precision image online analysis and visualization modeling.

[0149] By combining non-destructive testing such as X-rays with visual image pattern recognition, the reliable quality of smart meters, metering automation terminals, power electronic components, mutual inductors, PCB boards, and component welding connections can be achieved, and new needs such as high-frequency data collection, dynamic adjustment of time-of-use electricity prices, active real-time perception of power outages and restorations, and low-voltage monitoring can be realized. It solves the problems of the lack of comprehensive perception of the status of existing technical equipment and the lack of flexible expansion and upgrading capabilities, establishes a complete technical standard system for electrical measurement enterprises, and realizes full coverage and collection of metering and billing data for all links of power generation, transmission, transformation, distribution, use, and computing power integration, providing powerful measurement and energy utilization technical support for power computing power and related product marketing, production operation, planning and construction and other related businesses, and giving full play to the basic support and technical leading role of metering and billing in modern power supply service systems, power-carbon-computing-network integration, digital power grids and new power systems.

[0150] The rough surface, weak internal defects, and complex background of power electronics industry products make it difficult to effectively detect internal defects; the optical and non-destructive testing equipment is interfered with, making it difficult to detect internal defects, poor image clarity, insufficient information, small and chaotic information, and low spatial resolution. It can only provide the approximate location and situation of the defect, which is not conducive to computer detection, fault location and diagnosis. It is impossible to accurately detect and locate the tiny changes in the internal composition and structural defects of the product, such as Figure 1 The more detailed embodiment shown provides a multimodal CNN / Transformer image defect diagnosis and tracking decision method, which achieves the effects of image defect detection, measurement, diagnosis and tracking by combining multimodal imaging of power electronics industry products with internal structure visualization and multimodal optimization of CNN / Transformer deep learning.

[0151] In this embodiment, a multimodal CNN / Transformer image defect diagnosis and tracking decision method is provided, which can be used in computer equipment. Figure 2: is a flowchart of a multimodal CNN / Transformer image defect diagnosis and tracking decision method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0152] Step S101, obtaining defect location images of multiple multimodal data of the product to be diagnosed, wherein the multiple defect location images are defect location images containing the above multimodal data taken by different production batches, different process stages and different equipment; the multimodal data includes image data, electrical data and working condition data.

[0153] Specifically, from the defect position image acquisition range: multiple defect position images of the product to be diagnosed are obtained; the multiple defect position images are defect position images taken from different production batches, different process stages and different equipment.

[0154] Furthermore, multimodal data of multiple data types is included within the scope of the defect location image acquisition; the multimodal data definition of the multiple data types and the multimodal data fusion processing flow are as follows:

[0155] Definition of multimodal data: In the embodiments of the present invention, multimodal data refers to different types of data about the product to be diagnosed, collected from multiple dimensions and using multiple technical means, specifically covering the following categories:

[0156] (1) Image data:

[0157] Three-dimensional structural information: The three-dimensional structural information of the product to be decided is obtained through X-ray tomography, which can present the overall and internal spatial structure of the product.

[0158] Thermal radiation information: Thermal radiation information of the product to be decided is obtained using thermal infrared imaging, reflecting the heat distribution on the surface and inside of the product.

[0159] Internal structure information: Ultrasonic imaging can be used to obtain information about the internal structure of the product to be decided, and the physical structural characteristics inside the product can be detected.

[0160] Microstructure images: Microstructure images collected using a FIB-SEM dual-beam system show the organizational structure of the product material at a microscopic level.

[0161] (2) Electrical data:

[0162] Electrical anomaly point data: Electrical anomaly point data located using OBIRCH technology is used to determine the specific location of anomalies in the product's electrical system.

[0163] Electrical parameter data: Electrical parameter data such as current, voltage, and resistance collected by nanoprobes reflect the product's electrical performance indicators.

[0164] (3) Operating condition data: The real-time operating condition data of the product to be decided is collected, including load fluctuation and ambient temperature data, etc., reflecting the external conditions and load changes during the operation of the product.

[0165] These data of different modalities describe product characteristics from multiple aspects and scales, providing a rich information basis for comprehensive analysis of product status, diagnosis of defects and tracking decisions.

[0166] Step S102 : performing data preprocessing and data alignment on the defect position images of the plurality of multimodal data.

[0167] Specifically, if Figure 1 As shown, data preprocessing and data alignment of defect location images of multiple multimodal data include:

[0168] 1. Timestamp synchronization: Align data from different sources, such as three-dimensional structural information, thermal radiation information, and internal structure images, by timestamp to construct a multimodal dataset that is consistent in time and space.

[0169] 2. Image enhancement: Perform pre-processing such as noise reduction and contrast enhancement on three-dimensional structure information and microstructure images to improve the quality of feature extraction.

[0170] 3. Feature standardization: Normalize electrical data (current / voltage / resistance) to eliminate dimensional differences.

[0171] Step S103, based on the defect location image of each image type data, feature extraction is performed on the defect location images of multiple multimodal data using a position encoding method to obtain the image features after position encoding of each defect location image. At the same time, feature extraction is performed on the electrical data and the working condition data respectively to obtain electrical features and working condition features; the image features include contrast features, correlation features, energy features, entropy features, defect entropy value features and position texture parameter features.

[0172] Specifically, position encoding is used to add position information to the input sequence. Position encoding is used to convert each defect location image into a form that can be effectively processed by the neural network model. The contrast feature, denoted by C, refers to the brightness difference between the brightest and darkest areas in each defect location image. The correlation feature, denoted by R, refers to the similarity or correlation between different images or between different parts of the same image. The energy feature, denoted by E, is a concept based on neuropsychology or brain science. It refers to the psychological impact of an image. It is the result of the combined effects of factors such as image color, position, shape, and relationships. These factors determine the psychological and physiological effects of an image on a person. The entropy feature, denoted by H, is an estimate of the image's "busyness." It is expressed as the average number of bits in the image's grayscale set, measured in bits per pixel. It also describes the average information content of the image source. The defect entropy feature, denoted by D, measures the "busyness" or information content of the defect in the image. The positional texture parameter feature, denoted by P, includes texture size, texture coordinates, and texture pixel data.

[0173] like Figure 1 As shown, image feature extraction:

[0174] (1) Position code extraction; contrast features, correlation features, energy features, entropy features, defect entropy value features and position texture parameter features.

[0175] (2) Use CNN shallow network to extract basic image features, including: Extracting edge / texture features: Use ResNet-18 to extract edge features and texture features.

[0176] In step S104, based on a preset feature weight distribution scheme, each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature is subjected to feature weight fusion as an image feature, and the image feature, electrical feature and working condition feature are spliced ​​into a unified vector to obtain multiple multimodal fusion feature vectors.

[0177] Specifically, if Figure 1 As shown in the figure, time series feature conversion: convert electrical data into time series and extract frequency domain features through Fourier transform.

[0178] Working condition feature processing: Sliding window sampling is performed on real-time data such as load fluctuations and ambient temperature to extract statistical features. A 5-minute sliding window is used for statistics.

[0179] Based on the preset feature weight distribution scheme, feature weight fusion is performed on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature, and the image features, electrical features and working condition features are spliced ​​into a unified vector to obtain multiple multimodal fusion feature vectors.

[0180] In step S105, the convolutional neural network model is sequentially trained with independent sub-models and evaluated based on multiple multimodal fusion feature vectors to obtain a single-process image defect detection model, and the Transformer network model is sequentially trained with model evaluation based on multiple multimodal fusion feature vectors to obtain a multi-process image defect detection model.

[0181] Specifically, based on multiple multimodal fusion feature vectors, the convolutional neural network model for a single process and the network model for a multi-process Transformer network model are trained and evaluated in sequence to obtain an image defect detection model. Multiple images of the product to be diagnosed are actually taken and input into the image defect detection model for image defect diagnosis, obtaining a detection result indicating whether the corresponding area of ​​the image has defects or not.

[0182] The specific paths of the above two unified vectors with the help of multimodal fusion are: the multimodal fusion of the present invention is divided into two types: multimodal fusion CNN image defects and multimodal fusion Transformer image defects; the image defects of the multimodal fusion CNN and other solutions of the present invention constitute the first invention solution, and the image defects of the multimodal fusion Transformer can constitute the second invention solution with other solutions of the present invention; the difference between the two is:

[0183] In early fusion (feature level), image defects of multimodal fusion Transformer can be directly fused across modalities, while image defects without multimodal fusion CNN need to be spliced ​​first.

[0184] In the late fusion (decision level), the image defects of the multimodal fusion Transformer can directly realize the fusion of complex image defects of the multimodal fusion multi-channel CNN.

[0185] CNN image defect fusion path (simple image defects): Early fusion (feature level): image features, electrical features, and working condition features are spliced ​​into a unified vector.

[0186] Late fusion (decision level):

[0187] (1) Train sub-models independently and fuse their outputs.

[0188] (2) Train CNN to process image data.

[0189] (3) Train LSTM to process time series electrical data.

[0190] (4) Train MLP to process working condition data.

[0191] (5) Weighted fusion model output.

[0192] Transformer’s multi-process fusion path for complex image defects;

[0193] Early fusion (feature level),Early fusion uses the self-attention mechanism to directly fuse across modalities, which has the advantage of not requiring feature splicing and being able to capture semantic associations.

[0194] Decision-level late fusion:

[0195] Parallel processing of multi-channel CNN features, cross-attention fusion of complex defect features, actual detection output defect results based on multimodal fusion strategy and path output, and hybrid modeling and shape analysis.

[0196] Step S106, obtain multiple images of the product to be diagnosed that are actually taken and input them into a single-process image defect detection model and / or a multi-process image defect detection model for image defect diagnosis, and obtain a detection result of whether the corresponding area of ​​the product image has defects or no defects.

[0197] Step S107 , hybrid modeling and analysis are performed on the defective product image output by the image defect detection model input to be diagnosed to obtain hybrid modeled image defects, wherein the hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0198] Specifically, the shape model construction is second-order rectangular modeling, including:

[0199] (1) Construct a shape model based on the second-order moment of discrete data and the second-order moment of continuous function.

[0200] (2) Calculate the second-order moment matrix of discrete points.

[0201] (3) Fit the continuous function and calculate the second-order moment characteristics.

[0202] (4) Fusion of two moment features.

[0203] Defect quantification of the fusion of two moment feature shape models: Through geometric feature calculation: Calculate the defect quantification parameters such as the perimeter, area, and volume of the defect area based on the shape model, and use the defect quantification parameters for severity grading assessment based on the shape measurement results.

[0204] The severity is diagnosed based on the shape measurement results, and decision-level fusion and application are performed.

[0205] In step S108, the hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, realize defect area segmentation and positioning, and then track the position of the image defect area, make decisions based on the defect tracking and fault tracing results, and optimize the production process parameters.

[0206] Specifically, if Figure 1As shown in the figure, the hybrid modeling image defects are subjected to the improved U-Net model to obtain the defect detection results, realize the defect area segmentation and positioning, and then track the position of the image defect area. Decisions are made based on the defect tracking and fault tracing results, and the production process parameters are optimized. These are the decision-level fusion and application steps, including:

[0207] 1. Defect diagnosis network: Train and improve the U-Net defect detection model, and obtain defect detection results by fusion feature images through the improved U-Net model to achieve defect area segmentation and positioning.

[0208] 2. Defect location tracking and fault tracing;

[0209] Process improvement decision-making: Through defect tracking and fault tracing results, optimize and adjust the production process (such as water flow, gas flow, temperature, pressure, current, and time parameters).

[0210] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided by the present invention also has the following key technical features:

[0211] Multimodal data fusion: Spatiotemporal alignment ensures data consistency.

[0212] Feature extraction combination: CNN extracts local features + Transformer processes temporal associations.

[0213] Decision-level applications: closed-loop feedback from defect detection to process optimization.

[0214] Figure 1 It clearly demonstrates the entire process from data collection to decision optimization, highlights the technical features of the combination of multimodal data fusion and CNN / Transformer model, and can intuitively present the core logic of defect diagnosis and tracking decision-making. Figure 1 The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by the present invention is described in detail:

[0215] 1. Multimodal data acquisition:

[0216] Acquire multiple defect location images of the product to be diagnosed; the multiple defect location images are defect location images taken from different production batches, different process stages, and different equipment; the multimodal data is defined as follows:

[0217] In the embodiments of the present invention, multimodal data refers to different types of data about the product to be decided, collected from multiple dimensions and using multiple technical means, and specifically covers the following categories:

[0218] 1. Image data

[0219] (1) Three-dimensional structural information: The three-dimensional structural information of the product to be decided is obtained through X-ray tomography, which can present the overall and internal spatial structure of the product.

[0220] (2) Thermal radiation information: The thermal radiation information of the product to be decided is obtained by thermal infrared imaging, which reflects the heat distribution on the surface and inside of the product.

[0221] (3) Internal structure information: The internal structure information of the product to be decided is obtained with the help of ultrasonic imaging, which can detect the physical structural characteristics inside the product.

[0222] (4) Microstructure image: The microstructure image is collected by the FIB-SEM dual-beam system to show the organizational structure of the product material at the micro level.

[0223] 2. Electrical data:

[0224] (1) Electrical anomaly point data: Electrical anomaly point data located using OBIRCH technology is used to determine the specific location of anomalies in the product's electrical system.

[0225] (2) Electrical parameter data: Electrical parameter data such as current, voltage, and resistance collected by nanoprobes, reflecting the electrical performance indicators of the product.

[0226] 3. Operating condition data: The real-time operating condition data of the product to be decided is collected, including load fluctuation and ambient temperature data, etc., reflecting the external conditions and load changes during the product operation process.

[0227] These data of different modalities describe product characteristics from multiple aspects and scales, providing a rich information basis for comprehensive analysis of product status, diagnosis of defects and tracking decisions.

[0228] The overall multimodal data fusion framework of this embodiment includes a phased, integrated fusion framework and a multi-strategy, integrated approach to multimodal data fusion processing. The framework, from data acquisition to decision-making and application, is divided into five core steps: data preprocessing and alignment, feature extraction and transformation, multimodal fusion strategy, hybrid modeling and shape analysis, and decision-level fusion and application. This framework utilizes time alignment technology, a cross-modal feature fusion algorithm, and a dynamic weight allocation mechanism to accurately detect and track subtle defects within electronic products.

[0229] 2. Data Preprocessing and Alignment: Building a Unified Data Benchmark

[0230] 1. Timestamp synchronization: Data from different modal sources, such as X-ray 3D structural information, thermal infrared radiation information, and ultrasonic internal structure images, are spatiotemporally aligned based on acquisition timestamps to ensure consistency across the time dimension. For example, aligning 3D CT images and thermal imaging data of the same product taken within 30 minutes after the welding process can avoid defect location errors caused by time misalignment.

[0231] Technical advantages: By building spatiotemporal consistency, it solves the timing disorder problem of asynchronous data collection by multiple devices and provides a unified time base for subsequent feature fusion.

[0232] 2. Image enhancement preprocessing:

[0233] Preprocessing methods such as median filtering for noise reduction and histogram equalization for contrast enhancement are used on 3D CT images and FIB-SEM microstructure images to improve the edge clarity of characteristic defects. For example, in chip solder joint defect detection, enhancement processing can increase the grayscale contrast of the cold solder joint area by more than 40%.

[0234] Technical advantages: Eliminate noise interference during image acquisition and improve the accuracy of subsequent feature extraction.

[0235] 3. Feature standardization: The electrical parameters such as current (mA level), voltage (mV level), and resistance (kΩ level) collected by the nanoprobe are Z-score normalized and the data are mapped to the [-1,1] interval.

[0236] Technical advantages: Eliminate the impact of dimensional differences on the fusion model and avoid high-value range features (such as resistance) dominating the fusion.

[0237] 3. Feature extraction and conversion:

[0238] Image feature extraction: Based on the defect position of each defect position image, feature extraction is performed on the defect position image using a position encoding method to obtain a feature set of each defect position image after position encoding. The feature set includes contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features.

[0239] Load the pre-trained ResNet-18 model for CNN feature extraction, including: intercepting the first 4 layers as a shallow feature extractor (extracting basic features such as edges and textures).

[0240] Initialize feature weight configuration (the importance of each feature can be customized).

[0241] 1. Feature extraction:

[0242] (1) Position coding feature extraction, including:

[0243] Extract regions of interest based on defect masks.

[0244] Calculate the gray-level co-occurrence matrix features.

[0245] Calculate image entropy and defect entropy (the entropy difference between defective areas and normal areas).

[0246] Extract CNN shallow features (using the first 4 layers of ResNet-18).

[0247] Extract edge features (Canny operator) and texture features (LBP operator).

[0248] Returns the integrated multimodal feature set.

[0249] (2) Basic feature extraction methods, including:

[0250] Gray-level co-occurrence matrix features: Convert to grayscale image and calculate GLCM to extract three key features: contrast, correlation and energy.

[0251] CNN shallow features: Image preprocessing and the use of the ResNet-18 shallow convolutional layer to extract basic features such as image edges and textures. For example, in chip packaging defect detection, a 3×3 convolution kernel is used to capture crack edge features on the solder joint surface.

[0252] Edge features and texture features: Combine the Canny operator and the LBP texture operator to extract the geometric features of the defect area, such as the direction of the crack and the texture complexity of the wear area.

[0253] Among them, LBP (local binary pattern) is used to extract texture features: the LBP histogram is calculated as the texture feature vector.

[0254] 2. Feature fusion: Based on the preset feature weight distribution scheme, each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature is fused with feature weights to obtain multiple multimodal fusion feature vectors; specifically, it includes:

[0255] Fuse multimodal features based on preset weights, fuse various features by weight, process dictionary-type features (such as edge features containing multiple sub-features), and process scalar or vector features.

[0256] 3. Technical features and advantages:

[0257] (1) Multimodal fusion: Integrate image geometric features, CNN features, texture features, and statistical features to comprehensively describe defect characteristics.

[0258] (2) Position encoding: Accurately locate the region of interest based on the defect mask to improve the targeted feature extraction.

[0259] (3) Configurable weights: Supports customizing the weights of each feature to meet the needs of different industrial scenarios.

[0260] (4) Hybrid feature processing: unified processing of scalar features (such as contrast) and vector features (such as CNN features).

[0261] (5) Industrial-grade precision: Combining traditional image processing with deep learning, it is suitable for precision manufacturing defect detection such as chips.

[0262] 4. Typical application scenarios: solder joint defect detection in semiconductor chip manufacturing, circuit breakage detection on PCB circuit boards, surface wear analysis of precision mechanical parts, diagnosis of internal structural defects in battery production, and real-time monitoring of equipment in high-temperature and high-pressure environments.

[0263] 5. Time series feature conversion: Convert real-time current and voltage data into time series, and extract frequency domain features (such as the proportion of harmonic components) through Fourier transform. For example, in electrolytic capacitor defect detection, frequency domain features can reflect abnormal changes in internal dielectric loss.

[0264] 6. Operating condition feature processing: A sliding window (window size of 5 minutes) is used to extract statistical features such as mean and variance from data such as load fluctuations and ambient temperature. For example, in smart meter operating status monitoring, the variance of temperature fluctuations can be used as an early warning indicator for thermal defects.

[0265] 4. Multimodal Fusion Strategy: Technical Decomposition of CNN and Transformer:

[0266] The multimodal fusion of the present invention is divided into two types: single-process (single) defect multimodal fusion CNN image defect and multi-process multimodal fusion Transforme model image defect; the multimodal fusion CNN image defect and other solutions of the present invention constitute the first invention solution, and the multimodal fusion Transformer model image defect can constitute the second invention solution with other solutions of the present invention; the detailed technical decomposition of the two is shown in Table 1 below:

[0267] Table 1

[0268]

[0269]

[0270]

[0271]

[0272]

[0273]

[0274]

[0275] Typical application scenarios of multimodal fusion Transformer: implementation cases in smart meter manufacturing.

[0276] Taking the multimodal detection of flip-chip soldering of smart meter chips as an example, the multimodal fusion Transformer can be used to implement the following in the flip-chip soldering process of metering chips using a 0.13μm process:

[0277] Image mode: AOI detects solder ball offset (accuracy ±0.01mm).

[0278] Electrical mode: Microprobe test of solder joint contact resistance (resolution 1mΩ).

[0279] Thermal mode: Infrared thermal imager captures the instantaneous temperature field of welding (accuracy ±1°C).

[0280] After Transformer fusion, it is possible to identify compound defects of "solder ball offset + contact resistance critical value + local overheating" that are missed by traditional methods. For example, a combination defect of a solder ball offset of 0.03mm, a contact resistance of 15mΩ (critical value 10mΩ) and a soldering temperature of 230°C (standard 250°C) is found. The missed detection rate of such defects in traditional single-modal detection is as high as 30%, but the Transformer detection solution can reduce the missed detection rate to below 1%.

[0281] The Transformer model can be integrated to control the full life cycle quality of sampling component welding, that is, the welding of relay copper plates in the sampling circuit of smart meters.

[0282] Production stage: X-ray inspection of solder joint porosity (threshold <5%).

[0283] Testing phase: Contact resistance stability of 100A high current on-off test.

[0284] Operation phase: Temperature, humidity and load condition data collected on site.

[0285] By modeling long-range dependencies, Transformer can identify solder joints with a porosity of 4.5% (near the threshold) during production. After one year of operation, this porosity expands due to temperature cycling, ultimately leading to premature failure with a sudden increase in contact resistance. After implementing this technology, one electricity meter manufacturer reduced the field failure rate related to sampling circuit welding from 0.8% to 0.15%, reducing after-sales repair costs by 2 million yuan annually.

[0286] 5. Hybrid modeling and shape analysis, i.e. accurate characterization of defect geometric features:

[0287] The shape of the defective area of ​​the image is measured and calculated for defective product images to diagnose the severity level of the image defect and track the location of the image defect area.

[0288] 1. Second-order rectangular modeling:

[0289] Discrete data second-order moment: The second-order moment matrix of the contour is calculated based on the coordinates of the pixel points in the binary image of the defect area. The main direction and eccentricity of the defect can be obtained. For example, the discrete second-order moment of a crack defect can reflect its extension direction.

[0290] Second-order moment of continuous function: Fit the defect contour with a Gaussian function and calculate the second-order moment in continuous space to improve the accuracy of describing irregular defect shapes. For example, continuous modeling of wear areas can reduce shape distortion caused by discrete sampling.

[0291] 2. Fusion strategy: Use weighted average method to fuse two moment features. The formula is:

[0292] M fusion =0.6M C +0.4M d ;

[0293] Among them, M C is the second-order moment of a continuous function, M d It is the second-order moment of discrete data and can reduce the error of defect shape description by 30%.

[0294] 3. Geometric feature calculation: Calculate the perimeter, area, volume, and other parameters of the defect area based on the fused second-order moment model. For example, in multi-layer chip defects, the severity of the void defect can be quantified through three-dimensional volume calculation.

[0295] 6. Decision-level integration and application:

[0296] Defect diagnosis network: The fused features are input into the improved U-Net model (integrating residual connections and attention mechanisms) to achieve pixel-level segmentation of defect areas. For example, in lithium battery electrode defect detection, the model can accurately segment micron-level coating peeling areas.

[0297] 1. Process improvement decision:

[0298] Welding process optimization: Combining multimodal data (such as the three-dimensional morphology of X-ray welds and the stress distribution of ultrasonic welding), finite element simulation was used to determine the optimal laser power (error ±5W) and welding speed (error ±0.1mm / s), reducing the cold weld rate to below 0.3%.

[0299] Material durability assessment: Through accelerated aging tests (high temperature 85°C / high humidity 85% RH) combined with CNN image detection, a correlation model between the aging degree of metal film resistors and resistance value drift was established, providing data support for material selection.

[0300] 2. Summary of technological innovation:

[0301] Breakthrough in cross-modal fusion efficiency: Compared with traditional CNN, the Transformer strategy reduces feature splicing loss by 40% in multimodal feature association modeling, making it suitable for deep semantic understanding of complex defects.

[0302] Improved hybrid modeling accuracy: The discrete-continuous second-order moment fusion model solves the limitations of traditional single modeling methods in describing irregular defects, making the three-dimensional defect volume measurement error ≤5%.

[0303] Decision-level application closed loop: The full process from data collection to process optimization is integrated to achieve a complete closed loop of "detection-tracking-tracing-improvement" for electronic product defects. Compared with traditional single-modality detection solutions, the process improvement efficiency is more than doubled.

[0304] According to an embodiment of the present invention, an embodiment of a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0305] In this embodiment, a multimodal CNN / Transformer image defect diagnosis and tracking decision method is provided, which can be used in computer equipment, such as Figure 3 : is a flowchart of a multimodal CNN / Transformer image defect diagnosis and tracking decision method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0306] Step S201, obtain multiple defect location images of multimodal data of the product to be diagnosed, where the multiple defect location images are defect location images containing the above multimodal data taken from different production batches, different process stages, and different equipment; the multimodal data includes image data, electrical data, and working condition data.

[0307] Specifically, the products to be diagnosed include but are not limited to smart meters, metering automation terminals, mutual inductors, batteries, transformers and wires, power electronic components, multi-layer chips, transistors, mutual inductors and PCB boards, component welding connections and processes, existing welding structures and semiconductor packages such as SOP, QFP, BGA, CSP, IGBT, industrial devices, internal discontinuities in batteries (microcracks, porosity, etc.), cracks, voids, delamination, deformation, separation, bridging, open circuits in metal layers (wires, cores), fractures, dust, foreign matter and impurities, metal particles, looseness, dislocation, and damage to the insulation layer. Images of defect locations taken by different production batches, different process stages, and different equipment of the products to be diagnosed are obtained, and the label data corresponding to each image is collected, such as whether the product has a defect and the type of defect.

[0308] Step S202 : performing data preprocessing and data alignment on the defect position images of the plurality of multimodal data.

[0309] Specifically, the above step S202 includes:

[0310] Step S2021: Preprocess the defect location image in the following steps: data cleaning, data labeling, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization.

[0311] Among them, data cleaning is used to remove noise data, blurred defect location images and incomplete defect location images in defect location images; data labeling is to use a combination of manual labeling and semi-automatic labeling to label the defect location and defect type; image usage classification is to divide multimodal data into image data, electrical data and working condition data according to the production batch, process stage and equipment type to which the image belongs and the multimodal data type; image pixel normalization is to normalize the pixel values ​​of image data to a preset range to unify the scale of image data.

[0312] Image pixel normalization refers to normalizing image pixel values ​​to a specific range, such as [0, 1] or [-1, 1]. This helps speed up the training of subsequent convolutional neural network models and improve stability. For a certain pixel value x, the following formula can be used for normalization:

[0313]

[0314] Among them, x min and x max are the minimum and maximum values ​​of this pixel among all pixels respectively.

[0315] In step S2022, the image data, electrical data, and operating condition data are aligned, fused, and standardized according to timestamps to construct a temporally and spatially consistent data set.

[0316] Specifically, data from different modal sources, such as X-ray 3D structural information, thermal infrared radiation information, and ultrasonic internal structure images, are spatiotemporally aligned based on acquisition timestamps to ensure consistency across the time dimension. For example, 3D CT images of the same product taken within 30 minutes of welding can be aligned with thermal imaging data to avoid defect location errors caused by time misalignment.

[0317] Step S203, based on the defect location image of each image type data, feature extraction is performed on the defect location images of multiple multimodal data using a position encoding method to obtain the image features after position encoding of each defect location image. At the same time, feature extraction is performed on the electrical data and the working condition data respectively to obtain electrical features and working condition features; the image features include contrast features, correlation features, energy features, entropy features, defect entropy value features and position texture parameter features.

[0318] Specifically, the operating condition data includes load fluctuation and ambient temperature. The above step S203 includes:

[0319] Step S2031: Use a CNN shallow network to extract basic edge features and texture features of the defect location image.

[0320] Specifically, a CNN shallow network is used to extract basic image features, including: extracting edge / texture features: ResNet-18 is used to extract edge features and texture features.

[0321] Step S2032, determine the defect center position of each defect position image based on edge features and texture features; obtain the two-dimensional coordinate value of each defect center position, and discretize each two-dimensional coordinate value; convert each coordinate value after discretization into a feature vector through one-hot encoding, and use the feature vector as the image feature after the position encoding of each defect position image.

[0322] Specifically, a drawing tool such as CAD software is used to determine the defect center position of each defect position image and obtain the two-dimensional coordinate value of each defect center position. The two-dimensional coordinate value is expressed as (x d ,y d ) is expressed, and then the discrete coordinate method is used to calculate the two-dimensional coordinate value (x d ,y d ) for discretization.

[0323] Each coordinate value after discretization is converted into a vector form through One-Hot Encoding. For example, the defect location image is divided into grid areas, and one-hot encoding is performed according to the grid area where the defect center is located to obtain a vector of length m. 2The vector is used as the feature set after position encoding.

[0324] Step S2033: Convert the electrical data into a time series and extract frequency domain features as electrical features through Fourier transform; perform 5-minute sliding window sampling on the real-time data of load fluctuation and ambient temperature in the working condition data to extract working condition features.

[0325] In step S204, based on a preset feature weight distribution scheme, each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and position texture parameter feature is subjected to feature weight fusion as an image feature, and the image feature, electrical feature, and operating condition feature are spliced ​​into a unified vector to obtain multiple multimodal fusion feature vectors.

[0326] Specifically, the above step S204 includes:

[0327] In step S2041, the feature weight distribution scheme is set in the following manner: the contrast feature weight distribution is 20% to 30%, the correlation feature weight distribution is 10% to 25%, the energy feature weight distribution is 10% to 25%, the entropy feature weight distribution is 30% to 40%, the defect entropy value feature weight distribution is 10% to 20%, and the position texture parameter feature weight distribution is 10% to 30%, and the sum of the contrast feature weight, the correlation feature weight, the energy feature weight, the entropy feature weight, the defect entropy value feature weight and the position texture parameter feature weight is equal to 1.

[0328] Furthermore, let the input image of the product to be diagnosed be I d , the smart meter image is I B , the contrast feature is recorded as C(I d ), the correlation characteristic is recorded as R(I d ), the energy characteristic is recorded as E(I d ), the entropy characteristic is recorded as H(I d ), the defect entropy characteristic is recorded as D(I d ), the position texture parameter feature is recorded as P(I d ).

[0329] Contrast feature C(I B ): If the contrast C(I B ) has a strong ability to distinguish normal and defective areas. For example, when the meter display screen is blurred or has other defects, the contrast feature C(I B ) can clearly show the difference between the defect area and the surrounding normal area, then the contrast feature C(I B ) a relatively high initial weight range, such as 20% to 30%. For some minor defects (such as fine scratches), if the contrast feature C(I B) is not very significant, a lower value is taken within the weight range of 20% to 30%.

[0330] For the correlation feature R(I B ): When the correlation feature R(I B ) When the pixel correlation around the defect area is significantly reduced (e.g., the defect causes the destruction of texture continuity), a weight range of 15% to 25% can be assigned. B ) If the change is not significant for most defect types, the weight range is appropriately lowered to 10% to 20%.

[0331] For the energy characteristic E(I B ): If the energy E(I B ) presents a specific change pattern when a defect occurs (for example, a certain defect always causes the energy of a specific frequency component to increase or decrease), and is given a weight of 15% to 25%. If the energy feature E(I B ) changes are more complex and have no obvious pattern, and the weight can be set to 10% to 20%.

[0332] For the entropy feature H(I B ): Since the entropy characteristic is recorded as H(I B ) is related to the uncertainty of image information. For complex textures or mixed defects, if the entropy feature is recorded as H(I B ) can reflect these situations well and can be assigned a weight of 30% to 40%. If the entropy feature is recorded as H(I B ) If the performance is unstable in some simple defect scenarios, it should be appropriately adjusted within the weight range of 30% to 40%.

[0333] For the defect entropy feature D(I B If defect entropy is a key factor in distinguishing defects of varying severity (for example, a defect with a high entropy value may indicate a more serious fault in the smart meter's internal circuitry), assign a separate weight to the defect entropy value. A 10% to 20% weight can be allocated to adjust accuracy based on the defect entropy value. Defects with high entropy values ​​and a significant impact on meter functionality can be given a higher weight.

[0334] For the position texture parameter feature P(I B ): When the defect is located in the key part of the smart meter (such as the display screen, around the metering chip, etc.), its weight should be higher than the defect in the non-critical part. According to the importance of the position, the weight of the position factor can be allocated between 10% and 30%. At the same time, for areas with obvious texture features (such as the specific pattern of the meter shell), if the texture change is important for defect detection, the texture related parameters (such as the correlation feature R (I B ), entropy characteristics H(I B) to highlight the impact of texture changes on accuracy.

[0335] Detect different types of defects (such as scratches, wear, display anomalies, etc.), and then determine the preliminary weight range based on the different types of defects.

[0336] The preset feature weight allocation scheme is set according to the actual situation. For example, the input defect location image of the product to be diagnosed is a high-precision chip defect location image, which is recorded as I d , then its weight distribution scheme is: contrast feature C(I d ) weight is 0.25, and the correlation feature R(I d ) weight is 0.15, energy feature E(I d ) weight is 0.12, and the entropy feature H(I d ) weight is 0.20, the defect entropy feature D(I d ) weight is 0.25, position texture parameter feature P(I d ) weight is 0.03. In the feature fusion stage, the feature weight fusion is performed according to the given weight distribution scheme, and the feature vector after feature weight fusion is F(I d ).

[0337] Step S2042, respectively calculate the first product corresponding to each contrast feature and the contrast feature weight, the second product corresponding to each correlation feature and the correlation feature weight, the third product corresponding to each energy feature and the energy feature weight, the fourth product corresponding to each entropy feature and the entropy feature weight, the fifth product corresponding to each defect entropy value feature and the defect entropy value feature weight, and the sixth product corresponding to each position texture parameter feature and the position texture parameter feature weight; add each corresponding first product, second product, third product, fourth product, fifth product and sixth product to obtain multiple multimodal fusion feature vectors.

[0338] For smart meter images, in the feature fusion stage, the fusion is performed according to a given weight distribution scheme, and the fused feature vector is set as F(I B ),but:

[0339] F(I B )=(20% to 30%)×C(I B )+(10% to 25%)×R(I B )+E(I d )×(15% to 25)+

[0340] H(I B )×(30% to 40%)+D(I B )×(10% to 20%)+P(I B)×(10% to 30%))(2);

[0341] The principle of weight distribution is: the sum of the weights of each feature must be equal to 1.

[0342] For example, when the defect image of the product to be diagnosed is a high-precision chip, the final weight distribution scheme is: contrast feature C(I d ) weight is 0.25, and the correlation feature R(I d ) weight is 0.15, energy feature E(I d ) weight is 0.12, and the entropy feature H(I d ) weight is 0.20, the defect entropy feature D(I d ) weight is 0.25, position texture parameter feature P(I d ) weight is 0.03. In the feature fusion stage, the feature weight fusion is performed according to the given weight distribution scheme, and the feature vector after feature weight fusion is F(I d ), the formula is as follows: F(I d )=0.25×C(I d )+0.15×R(I d )+E(I d )×0.12+H(I d )×0.2+D(I d )×0.25+P(I d )×0.03(3).

[0343] Step S2043: Early fusion of multiple multimodal fusion feature vectors by splicing electrical features and operating condition features into a unified vector.

[0344] The multimodal CNN / Transformer image defect diagnosis and tracking decision-making method provided in this embodiment assigns weights to multiple features, such as contrast, correlation, energy, entropy, and defect entropy value, before fusing them together, fully leveraging the role of each feature in defect detection. Weights are assigned based on the effectiveness of different features in distinguishing different types of defects, making the fused feature vector more discriminative and meeting diagnostic requirements. This approach considers the characteristics of different image types and the performance of different features for different defect types, making the fusion solution more targeted and better suited to specific detection tasks, meeting the need for high adaptability.

[0345] In step S205, the convolutional neural network model is sequentially trained with independent sub-models and evaluated based on multiple multimodal fusion feature vectors to obtain a single-process image defect detection model, and the Transformer network model is sequentially trained with model evaluation based on multiple multimodal fusion feature vectors to obtain a multi-process image defect detection model.

[0346] Specifically, the convolutional neural network model includes an input layer, multiple convolutional layers and pooling layers, multiple fully connected layers and an output layer. The convolutional neural network model (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. The convolutional neural network model mainly analyzes visual images by extracting and identifying image features. After the convolutional neural network model is trained and evaluated in sequence through multiple multimodal fusion feature vectors, the convolutional neural network model has the function of image defect detection, thereby obtaining an image defect detection model.

[0347] The above step S205 includes:

[0348] In step S2051, multiple multimodal fusion feature vectors are transmitted through the input layer to multiple convolutional layers and pooling layers for deep learning, wherein the multiple convolutional layers use convolution kernels to perform convolution operations on the multimodal fusion feature vectors to extract local features, and the pooling layer obtains multiple key features through downsampling operations; multiple fully connected layers are used to integrate the multiple key features, and the extracted features are converted from local information to global information, and mapped to the output probability distribution through the activation function; the output layer is used to convert the probability distribution mapped to the output into multiple fused probability distribution images and output them, and the value of each pixel in the fused probability distribution image represents the probability of a defect at that position.

[0349] Specifically, if Figure 5 As shown, the input layer receives the multimodal fusion feature vector The number of nodes is equal to The number of dimensions.

[0350] Convolution layer and pooling layer: Set up multiple convolution layers and pooling layers to automatically extract high-level information from features, that is, key features. This embodiment sets three convolution layers, and the convolution kernel sizes are 3×3, 3×3, and 5×5, respectively, with a step size of 1. Each convolution layer is followed by a pooling layer, and the pooling kernel size is 2×2, with a step size of 2. The activation function of the convolution layer uses the ReLU function (Rectified Linear Unit, linear rectification function), which is defined as: f(x) = max(0, x). In this formula, if x is less than or equal to 0, the function value f(x) is 0, and if x is greater than 0, the function value f(x) is x. Such an activation function can increase the nonlinear expression ability of the model, and x represents the feature vector Every feature.

[0351] Fully connected layer: After the key features are extracted by the convolution layer and the pooling layer, several fully connected layers are set to further integrate the information and map it to the output probability distribution. In this embodiment, two fully connected layers are set. The number of nodes in the first fully connected layer is k (the appropriate value can be determined through experiments, such as 128), and the number of nodes in the second fully connected layer is n, corresponding to the number of elements P of the probability distribution of the output image type of the neural network model = [p0, p1…p n ], where n is the number of image types, P i Indicates the probability that the image belongs to the i-th type. The type with the highest probability is the predicted image type. In order to alleviate the gradient disappearance and improve computational efficiency, neuron death and prevent model overfitting in multimodal and high-resolution images, and filter out negative signals, the activation function of the fully connected layer can use the RReLU function, f(x) = max: (ax, x), where a is a learnable, adaptive, and automatically adjusted slope parameter and is averaged after training. x represents the feature vector Every feature.

[0352] The output layer uses the Softmax function (an activation function commonly used in multi-class classification problems) as the activation function to convert the output of the fully connected layer into a form representing probability. For the output Z of the i-th neuron in the output layer i , the output after processing by the Softmax function is: This ensures that the sum of all output probability values ​​is 1, which meets the requirements of probability distribution.

[0353] Where: P i =(i, 1) represents the probability value output by the i-th neuron after processing by the Softmax function. This probability value represents the likelihood that the input sample belongs to the i-th category. For example, in an image classification task with 10 categories, when i = 3, it means the probability that the image belongs to category 3.

[0354] z j It represents the raw output value of the jth neuron in the fully connected layer before the Softmax function is input. When calculating the jth output probability, all (j ranges from 1 to n) need to be calculated. These raw output values ​​contain information about the model's judgment of each category, but they are not in the form of probabilities. n represents the total number of categories. In classification tasks, the total number of categories is fixed and represents the number of all possible categories. For example, for a five-category problem, n = 5, which means that when calculating the probability of each category, the raw output values ​​of all five categories (j = 1, 2, 3, 4, 5) need to be considered.

[0355] Initialization parameters: Initialize the convolution kernel, bias, and other parameters in the network. Common initialization methods include random initialization (such as Xavier initialization (neural network weight initialization) and He initialization (initialization of the ReLU activation function). These initialization methods can help the model converge faster.

[0356] In step S2052, multiple fused probability distribution images are divided into a training set, a validation set, and a test set according to a preset ratio; the training set is input into the convolutional neural network model to perform independent sub-model training in sequence, and the outputs of each independent sub-model are weightedly fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and an evaluation result is output. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as a single process image defect detection model.

[0357] In an optional implementation, the above step S2052 includes:

[0358] The training set, validation set, and test set are divided into 7:2:1 or 8:1:1 ratios.

[0359] The unified vector of defect features of the multimodal fusion early fusion image is input into the convolutional neural network for later fusion; the convolutional neural network model is trained to specifically process image data to extract defect features in the image; the long short-term memory network model is trained to process time-series electrical data to capture the law of change of electrical data over time; the multi-layer perceptron model is trained to process operating condition data to analyze the impact of operating conditions on defects; the outputs of each independent sub-model obtained from the above training are weightedly fused to obtain the detection model to be evaluated, and the test set is input into the detection model to be evaluated. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model, and the evaluation result is used as the final defect diagnosis result.

[0360] Step S2053, during the training process, the loss is calculated based on the preset loss function, and the Adam optimizer is used for back propagation to adjust the weights and biases of the convolutional neural network model, and the parameters of multiple convolutional layers, pooling layers, and multiple fully connected layers are adjusted based on the verification set to obtain the convolutional neural network model to be evaluated; the Adam optimizer combines the idea of ​​momentum method and the idea of ​​Adagrad optimization algorithm, dynamically adjusts the learning rate of each parameter in the model based on the first-order moment estimator and the second-order moment estimator of the gradient, and corrects the first-order moment estimator and the second-order moment estimator through the deviation.

[0361] Specifically, the preset loss function uses the cross-entropy loss function to measure the difference between the probability distribution predicted by the model and the actual probability distribution. Its expression is:

[0362]

[0363] Among them, y i is the element in the actual probability distribution p, that is, the actual label. In the classification problem, if it is in the form of one-hot encoding, for example, for a three-category problem, the category label is [0,0,1], indicating that it belongs to the third category, then y i It is 1 when i=3 and 0 at other positions. It represents the probability corresponding to the category to which the sample actually belongs (true probability distribution), and i is the sum from 1 to j, which is used to calculate the cross entropy loss.

[0364] Among them, p i , predicted represent the true probability value and the probability value predicted by the model respectively. For example, the model predicts that the probability of a sample belonging to the first category is 0.2, the probability of belonging to the second category is 0.3, and the probability of belonging to the third category is 0.5. These probability values ​​are p i , predicted(i=1,2,3) is the output after the convolutional neural network and the Softmax function, which is used to compare with the probability corresponding to the actual label to measure the quality of the model prediction.

[0365] After the Softmax function and minimizing the cross entropy loss, the convolutional neural network model can effectively learn the correct classification boundaries.

[0366] The training process is as follows: input the training set data into the constructed single process image defect detection model, calculate the output through forward propagation, calculate the loss according to the loss function, and then use the optimizer to back propagate and update the network parameters. During the training process, the model parameters are saved regularly, and the model performance is evaluated on the validation set, and the training parameters (such as learning rate, number of training rounds, etc.) are adjusted according to the validation results. In this embodiment, the convolutional neural network model is trained using the training set data, and the weights and biases of the model are adjusted by combining the back propagation algorithm with the optimization algorithm and the Adam optimizer (improved stochastic gradient descent method) so that the value of the loss function is continuously reduced, that is, the difference between the probability distribution predicted by the model and the actual probability distribution is continuously reduced. During the training process, the parameters of the convolution layer, pooling layer, and fully connected layer of the model (such as convolution kernel size, step size, number of nodes, etc.) can be adjusted according to the training effect (such as the accuracy rate, loss value, etc. on the validation set) to improve the performance of the model.

[0367] A detailed description of the Adam optimizer (improved stochastic gradient descent method):

[0368] The Adam optimizer maintains two moving average estimators, namely the first-order moment estimator m and the second-order moment estimator v.

[0369] First, at each iteration t, a weight of the convolutional layer with parameter θ is The gradient calculation formula is as follows:

[0370]

[0371] Among them, g t is the gradient, θ is the parameter, It usually represents the gradient of the loss function L with respect to the parameter θ (here it is some kind of averaged gradient, or the average gradient over a small batch of data or other forms of gradient processing). The loss function L measures the difference between the model's prediction results and the true label. When training machine learning models such as neural networks, our goal is to minimize this loss function. It represents the rate of change of the loss function in the parameter space, which indicates how the parameters should be adjusted to reduce the loss function.

[0372] Calculate the gradient First calculate the gradient g of the loss function L with respect to the parameter θ (here specifically a weight w of the convolutional layer) t This gradient calculation is based on the data of the current batch, which points out the direction in which the loss function grows fastest at the current position of the parameter.

[0373] Update the first-order moment estimator (momentum term):

[0374] m t =β t m t-1 +(1-β1)g t (6);

[0375] Among them, β1 is a hyperparameter, which is set to 0.9, β t is a parameter used to control the previous momentum information m t-1 The weight parameter when updating the current momentum. t When it is close to 1, it means that m is being updated. t When , it depends more on the previous momentum m t-1 ; When β t When it is close to 0, the current gradient information (1-β1)g will be more important. t .

[0376] In the given formula, the hyperparameter is defined only as β1 = 0.9, where β t It is a parameter similar to β1, and in practical application scenarios (such as optimization algorithms such as Adam, etc.), β t It is usually a number in the interval (0,1) to balance the contribution of historical information and current information to the momentum term update. If this is in a specific algorithm iteration process, βt It may also change with the number of iterations. The specific change rules need to be determined according to the detailed settings of the algorithm. Formula (6) g t Used to update the first-order moment estimator m t . m t It can be regarded as the gradient g t A weighted moving average of m0 that accumulates information about past gradients, similar to momentum with friction. This momentum term helps accelerate convergence, especially when dealing with parameter updates with similar gradient directions. Initially, m0 is usually set to 0.

[0377] Update the second-order moment estimator:

[0378]

[0379] Among them, β1 is a hyperparameter and is set to 0.999.

[0380] The above first-order moment estimator and second-order moment estimator are bias-corrected as follows:

[0381]

[0382] Finally, update the parameters, the formula is as follows:

[0383]

[0384] Among them, η is the learning rate, and ∈ is a very small number to prevent the denominator from being zero.

[0385] The improvement of the Adam optimizer combines the momentum method (through the first-order moment estimator m) and the idea of ​​Adagrad (through the second-order moment estimator v). It can dynamically adjust the learning rate of each parameter in the model according to the first-order moment and second-order moment of the gradient, and through deviation correction, it can achieve better learning effects in the early stages of training.

[0386] In step S2054, the image defect fusion path of the multimodal fusion Transformer is early fused, and the self-attention mechanism is used to effectively capture the semantic association between different modal features through this mechanism, and the multimodal features are directly cross-modally fused early; the cross-attention mechanism is used to parallelly process the local features extracted by the multi-channel convolutional neural network from different sources to fuse the complex defect feature detection model, and the complex defect feature detection model and the Transformer processing time-series correlation detection model are combined to form a combined detection model, and the image to be diagnosed is input into the combined detection model to output the final defect diagnosis result.

[0387] For example, the obtained image defect detection model is deployed to the production and application environment, and the image of the product to be diagnosed taken by the detection equipment is input into the image defect detection model to realize real-time detection. Taking the binary classification to determine whether there is a defect as an example, the detection result of whether the corresponding area of ​​the image has a defect or not is obtained.

[0388] Use the test set for evaluation. That is, after training is completed, use the test set to evaluate the trained model to be evaluated, calculate evaluation indicators such as accuracy, precision, recall, average intersection-over-union, and F1-score, and evaluate the performance of the model on unseen data.

[0389] Taking the accuracy performance evaluation indicators of the product image to be diagnosed as an example, the calculation formulas for evaluation indicators such as accuracy, precision, recall rate, and F1-score are as follows:

[0390] Accuracy:

[0391]

[0392] Among them, TP (True Positive) represents the number of samples correctly detected as positive (such as defective chip images correctly detected); TN (True Negative) represents the number of samples correctly detected as negative (such as non-defective chip images correctly detected); FP (False Positive) represents the number of samples incorrectly detected as positive (actually non-defective but detected as defective); FN (False Negative) represents the number of samples incorrectly detected as negative (actually defective but detected as non-defective).

[0393] Precision:

[0394]

[0395] The precision rate measures the proportion of positive samples detected as true positive samples, reflecting the reliability of the detection results.

[0396] Recall:

[0397]

[0398] The recall rate indicates the proportion of actual positive samples that are correctly detected, reflecting the model's ability to capture positive samples.

[0399] Mean Intersection over Union (mIoU):

[0400]

[0401] Assume that k categories (e.g., different types of defects or normal areas are considered different categories) are detected in the product image to be diagnosed. Intersection(i) represents the intersection area of ​​the predicted result and the true label in category i, and Union(i) represents the union area of ​​the predicted result and the true label in category i.

[0402] Step S206: Obtain multiple images of the product to be diagnosed that are actually taken and input into the single-process image defect detection model and / or the multi-process image defect detection model to perform image defect diagnosis, and obtain a detection result indicating whether the corresponding area of ​​the product image has defects or not. Figure 1 Step S106 of the illustrated embodiment will not be described in detail here.

[0403] Step S207: Perform hybrid modeling and analysis on the defective product image output by the image defect detection model input to be diagnosed to obtain hybrid modeled image defects. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis. Figure 1 Step S107 of the illustrated embodiment will not be described in detail here.

[0404] Step S208: The hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, achieve defect area segmentation and positioning, and then track the position of the image defect area. Based on the defect tracking and fault tracing results, decisions are made to optimize production process parameters. For details, please refer to Figure 1 Step S108 of the illustrated embodiment will not be described in detail here.

[0405] The multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method provided in this embodiment determines the encoded feature set through position encoding, and uses a preset feature weight distribution scheme to fuse the feature weights of each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature in the feature set to obtain multiple multimodal fusion feature vectors; based on the multiple multimodal fusion feature vectors combined with a convolutional neural network model, an image defect detection model is obtained. The image defect detection model is applied to actual scenarios to obtain a detection result indicating whether the corresponding area of ​​the image has defects or no defects, thereby achieving accurate detection and positioning of minor component changes and structural defects within the product, improving detection accuracy, and performing image defect area shape measurement calculation on images of defective products to be diagnosed, so as to diagnose the severity level of the image defects and track the location of the image defect areas, thereby achieving diagnosis and tracking of the image defect locations and solving the problem of being unable to accurately detect, locate, diagnose, track and make decisions on minor component changes and structural defects within electronic products.

[0406] In this embodiment, a multimodal CNN / Transformer image defect diagnosis and tracking decision method is provided, which can be used in computer terminals such as central processing units, servers, etc. Figure 4 FIG. 1 is a flowchart of a multimodal CNN / Transformer image defect diagnosis and tracking decision method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0407] Step S301: Acquire multiple multimodal defect location images of the product to be diagnosed. The multiple defect location images are defect location images containing the above multimodal data taken from different production batches, different process stages, and different equipment. The multimodal data includes image data, electrical data, and working condition data. For details, please refer to Figure 3 Step S201 of the illustrated embodiment will not be described in detail here.

[0408] Step S302: perform data preprocessing and data alignment on the defect location images of multiple multimodal data. Figure 3 Step S202 of the illustrated embodiment will not be described in detail here.

[0409] Step S303: Based on the defect location image of each image data, feature extraction is performed on the defect location images of multiple multimodal data using a position encoding method to obtain the image features of each defect location image after position encoding. At the same time, feature extraction is performed on the electrical data and the working condition data respectively to obtain electrical features and working condition features. Image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features. For details, please refer to Figure 3 Step S203 of the illustrated embodiment will not be described in detail here.

[0410] Step S304: Based on the preset feature weight allocation scheme, each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature, and position texture parameter feature is subjected to feature weight fusion as image features, and the image features, electrical features, and operating condition features are spliced ​​into a unified vector to obtain multiple multimodal fusion feature vectors. For details, please refer to Figure 3 Step S204 of the illustrated embodiment will not be described in detail here.

[0411] In step S305, the convolutional neural network model is trained and evaluated based on multiple multimodal fusion feature vectors to obtain a single-process image defect detection model, and the Transformer network model is trained and evaluated based on multiple multimodal fusion feature vectors to obtain a multi-process image defect detection model. Figure 3 Step S205 of the illustrated embodiment will not be described in detail here.

[0412] In step S306, visualization tools are used to visualize the changes in loss and evaluation results during the training process of the convolutional neural network model, the key features deeply learned by multiple convolutional layers and pooling layers, and the feature integration and mapping of multiple fully connected layers to the probability distribution of the output.

[0413] Specifically, visualization tools (such as TensorBoard) can be used to analyze curves such as loss and accuracy during model training to gain an intuitive understanding of the model's training progress. Feature maps learned by the convolutional layer can also be visualized to verify whether the model has truly learned valuable features in the chip (smart meter) image.

[0414] Step S307 , obtain multiple images of the product to be diagnosed that are actually taken and input them into a single-process image defect detection model and / or a multi-process image defect detection model for image defect diagnosis, and obtain a detection result of whether the corresponding area of ​​the product image has defects or no defects.

[0415] Specifically, deploy to the production and application environment: deploy the trained and evaluated model to the actual product inspection production line and application, and input the actual product images to be diagnosed taken by the inspection equipment into the tested image defect detection model to achieve real-time inspection, measurement, diagnosis, and tracking decisions.

[0416] The above step S307 includes:

[0417] In step a, image defect diagnosis is performed on multiple product images to be diagnosed that are input into the image defect detection model using a decision function. The decision function is expressed by the following formula:

[0418]

[0419] Here, G(F)=1 indicates that a defect is detected in the corresponding area of ​​the image, G(F)=0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that a preset defect condition is met.

[0420] Specifically, the value of G(F) is a binary classification value. The preset defect condition can be a judgment rule such as the value of F being greater than a certain threshold. The threshold can be determined through training or other methods. In other words, the preset defect condition can be a judgment rule such as the value of F being greater than a certain threshold. This is a variable related to the relevant features and parameters of the image of the product to be diagnosed. Its reference needs to be further clarified in conjunction with more specific background information about the image defect detection model and related settings.

[0421] The multimodal fusion feature vector of the input product image to be diagnosed is matched with the defect feature threshold set during model training. That is, when the feature parameters such as contrast, energy, and entropy in the feature vector exceed the preset threshold, it is judged that there is a defect, otherwise it is judged as no defect.

[0422] Step S308 , hybrid modeling and analysis are performed on the defective product image output by the image defect detection model input to be diagnosed to obtain hybrid modeled image defects, wherein the hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0423] Specifically, a semantic segmentation model based on a CNN deep learning-based improved U-Net model is used to segment the image defect area, generating a binary image of the defect area. Contours are extracted from the binary image using image processing library functions to obtain the defect area contour. The geometric features of the defect area contour are calculated to determine the preliminary defect area shape; these geometric features include the contour perimeter, the contour convex hull perimeter, and the shape factor.

[0424] The above step S308 includes:

[0425] Step S3081, calculate the geometric features of the image defect area contour and determine the preliminary image defect area shape; the geometric features of the preliminary image defect area shape include contour perimeter, contour convex hull perimeter and shape factor.

[0426] Specifically, the first step is to accurately segment the defective area from the image. This can be achieved by using a semantic segmentation model based on a modified U-Net model based on CNN deep learning. The modified U-Net model has excellent segmentation performance and is particularly suitable for multi-task and multimodal segmentation of complex images such as welds, crystal oscillators, chips, IGBTs, GBAs (Game Boy Advance, the core processor chip), and the internal details of artificial intelligence. To establish a coordinate system, a suitable coordinate system xy must be established. Assuming the imaging plane is used as a reference, let the point within the defective area of ​​the image be (x, y), and the corresponding grayscale value be I(x, y).

[0427] Step S3082 : Based on the preliminary defect area shape of the image, a shape hybrid model is performed using the second-order moment of discrete data and the second-order moment of continuous function to obtain a mathematical model for measuring the shape of the defect area of ​​the image.

[0428] In some optional embodiments, the semantic segmentation model of the improved U-Net model based on CNN deep learning introduces residual connections in the encoder (downsampling path) and introduces an attention mechanism in the decoder (upsampling path), that is, the semantic segmentation model of the improved U-Net model based on CNN deep learning includes an encoder, a decoder and skip connections, the encoder includes residual connections, and the decoder includes an attention mechanism.

[0429] The above step S3082 includes:

[0430] Determine the discrete data second-order moment and continuous function second-order moment of the image defect area based on the shape of the preliminary defect area of ​​the image;

[0431] The discrete data second-order moment and the continuous function second-order moment of the image defect area are fused and modeled to obtain the mathematical model for shape measurement of the image defect area.

[0432] In an optional embodiment, the discrete data second-order moment of the image defect area and the continuous function second-order moment are fused and modeled to obtain a mathematical model for shape measurement of the image defect area, including:

[0433] The first moment of inertia is calculated based on the second-order moment of discrete data, and the first equivalent ellipse parameters are calculated based on the first moment of inertia. The second moment of inertia is calculated based on the second-order moment of the continuous function, and the second equivalent ellipse parameters are calculated based on the second moment of inertia. The first and second moments of inertia, as well as the first and second equivalent ellipse parameters, are fused using a weighted average method to obtain a mathematical model for measuring the shape of the image defect area.

[0434] Furthermore, the weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and the first equivalent ellipse parameters and the second equivalent ellipse parameters, respectively, to obtain a mathematical model for measuring the shape of the image defect area, including:

[0435] Set the weight of the second-order moment of the continuous function, and calculate the weight of the second-order moment of the discrete function based on the weight of the second-order moment of the continuous function;

[0436] Based on the second-order moment weight of the continuous function and the second-order moment weight of the discrete function, the first moment of inertia and the second moment of inertia are fused by the weighted average method to obtain the fused second-order moment; the first equivalent ellipse parameters and the second equivalent ellipse parameters are fused to obtain the fused equivalent ellipse parameters;

[0437] The mathematical model for measuring the shape of the image defect area is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters;

[0438] The fused second-order moment formula is as follows:

[0439] M fusion =aM c +(1-a)M d ;

[0440] Among them, M c is the second-order moment of a continuous function, M dis the second-order moment of discrete data; a (0≤a≤1) is a weight coefficient used to measure the proportion of the second-order moment of the continuous function in the fused second-order moment; when a=0, it means that when calculating the fused second-order moment, the contribution of the second-order moment of the continuous function is completely ignored, and only the second-order moment of the discrete data is used; when a=1, it completely relies on the second-order moment of the continuous function, and does not consider the second-order moment of the discrete data.

[0441] Step S3083 , measuring the degree of the defect in the image defect area based on the image defect area shape measurement mathematical model, and obtaining the final shape, width, length, area and volume of the image defect area.

[0442] The final shape, width, length, area, and volume of the image defect area are calculated based on the major axis, minor axis, and orientation angle.

[0443] The following is an example of a common method for calculating the parameters related to the image defect area by fusing the second-order moment of the continuous function and the second-order moment of discrete data to obtain the major axis, minor axis and direction angle (major axis a, minor axis b, direction angle Э) based on the weighted average method. d )as follows:

[0444] Final shape: The approximate elliptical shape of the defect area can be roughly determined based on the ratio of the major axis to the minor axis and the direction angle (in the case of an elliptical-like defect). For example, when the major axis is significantly longer than the minor axis, it appears as a narrow and long ellipse. The direction angle determines the inclination angle direction within the plane, together outlining the final shape characteristics on the two-dimensional plane.

[0445] Width: The short axis length b is usually regarded as the width of the defect area, which represents the size of the defect in the direction perpendicular to the long axis.

[0446] Length: The major axis length a is considered to be the length of the defect area, that is, the distance range spanned by the defect in its main extension direction.

[0447] Area: For defect areas that are approximately elliptical in shape, the area S is calculated as S = πaab. The area occupied by the defect on a two-dimensional plane is obtained by multiplying the values ​​of the major axis and the minor axis and then multiplying them by pi.

[0448] Volume: If the three-dimensional case is to be considered (assuming that the defect also extends in a certain thickness direction, and the thickness is set to h), the volume V can be simply approximated according to the volume of an elliptical cylinder (when the defect morphology meets such characteristics). The calculation formula is V = πabh, that is, the volume size is obtained by multiplying the two-dimensional area calculated above by the dimension in the thickness direction.

[0449] In practical applications, these calculation methods can be appropriately adjusted based on factors such as the specific physical characteristics of the defects and the degree of shape conformity, or more practical professional models can be used to measure relevant parameters.

[0450] Step S3084: diagnose and grade the severity of the image defect based on the final shape, width, length, area, and volume of the image defect area.

[0451] The geometric features of the image defect area contour are calculated to determine the preliminary shape of the image defect area; the geometric features of the preliminary shape of the image defect area include contour perimeter, contour convex hull perimeter and shape factor.

[0452] Specifically, after obtaining the contour of the defect area, the shape of the defect can be described by calculating some geometric features of the contour. For defects in a two-dimensional image, the perimeter C of the contour can be calculated. p 、Convex hull perimeter C hc , shape factor and other indicators to describe the shape of image defects from different angles.

[0453] Among them, the perimeter C p calculate:

[0454] Perform contour extraction on the binary image of the image defect area, and set the extracted contour point set as N is the number of contour points, then the contour perimeter C p It can be obtained by calculating the sum of the Euclidean distances between adjacent contour points. The calculation formula is:

[0455]

[0456] Among them, x i+1 with x i are adjacent contour points, y i+1 with y i are adjacent contour points.

[0457] Convex hull perimeter C hc calculate:

[0458] First calculate the convex hull of the defect area contour, and set the convex hull point set as M is the number of convex hull points. The perimeter of the convex hull is C. hc It is also determined by calculating the sum of the Euclidean distances between adjacent convex hull points. The calculation formula is:

[0459]

[0460] Shape factors such as circularity are calculated as:

[0461]

[0462] Where A is the area of ​​the image defect region. The closer the circularity value is to 1, the closer the defect shape is to a circle. The smaller the circularity value is, the more irregular the shape is.

[0463] Calculate the moment characteristics of the defect area, such as the zero-order moment M0 (whose value is equal to the area of ​​the defect area), the first-order moment M1 (which can be used to determine the center of gravity of the defect area), and the second-order moment M2 (related to the shape of the defect area). These moment characteristics can also describe the defect shape to a certain extent. Calculate the second-order moment of the defect area, including the central moment.

[0464] The mean squared error (MSE) is used to evaluate the measurement task. The calculation formula is as follows:

[0465]

[0466] When measuring certain features in product images (such as component size, line width, etc.), is the true value, is the predicted value, and n is the number of measurement samples. The mean square error can be used to assess the degree of deviation between the measured value and the true value. The smaller the value, the higher the measurement accuracy.

[0467] Measurement step (taking the measurement of a target size as an example, let the measurement function be M:

[0468] M(F)=size value calue calculated on F(20);

[0469] That is, the size value of the target to be measured (such as the width, length, area, volume, etc. of the chip circuit (smart meter defects, faults)) is calculated based on the fused feature vector F.

[0470] In step S309, the hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, realize defect area segmentation and positioning, and then track the position of the image defect area, make decisions based on the defect tracking and fault tracing results, and optimize the production process parameters.

[0471] Specifically, the above step S309 includes:

[0472] In step S3091, the hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, and the defect area is segmented and located based on the defect detection results.

[0473] In an optional implementation, the above step S3091 includes:

[0474] After convolution processing on the image defect area in the encoder, the output feature map and the input feature map are obtained, and the output feature map and the input feature map are added through residual connection to obtain a new output feature map; the new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, which is then convolved to obtain a TFT ray map; the TFT ray map is input into the improved U-Net model for image area segmentation to obtain a binary image of the image defect area.

[0475] The semantic segmentation model of the improved U-Net model based on CNN deep learning introduces residual connections in the encoder (downsampling path) and introduces an attention mechanism in the decoder (upsampling path), that is, the semantic segmentation model of the improved U-Net model based on CNN deep learning includes an encoder, a decoder and skip connections, the encoder includes residual connections, and the decoder includes an attention mechanism.

[0476] Introduce residual connection in the encoder, and set the weight matrix of residual connection to be The output feature map after two convolutional layers is With input features Figure X i (x, y, z), add the residual connection to get the new output feature map The formula is as follows:

[0477]

[0478] The new output feature map is input into the attention mechanism of the decoder to obtain the weighted feature map, and then a convolution operation is performed to obtain the TFT ray map.

[0479] Specifically, the attention mechanism is introduced in the decoder, and the attention weight matrix is ​​set as Feature map after upsampling and concatenation by the encoder The weighted features are obtained through the attention mechanism That is the TFT ray diagram, the formula is as follows:

[0480]

[0481] Among them, (x, y, z) are image coordinates, a represents the relative position offset involved when the feature map performs a weighted operation using the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[0482] Then replace Perform subsequent convolution layer operations. And (C i-1 =C i+1 ), the number of channels may change after passing through the convolutional layer.

[0483] The TFT radiographic image is input into the trained improved U-Net model to obtain a binary image of the defect area, where the pixel value of the defect area is 1 and the pixel value of the background area is 0.

[0484] Specifically, the calculation process of discrete second-order moment is as follows:

[0485] Relationship between zero-order moment and area: Assume that the binary image function of the defect area is f(x, y), where the pixel value of the defect area is 1 and the pixel value of the background area is 0. The calculation formula of the zero-order moment M0 is:

[0486] M0=∑∑ f f(x, y) (23);

[0487] From the above formula, we can see that the value of the zero-order moment is equal to the area A of the defect area, that is, A=M0.

[0488] The first-order moment is represented by M1, and the first-order moment and the center of gravity position are calculated as follows:

[0489] The calculation formulas for the first-order moment M1x (about the x-axis) and M1y (about the y-axis) are:

[0490] M1x=∑ x ∑ y xf(x,y) (24);

[0491] M1y=∑ x ∑ y yf(x,y) (25);

[0492] Then the center of gravity position of the defect area (x g -y g ) can be calculated using the following formula:

[0493]

[0494] Relationship between second-order moment and defect shape:

[0495] The calculation formulas for the second-order moment M2xx (the second-order moment about the x-axis), M2yy (the second-order moment about the y-axis), and M2xy (the mixed second-order moment about the x-axis and y-axis) are:

[0496] M2xy=∑ x ∑ y xyf(x, y) (27);

[0497] M2xx=∑ x ∑ y x 2 f(x, y) (28);

[0498] M2yy=∑ x∑ y y 2 f(x, y) (29);

[0499] The second moment of a continuous function is calculated as follows:

[0500] M 00 =∫∫ D xf(x,y)dxdy(30);

[0501] For the continuous function z=f(x,y) defined on the defect location area D.

[0502] Zero-order moment M 00 calculate:

[0503] M 00 =∫∫ D f(x, y)dx dy (31);

[0504] The above formula is a double integral, and the integration region is the domain D of the function f(x, y).

[0505] First-order moment calculation:

[0506] M 10 =∫∫ D xf(x,y)dxdy(32);

[0507] M 01 =∫∫ D yf(x,y)dxdy (33);

[0508] The coordinates (x, y) of the center of gravity are given by and Sure.

[0509] The second-order central moment calculations are as follows:

[0510] M 2xy =∫∫ D xf(x,y)dxdy (34);

[0511] M 2xx =∫∫ D x 2 f(x, y)dx dy (35);

[0512] M 2yy =∫∫ D y 2 f(x,y)dxdy (36).

[0513] The calculation of these second-order central moments requires first finding the coordinates of the center of gravity and then completing it through double integration.

[0514] A hybrid second-order model is constructed, combining the second-order moments of continuous functions and discrete data, combining discrete data and continuous functions. For defect shape, it is preferred that the main portion of the defect be approximated by a continuous function. In this embodiment, a polynomial function or a combination of trigonometric functions is preferred. Furthermore, discrete data points are used to supplement details at key boundary locations or local irregularities.

[0515] In some optional implementations, this embodiment also uses a weighted average method to fuse the second-order moment of the continuous function and the second-order moment of discrete data, and the second-order moment is used to calculate the mathematical model of the major axis, minor axis and direction angle of the defect shape:

[0516] The required defect shape is represented by a continuous function z = f(x, y) on a two-dimensional plane, whose domain is Ω. Calculate the moment of inertia The formulas are as follows:

[0517]

[0518] M 2xx =∫∫ D x 2 f(x, y)dx dy (38);

[0519] M 2yy =∫∫ D y 2 f(x, y)dx dy (39);

[0520] In an optional embodiment, the above includes:

[0521] A first moment of inertia is calculated based on the second-order moment of the discrete data, and a first equivalent ellipse parameter is calculated based on the first moment of inertia.

[0522] Specifically, there are n discrete data points (x i -y i ), weight is w i (If the weights are the same, w i =1).

[0523] Calculating the moment of inertia and The formula is as follows:

[0524]

[0525] Let the long axis k corresponding to the discrete data part d 、Short axis b d and direction angle Э d . First calculate:

[0526]

[0527] The long axis short axis Direction

[0528] Among them, j d is an intermediate variable, which is obtained through the moment of inertia and Calculated. From the subsequent calculation of the major axis k d 、Short axis b d and direction angle Э d According to the formula, j d With k d The values ​​of are first obtained through a fixed calculation method (Formulas (43) and (44)), and then they are used to calculate the lengths of the major and minor axes of the equivalent ellipse.

[0529] j d With k d The calculation method is similar to combining the second-order moments of discrete data to more conveniently obtain the parameters (major axis, minor axis and direction angle) of the equivalent ellipse that can represent the distribution shape of discrete data.

[0530] A second moment of inertia is calculated based on the second-order moment of the continuous function, and a second equivalent ellipse parameter is calculated based on the second moment of inertia.

[0531] Suppose the continuous function is y=f(x), its domain is [a,b], and the discrete data points are (x i ,y i )i=1,2,…,n. Calculate the second moment of the continuous function part:

[0532]

[0533] in,

[0534] For the discrete data part, its second-order moment is:

[0535]

[0536] in, By adjusting the parameters of the continuous function and the positions of the discrete data points, M is minimized to optimize the description of the defect shape.

[0537] Let the major axis k corresponding to the continuous function part be c 、Short axis b c and direction angle Э c .

[0538] First calculate:

[0539]

[0540] Long axis short axis Direction

[0541] The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent ellipse parameters and the second equivalent ellipse parameters, to obtain the mathematical model for shape measurement of the image defect area.

[0542] Assuming the weight of the second-order moment of the continuous function is a (0≤a≤1), the weight of the second-order moment of discrete data is 1-a. The moment of inertia after fusion is:

[0543]

[0544] Calculate the equivalent ellipse parameters after fusion (major axis a, minor axis b, direction angle Э d ), first calculate:

[0545]

[0546] Long axis short axis Direction

[0547] Weighted average fusion second-order moment: M fusion＝ =aM c +(1-a)M d , where M c is the second-order moment of a continuous function, M d is the second-order moment of discrete data;

[0548]

[0549] In this model, a (0≤a≤1) is a weight coefficient. It is used to measure the proportion of the second-order moment of the continuous function in the fused second-order moment. When a = 0, it means that when calculating the fused second-order moment, the contribution of the second-order moment of the continuous function is completely ignored, and only the second-order moment of discrete data is used; when a = 1, it completely relies on the second-order moment of the continuous function and ignores the second-order moment of discrete data. By adjusting the value of a, the influence of the second-order moment of the continuous function and the second-order moment of discrete data on the final result can be flexibly balanced according to the characteristics and accuracy requirements of the data, thereby better adapting to the description requirements of different image defect area shapes. b mainly appears in the second-order moment of the continuous function M c When the continuous function is y=f(x), its domain is [a, b], where b is the upper limit of the domain of this function. c In the process, the integration interval is from a to b, which determines the range of values ​​considered when calculating the second-order moment of the continuous function.

[0550] For example, in computing (This is used to calculate M c When ba is used as the denominator to normalize the integral result, the value of b directly affects the integral interval and the final calculation result. Together with a, it defines the range involved in the calculation of the second-order moment of the continuous function.

[0551] Through the above model, the value of weight a is adjusted according to the characteristics and accuracy of the data, and the major axis, minor axis and direction angle of the defect shape are determined based on the fusion results of the second-order moment of the continuous function and the second-order moment of the discrete data.

[0552] For example, taking actual meter sampling components as an example, the weighted average method is used to fuse the second-order moment of the continuous function and the second-order moment of discrete data to express the defect shape's major axis, moment axis, and direction angle to reconstruct the defect shape.

[0553] Electricity meter sampling components are crucial for accurately measuring electricity consumption. During production and use, defects may occur due to material problems, processing errors such as weld cracks, voids, and cold solder joints, or due to long-term wear and tear. These defects can affect sampling accuracy and, in turn, lead to inaccurate meter readings. For example, sampling resistors may exhibit localized wear or uneven conductive layers, hard-wired connections may exhibit cracks, and capacitors may exhibit dielectric defects.

[0554] Calculation of the second moment of a continuous function:

[0555] For a typical defect in meter sampling components, such as material loss due to wear on the surface of sampling resistors and capacitors, its shape on a two-dimensional plane can be approximately represented by a parabolic function based on material physics and electrical principles.

[0556] Let function f(x, y) = AB[(x-x0) 2 +(y-y0) 2 ](within the defect area);

[0557] Among them, (x0, y0) is a parameter related to the resistor material and the degree of wear, and is the wear center position.

[0558] According to the continuous function second-order moment model, the moment of inertia is obtained by integration operation (the integration area is the wear and crack area on the resistor surface, and the wear radius of the circular area is r):

[0559]

[0560] Among them, function x0 is symmetrical to function y0, and their direction angles are both 0.

[0561] Discrete data second moment calculation:

[0562] Data acquisition: Figure 6As shown in the figure, the sampled components were inspected by high-precision optical microscope and electron microscope, and discrete data points (x i ,y i ). Weights w are assigned according to the clarity and credibility of discrete data points. i (For example, data points that are clear and have been confirmed by multiple tests) have a higher weight. Weight determination Based on the accuracy evaluation of the ideal model and the test data, the weight of the second-order moment of the continuous function is determined to be a=0.3, and the weight of the second-order moment of the discrete data is 1-a=0.7. The calculated moment of inertia and equivalent ellipse parameters are then subjected to weighted average fusion calculation to obtain the fused moment of inertia and equivalent ellipse parameters (major axis a, minor axis b, direction angle Э d ), through this weighted average fusion method, the defect shapes of meter sampling components are reproduced and constructed, providing a basis for evaluating component quality and improving production processes.

[0563] Taking multi-layer chip manufacturing and testing as an example, during the production of multi-layer chips, various defects may arise due to complex process steps such as lithography, etching, and deposition. These defects may exist between or within different layers, severely impacting chip performance. To detect these defects, a variety of techniques are typically used, including electron microscopy imaging and X-ray inspection. These detection methods can provide theoretical models of the defects in the form of continuous functions as well as discrete data points.

[0564] For a certain type of conductive layer short-circuit defect in a multi-layer chip, based on electromagnetic field theory and circuit models, its shape on a two-dimensional plane can be approximately represented by a function that combines a log-normal distribution and a trigonometric function. For example, where A is a coefficient related to the defect current density, (x0, y) is the estimated position of the defect center on the x-axis, and a and b are parameters related to the defect extent.

[0565] Weight determination: Based on the accuracy of the theoretical model (based on previous research and verification of similar defects) and the reliability evaluation of the electron microscope data, the weight of the second-order moment of the continuous function is determined to be 0.4, and the weight of the second-order moment of the discrete data is determined to be 0.6.

[0566] Through the steps of the present invention, the second-order moment of the continuous function and the second-order moment of discrete data are fused using the weighted average method to obtain the major axis, minor axis and directional angle parameters that can reproduce the shape of the chip defect, thereby more accurately describing the defects in the chip and providing a basis for subsequent chip repair or quality improvement.

[0567] Step S3092: Track the image defect position based on the segmented and located defect area.

[0568] Specifically, in the diagnosis phase (taking the determination of defect severity as an example, let the diagnosis function be J:

[0569] J(F)= severity level determined by F (58);

[0570] The severity level of the defect is determined based on the fusion feature F. For example, different F ranges can be set to correspond to different severity levels (such as mild, moderate, severe, etc.).

[0571] Tracking: Taking the position change of the tracking target at different times as an example, let the tracking function be T, and different times be t1, t2..., the tracking function can be expressed as:

[0572] T(F,t,)= position of target at time t: calculated based on F (59);

[0573] By fusing features F at different time t j To calculate the position of the target (such as the position movement of the defect, the displacement of the component, etc.), and then track the target.

[0574] In step S310, the hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, realize defect area segmentation and positioning, and then track the position of the image defect area, make decisions based on the defect tracking and fault tracing results, and optimize the production process parameters.

[0575] Decision-making stage: Discrete data second-order moment calculation Data acquisition: The sampled originals were inspected by high-precision optical microscope, electron microscope, optical current emission microscope, etc., and the discrete data points (x i ,y i ). Assign weights w based on the clarity and credibility of the data points. i (For example, data points that are clear and have been confirmed through multiple inspections have a higher weight.) Weight determination: Based on an accuracy assessment of the ideal model and the inspection data, the weight of the second-order moment of the continuous function is determined to be a = 0.3, while the weight of the second-order moment of the discrete data is 1-a = 0.7. The parameters of the equivalent ellipse of the moment of inertia are calculated using the previous formula, and the weighted average fusion is used to calculate the parameters of the fused equivalent ellipse of the moment of inertia (major axis k, minor axis b, direction angle Эd). This weighted average fusion method recreates the defect shapes of the meter sampling components, providing a basis for evaluating component quality and improving production processes.

[0576] Specifically, the above step S310 includes:

[0577] A defect evolution dynamics model is constructed based on discrete data points of defect area and perimeter and a continuous function second-order moment model;

[0578] Based on the defect location image, a preset tracking function is used to analyze the morphological change characteristics of the crack propagation angle and wear depth increment of the defect in the defect area of ​​the image at different times, and the defect evolution process is visualized and quantitatively analyzed;

[0579] Obtain the physical defect locations in the 3D structural information and use a spatial mapping algorithm to match the physical defect locations in the 3D structural information with the electrical anomaly data and microstructure images, respectively, to establish a correlation map between physical defects and electrical faults.

[0580] Based on the correlation map between physical defects and electrical faults and the morphological change characteristics, combined with the abnormal current distribution analysis in the wear area of ​​the resistor surface, the fault is traced to the location of the defective area in the image;

[0581] Based on defect tracking and fault tracing, the welding process, material replacement process and online monitoring process of the decision-making products are improved.

[0582] Among them, the welding process of the decision-making products will be improved, including:

[0583] Finite element simulation is used to analyze the effects of different welding parameters such as laser power, welding speed, and spot diameter on the residual stress, fatigue life, and resistance value of the welded joint for the product to be determined. The simulation results are verified by combining multimodal data from ultrasonic scanning results to determine the optimal process parameters for laser welding and resistance brazing. The effects of multiple power sources such as photovoltaic, green electricity, energy storage, mains electricity, combined electricity, and nuclear power on the welding parameters are also considered. The defect location images of weld cracks and cold welds detected by convolutional neural networks are combined with multimodal data from ultrasonic scanning to quantify the welding quality.

[0584] Improve the material replacement process for the products to be decided, including:

[0585] Establish a material performance database to record the resistance change rate and capacitance stability performance parameters of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments. Through accelerated aging tests simulating extreme working conditions, use convolutional neural networks to detect defects in images of aged materials, and evaluate material durability based on performance data.

[0586] Improve the online monitoring process of decision-making products, including:

[0587] Deploy a real-time image recognition system based on deep learning to perform millisecond-level analysis of X-ray or OBIRCH inspection images to automatically mark and classify defects; use deep learning algorithms to train X-ray or OBIRCH inspection images to establish a defect recognition model.

[0588] The multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method provided in this embodiment obtains accurate internal three-dimensional structural information through multi-parameter data fusion and reconstruction technology, so that the second-order moment of discrete data is combined with the second-order moment of continuous function for fusion, solving the difficulties in combining the second-order moment of discrete data with the second-order moment of continuous function and weight distribution in image accuracy, which restricts high-precision image online analysis and visual modeling. It is necessary to solve the problems of attenuation of depth imaging technology at different depths and the limitations of a single modality, break through the difficulties in establishing a product defect model and weight distribution by combining the second-order moment of discrete data with the second-order moment of continuous function, and realize high-precision image online analysis and visual modeling. The weighted average method is used to fuse the first moment of inertia with the second moment of inertia, and the first equivalent ellipse parameters with the second equivalent ellipse parameters, respectively, to obtain a mathematical model for measuring the shape of the image defect area, realize the construction of the shape of the image defect area, and provide conditions for tracking the image defect area.

[0589] Combine Figure 5 The multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by the present invention is further described in detail as follows:

[0590] like Figure 5 As shown, the overall process of the multimodal CNN / Transformer image defect diagnosis and tracking decision method provided by the present invention is as follows:

[0591] Step 1: Multimodal data collection: Acquire multiple layers of chip (smart meter and components) images from various channels (data types include the image, electrical, and operating condition categories described above), including images of chips (smart meters and components) from different production batches, different process stages, and captured by different equipment. At the same time, collect corresponding label data, such as whether the chip (smart meter and components) has defects and the defect type.

[0592] Step 2, data cleaning: remove unclear, incorrectly labeled or damaged image data to ensure data quality.

[0593] Step 3, Data Labeling: For images that are not labeled or are incompletely labeled, they can be labeled by professionals or using labeling tools. The labeled content can be the different layer structures of the chip, defect locations and categories, etc.

[0594] Step 4: Clarify the usage classification operation:

[0595] A. Image data of different modalities have significant differences in data release in terms of content, scenes, target objects, etc.; appearance inspection and internal structure inspection, key or critical position defects or product inspection and measurement are classified; classification is convenient for labeling and targeted model training.

[0596] B. Image data of different modalities are relatively similar in content and usage, and the data homogeneity is high. When planning to use a unified tool method architecture and feature extraction strategy, and emphasizing the correlation and complementarity between multiple modalities, classification is not required.

[0597] Data preprocessing and alignment (the fusion processing, image preprocessing and electrical parameter processing described above).

[0598] Step 5, feature normalization: Normalize the image pixel values ​​to a specific range, such as [0, 1] or [-1, 1]. This helps speed up model training and improve stability. For a certain feature value x, the formula can be used:

[0599]

[0600] Among them, x min and x max are the minimum and maximum values ​​of the feature in all samples respectively.

[0601] Step 6, position encoding: For the defect position texture parameters (in two-dimensional coordinates (x d ,y d ) represents the center position of the defect), and position encoding is used to convert it into a form that can be effectively processed by the neural network. A simple method is to discretize the coordinate values ​​and then convert them into an mxm vector form through one-hot encoding. For example, the image is divided into grid areas, and one-hot encoding is performed according to the grid area where the defect center is located, resulting in a vector of length m. 2 The vector is used as the feature after position encoding.

[0602] Step 7, weight allocation and feature fusion: When x min and x max It is used to define a simple range. All data have the same function at this stage and no weight distribution is required. min and x max When used for further comparison and classification, as a threshold judgment, each feature has different main or defect, important severity of feature changes in subsequent operations, and weights of contrast C, correlation R, energy E, entropy H and defect entropy value are assigned.

[0603] Different sensors, different defects or different equipment for product visual inspection have different detection conditions and rules. The image information of the same defect or product varies greatly and is difficult to integrate. The entropy weight method is used to calculate the minimum and maximum values ​​in the sample and take their respective lowest entropy weights for splicing. The gaps between the splicing are spliced ​​by taking the average value.

[0604] The relationship between each parameter and the accuracy of the multi-layer chip image is analyzed as follows: Assume that the input multi-layer chip image is I d , the smart meter image is I B , the contrast feature is recorded as C(I d ), the correlation characteristic is recorded as R(I d ), the energy characteristic is recorded as E(I d ), the entropy characteristic is recorded as H(I d ), the defect entropy characteristic is recorded as D(I d ) (Here D is used to represent the defect entropy value for the convenience of subsequent description), the position texture parameter feature is recorded as P(I d ).

[0605] The contrast feature is recorded as C(I B ): If the contrast C(I B ) has a strong ability to distinguish normal and defective areas. For example, when the meter display screen is blurred or has other defects, the contrast C(I B ) can clearly show the difference between the display area and the surrounding normal area, then the contrast C(I B ) a relatively high initial weight range, such as 20% to 30%. For some minor defects (such as fine scratches), if the contrast C(I B ) changes is not obvious, then take a lower value within this weight range.

[0606] The correlation characteristic is recorded as R(I B ): When the correlation R(I B ) When the pixel correlation around the defect area is significantly reduced (such as the defect causes the destruction of texture continuity), a weight range of 15% to 25% can be given. B ) If the change is not significant for most defect types, the weight range is appropriately adjusted down to 10% to 20%.

[0607] Energy characteristic is recorded as E(I B ): If the energy E(I B ) presents a specific change pattern when the defect occurs (for example, a certain defect always causes the increase or decrease of the energy of a specific frequency component), and is given a weight of 15% to 25%. If the energy E(I B ) changes are more complex and have no obvious pattern, and the weight can be set to 10% to 20%.

[0608] The entropy characteristic is denoted as H(IB ): Since the entropy characteristic is recorded as H(I B ) is related to the uncertainty of image information. For complex textures or mixed defects, if the entropy feature is recorded as H(I B ) can reflect these situations well and can be assigned a weight of 30% to 40%. If the entropy feature is recorded as H(I B ) may behave unstable in some simple defect scenarios, so make appropriate adjustments within this range.

[0609] The defect entropy characteristic is recorded as D(I B (D is used here to represent the defect entropy value for ease of explanation): If the defect entropy value is the key factor in distinguishing defects of different severity (for example, a high entropy defect may indicate a more serious internal circuit failure in the meter), assign a separate weight to the defect entropy value. Based on the total weight, a weight of 10% to 20% can be allocated specifically to adjust the accuracy based on the defect entropy value. Defects with high entropy values ​​and a significant impact on the meter's function should be given a higher weight.

[0610] The position texture parameter feature is recorded as P(I B ): When the defect is located in a key part of the smart meter (such as the display screen, around the metering chip, etc.), its weight should be higher than that of the defect in a non-critical part. According to the importance of the position, the weight of the position factor can be allocated between 10% and 30%. At the same time, for areas with obvious texture features (such as the specific pattern of the meter shell), if the texture change is important for defect detection, the texture related parameters (such as the correlation feature is recorded as R(I B ), entropy characteristics are recorded as H(I B ) to highlight the impact of texture changes on accuracy.

[0611] The principle of the weight distribution scheme is that the sum of the weights must be equal to 1.

[0612] Detect different types of defects (such as scratches, wear, display abnormalities, etc.), and then determine the initial weight range according to different types of defects. In the feature fusion stage, the fusion is performed according to the given weight distribution scheme, and the fused feature vector is set as F(I B ),but:

[0613] F(I B )=(20% to 30%)×C(I B )+(10% to 25%)x R(I B )+E(I d )x(15% to 25)+H(I B )x(30%

[0614] to 40%)+D(I B)x(10% to 20%)+P(I B )x(10% to 30%).

[0615] The advantage of performing multimodal fusion after position encoding is that, after encoding the position information and then participating in the fusion, the position features can be better aligned with other modal features in terms of space and other concepts. This allows the fused feature vector to comprehensively consider the relationship between position and other physical attribute features, which may be more beneficial for tasks such as defect identification with positional dependencies. In some industrial inspection scenarios, if there is a certain pattern of correlation between the defect location and the defect's own contrast, energy, and other characteristics, this method can better capture this relationship. In the power electronics industry, the first requirement is to accurately locate the defect fault and quickly find a countermeasure, and the shape and size are the second requirement. This improvement meets these needs.

[0616] For example, after position encoding, the X-ray image is multimodally fused, including: transforming the normalized contrast feature C(I d ), correlation characteristics R(I d ), energy characteristics E(I d ), entropy feature H(I d ), defect entropy value feature D(I d ) (Here D is used to represent the defect entropy value for convenience of subsequent expression), the position texture parameter feature is recorded as P(I d ), and the defect position features after position encoding p are spliced ​​and fused to form a multimodal fusion feature vector or After the above processing, the features are C_{normalized}, R_{normalized}, E_{normalized}, H_{normalization}, H_d_{normalized} and the position encoding vector p, then:

[0617] \mathbf{F}=[C_{normalized},R_{normalized},E_{normalized},H_{normalization},H_d_{normalized},\mathbf{P}].

[0618] The advantages and effects of these innovations include: Considering location information: Through location encoding, defect location features are incorporated into multimodal fusion, enabling the model to better utilize location information for defect detection. For devices like smart meters, defects in different locations may have different impacts. This approach can more accurately address location-related issues.

[0619] Comprehensive feature utilization: Multiple features, such as contrast, correlation, energy, entropy, and defect entropy, are weighted and then fused to fully leverage each feature in defect detection. Weights are assigned based on the effectiveness of different features in distinguishing different types of defects, making the fused feature vector more discriminative and meeting diagnostic requirements.

[0620] Normalization advantage: Normalizing features before fusion can prevent certain features from dominating the fusion process due to their large value range, ensuring that each feature can participate in the fusion equally, improving the fusion effect and model stability.

[0621] Targeted weight allocation: Weight allocation schemes are determined separately for multi-layer chip images and smart meter images, taking into account the characteristics of different image types and the performance of different features under different defect types. This makes the fusion solution more targeted and can better adapt to specific inspection tasks and meet the needs of high adaptability.

[0622] Step 8: Description of the model architecture based on CNN (Convolutional Neural Networks):

[0623] Basic CNN: Choose an appropriate convolutional neural network architecture as the foundation, such as ResNet (Residual Network) or VGG (Visual Geometry Group), and determine the network's depth and width based on the complexity of the chip image and the task requirements. For example, if the chip image is rich in detail and requires more convolutional layers to extract features, choose ResNet-50 or a more complex network. A CNN model consists of an input layer, multiple convolutional and pooling layers, multiple fully connected layers, and an output layer.

[0624] The input layer receives the multimodal fusion feature vector The number of nodes is equal to dimension.

[0625] Convolutional and pooling layers: Multiple convolutional and pooling layers are used to automatically extract high-level information from features. This example uses three convolutional layers with kernel sizes of 3x3, 3x3, and 5x5, respectively, and a stride of 1. Each convolutional layer is followed by a pooling layer with a kernel size of 2x2 and a stride of 2. The activation function for the convolutional layer is the Randomized ReLU (RReLU) activation function, defined as: f(x) = max(0.x).

[0626] Fully connected layer: After the feature extraction of the convolution layer and the pooling layer, several fully connected layers are set to further integrate the information and map it to the output probability distribution. In this embodiment, two fully connected layers are set. The number of nodes in the first fully connected layer is k (the appropriate value can be determined through experiments, such as 128), and the number of nodes in the second fully connected layer is n, corresponding to the number of elements P of the probability distribution of the model output image type = [p0, o1…p n ], where n is the number of image types, P i represents the probability that the image belongs to the i-th type. The type with the highest probability is the predicted image type. To alleviate the gradient vanishing problem in multimodal and high-resolution images, improve computational efficiency, prevent neuron death, and prevent model overfitting, and filter out negative signals, the activation function of the fully connected layer can be the RReLU function (,f(x)=max:(ax,x)), where a is a learnable, adaptive, and automatically adjusted slope parameter that is averaged after training.

[0627] The output layer uses the Softmax function (an activation function used in multi-class classification problems) as the activation function to convert the output of the fully connected layer into a form representing probability. i , the output after processing by the Softmax function is: This ensures that the sum of all output probability values ​​is 1, which meets the requirements of probability distribution.

[0628] Step 9, Initialize parameters: Initialize the convolution kernel, bias and other parameters in the network. Common initialization methods include random initialization (such as Xavier initialization and He initialization), which can help the model converge faster.

[0629] Step 10: Divide the model training data set: Divide the collected image data into training set, validation set and test set in a ratio of 7:2:1 or 8:1:1.

[0630] Step 11, loss function selection: Cross-Entropy Loss is used to measure the difference between the probability distribution predicted by the model and the actual probability distribution. Its expression is:

[0631]

[0632] Among them, y i is the element in the actual probability distribution p, that is, the actual label. In the classification problem, if it is in the form of one-hot encoding, for example, for a three-category problem, the category label is [0,0,1], indicating that it belongs to the third category, then y iIt is 1 when i=3 and 0 at other positions. It represents the probability corresponding to the category to which the sample actually belongs (true probability distribution). i is the sum from 1 to j and is used to calculate the cross entropy loss. i , predicted represent the true probability value and the probability value predicted by the model respectively. For example, the model predicts that the probability of a sample belonging to the first category is 0.2, the probability of belonging to the second category is 0.3, and the probability of belonging to the third category is 0.5. These probability values ​​are p i , predicted(i=1,2,3) is the output after the convolutional neural network and the Softmax function. It is used to compare the probability corresponding to the actual label to measure the quality of the model's prediction. By using the Softmax function and minimizing the cross-entropy loss, the convolutional neural network model can effectively learn the correct classification boundaries.

[0633] Step 12, multimodal fusion CNN training: training set data is input into the multimodal fusion CNN model constructed, output is calculated by forward propagation, loss is calculated according to loss function, and then optimizer is used to back propagate and update network parameters. During the training process, model parameters are regularly saved, and model performance is evaluated on the validation set, and training parameters (such as learning rate, number of training rounds, etc.) are adjusted according to the verification results. In the present embodiment, training set data is used to train the convolutional neural network model, and the weights and bias of the model are adjusted by back propagation algorithm in combination with optimization algorithm and Adam optimizer (improved stochastic gradient descent method) so that the value of loss function is continuously reduced, that is, the difference between the probability distribution predicted by the model and the actual probability distribution is continuously reduced. During the training process, the parameters (such as convolution kernel size, step length, number of nodes, etc.) of the convolutional layer, pooling layer, and fully connected layer of the model can be adjusted according to training effect (such as the accuracy rate, loss value and other indicators on the validation set) to improve the performance of the model.

[0634] Step 13: Describe the Adam optimizer (improved stochastic gradient descent method) in detail:

[0635] The Adam optimizer maintains two moving average estimators, namely the first-order moment estimator m and the second-order moment estimator v. First, at each iteration t, a weight of the convolutional layer with parameter θ is The gradient calculation formula is as follows:

[0636]

[0637] Among them, g t is the gradient, θ is the parameter, It usually represents the gradient of the loss function L with respect to the parameter θ (here it is some kind of averaged gradient, or the average gradient over a small batch of data or other forms of gradient processing). The loss function L measures the difference between the model's prediction results and the true label. When training machine learning models such as neural networks, our goal is to minimize this loss function. It represents the rate of change of the loss function in the parameter space, which indicates how the parameters should be adjusted to reduce the loss function.

[0638] Calculate the gradient First calculate the gradient g of the loss function L with respect to the parameter θ (here specifically a weight w of the convolutional layer) t This gradient calculation is based on the data of the current batch, which points out the direction in which the loss function grows fastest at the current position of the parameter.

[0639] Update the first-order moment estimator (momentum term):

[0640] m t =β t m t-1 +(1-β1)g t (6);

[0641] Among them, β1 is a hyperparameter, which is set to 0.9, β t is a parameter used to control the previous momentum information m t-1 The weight parameter when updating the current momentum. t When it is close to 1, it means that m is being updated. t When , it depends more on the previous momentum m t-1 ; When β t When it is close to 0, the current gradient information (1-β1)g will be more important. t .

[0642] In the given formula, the hyperparameter is defined only as β1 = 0.9, where β t It is a parameter similar to β1, and in practical application scenarios (such as optimization algorithms such as Adam, etc.), β t It is usually a number in the interval (0,1) to balance the contribution of historical information and current information to the momentum term update. If this is in a specific algorithm iteration process, β t It may also change with the number of iterations. The specific change rules need to be determined according to the detailed settings of the algorithm. Formula (6) g t Used to update the first-order moment estimator m t . m t It can be regarded as the gradient g tA weighted moving average of m0 that accumulates information about past gradients, similar to momentum with friction. This momentum term helps accelerate convergence, especially when dealing with parameter updates with similar gradient directions. Initially, m0 is usually set to 0.

[0643] Update the second-order moment estimator:

[0644]

[0645] Among them, β1 is a hyperparameter and is set to 0.999.

[0646] The above first-order moment estimator and second-order moment estimator are bias-corrected as follows:

[0647]

[0648] Finally, update the parameters, the formula is as follows:

[0649]

[0650] Among them, η is the learning rate, and ∈ is a very small number to prevent the denominator from being zero.

[0651] The improvement of the Adam optimizer combines the momentum method (through the first-order moment estimator m) and the idea of ​​Adagrad (through the second-order moment estimator v). It can dynamically adjust the learning rate of each parameter in the model according to the first-order moment and second-order moment of the gradient, and through deviation correction, it can achieve better learning effects in the early stages of training.

[0652] Step 14, image performance evaluation: Use the test set for evaluation. That is, after training is completed, use the test set to evaluate the trained model to be evaluated, calculate evaluation indicators such as accuracy, precision, recall, average intersection over union, and F1-score, and evaluate the performance of the model on unseen data.

[0653] Taking the multi-layer chip image accuracy performance evaluation indicators as an example, the calculation formulas for evaluation indicators such as accuracy, precision, recall rate, and F1-score are as follows:

[0654] Accuracy:

[0655]

[0656] Among them, TP (True Positive) represents the number of samples correctly detected as positive (such as defective chip images correctly detected); TN (True Negative) represents the number of samples correctly detected as negative (such as non-defective chip images correctly detected); FP (False Positive) represents the number of samples incorrectly detected as positive (actually non-defective but detected as defective); FN (False Negative) represents the number of samples incorrectly detected as negative (actually defective but detected as non-defective).

[0657] Precision:

[0658]

[0659] The precision rate measures the proportion of positive samples detected as true positive samples, reflecting the reliability of the detection results.

[0660] Recall:

[0661]

[0662] The recall rate indicates the proportion of actual positive samples that are correctly detected, reflecting the model's ability to capture positive samples.

[0663] Mean Intersection over Union (mIoU):

[0664]

[0665] Assume that k categories (e.g., different types of defects or normal regions are considered different categories) are detected in a multi-layer chip image. Intersection(i) represents the intersection area between the predicted result and the true label in category i, and Union(i) represents the union area between the predicted result and the true label in category i.

[0666] Step 15: Model Visualization Analysis: Use visualization tools (such as TensorBoard) to analyze the loss and accuracy curves during model training to gain an intuitive understanding of the model's training progress. You can also visualize the feature maps learned by the convolutional layer to verify whether the model has truly learned the valuable features in the chip (smart meter) image.

[0667] Step 16: Determination of defects based on the fused feature image:

[0668] Deployment to production and application environments: Deploy the trained and evaluated model to the actual product inspection production line and application, input the actual multi-layer chip images taken by the inspection equipment into the tested image defect detection model, and achieve real-time inspection, measurement, diagnosis and tracking. Inspection phase (taking the binary classification to determine whether there is a defect as an example, let the decision function be G, and the decision function is expressed by the following formula:

[0669]

[0670] Here, G(F)=1 indicates that a defect is detected in the corresponding area of ​​the image, G(F)=0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that a preset defect condition is met.

[0671] The value of G(F) is a binary classification value. The preset defect condition can be a judgment rule such as the value of F being greater than a certain threshold. The threshold can be determined through training or other methods. In other words, the preset defect condition can be a judgment rule such as the value of F being greater than a certain threshold. It is a variable related to the characteristics and parameters related to the multi-layer chip image. More specific background information about the image defect detection model and related settings is needed to further clarify its reference. For example, if G(F) = 1, it means that a defect is detected in the corresponding area of ​​the image, and G(F) = 0 means that there is no defect.

[0672] Step 17, image defect segmentation: First, the defect area needs to be accurately segmented from the image. The semantic segmentation model of the improved U-Net model based on CNN deep learning can be used. The improved U-Net model has good segmentation performance and is particularly suitable for multi-task and multi-modal segmentation of complex images such as welds, crystal oscillators, chips, IGBTs, GBAs, and the internal details of artificial intelligence multi-layer chips.

[0673] The semantic segmentation model of the improved U-Net model based on CNN deep learning introduces residual connections in the encoder (downsampling path) and an attention mechanism in the decoder (upsampling path).

[0674] Residual connection (improved) (x, y, z): Introduce residual connection in the encoder, and set the weight matrix of residual connection to be The output feature map after two convolutional layers is With input features Figure X i (x, y, z), add the residual connection to get the new output feature map The formula is as follows:

[0675]

[0676] Attention mechanism (improved): Introduce the attention mechanism in the decoder and set the attention weight matrix as Feature map after upsampling and concatenation by the encoder Obtain weighted features through the attention mechanism That is the TFT ray diagram, the formula is as follows:

[0677]

[0678] Among them, (x, y, z) are image coordinates, a represents the relative position offset involved when the feature map performs a weighted operation using the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[0679] Then replace Perform subsequent convolutional layer operations. Once And (C i-1 =C i+1 ), the number of channels may change after the convolutional layer. The TFT radiographic image is input into the trained improved U-Net model to obtain a binary image of the defect area, where the pixel value of the defect area is 1 and the pixel value of the background area is 0.

[0680] Step 18, defect shape contour analysis: Contour extraction is performed on the binary image of the defect area. This can be achieved using functions provided by image processing libraries such as OpenCV. After obtaining the contour of the defect area, the shape of the defect can be described by calculating some geometric features of the contour. For defects in a two-dimensional image, the perimeter C of the contour can be calculated. p 、Convex hull perimeter C hc , shape factors (such as circularity A is the area of ​​the image defect region. The closer the circularity value is to 1, the closer the defect shape is to a circle; the smaller the circularity value is, the more irregular the shape is. These indicators are used to describe the image defect shape from different angles.

[0681] Circumference C p Calculation: Extract the contour of the binary image of the defect area of ​​the image, and set the extracted contour point set as N is the number of contour points, then the contour perimeter C p It can be obtained by calculating the sum of the Euclidean distances between adjacent contour points. The calculation formula is:

[0682]

[0683] Among them, x i+1 with x i are adjacent contour points, y i+1 with y i are adjacent contour points.

[0684] Convex hull perimeter C hc Calculation: First calculate the convex hull of the defect area contour, and set the convex hull point set as M is the number of convex hull points. The perimeter of the convex hull is C. hc It is also determined by calculating the sum of the Euclidean distances between adjacent convex hull points. The calculation formula is:

[0685]

[0686] Step 19: Shape hybrid detection modeling for high-precision defect features:

[0687] Calculate the moment characteristics of the defect area, such as the zero-order moment M0 (whose value is equal to the area of ​​the defect area), the first-order moment M1 (which can be used to determine the center of gravity of the defect area), and the second-order moment M2 (related to the shape of the defect area). These moment characteristics can also describe the defect shape to a certain extent. Calculate the second-order moment of the defect area, including the central moment.

[0688] The mean squared error (MSE) is used to evaluate the measurement task. The calculation formula is as follows:

[0689]

[0690] When measuring certain features in product images (such as component size, line width, etc.), is the true value, is the predicted value, and n is the number of measurement samples. The mean square error can be used to assess the degree of deviation between the measured value and the true value. The smaller the value, the higher the measurement accuracy.

[0691] Step 20, calculate the measured target measurement based on the fused feature F:

[0692] Measurement step (taking the measurement of a target size as an example, let the measurement function be M:

[0693] M(F)=size value calue calculated on F (22);

[0694] That is, the size value of the target to be measured (such as the width, length, area, volume, etc. of the chip circuit (smart meter defects, faults)) is calculated based on the fused feature vector F.

[0695] Step 21: Output the width, length, area, volume, etc. of defects and faults:

[0696] 1. Based on the second-order moment of discrete data: define the second-order moment of discrete data in the defect location area.

[0697] Relationship between zero-order moment and area: Assume that the binary image function of the defect area is f(x, y), where the pixel value of the defect area is 1 and the pixel value of the background area is 0. The calculation formula of the zero-order moment M0 is:

[0698] M0=∑∑ f f(x, y) (23);

[0699] From the above formula, we can see that the value of the zero-order moment is equal to the area A of the defect area, that is, A=M0.

[0700] The first-order moment is represented by M1, and the first-order moment and the center of gravity position are calculated as follows:

[0701] The calculation formulas for the first-order moment M1x (about the x-axis) and M1y (about the y-axis) are:

[0702] M1x=∑ x ∑ y xf(x,y) (24);

[0703] M1y=∑ x ∑ y yf(x,y) (25);

[0704] Then the center of gravity position of the defect area (x g -y g ) can be calculated using the following formula:

[0705]

[0706] Relationship between second-order moment and defect shape:

[0707] The calculation formulas for the second-order moment M2xx (the second-order moment about the x-axis), M2yy (the second-order moment about the y-axis), and M2xy (the mixed second-order moment about the x-axis and y-axis) are:

[0708] M2xy=∑ x ∑ y xyf(x, y) (27);

[0709] M2xx=∑ x ∑ y x 2 f(x, y) (28);

[0710] M2yy=∑ x ∑ y y 2 f(x, y) (29).

[0711] 2. The calculation of the second-order moment of a continuous function (taking two dimensions as an example) is as follows:

[0712] M 00 =∫∫D xf(x,y)dxdy(30);

[0713] For the continuous function z=f(x,y) defined on the defect location area D.

[0714] Zero-order moment M 00 calculate:

[0715] M 00 =∫∫ D f(x, y)dx dy (31);

[0716] The above formula is a double integral, and the integration region is the domain D of the function f(x, y).

[0717] First-order moment calculation:

[0718] M 10 =∫∫ D xf(x,y)dxdy(32);

[0719] M 01 =∫∫ D yf(x,y)dxdy (33);

[0720] The coordinates (x, y) of the center of gravity are given by and Sure.

[0721] The second-order central moment calculations are as follows:

[0722] M 2xy =∫∫ D xf(x,y)dxdy (34);

[0723] M 2xx =∫∫ D x 2 f(x, y)dx dy (35);

[0724] M 2yy =∫∫ D y 2 f(x, y)dx dy (36). The calculation of these second-order central moments requires first finding the coordinates of the center of gravity and then completing it through double integration.

[0725] 3. Hybrid Second-Order Model: A hybrid second-order model is constructed by combining the second-order moments of continuous functions and discrete data, combining discrete data and continuous functions. For defect shapes, it is preferred that the main portion of the defect be approximated by a continuous function. In this embodiment, a polynomial function or a combination of trigonometric functions is preferred. Furthermore, discrete data points are used to supplement details at key boundary locations or local irregularities.

[0726] Suppose the continuous function is y=f(x), its domain is [a,b], and the discrete data points are (x i ,y i )i=1,2,…,n. Calculate the second moment of the continuous function part:

[0727]

[0728] in,

[0729] For the discrete data part, its second-order moment is:

[0730]

[0731] in, By adjusting the parameters of the continuous function and the positions of the discrete data points, M is minimized to optimize the description of the defect shape.

[0732] 4. The following is a mathematical model that uses the weighted average method to fuse the second-order moment of the continuous function and the second-order moment of discrete data to calculate the major axis, minor axis and direction angle of the defect shape:

[0733] The required defect shape is represented by a continuous function z=f(x, y) on a two-dimensional plane, and its domain is Ω.

[0734] Calculating the moment of inertia The formulas are as follows:

[0735]

[0736] M 2xx =∫∫ D x 2 f(x, y)dx dy (38);

[0737] M 2yy =∫∫ D y 2 f(x, y)dx dy (39);

[0738] 4.1 Calculation of equivalent ellipse parameters (continuous function part):

[0739] Let the major axis k corresponding to the continuous function part be c 、Short axis b c and direction angle Э c .

[0740] First calculate:

[0741]

[0742]

[0743] Long axis short axis Direction

[0744] 4.2 Second-order moment of discrete data:

[0745] There are n discrete data points (x i -y i ), weight is w i (If the weights are the same, w i =1).

[0746] Calculating the moment of inertia and The formula is as follows:

[0747]

[0748] 4.3 Calculate the equivalent ellipse parameters (discrete data part):

[0749] Let the long axis k corresponding to the discrete data part d 、Short axis b d and direction angle Э d . First calculate:

[0750]

[0751] The long axis short axis Direction

[0752] 4.4 Weighted average fusion calculation:

[0753] Assume that the weight of the second-order moment of the continuous function is a (0≤a≤1), then the weight of the second-order moment of discrete data is 1-a. The moment of inertia after fusion is:

[0754]

[0755] Calculate the equivalent ellipse parameters after fusion (major axis k, minor axis b, direction angle Э d ), first calculate:

[0756]

[0757] Long axis short axis Direction

[0758] Weighted average fusion second-order moment: M fusion＝ =aM c +(1-a)M d , where M c is the second-order moment of a continuous function, Md is the second-order moment of discrete data.

[0759] Through the above model, the weight a is adjusted according to the characteristics and accuracy of the data. The major axis, minor axis and orientation angle of the defect shape are determined by fusing the continuous function and the second-order moment of the discrete data. The shape is visualized online.

[0760] Final shape: The approximate elliptical shape of the defect area can be roughly determined based on the ratio of the major axis to the minor axis and the direction angle (in the case of an elliptical-like defect). For example, when the major axis is significantly longer than the minor axis, it appears as a narrow and long ellipse. The direction angle determines the inclination angle direction within the plane, together outlining the final shape characteristics on the two-dimensional plane.

[0761] Width: The short axis length b is usually regarded as the width of the defect area, which represents the size of the defect in the direction perpendicular to the long axis.

[0762] Length: The major axis length a is considered to be the length of the defect area, that is, the distance range spanned by the defect in its main extension direction.

[0763] Area: For defect areas that are approximately elliptical in shape, the area S is calculated as S = πaab. The area occupied by the defect on a two-dimensional plane is obtained by multiplying the values ​​of the major axis and the minor axis and then multiplying them by pi.

[0764] Volume: If the three-dimensional case is to be considered (assuming that the defect also extends in a certain thickness direction, and the thickness is set to h), the volume V can be simply approximated according to the volume of an elliptical cylinder (when the defect morphology meets such characteristics). The calculation formula is V = πabh, that is, the volume size is obtained by multiplying the two-dimensional area calculated above by the dimension in the thickness direction.

[0765] In practical applications, these calculation methods can be appropriately adjusted based on factors such as the specific physical characteristics of the defects and the degree of shape conformity, or more practical professional models can be used to measure relevant parameters.

[0766] Step 22, defect and fault diagnosis:

[0767] Diagnosis (taking the determination of defect severity as an example, let the diagnosis function be J:

[0768] J(F)=severity level determined by F (58);

[0769] The severity level of the defect is determined based on the fusion feature F. For example, different F ranges can be set to correspond to different severity levels (such as mild, moderate, severe, etc.).

[0770] Step 23, tracking phase: Taking the position change of the tracking target at different times as an example, let the tracking function be T, and different times be t1, t2, etc., the tracking function can be expressed as:

[0771] T(F,t,)=position of target at time t:calculated based on F (59);

[0772] By fusing features F at different time t i To calculate the position of the target (such as the position movement of the defect, the displacement of the component, etc.), and then track the target.

[0773] Step 24, decision-making phase:

[0774] like Figure 6As shown, the detection system is a flexible configuration detection system that integrates x-rays with nano-probe-based FIB (an algorithm for calculating Fibonacci numbers), TEM, EMM, and OBIRCH (OBIRCH stands for Optical Beam Induced Resistance Change). The integrated detection, measurement, diagnosis, tracking, and decision-making scheme for defects and faults realizes non-destructive detection of physical and electrical defects and faults in circuits and devices. The detection system includes an x-ray source and a flexible fusion configuration of FIB, TEM, EMM, and OBIRCH. The inspected object is inspected by x-rays, electrons, photocurrents, and ion beams. When defects are detected, the synchronous nano-probes detect the current, voltage, resistance, etc. in the circuit to determine the electrical performance faults and defects. The tiny defects of the inspected object are further detected by FIB, TEM, EMM, and OBIRCH. The transmitted x-rays, electrons, photocurrents, and ion beams hit the detector. The information on the detector is imaged and transmitted to the image acquisition and transmission circuit. The image acquisition and transmission circuit transmits the signal unidirectionally and the automatic control unit transmits the signal bidirectionally to the computer's management data correction unit. The data from the correction unit is then processed through a high-precision image reconstruction unit (implemented by a multi-dimensional prediction model of image types, detection, measurement images, and evaluation based on the multi-modal CNN / Transformer image defect diagnosis and tracking decision-making method) to perform layer-by-layer electrical fault detection and performance assessment. This unit also includes image storage, display, and output, defect image visualization, fault diagnosis, and fault tracking. Based on the detection, measurement, and diagnosis results, improvement decisions or remaining life performance for the process, materials, structure, electrical system, quality, production, in-use, and global circuits, systems, products, and devices are made. The multi-modal fusion CNN high-precision image reconstruction unit exchanges information with the automatic control unit. The automatic control unit controls the positioning of different inspected items on one side, while the nanoprobe coordinates defects with FIB, TEM, EMM, and OBIRCH detection, moving in different directions to inspect different layers of the item and device for layered and faceted inspection. Another control unit controls the high-voltage generator on and off, generating X-ray detection energy and simultaneously controlling the automatic detection and safety protection of the FIB, TEM, EMM, and OBIRCH.

[0775] Discrete data second-order moment calculation data acquisition: The sampling components are inspected by high-precision optical microscope, electron microscope, optical current emission microscope, etc., and the discrete data points (x i ,y i ). Assign weights w based on the clarity and credibility of the data points. i(For example, data points that are clear and have been confirmed through multiple inspections are given a higher weight.) Weight determination is based on an assessment of the accuracy of the ideal model and the inspection data. The weight of the second-order moment of the continuous function is determined to be a = 0.3, while the weight of the second-order moment of the discrete data is 1-a = 0.7. The parameters of the equivalent ellipse of the moment of inertia are calculated using the previous formula, and the weighted average fusion method is used to calculate the parameters of the fused equivalent ellipse of the moment of inertia (major axis k, minor axis b, direction angle Эd). This weighted average fusion method recreates the defect shapes of the meter sampling components, providing a basis for evaluating component quality and improving production processes.

[0776] In combination with the above description, the invention details of the improved decision-making solution for tracking defects and faults of smart meter sampling components are further described in detail: a systematic improvement and prediction decision-making solution for faults and defects of meter sampling components based on multimodal fusion and second-order moment analysis.

[0777] 1. Deepening of the fault and defect analysis system:

[0778] 1. Mapping of defect types and failure mechanisms:

[0779] Material defects: The wear of the resistor surface leads to increased contact resistance, and the defects of the capacitor medium cause fluctuations in the dielectric constant. Both directly affect the voltage / current conversion accuracy of the sampling circuit. Specifically, when the resistor material wears, its conductive layer becomes thinner. According to the law of resistance, A decrease in cross-sectional area S will lead to an increase in resistance; defects in the capacitor dielectric will change the internal electric field distribution, causing the dielectric constant ∈ to change, affecting the capacitance The stability of the sampling circuit will eventually affect the voltage / current conversion accuracy.

[0780] Welding defects: Hard connection cracks and cold joints lead to unstable contact resistance, causing intermittent loss or distortion of the sampling signal. Cracks and cold joints in the welds form an unstable contact interface, causing the contact resistance to fluctuate under different operating conditions, resulting in intermittent abnormal sampling signals.

[0781] Structural defects: Voids disrupt the electric field distribution within components, causing local overheating or electrical parameter drift. The presence of voids alters the dielectric distribution within the component, concentrating the electric field at the edge of the void. This results in excessively high local electric field strength, causing local overheating and, in turn, affecting the component's electrical parameters.

[0782] By establishing a defect-fault causal matrix and fusion-analyzing defect shape parameters (major axis, minor axis, and orientation angle) using the second-order moments of continuous functions and discrete data, the contribution of different defect types to metrological error is quantified. For example, for resistance wear defects, by calculating the major and minor axis parameters of their equivalent ellipse and combining historical data to establish a correlation model between wear degree and metrological error, the weight of the defect's impact on metrological error is determined.

[0783] 2. Dynamic assessment model of defect severity:

[0784] J(F) function expansion: The fused feature F is decomposed into three sub-features: geometric features (defect area, perimeter), physical features (resistance value change rate, dielectric loss angle), and spatiotemporal features (defect expansion speed). The analytic hierarchy process (AHP) is used to determine the weights and construct a weighted diagnostic function: Among them, the geometric features are obtained by i ,y i ) is processed and the moment of inertia and equivalent ellipse parameters calculated by fusing the second-order moment of continuous function and the second-order moment of discrete data are used to further determine the area and perimeter of the defect; the physical characteristics are calculated by parameters such as current, voltage, and resistance in the online detection circuit; the spatiotemporal characteristics are obtained by analyzing the position and morphological changes of the defect at different times through the tracking function T(F,t).

[0785] Dynamic threshold adjustment: A threshold correction model is established through historical data training based on meter operating conditions (such as load fluctuations and ambient temperature) to avoid misjudgments. Defect image information acquired by the multimodal fusion CNN high-precision image reconstruction unit is combined with real-time operating data to dynamically adjust the defect severity threshold under different operating conditions. For example, in high-temperature environments, the defect severity threshold can be appropriately lowered to identify potential risks in advance.

[0786] 3. Fault tracing and location optimization:

[0787] T(F,t) function enhancement: Introducing a defect evolution dynamics model, the tracking target is expanded from a single location to defect morphological changes (such as crack propagation angle and wear depth increment). Based on discrete data points (xi,yi) and a continuous function second-order moment model, the morphological changes of the defect at different times are analyzed, and dynamic tracking of defect morphological changes is achieved through the tracking function \(T(F,t)\).

[0788] Multimodal data collaboration: A three-dimensional defect model is constructed based on X-ray tomography (CT), microstructural information is acquired through a dual-beam FIB-SEM system, and electrical anomalies are located using OBIRCH, achieving spatial mapping between physical defects and electrical faults. When a defect is detected, a synchronous nanoprobe detects parameters such as current, voltage, and resistance in the circuit, and combines this with multimodal detection data to precisely locate the fault location. For example, an X-ray CT scan determines the macroscopic location of the defect, followed by a FIB-SEM microstructural analysis of the defect area. Finally, OBIRCH is used to detect electrical properties, correlating the physical defect with the electrical fault.

[0789] 2. Improve and optimize decision-making plans:

[0790] 1. Quantitative verification of process improvement:

[0791] Welding Process: Finite element simulation (FEM) was used to optimize laser welding parameters, with residual stress distribution and fatigue life of the welded joint used as evaluation indicators. Ultrasonic C-scan technology was introduced to enable non-contact quantitative testing of weld quality. Finite element simulation was used to analyze the effects of different laser welding parameters (such as power, welding speed, and spot diameter) on the residual stress distribution and fatigue life of the welded joint, and the optimal welding parameters were determined.

[0792] Welding Process: Finite element modeling (FEM) was used to optimize resistance brazing parameters, with residual stress distribution and fatigue life of the weld joint used as evaluation indicators. Ultrasonic C-scan technology was introduced to enable non-contact quantitative testing of weld quality. Finite element simulations were used to analyze the effects of different multi-source welding parameters (such as power, welding speed, and spot diameter) (photovoltaic, green power, energy storage, mains, combined power, and nuclear power) on the residual stress distribution and fatigue life of the weld joint, as well as the weld joint area and resistance, to determine the optimal welding parameters.

[0793] Ultrasonic C-scan technology is used to perform non-destructive testing on welded joints to obtain defect information (such as cracks and pores) inside the welded joints and quantify the welding quality.

[0794] Material Replacement: Establish a material performance database to compare the long-term stability of metal film and carbon film resistors under different temperature and humidity conditions. Verify the wear resistance of ceramic dielectric capacitors through accelerated aging tests. The material performance database records the performance parameters of different materials under various environmental conditions (such as resistance change rate and capacitance change rate). Accelerated aging tests simulate the aging process of materials over long-term use, evaluate the wear resistance of ceramic dielectric capacitors, and provide a basis for material replacement.

[0795] Online inspection: A real-time image recognition system based on deep learning is deployed to analyze X-ray and OBIRCH inspection data in milliseconds, enabling automatic defect labeling and grading. Deep learning algorithms are used to train X-ray and OBIRCH inspection images to build a defect recognition model. This model can quickly identify defects in images and automatically label and grade them based on their shape parameters (derived through second-order moment fusion analysis).

[0796] 2. Intelligent quality inspection system:

[0797] Automation of multimodal inspection processes: Develop collaborative control software for inspection equipment to automatically switch inspection modes according to preset rules; X-ray coarse positioning of defect areas: Utilize the penetrating properties of X-rays to perform an overall scan of the sampled components to quickly locate the approximate location of the defect.

[0798] FIB for micro-area sampling and composition analysis: Based on X-ray positioning, FIB is used to prepare micro-area samples in the defect area, and the sample composition is analyzed by electron microscopy to obtain material information of the defect area.

[0799] OBIRCH scanning locates electrical anomalies: Using OBIRCH technology, scan the electrical performance of components, locate electrical anomalies, and determine the impact of defects on electrical performance. TEM obtains crystal structure information: TEM performs high-resolution imaging of defect areas to obtain crystal structure information and conduct in-depth analysis of the defect's microstructure.

[0800] Data fusion decision-making: The weighted average fused second-order moment parameters (major axis k, minor axis b, and direction angle θ) are input into the support vector machine (SVM) classifier to achieve a joint determination of defect shape and severity, and output improvement suggestions (such as rework or scrapping). The fused second-order moment parameters are trained and classified by the SVM classifier, establishing a mapping relationship between defect shape, severity, and improvement suggestions. The corresponding improvement suggestions are automatically generated based on the classification results.

[0801] 3. Enhanced reliability through design improvements:

[0802] Redundancy design optimization: A dynamic redundancy strategy is used to automatically switch to a backup component based on real-time monitoring of the primary component's performance parameters, avoiding metering interruptions during the switching process. By setting up redundant backup components in the circuit and monitoring the primary component's performance parameters (such as resistance and capacitance) in real time, when the primary component's performance drops below a certain threshold, the system automatically switches to the backup component, ensuring the normal operation of the sampling circuit.

[0803] Environmentally Adaptable Design: Components are coated with a nano-scale moisture-proof coating and integrated with micro-temperature sensors, enabling adaptive heat dissipation to regulate operating temperature. The nano-scale moisture-proof coating effectively prevents moisture from entering the components, protecting their electrical performance. The micro-temperature sensor monitors the operating temperature of the components in real time. If the temperature is too high, the adaptive heat dissipation structure automatically activates to adjust the operating temperature, improving the component's environmental adaptability.

[0804] 3. Strengthening the prediction and decision-making model:

[0805] 1. Hybrid modeling for remaining life prediction: This approach combines physical models (such as the Arrhenius equation for wear rate) with data-driven models (LSTM neural networks). It inputs multi-source data (defect size, operating temperature, and voltage fluctuation) and outputs a probability distribution for remaining life. The Arrhenius equation is used to describe the relationship between material wear and temperature, establishing a physical model. Simultaneously, an LSTM neural network is used to learn from historical inspection data to capture the nonlinear patterns of defect development. Multi-source data is fed into the hybrid model, and through the synergistic effect of the physical and data-driven models, a probability distribution for remaining life is output.

[0806] Monte Carlo simulation is used to quantify prediction uncertainty and generate risk warning intervals. Through Monte Carlo simulation, model input parameters are randomly sampled, and remaining life prediction results under different scenarios are simulated. The distribution of prediction results is statistically analyzed to quantify prediction uncertainty and generate risk warning intervals, providing a reference for maintenance decision-making.

[0807] 2. Intelligent maintenance decision support system:

[0808] Grading Maintenance Strategy: Minor defects: Activate an adaptive filtering algorithm to compensate for measurement errors and extend the inspection cycle. Based on the defect severity assessment results, if the defect is considered minor, an adaptive filtering algorithm is used to process the sampled data to compensate for measurement errors and appropriately extend the inspection cycle to reduce maintenance costs.

[0809] Moderate defects: Triggering a reinforcement learning algorithm to optimize operating parameters and schedule planned maintenance. For moderate defects, the reinforcement learning algorithm optimizes the operating parameters of the sampled circuit to improve circuit performance. Based on the remaining life prediction results, planned maintenance is scheduled to prevent failures.

[0810] Severe defects: Linked to the grid dispatch system to achieve "zero power outage" replacement. When a severe defect is detected, the grid dispatch system is immediately linked to adjust the grid operation mode, achieving "zero power outage" replacement of the sampled components without affecting user electricity.

[0811] Dynamic Inventory Management: Based on remaining life predictions, we use multimodal CNN-based image defect diagnosis and tracking, along with Markov decision processes (MDPs) for products and components, to optimize spare parts stocking strategies, balancing inventory costs and downtime risks. The MDP model analyzes inventory costs and downtime risks under different spare parts stocking strategies. Based on remaining life predictions, we dynamically adjust spare parts stockpiles to achieve the optimal balance between inventory costs and downtime risks.

[0812] 3. Full Lifecycle Digital Twin Drive: Build digital twins of meter sampling components, synchronizing detection data, operating status, and maintenance records in real time. Leveraging multimodal detection data and sensor data, a digital twin model of meter sampling components is constructed. This model reflects the actual status of the components in real time, including defect information, operating parameters, and maintenance records.

[0813] Through simulation optimization, we evaluate the impact of different improvement options (such as material replacement and process adjustments) on product lifecycle costs, providing a quantitative basis for long-term decision-making. Within the digital twin model, we simulate different improvement options to evaluate their impact on product performance, reliability, and lifecycle costs. Through comparative analysis, we select the optimal improvement option, providing a scientific basis for long-term decision-making.

[0814] 4. Improve the implementation guarantee system:

[0815] 1. Technical Verification and Standardization: Establish a verification test platform to simulate extreme operating conditions (high temperature, high humidity, and strong electromagnetic interference) to verify the effectiveness of the improvement plan. On the verification test platform, conduct various extreme operating condition tests on the improved meter sampling components, monitor changes in component performance, and evaluate the actual effect of the improvement plan. Develop the "Meter Sampling Component Defect Detection and Treatment Specification" to unify data collection, analysis, and decision-making processes. The specification clearly stipulates the collection method of multimodal detection data, the calculation process of second-order moment analysis, the evaluation criteria for defect severity, and the corresponding decision-making measures to ensure the consistency and standardization of the detection and treatment process.

[0816] 2. Intelligent Inspection System Integration: Develop a multimodal data management platform to seamlessly integrate inspection equipment, analysis algorithms, and decision-making systems. This platform integrates data from X-ray, FIB, TEM, OBIRCH, and other inspection equipment, transmits the data to the analysis algorithm module for processing, and then feeds the analysis results back to the decision-making system, achieving efficient data flow and system collaboration.

[0817] Deploy edge computing nodes to perform data preprocessing and preliminary diagnosis locally, reducing transmission pressure on the cloud. Edge computing nodes can preprocess detection data in real time, extract key feature information, perform preliminary diagnosis, and transmit the diagnosis results and necessary data to the cloud, reducing data transmission volume and improving system response speed.

[0818] 3. Talent and Organizational Support: A multidisciplinary team (materials science, electronic engineering, and data analysis) will be formed to oversee technical research and solution implementation. Members of this multidisciplinary team possess diverse expertise and are able to research and address faults and defects in meter sampling components from multiple perspectives, ensuring the feasibility and effectiveness of improvement plans.

[0819] Conduct AI-assisted inspection training to enhance operators' ability to interpret multimodal data and improve their efficiency in handling exceptions. Through training, operators will master the use of multimodal inspection equipment, the principles and operational procedures of AI analysis algorithms, and be able to accurately interpret multimodal data, promptly detect and handle anomalies, and improve the efficiency and quality of inspection and maintenance work.

[0820] Through the above solution, full-chain closed-loop management from defect detection, fault diagnosis to predictive maintenance can be achieved, significantly improving the reliability and intelligence level of meter sampling components.

[0821] In the case of actual meter sampling components, the weighted average method is used to fuse the second-order moment of continuous function and the second-order moment of discrete data to express the major axis, moment axis, and direction angle of the defect shape to reconstruct the defect shape.

[0822] Background on the Application and Testing of Electricity Meter Sampling Components: Electricity meter sampling components are crucial for accurately measuring electricity consumption. During production and use, defects may occur due to material problems, processing errors such as weld cracks, voids, and cold solder joints, or due to long-term wear and tear. These defects can affect sampling accuracy, leading to inaccurate meter readings.

[0823] For example, the sampling resistor may have local wear or uneven conductive layer, the sampling hard connection may have welding cracks, and the capacitor may have dielectric defects.

[0824] Construction of a theoretical model for calculating the second-order moment of a continuous function: For a typical defect in meter sampling components, such as material loss due to wear on the surface of a sampling resistor or capacitor, its shape on a two-dimensional plane can be approximately represented by a parabolic function based on the principles of material physics and electricity.

[0825] Let function f(x, y) = AB[(x-x0) 2 +(y-y0) 2 ] (within the defect area), where (x0, y0) is a parameter related to the resistor material and the degree of wear, and is the wear center position. According to the continuous function second-order moment model, the moment of inertia is obtained by integration (the integration area is the wear and crack area on the resistor surface, and the circular area wear radius is r):

[0826] (Function x0 is symmetrical to function y0, and the direction angle is both 0) For the calculation of equivalent ellipse parameters (continuous function part), refer to step 21.

[0827] Example of multi-layer chip defect analysis:

[0828] Background of multi-layer chip manufacturing and testing: During the production process of multi-layer chips, various defects may occur due to complex process steps such as photolithography, etching, deposition, etc. These defects may exist between different layers or within layers, which have a serious impact on the performance of the chip. In order to detect these defects, a variety of technologies are usually used, including electron microscope imaging, X-ray detection, etc. These detection methods can provide theoretical models in the form of continuous functions of the defects and discrete data points. Model establishment: For a certain type of conductive layer short-circuit defects in multi-layer chips, based on electromagnetic field theory and circuit models, its shape on a two-dimensional plane can be approximately represented by a function that is a combination of a log-normal distribution and a trigonometric function. For example, where A is a coefficient related to the defect current density, (x0, y) is the estimated position of the defect center on the x-axis, and k and b are parameters related to the defect range.

[0829] Weight determination: Based on the accuracy of the theoretical model (based on previous research and verification of similar defects) and the reliability evaluation of the electron microscope data, the weight of the second-order moment of the continuous function is determined to be 0.4, and the weight of the second-order moment of the discrete data is determined to be 0.6.

[0830] Through the steps of the present invention, the second-order moment of the continuous function and the second-order moment of discrete data are fused using the weighted average method to obtain the major axis, minor axis and directional angle parameters that can reproduce the shape of the chip defect, thereby more accurately describing the defects in the chip and providing a basis for subsequent chip repair or quality improvement.

[0831] For a better understanding Figure 5 and Figure 6 The above invention features are used as an application for smart meter sampling component defect tracking and decision-making, highlighting the overall picture and technical effects of the present invention as follows:

[0832] 1. Deepening of the multimodal data collection system:

[0833] Defect location image acquisition scope: Acquire defect location images of smart meter sampling components (such as resistors, capacitors, and welding points) under different production batches, process stages (photolithography, welding, and packaging), and detection equipment (X-ray, electron microscope, and OBIRCH).

[0834] Data collection scope covers:

[0835] Image data: 3D structure from X-ray tomography, thermal radiation distribution from thermal infrared imaging, internal structure from ultrasonic imaging, and microscopic crack images from FIB-SEM;

[0836] Electrical data: current anomalies located by OBIRCH, resistance / voltage fluctuation parameters collected by nanoprobes;

[0837] Working condition data: real-time load fluctuation, ambient temperature (-20℃~70℃) and humidity data.

[0838] Fundamentals of multimodal data definition and fusion.

[0839] Three-dimensional structural information: X-ray CT scanning is used to obtain the spatial distribution of defects such as voids and interlayer short circuits inside components;

[0840] Thermal radiation information: Abnormally hot areas in thermal infrared images (such as sudden temperature rise when a resistor is overloaded) can map electrical faults;

[0841] Electrical parameter data: Sudden changes in contact resistance detected by the nanoprobe (such as a cold solder joint causing a resistance fluctuation of >10%) are directly associated with soldering defects;

[0842] Operating condition data: Load fluctuations (such as sudden increases and decreases in current) and changes in ambient temperature will accelerate defect evolution and need to be collected synchronously.

[0843] 2. Data preprocessing and feature fusion process:

[0844] Timestamp synchronization and standardization:

[0845] X-ray images (acquisition interval 10ms), electrical parameters (sampling frequency 1kHz), and operating condition data (5-minute sliding window) are aligned by timestamp to construct a temporally and spatially consistent dataset.

[0846] The electrical parameters such as resistance value, current, water flow, pressure, gas flow, temperature, etc. are normalized using the formula x_normalized = (x-x_min) / (x_max-x_min) to eliminate the dimension effect.

[0847] Feature extraction and position encoding:

[0848] Defect location code:

[0849] For the defect center coordinates (x_d, y_d), the image is divided into m×m grids and converted into length m by one-hot encoding 2 For example, when m=16, the vector dimension is 256;

[0850] Image feature extraction:

[0851] ResNet-18 is used to extract edge / texture features and calculate contrast C (grayscale difference between defect and background), correlation R (texture continuity), energy E (grayscale distribution uniformity), and entropy H (information uncertainty);

[0852] The defect entropy value D is calculated through the grayscale histogram and reflects the severity of the defect (e.g., the entropy value of a deep crack is higher than that of surface wear);

[0853] Electrical and Timing Characteristics Conversion:

[0854] Perform Fourier transform on current / voltage data to extract frequency domain features (e.g., abnormal 100Hz harmonic component indicates poor contact);

[0855] The statistical characteristics of working condition data (temperature, load) such as mean and variance are calculated through a sliding window.

[0856] Weight distribution and multimodal fusion:

[0857] Dynamically adjust feature weights based on defect type:

[0858] Welding crack: contrast C (25% to 30%) + correlation R (20% to 25%) + position texture P (25% to 30%);

[0859] Resistance wear: entropy H (35% to 40%) + defect entropy D (15% to 20%) + electrical parameters (20% to 25%);

[0860] Fusion formula: F = w1·C+w2·R+w3·E+w4·H+w5·D+w6·P, where the sum of the weights is 1.

[0861] 3. Comparison of CNN and Transformer Fusion Strategies:

[0862] Solution 1: Multimodal fusion CNN (suitable for simple defects).

[0863] Early fusion (feature level): concatenates image features (256 dimensions), electrical features (16 dimensions), and operating condition features (8 dimensions) into a 280-dimensional unified vector;

[0864] Network architecture:

[0865] Input layer → 3 convolution layers (kernel size 3×3, 3×3, 5×5) → pooling layer → 2 fully connected layers (number of nodes 128→n) → Softmax output;

[0866] The activation function uses RReLU to alleviate the gradient disappearance problem;

[0867] Loss function: cross entropy loss L = -∑yi·log(pi), trained with Adam optimizer (β1 = 0.9, β2 = 0.999).

[0868] Solution 2: Multimodal fusion Transformer (suitable for complex defects).

[0869] Early fusion (feature level): Direct cross-modal fusion through the self-attention mechanism, without feature splicing, capturing semantic associations (e.g., "high temperature + abnormal resistance" indicates material aging);

[0870] Decision-level fusion:

[0871] Parallel processing of multi-channel CNN features (such as X-ray images + thermal infrared images);

[0872] Cross-attention mechanism to fuse features from different modalities, such as spatial mapping of OBIRCH electrical anomalies and FIB microstructure images;

[0873] Advantages: It has a stronger ability to model long-term dependencies of time series data (such as defect expansion process).

[0874] 4. Hybrid modeling and defect shape analysis, including:

[0875] Second-order rectangular modeling process:

[0876] Second-order moment of discrete data: extract n defect edge points (xi,yi) from the electron microscope image and calculate the contour perimeter Cp = ∑√[(xi+1-xi) 2 +(yi+1-yi) 2 ], circularity Rc=4πA / Cp 2 (A is the defect area);

[0877] Second moment of a continuous function:

[0878] For resistance wear defects, fitting parabolic function f(x,y)=AB[(x-x0) 2 +(y-y0) 2 ], calculate the moment of inertia by integration Ixx^c=πA / 4·r^4-πB / 6·r^6;

[0879] Weighted fusion:

[0880] The weight a = 0.3 (continuous function) + 0.7 (discrete data) is used to calculate the fused moment of inertia: Ixx = a·Ixx^c + (1-a)·Ixx^d, and then the major axis a, minor axis b, and direction angle θ of the equivalent ellipse are obtained;

[0881] Defect quantification parameters:

[0882] Length = major axis a, width = minor axis b, area = π·a·b, volume = π·a·b·h (h is the defect depth).

[0883] 5. Decision-making level applications and improvement plans:

[0884] Defect diagnosis and tracking:

[0885] Improved U-Net model: The encoder introduces residual connections, the decoder adds an attention mechanism, and segmentation of defect areas (such as binary images of weld cracks);

[0886] Tracking function T(F,t): By fusing features F at different times, the defect position offset (such as crack growth rate 0.1mm / month) and morphological changes (long axis growth rate) are calculated.

[0887] Process improvement decisions:

[0888] Welding process optimization:

[0889] Based on finite element simulation, the laser welding power was adjusted from 20W to 25W, which reduced the joint residual stress (from 300MPa to 150MPa), and the weld area increased by 20% as verified by ultrasonic C-scan;

[0890] Material replacement strategy:

[0891] Replacing carbon film resistors with metal film resistors reduces resistance fluctuation from ±5% to ±1% at high temperatures (70°C). Accelerated aging tests (85°C / 85% RH, 1000 hours) verify that the lifespan is extended by three times.

[0892] Smart maintenance strategy:

[0893] Minor defects (such as surface wear): The adaptive filtering algorithm is activated to compensate for measurement errors, and the inspection cycle is extended from 1 month to 3 months;

[0894] Severe defects (such as interlayer short circuits): Link the power grid dispatching system to achieve "zero power outage" replacement, and use digital twin simulation to measure the interruption time of the switching process to <50ms.

[0895] VI. Key Innovations and Implementation Guarantees:

[0896] Technological innovation:

[0897] Cross-modal complementarity: combining image geometric features (such as crack length) with electrical features (resistance mutation), the diagnostic accuracy is increased to 98.5%;

[0898] Hybrid modeling: discrete point cloud (edge ​​detection) and continuous function (physical model) are integrated to achieve a defect shape description error of <5%;

[0899] Multi-stage fusion: Simple defects are fused using CNN early stage, while complex defects are fused using Transformer decision-level, improving computational efficiency by 40%.

[0900] Implementation of the guarantee system:

[0901] Establish an extreme operating condition test platform (-40°C to 125°C, humidity 0-100%) to verify the reliability of the improvement plan;

[0902] Formulate the "Smart Meter Defect Detection Specification" to standardize multimodal data acquisition frequency (10fps for images and 1kHz for electrical parameters) and analysis processes;

[0903] An interdisciplinary team (materials science + AI algorithms + power engineering) collaborates to visualize the model training process through TensorBoard and optimize parameters in real time.

[0904] The present invention also has the following outstanding features:

[0905] A multimodal CNN / Transformer image defect diagnosis and tracking decision method, the method comprising:

[0906] Multimodal data acquisition: Acquires multiple types of data, including 3D structural information (X-ray CT), thermal radiation information (infrared imaging), and electrical anomaly point data (OBIRCH), of smart meter sampling components, covering different production batches and process stages.

[0907] Data preprocessing and alignment: Constructing a temporally and spatially consistent multimodal dataset through timestamp synchronization, image enhancement, and feature normalization;

[0908] Feature extraction and conversion: Position encoding of defect locations, combined with CNN to extract image features (contrast, correlation, etc.), and convert electrical parameters into frequency domain features;

[0909] Multimodal fusion strategy:

[0910] CNN path: Early fusion and splicing of feature vectors, suitable for simple defects;

[0911] Transformer path: The self-attention mechanism directly integrates across modalities, making it suitable for temporal correlation analysis of complex defects.

[0912] Hybrid modeling and shape analysis: Based on the weighted fusion of the second-order moment of discrete data and the second-order moment of continuous function, the major and minor axis parameters of the defect are calculated to quantify the shape characteristics;

[0913] Decision-level application: Defect segmentation is achieved by improving U-Net, combining the tracking function T(F,t) with process optimization strategies to form a "detection-diagnosis-decision-making" closed loop.

[0914] As one or more specific application embodiments of the present invention, combined with Figure 6 to Figure 1 1 The multimodal CNN / Transformer-based image defect diagnosis and tracking decision method provided by the present invention is further described in detail as follows:

[0915] Example 1: Figure 7As shown, the present invention is used to identify and detect the TVS diode (SMBJ10CA) of the smart meter, detect the location of the welding void through X-ray, measure the defect shape, size, area, and the total area of ​​the welding void is greater than 5%, which is a schematic diagram of the defect criterion of exceeding the standard; the present invention is used to identify, diagnose and track unqualified products through X-ray detection imaging, and provide a specific basis for the repair welding difference for the welding process; this can re-weld unqualified products online into qualified products during the welding process.

[0916] As shown in Figure 8(a), Figure 8(b), Figure 8(c) and Figure 8(d), Figure 7 Schematic diagram of continuous tracking of defective products (void rate 5.1%). The devices in Figures 8(a), 8(b), 8(c) and 8(d) are all TVS diodes (SMBJ10CA), referred to as sample 107#. Figure 8(a) is the front X-ray image of sample 107#, Figure 8(b) is the locally enlarged X-ray image of Figure 8(a), and there are voids in the contact area (total void rate is 5.1%). Figure 8(c) is the side X-ray image of sample 107#, and Figure 8(d) is the locally enlarged X-ray image of Figure 8(c). The shape, size, and area of ​​TVS diode (SMBJ10CA) voids in different time periods are continuously tracked, and the process capability of the TVS diode (SMBJ10CA) is simultaneously calculated (the mean, upper and lower limits, CP, CPL, and CPK of the process capability are calculated). The process capability prediction of void development also determines the consistency of the TVS diode (SMBJ10CA) (the process capability of VBR minimum breakdown voltage and IR reverse operating voltage is insufficient, and a large number of specifications may be unqualified. This provides a basis for improving the performance of process production products and the consistency of VBR minimum breakdown voltage and IR reverse operating parameters).

[0917] The assessment criteria for the process capability index and void rate of TVS diode SMBJ10CA are as follows: Process capability assessment of VBR(+). The qualified criteria are: Cp ≥ 1.33 and Cpk ≥ 1.0 are usually required. Cp measures the potential production capability of the process. Cp ≥ 1.33 means that the natural fluctuation of the process is small and can better meet the specification requirements. Cpk takes into account the center offset of the process. Cpk ≥ 1.0 means that the process still has a high ability to meet the specification limit while taking the offset into account, and the product consistency is good. Failure criteria: When Cp < 1.33 or Cpk < 1.0, the process capability of VBR(+) is considered unqualified, that is, there is a large variation or center offset in the process, which leads to unstable breakdown voltage characteristics of the product and may not be able to effectively protect the circuit.

[0918] The IR(+) process capability assessment has the following acceptance criteria: CP ≥ 1.0, CPL ≥ 0.8, and CPK ≥ 0.8 are generally expected. CP reflects the overall capability of the process, and CP ≥ 1.0 indicates that the process has a certain degree of stability and reliability. CPL focuses on the lower limit capability. CPL ≥ 0.8 means that the process has good control capabilities in terms of the lower limit and can ensure that the reverse leakage current does not exceed the specified lower limit value. CPK comprehensively considers the offset. CPK ≥ 0.8 indicates that the overall capability of the process is still acceptable when considering the center offset, and the reverse leakage current performance of the product is relatively stable. Failure criteria: If CP < 1.0, CPL < 0.8, or CPK < 0.8, the IR(+) process capability is judged to be unqualified, that is, the process may have large fluctuations or center offsets, resulting in an increased risk of reverse leakage current exceeding the specification limit, affecting the normal operation of the TVS diode.

[0919] Regarding the void ratio assessment, the acceptance standard is: In the electronic component industry, the void ratio is generally required to be controlled below 5%, and this is no exception for the TVS diode SMBJ10CA. A lower void ratio helps maintain the internal structural integrity and performance stability of the diode, reducing electrical performance degradation and reliability issues caused by voids. The failure standard is: A void ratio exceeding 5% is generally considered unacceptable. Excessively high void ratios may make the diode more susceptible to localized overheating and electric field concentration during operation, further affecting key performance parameters such as breakdown voltage and reverse leakage current, reducing product lifespan and reliability.

[0920] Example 2: Figure 9 As shown, the present invention is used to identify, detect, and measure the location of solder voids in the rectifier diode (M7) of a smart meter, and track the development of defect shapes, sizes, and areas from qualified products. The total area of ​​the solder voids is less than 2.4%, and does not exceed the standard defect criteria. The location indicated by the arrow is the location of the void. After identifying and detecting the location of the solder voids in the rectifier diode (M7), and measuring the shape, size, and area of ​​the voids, the product is judged to be qualified according to the standard and no re-soldering is required. At the same time, by continuously tracking the shape, size, and area of ​​the voids at different time periods, the process capability of the rectifier diode (M7) is simultaneously calculated (the mean, upper and lower limits, CP, CPL, and CPK of the process capability are calculated). Based on the process capability prediction of the void development, the consistency of the rectifier diode (M7) is determined (the VBR minimum breakdown voltage and IR reverse operating voltage process capabilities are sufficient, and qualified products will not fail to meet the specifications and can be continuously produced. If a qualified CPK exceeds 1.67, the performance of the product produced by the process is predicted in advance, providing a basis for improving the consistency of the VBR minimum breakdown voltage and IR reverse operating parameters).

[0921] Example 3: Figure 10As shown, the present invention is used to identify, detect and measure the location of welding cracks in the current transformer of a smart meter, track the shape, size and area of ​​crack defects that develop from unqualified products, and the total area of ​​welding cracks is less than 2.8%. It is a schematic diagram of the process of judging the defect exceeding the standard. The left figure is an X-ray image of the side of the current transformer, and the right figure is a partial enlarged image of the left figure. After identifying the welding cracks of the current transformer, detecting the location, and measuring the shape, size and area of ​​the cracks, the product is judged as unqualified according to the standard and isolation is recommended. At the same time, by continuously tracking the shape, size and area of ​​cracks in different time periods, the process capability of the current transformer welding crack is synchronously calculated (the mean, upper and lower limits, CP, CPL, CPK of the calculated process capability are calculated). The crack development process capability is predicted and judged. The current transformer parameter consistency process capability is sufficient. Even if the specifications are unqualified, qualified products can be continuously produced. Once the qualified CPK is less than 1.67, it is predicted in advance that the performance of the process production product is unqualified. The cracks in the signal line and other adverse effects have poor conductive performance and safety and reliability. It provides a basis for improving the consistency of its working parameters, equipment maintenance and updating, equipment parameter adjustment, and raw material and process optimization.

[0922] Example 4: As shown in Figures 11(a), 11(b) and 11(c), a schematic diagram of the defect judgment process for applying the present invention to identify, detect and measure the position of solder voids between the chip and leads, terminal and substrate of a smart meter, detect and track the development of defect shape, size and area of ​​qualified products, and the total area of ​​solder voids is less than 2.4%, which does not exceed the standard; after identifying and detecting the position of solder voids between the chip and leads, terminal and substrate, and measuring the shape, size and area of ​​the voids, it is judged as a qualified product according to the standard and no re-soldering is recommended. By continuously tracking the shape, size, and area of ​​voids over different time periods, the chip-to-lead, terminal, and substrate soldering void process capability (including the mean, upper and lower limits, CP, CPL, and CPK) is simultaneously calculated. This process capability prediction of void development indicates sufficient process capability for parameter consistency in chip-to-lead, terminal, and substrate soldering, preventing substandard specifications and enabling consistent production of qualified products. If a qualified CPK value exceeds 1.33, this indicates early prediction of product performance failure and provides a basis for improving operating parameter consistency. Generally speaking, in the chip soldering field, a void rate exceeding 5%-10% may be a cause for concern. A void rate of approximately 20%-30% is likely to have a significant negative impact on the chip's electrical, mechanical, and thermal performance, typically resulting in a failure. When a defective chip is detected by X-ray, the chip material, the thickness of the gate layer of the transistor, the cracks and voids of the metal interconnection are displayed by TEM detection, the FIB realizes fast and accurate positioning of defects, the nanoprobe detects the source, drain, gate of the transistor and defects below 10 nanometers or key parameters such as metal interconnection current, voltage, resistance, voids, cracks, atomic-level failure, etc., and detects saturation current failure, leakage, open circuit, short circuit and other fault defects. It provides a basis for flexible process improvement measures for fault defects in different layers and surfaces, and achieves non-destructive detection and diagnosis of circuit devices in the chip.

[0923] Example 6, such as Figure 12The figure shows a schematic diagram of detecting bumps, voids, cracks, and insufficient soldering in the PCB, crystal oscillator, and seven major chips used in smart meters. Faults and defects such as bumps, voids, cracks, and insufficient soldering in the seven major chips used in smart meters—the metering chip, main control chip (MCU), communication chip, security and encryption chip (ESAM chip), clock chip, memory chip, RF front-end chip, and crystal oscillator—can affect metering performance, accuracy, reliability, stability, and lifespan. The present invention's real-time detection method is not limited to X-rays (thermal infrared and ultrasonic testing can also be used). A multimodal CNN-based approach, combined with electron microscopy, FIB, TEM, EMM, and OBIRCH, can be used to detect internal chip circuit defects. First, X-rays (thermal infrared and ultrasonic testing can also be used) penetrate the chip to obtain images of the internal structure, revealing macroscopic defect information such as circuit breaks and short circuit locations. The multimodal CNN performs high-precision analysis of chip surface and subsurface images, identifying subtle fault features such as tiny cracks, scratches, bumps, cracks, voids, and insufficient soldering, and then detects, measures, and tracks them. When the two are integrated, the X-ray image (thermal infrared and ultrasonic testing can also be used) and the chip surface image are input into the multimodal CNN model of the present invention. After training with a large number of samples, the model can comprehensively determine the type and severity of the defect fault. For example, it can accurately distinguish between circuit defects caused by the manufacturing process and physical damage during use. Based on the results, it can make subsequent repair or replacement decisions, thus achieving comprehensive and accurate detection and diagnosis of chip defects. After long-term online testing with sufficient sample preparation of 6,000 cases and meticulous training, the accuracy rate of detecting and measuring voids and cracks reached over 99.9%. However, for the detection of bumps and insufficient soldering with fewer samples, the accuracy rate dropped to around 98.5%.

[0924] The key innovations in the embodiments of the present invention include:

[0925] Cross-modal feature complementarity: Combining image geometric features with electrical timing features to improve defect diagnosis accuracy.

[0926] Hybrid modeling method: Combine discrete point cloud with continuous function modeling to accurately describe the defect shape.

[0927] Multi-stage fusion strategy: select early or late fusion based on data characteristics, balancing computational efficiency and information retention.

[0928] This fusion solution, in its practical application, achieves a 93.5% real-time synchronization rate and 99.9% conversion efficiency for photovoltaic and green power control, significantly improving the fault diagnosis capabilities of photovoltaic systems. Multi-modal, multi-parameter fusion can significantly enhance intelligent welding energy (weld heat temperature) balance regulation.

[0929] The multimodal data fusion processing described in the present invention has significant advantages in the fields of photovoltaic system fault diagnosis and industrial quality inspection:

[0930] Improve diagnostic accuracy and robustness:

[0931] Information complementarity: Combining image data (such as X-ray tomography and thermal infrared imaging) with electrical parameters (current / voltage) enables multi-dimensional defect detection. For example, temperature anomalies detected by thermal imaging can be corroborated with electrical parameter anomalies, reducing false alarm rates.

[0932] Interference resistance: Multimodal data fusion maintains diagnostic performance by leveraging data from other modalities when a single data source is subject to interference (e.g., image blur or electrical noise). Experiments have shown that this fusion method can reduce harmonic detection error from 20% to below 2.23%.

[0933] Fusion algorithm performance: data fusion accuracy ≥ 98%.

[0934] Model prediction accuracy: welding parameter (temperature, energy) prediction error ≤ ±3%, abnormal state recognition accuracy ≥ 99%.

[0935] Enhanced fault location and tracing capabilities:

[0936] Spatial-temporal correlation: Using timestamp alignment technology, 3D structural information is correlated with real-time operating condition data (load fluctuations, ambient temperature) to achieve spatiotemporal fault location. For example, combining internal structural information from ultrasonic imaging with temperature data can pinpoint the physical location of a hotspot.

[0937] Causal relationship mining: The fusion of electrical parameters and microstructural images can reveal the causal relationship between process defects (such as weak solder joints) and electrical performance degradation, supporting root cause analysis.

[0938] Optimize system real-time performance and efficiency:

[0939] Hybrid modeling acceleration: Using a hybrid modeling method that combines the second-order moment of discrete data and the second-order moment of continuous functions, the computational efficiency is increased by more than 50% while ensuring the accuracy of shape description.

[0940] Intelligent decision support: Dynamically select the optimal feature combination through early / late fusion strategies, reduce redundant calculations, and achieve a system real-time performance of 93.5%, meeting the rapid response requirements of smart grids.

[0941] Support standardization and compatibility:

[0942] Multi-protocol adaptation: Integrates distributed power access unit data and equipment operation data that comply with the T / CIMA 0142 standard to achieve collaborative diagnosis of cross-vendor equipment.

[0943] Data interoperability: Unified feature vector representation enables different modal data to be processed in the same model, reducing system integration costs and improving compatibility.

[0944] Expand application scenarios and value:

[0945] Full life cycle management: Multi-scale data fusion from microstructure to macro performance supports full life cycle quality monitoring of products from production to retirement.

[0946] Predictive maintenance: By integrating historical fault data with real-time operation data, potential faults can be predicted in advance, reducing downtime. For example, unexplained switching losses can be reduced by 90%.

[0947] In practical applications, in photovoltaic systems, multimodal fusion technology improves the inverter's low voltage ride-through capability by 30%, the harmonic suppression capability by 85%, and the system conversion efficiency reaches 99.9%, which is significantly better than the traditional single-modal method. This technology has become the core support for smart grid construction and industrial intelligent upgrades.

[0948] The multimodal CNN / Transformer image defect diagnosis, tracking and decision-making method provided in this embodiment solves the problem of deep learning-based X-ray (thermal infrared and ultrasonic detection can also be used) smart meters, metering automation terminals, mutual inductors, batteries, transformers and wires, power electronic components, multi-layer chips, transistors, mutual inductors and PCB boards, component welding connections and processes, existing welding structures and SOP, QFP, BGA, CSP, IGBT and other semiconductor packages, industrial devices, internal discontinuities in batteries (microcracks, porosity, etc.), cracks, voids, delamination, deformation, separation, bridging, open circuits in metal layers (wires, cores), fractures, dust, foreign matter and impurities, metal particles, looseness, dislocation, and damage to insulation layers. The automated detection method relies on large amounts of labeled data sets, long training times, and high computing resources, making it suitable for industrial scenarios where data is scarce and real-time processing is required.

[0949] In this embodiment, a multimodal CNN / Transformer image defect diagnosis, tracking and decision-making device is also provided. The device is used to implement the above-mentioned embodiments and preferred implementation methods. The details that have been explained will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0950] This embodiment provides a multimodal CNN / Transformer image defect diagnosis and tracking decision-making device, such as Figure 13 As shown, including:

[0951] The multimodal data acquisition module 1301 is used to obtain defect location images of multiple multimodal data of the product to be diagnosed. The multiple defect location images are defect location images containing the above multimodal data taken from different production batches, different process stages, and different equipment; the multimodal data includes image data, electrical data, and operating condition data.

[0952] The data preprocessing and data alignment module 1302 is used to perform data preprocessing and data alignment on the defect position images of the plurality of multimodal data.

[0953] The feature extraction module 1303 is used to extract features from defect location images of multiple multimodal data using a position encoding method based on the defect location image of each image type data, and obtain the image features after position encoding of each defect location image. At the same time, feature extraction is performed on electrical data and operating condition data respectively to obtain electrical features and operating condition features; image features include contrast features, correlation features, energy features, entropy features, defect entropy value features and position texture parameter features.

[0954] The multimodal feature fusion module 1304 is used to perform feature weight fusion on each contrast feature, correlation feature, energy feature, entropy feature, defect entropy value feature and position texture parameter feature as an image feature based on a preset feature weight distribution scheme, and to splice the image features, electrical features and working condition features into a unified vector to obtain multiple multimodal fusion feature vectors.

[0955] The multimodal detection model construction and detection module 1305 is used to obtain a single-process image defect detection model by performing independent sub-model training and model evaluation on the convolutional neural network model based on multiple multimodal fusion feature vectors, and to obtain a multi-process image defect detection model by performing model training and model evaluation on the Transformer network model based on multiple multimodal fusion feature vectors; obtain multiple actual product images to be diagnosed and input them into the single-process image defect detection model and / or the multi-process image defect detection model for image defect diagnosis, and obtain a detection result of whether the corresponding area of ​​the product image has defects or no defects.

[0956] The hybrid modeling and analysis module 1306 is used to perform hybrid modeling and analysis on the defective product image output by the image defect detection model input to be diagnosed to obtain hybrid modeling image defects. The hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis.

[0957] The tracking decision module 1307 is used to obtain defect detection results by using the improved U-Net model for hybrid modeling image defects, realize defect area segmentation and positioning, and then track the position of the image defect area, make decisions based on the defect tracking and fault tracing results, and optimize production process parameters.

[0958] In some optional implementations, the data preprocessing and data alignment module 1302 includes:

[0959] A preprocessing unit is used to perform data cleaning, data labeling, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization on defect location images in sequence;

[0960] Among them, data cleaning is used to remove noise data, blurred defect location images and incomplete defect location images in defect location images; data labeling is to use a combination of manual labeling and semi-automatic labeling to label the defect location and defect type; image usage classification is to divide multimodal data into image data, electrical data and working condition data according to the production batch, process stage and equipment type to which the image belongs and the multimodal data type; image pixel normalization is to normalize the pixel values ​​of image data to a preset range to unify the scale of image data.

[0961] The data alignment and standardization unit is used to align, fuse and standardize image data, electrical data and working condition data according to timestamps to build a temporally and spatially consistent data set.

[0962] In some optional implementations, the operating condition data includes load fluctuation and ambient temperature; the feature extraction module 1303 includes:

[0963] The edge feature and texture feature extraction unit is used to extract the basic edge features and texture features of the defect position image using a CNN shallow network.

[0964] An image feature extraction unit is used to determine the defect center position of each defect location image based on edge features and texture features; obtain the two-dimensional coordinate value of each defect center position and discretize each two-dimensional coordinate value; convert each discretized coordinate value into a feature vector through one-hot encoding, and use the feature vector as the image feature after the position encoding of each defect location image.

[0965] The electrical feature and operating condition feature extraction unit is used to convert electrical data into time series and extract frequency domain features as electrical features through Fourier transform; it performs 5-minute sliding window sampling on the real-time data of load fluctuations and ambient temperature in the operating condition data to extract operating condition features.

[0966] In some optional implementations, the multimodal feature fusion module 1304 includes:

[0967] A feature weight allocation scheme setting unit is used to allocate a contrast feature weight of 20% to 30%, a correlation feature weight of 10% to 25%, an energy feature weight of 10% to 25%, an entropy feature weight of 30% to 40%, a defect entropy value feature weight of 10% to 20%, and a position texture parameter feature weight of 10% to 30%, and the sum of the contrast feature weight, the correlation feature weight, the energy feature weight, the entropy feature weight, the defect entropy value feature weight, and the position texture parameter feature weight is equal to 1.

[0968] The multimodal fusion feature vector calculation unit is used to respectively calculate the first product corresponding to each contrast feature and the contrast feature weight, the second product corresponding to each correlation feature and the correlation feature weight, the third product corresponding to each energy feature and the energy feature weight, the fourth product corresponding to each entropy feature and the entropy feature weight, the fifth product corresponding to each defect entropy value feature and the defect entropy value feature weight, and the sixth product corresponding to each position texture parameter feature and the position texture parameter feature weight, and add each corresponding first product, second product, third product, fourth product, fifth product and sixth product to obtain multiple multimodal fusion feature vectors.

[0969] In some optional embodiments, the convolutional neural network model includes an input layer, multiple convolutional layers and pooling layers, multiple fully connected layers, and an output layer. The multimodal detection model construction and detection module 1305 includes:

[0970] The convolutional neural network model initialization unit transmits multiple multimodal fusion feature vectors through the input layer to multiple convolution layers and pooling layers for deep learning. Among them, multiple convolution layers use convolution kernels to perform convolution operations on the multimodal fusion feature vectors to extract local features, and the pooling layer obtains multiple key features through downsampling operations; multiple fully connected layers are used to integrate multiple key features, convert the extracted features from local information into global information, and map them to the output probability distribution through the activation function; the output layer is used to convert the probability distribution mapped to the output into multiple fused probability distribution images and output them. The value of each pixel in the fused probability distribution image represents the probability of a defect at that location.

[0971] A single-process image defect detection model construction unit is used to divide multiple fused probability distribution images into a training set, a validation set, and a test set according to a preset ratio; the training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weightedly fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and an evaluation result is output. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single-process image defect detection model.

[0972] A training constraint unit is used to calculate the loss based on a preset loss function during training, and use the Adam optimizer for backpropagation to adjust the weights and biases of the convolutional neural network model, and adjust the parameters of the multiple convolutional layers, pooling layers, and multiple fully connected layers based on the validation set to obtain the convolutional neural network model to be evaluated; the Adam optimizer combines the ideas of the momentum method and the Adagrad optimization algorithm, dynamically adjusts the learning rate of each parameter in the model based on the first-order moment estimator and the second-order moment estimator of the gradient, and corrects the first-order moment estimator and the second-order moment estimator through the deviation.

[0973] The multi-process image defect detection model construction unit is used for the early fusion of the image defect fusion path of the multimodal fusion Transformer. It uses the self-attention mechanism to effectively capture the semantic associations between different modal features and directly perform cross-modal early fusion of multimodal features. The cross-attention mechanism is used to parallelly process the local features extracted by multi-channel convolutional neural networks from different sources to fuse the complex defect feature detection model. The complex defect feature detection model and the Transformer processing time-series correlation detection model form a combined detection model. The image to be diagnosed is input into the combined detection model to output the final defect diagnosis result.

[0974] The defect diagnosis unit is used to perform image defect diagnosis on multiple multi-layer chip images input into the image defect detection model using a determination function. The determination function is expressed by the following formula:

[0975]

[0976] Here, G(F)=1 indicates that a defect is detected in the corresponding area of ​​the image, G(F)=0 indicates that there is no defect, and ifsatisfies certain defect condition indicates that a preset defect condition is met.

[0977] In some optional embodiments, the single process image defect detection model building unit includes:

[0978] The independent sub-model training sub-unit is used to input the unified vector of defect features of the multimodal fusion early fusion image into the convolutional neural network for later fusion; the convolutional neural network model is trained to specifically process image data to extract defect features in the image; the long short-term memory network model is trained to process time-series electrical data to capture the law of change of electrical data over time; the multi-layer perceptron model is trained to process operating condition data to analyze the impact of operating conditions on defects.

[0979] The defect diagnosis subunit is used to perform weighted fusion on the outputs of the independent sub-models obtained from the above training to obtain the detection model to be evaluated, input the test set into the detection model to be evaluated, and when the evaluation result meets the preset threshold range, use the detection model to be evaluated corresponding to the evaluation result as the single process image defect detection model, and use the evaluation result as the final defect diagnosis result.

[0980] In some optional embodiments, the multimodal CNN / Transformer image defect diagnosis, tracking and decision-making device further includes:

[0981] The visualization display module is used to use visualization tools to visualize the changes in loss and evaluation results during the training process of the convolutional neural network model, the key features deeply learned by multiple convolutional layers and pooling layers, and the feature integration and mapping of multiple fully connected layers to the probability distribution of the output.

[0982] In some optional implementations, the hybrid modeling and analysis module 1306 includes:

[0983] The geometric feature calculation unit is used to calculate the geometric features of the image defect area contour and determine the preliminary defect area shape of the image; the geometric features of the preliminary defect area shape of the image include the contour perimeter, the contour convex hull perimeter and the shape factor.

[0984] The shape hybrid modeling unit is used to perform shape hybrid modeling based on the shape of the preliminary defect area of ​​the image using the second-order moment of discrete data and the second-order moment of continuous function to obtain a mathematical model for measuring the shape of the defect area of ​​the image.

[0985] The defect measurement unit is used to measure the defect degree of the image defect area based on the image defect area shape measurement mathematical model, and obtain the final shape, width, length, area and volume of the image defect area.

[0986] The diagnosis and grading unit is used to diagnose and grade the severity level of image defects based on the final shape, width, length, area and volume of the image defect area.

[0987] In some optional embodiments, the shape blending modeling unit includes:

[0988] The second-order moment calculation subunit is used to determine the discrete data second-order moment and the continuous function second-order moment of the image defect area based on the shape of the preliminary defect area of ​​the image.

[0989] A fusion modeling unit is used to fuse the discrete data second-order moment and the continuous function second-order moment of the image defect area to obtain a mathematical model for shape measurement of the image defect area. The first moment of inertia is calculated based on the discrete data second-order moment, and the first equivalent ellipse parameters are calculated based on the first moment of inertia.

[0990] The second moment of inertia is calculated based on the second-order moment of the continuous function, and the second equivalent ellipse parameters are calculated based on the second moment of inertia. The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent ellipse parameters and the second equivalent ellipse parameters, respectively, to obtain a mathematical model for shape measurement of the image defect area.

[0991] The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, and to fuse the first equivalent ellipse parameters and the second equivalent ellipse parameters, respectively, to obtain a mathematical model for shape measurement of the image defect area, including:

[0992] Set the weight of the second-order moment of the continuous function, and calculate the weight of the second-order moment of the discrete function based on the weight of the second-order moment of the continuous function;

[0993] Based on the second-order moment weight of the continuous function and the second-order moment weight of the discrete function, the first moment of inertia and the second moment of inertia are fused by the weighted average method to obtain the fused second-order moment; the first equivalent ellipse parameters and the second equivalent ellipse parameters are fused to obtain the fused equivalent ellipse parameters;

[0994] The mathematical model for measuring the shape of the image defect area is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters;

[0995] The fused second-order moment formula is as follows:

[0996] M fusion =aM c +(1-a)M d ,

[0997] Among them, M c is the second-order moment of a continuous function, M d is the second-order moment of discrete data; a (0≤a≤1) is a weight coefficient used to measure the proportion of the second-order moment of the continuous function in the fused second-order moment; when a=0, it means that when calculating the fused second-order moment, the contribution of the second-order moment of the continuous function is completely ignored, and only the second-order moment of the discrete data is used; when a=1, it completely relies on the second-order moment of the continuous function, and does not consider the second-order moment of the discrete data.

[0998] In some optional implementations, the tracking decision module 1307 includes:

[0999] The defect area segmentation and positioning unit is used to obtain defect detection results by using the improved U-Net model for hybrid modeling image defects, and to segment and locate the defect area based on the defect detection results.

[1000] The defect position tracking unit is used to track the image defect position based on the segmented and located defect area.

[1001] The fault tracing unit is used to construct a defect evolution dynamics model based on discrete data points of defect area and perimeter and a second-order moment model of a continuous function; based on the defect position image, a preset tracking function is used to analyze the morphological change characteristics of the crack propagation angle and wear depth increment of the defect in the defect area position of the image at different times, and the defect evolution process is visualized and quantitatively analyzed; the physical defect position in the three-dimensional structural information is obtained, and the spatial mapping algorithm is used to match the physical defect position in the three-dimensional structural information with the electrical anomaly point data and the microstructure image respectively, and a physical defect and electrical fault correlation map is established; based on the physical defect and electrical fault correlation map and morphological change characteristics, and combined with the current distribution anomaly analysis in the wear area of ​​the resistor surface, the image defect area position is fault traced.

[1002] The process improvement unit is used to improve the welding process, material replacement process and online monitoring process of the decision-making products based on defect tracking and fault tracing.

[1003] In some optional embodiments, the semantic segmentation model of the improved U-Net model based on CNN deep learning includes an encoder, a decoder and a skip connection, the encoder includes a residual connection, and the decoder includes an attention mechanism; the defect area segmentation and positioning unit includes:

[1004] A downsampling subunit is configured to obtain an output feature map and an input feature map after performing convolution processing on the image defect area in the encoder, and add the output feature map and the input feature map through a residual connection to obtain a new output feature map, wherein the output feature map and the input feature map are obtained after performing convolution processing on the image defect area in the encoder, and the output feature map and the input feature map are added through a residual connection to obtain a new output feature map, including:

[1005] Residual connection is introduced in the encoder, and the weight matrix of the residual connection is The output feature map after two convolutional layers is With input features Figure X i (x, y, z), and add the output feature map and the input feature map through the residual connection to obtain a new output feature map The formula is as follows:

[1006]

[1007] Where (x, y, z) is the image coordinate.

[1008] The upsampling subunit is used to input the new output feature map into the attention mechanism of the decoder to obtain a weighted feature map and then perform a convolution operation to obtain a TFT ray map. The new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map and then perform a convolution operation to obtain a TFT ray map, including:

[1009] The attention mechanism is introduced in the decoder, and the attention weight matrix is Feature map after upsampling and concatenation by the encoder The weighted features are obtained through the attention mechanism That is the TFT ray diagram, the formula is as follows:

[1010]

[1011] Among them, (x, y, z) are image coordinates, a represents the relative position offset involved when the feature map performs a weighted operation using the attention mechanism, c is the output channel index, and d and e are both input channel indices.

[1012] In some optional embodiments, the process improvement unit includes:

[1013] The first process improvement sub-unit is used to improve the welding process of the product to be decided, including:

[1014] Finite element simulation is used to analyze the influence of different welding parameters such as laser power, welding speed and spot diameter of the product to be decided on on the residual stress, fatigue life and resistance value of the welded joint. The simulation effect is verified by combining the multimodal data of the ultrasonic scanning results to determine the optimal process parameters of laser welding and resistance brazing, and the influence of multiple power supplies such as photovoltaic, green electricity, energy storage, mains electricity, converged electricity and nuclear power on the welding parameters is considered; the defect position images of welding cracks and cold welds detected by convolutional neural network are combined with the multimodal data of ultrasonic scanning to quantify the welding quality.

[1015] The second process improvement sub-unit is used to improve the material replacement process of the decision-making products, including: establishing a material performance database to record the performance parameters of the resistance change rate and capacitance stability of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments; through accelerated aging tests simulating extreme working conditions, using convolutional neural networks to detect defects in images of aged materials, and evaluating material durability in combination with performance data.

[1016] The third process improvement sub-unit is used to improve the online monitoring process of the decision-making products, including: deploying a real-time image recognition system based on deep learning to perform millisecond-level analysis of X-ray or OBIRCH inspection images to automatically mark and classify defects; using deep learning algorithms to train X-ray or OBIRCH inspection images to establish a defect recognition model.

[1017] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[1018] The multimodal CNN / Transformer image defect diagnosis, tracking and decision-making device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[1019] The embodiment of the present invention also provides a computer device having the above Figure 12 The image defect diagnosis, tracking and decision-making device based on multimodal fusion CNN is shown.

[1020] See also Figure 14 , Figure 14 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 14As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 14 A processor 10 is taken as an example.

[1021] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[1022] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[1023] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[1024] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[1025] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 can be connected via a bus or other means. Figure 13 The bus connection is taken as an example.

[1026] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[1027] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a...

Claims

1. A multimodal CNN / Transformer image defect diagnosis and tracking decision method, characterized by: The method comprises: Acquire defect location images of multiple multimodal data of the product to be diagnosed, wherein the multiple defect location images are defect location images containing the above multimodal data and are taken from different production batches, different process stages, and different equipment; the multimodal data includes image data, electrical data, and operating condition data; performing data preprocessing and data alignment on the defect position images of the plurality of multimodal data; Based on the defect location image of each of the image data, feature extraction is performed on the defect location images of the multiple multimodal data using a position encoding method to obtain image features after position encoding of each defect location image. At the same time, feature extraction is performed on the electrical data and the operating condition data respectively to obtain electrical features and operating condition features; the image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features; Based on a preset feature weight distribution scheme, each of the contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features is subjected to feature weight fusion as an image feature, and the image features, electrical features, and operating condition features are spliced ​​into a unified vector to obtain multiple multimodal fusion feature vectors; A single-process image defect detection model is obtained by sequentially performing independent sub-model training and model evaluation on a convolutional neural network model based on multiple multimodal fusion feature vectors, and a multi-process image defect detection model is obtained by sequentially performing model training and model evaluation on a Transformer network model based on the multiple multimodal fusion feature vectors; Acquire multiple images of the product to be diagnosed that are actually taken and input them into the single-process image defect detection model and / or the multi-process image defect detection model to perform image defect diagnosis, and obtain a detection result indicating whether the corresponding area of ​​the product image has defects or not; Perform hybrid modeling and analysis on the defective product image output by the image defect detection model input to be diagnosed to obtain hybrid modeled image defects, wherein the hybrid modeling and analysis includes shape model construction, defect quantification and severity diagnosis; The hybrid modeling image defects are used to obtain defect detection results through the improved U-Net model, which realizes the segmentation and positioning of the defect area, and then tracks the position of the image defect area. Decisions are made based on the defect tracking and fault tracing results to optimize the production process parameters.

2. The method according to claim 1, characterized in that Performing data preprocessing and data alignment on the defect location images of the plurality of multimodal data includes: The defect location image is sequentially subjected to data preprocessing including data cleaning, data labeling, image usage classification, multimodal data timestamp synchronization and standardization, image enhancement, and image pixel normalization; The data cleaning is used to remove noise data, blurred defect location images, and incomplete defect location images from defect location images; the data annotation is to use a combination of manual annotation and semi-automatic annotation to annotate defect locations and defect types; the image usage classification is to divide multimodal data into image data, electrical data, and working condition data based on the production batch, process stage, equipment type, and multimodal data type to which the image belongs; the image pixel normalization is to normalize the pixel values ​​of the image data to a preset range to unify the scale of the image data; Image data, electrical data, and operating condition data are aligned, fused, and standardized by timestamp to construct a temporally and spatially consistent dataset.

3. The method according to claim 1, characterized in that The working condition data includes load fluctuation and ambient temperature; Based on the defect location image of each of the image data, feature extraction is performed on the defect location images of the plurality of multimodal data using a position encoding method to obtain image features after position encoding of each defect location image. Feature extraction is also performed on the electrical data and the operating condition data respectively to obtain electrical features and operating condition features, including: Use CNN shallow network to extract basic edge features and texture features of defect location images; Determining the defect center position of each defect location image based on the edge features and texture features; obtaining a two-dimensional coordinate value of each defect center position, and discretizing each of the two-dimensional coordinate values; converting each discretized coordinate value into a feature vector through one-hot encoding, and using the feature vector as an image feature after position encoding of each defect location image; The electrical data is converted into a time series, and the frequency domain features are extracted as electrical features through Fourier transform. The real-time data of load fluctuations and ambient temperature in the operating condition data are sampled in a 5-minute sliding window to extract the operating condition features.

4. The method according to claim 1, wherein The feature weight distribution scheme is set in the following way: The weight distribution of contrast features is 20% to 30%, the weight distribution of correlation features is 10% to 25%, the weight distribution of energy features is 10% to 25%, the weight distribution of entropy features is 30% to 40%, the weight distribution of defect entropy value features is 10% to 20%, and the weight distribution of position texture parameter features is 10% to 30%, and the sum of the contrast feature weight, correlation feature weight, energy feature weight, entropy feature weight, defect entropy value feature weight and position texture parameter feature weight is equal to 1.

5. The method according to claim 4, characterized in that The method of performing feature weight fusion on each of the contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features based on a preset feature weight allocation scheme, and splicing the image features, electrical features, and operating condition features into a unified vector, includes: Calculate respectively the first product corresponding to each contrast feature and the contrast feature weight, the second product corresponding to each correlation feature and the correlation feature weight, the third product corresponding to each energy feature and the energy feature weight, the fourth product corresponding to each entropy feature and the entropy feature weight, the fifth product corresponding to each defect entropy value feature and the defect entropy value feature weight, and the sixth product corresponding to each position texture parameter feature and the position texture parameter feature weight; Adding each corresponding first product, second product, third product, fourth product, fifth product and sixth product to obtain a plurality of multimodal fusion feature vectors; Early fusion of multiple multimodal fusion feature vectors by splicing electrical features and operating condition features into a unified vector.

6. The method according to claim 5, characterized in that The convolutional neural network model includes an input layer, multiple convolutional layers and pooling layers, multiple fully connected layers, and an output layer. Based on multiple multimodal fusion feature vectors, the convolutional neural network model is sequentially trained as an independent sub-model and evaluated to obtain a single process image defect detection model, including: The multiple multimodal fusion feature vectors are transmitted through the input layer to multiple convolutional layers and pooling layers for deep learning, wherein the multiple convolutional layers use convolution kernels to perform convolution operations on the multimodal fusion feature vectors to extract local features, and the pooling layer obtains multiple key features through downsampling operations; Multiple fully connected layers are used to integrate the multiple key features, convert the extracted features from local information into global information, and map them to the output probability distribution through an activation function; The output layer is used to convert the probability distribution mapped to the output into multiple fused probability distribution images and output them. The value of each pixel in the fused probability distribution image represents the probability of a defect at that position.

7. The method according to claim 6, characterized in that The single process image defect detection model is obtained by sequentially performing independent sub-model training and model evaluation on the convolutional neural network model based on multiple multimodal fusion feature vectors, and further includes: Dividing the multiple fused probability distribution images into a training set, a validation set, and a test set according to a preset ratio; The training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weightedly fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and an evaluation result is output. When the evaluation result meets a preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as a single process image defect detection model; During the training process, the loss is calculated based on a preset loss function, and the Adam optimizer is used for backpropagation to adjust the weights and biases of the convolutional neural network model. The parameters of the multiple convolutional layers, pooling layers, and multiple fully connected layers are adjusted based on the verification set to obtain the convolutional neural network model to be evaluated. The Adam optimizer combines the ideas of the momentum method and the Adagrad optimization algorithm to dynamically adjust the learning rate of each parameter in the model based on the first-order moment estimator and the second-order moment estimator of the gradient, and corrects the first-order moment estimator and the second-order moment estimator through the deviation.

8. The method according to claim 7, characterized in that The training set is input into the convolutional neural network model to train independent sub-models in sequence, and the outputs of each independent sub-model are weightedly fused to obtain the detection model to be evaluated; the detection model to be evaluated is evaluated based on the test set and an evaluation result is output. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as a single process image defect detection model, including: The unified vector of defect features of multimodal fusion early fusion images is input into the convolutional neural network for late fusion; The convolutional neural network model is trained to process image data specifically to extract defect features in images; Train the long short-term memory network model to process time-series electrical data and capture the patterns of electrical data changes over time; Train a multi-layer perceptron model to process operating condition data and analyze the impact of operating conditions on defects; The outputs of the independent sub-models obtained from the above training are weightedly fused to obtain the detection model to be evaluated. The test set is input into the detection model to be evaluated. When the evaluation result meets the preset threshold range, the detection model to be evaluated corresponding to the evaluation result is used as the single process image defect detection model, and the evaluation result is used as the final defect diagnosis result.

9. The method according to claim 7, characterized in that Based on the multiple multimodal fusion feature vectors, the Transformer network model is sequentially trained and evaluated to obtain a multi-process image defect detection model, including: The early fusion of the image defect fusion path of the multimodal fusion Transformer uses the self-attention mechanism to effectively capture the semantic associations between different modal features and directly perform early cross-modal fusion of multimodal features; A cross-attention mechanism is used to parallelly process local features extracted by multi-channel convolutional neural networks from different sources to fuse the complex defect feature detection model. The complex defect feature detection model and the Transformer processing time-series associated detection model are combined to form a combined detection model. The image to be diagnosed is input into the combined detection model to output the final defect diagnosis result.

10. The method according to claim 7, characterized in that The method also includes: using visualization tools to visualize the loss changes and evaluation result changes during the training process of the convolutional neural network model, the key features deeply learned by multiple convolutional layers and pooling layers, and the feature integration and mapping of multiple fully connected layers to the probability distribution of the output.

11. The method according to claim 1, characterized in that The step of obtaining a plurality of images of the product to be diagnosed that are actually taken and inputting them into the image defect detection model for image defect diagnosis to obtain a detection result of whether there are defects or no defects includes: Image defect diagnosis is performed on multiple product images to be diagnosed that are input into the image defect detection model using a judgment function, where the judgment function is expressed by the following formula: Wherein, G(F)=1 indicates that a defect is detected in the corresponding area of ​​the image, G(F)=0 indicates that there is no defect, and if satisfiescertain defect condition indicates that the preset defect condition is met; The multimodal fusion feature vector of the input product image to be diagnosed is matched with the defect feature threshold set during model training. That is, when the feature parameters such as contrast, energy, and entropy in the feature vector exceed the preset threshold, it is judged that there is a defect, otherwise it is judged as no defect.

12. The method according to claim 1, characterized in that Performing hybrid modeling and analysis on the defective product image output by the image defect detection model for the image to be diagnosed, including: Calculating the geometric features of the image defect area contour and determining the shape of the preliminary image defect area; the geometric features of the shape of the preliminary image defect area include contour perimeter, contour convex hull perimeter and shape factor; Based on the shape of the initial defect area in the image, the shape hybrid modeling is performed using the second-order moment of discrete data and the second-order moment of continuous function to obtain the mathematical model for measuring the shape of the defect area in the image; The defect degree of the image defect area is measured based on the image defect area shape measurement mathematical model, and the final shape, width, length, area and volume of the image defect area are obtained; The severity level of the image defect is diagnosed and graded based on the final shape, width, length, area and volume of the image defect area.

13. The method according to claim 12, characterized in that The shape hybrid modeling is performed based on the shape of the preliminary defect area in the image using the second-order moment of discrete data and the second-order moment of continuous function to obtain a mathematical model for measuring the shape of the defect area in the image, including: Determine the discrete data second-order moment and continuous function second-order moment of the image defect area based on the shape of the preliminary defect area of ​​the image; The discrete data second-order moment and the continuous function second-order moment of the image defect area are fused and modeled to obtain a mathematical model for shape measurement of the image defect area.

14. The method according to claim 12, characterized in that The discrete data second-order moment and the continuous function second-order moment of the image defect area are fused and modeled to obtain a mathematical model for shape measurement of the image defect area, including: Calculating a first moment of inertia based on the second-order moment of the discrete data, and calculating a first equivalent ellipse parameter based on the first moment of inertia; Calculating a second moment of inertia based on the second-order moment of the continuous function, and calculating a second equivalent ellipse parameter based on the second moment of inertia; The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent ellipse parameters and the second equivalent ellipse parameters, to obtain the mathematical model for shape measurement of the image defect area.

15. The method according to claim 14, characterized in that The weighted average method is used to fuse the first moment of inertia and the second moment of inertia, as well as the first equivalent ellipse parameters and the second equivalent ellipse parameters, to obtain a mathematical model for shape measurement of the image defect area, including: Setting the second-order moment weight of the continuous function, and calculating the second-order moment weight of the discrete function based on the weight of the second-order moment of the continuous function; Based on the second-order moment weight of the continuous function and the second-order moment weight of the discrete function, the first moment of inertia and the second moment of inertia are fused by a weighted average method to obtain a fused second-order moment; the first equivalent ellipse parameter and the second equivalent ellipse parameter are fused to obtain a fused equivalent ellipse parameter; The mathematical model for measuring the shape of the image defect area is obtained based on the fused moment of inertia and the fused equivalent ellipse parameters; The fused second-order moment formula is as follows: M fusion =aM c +(1-a)M d ; Among them, M c is the second-order moment of a continuous function, M d is the second-order moment of discrete data; a (0≤a≤1) is a weight coefficient used to measure the proportion of the second-order moment of the continuous function in the fused second-order moment; when a=0, it means that when calculating the fused second-order moment, the contribution of the second-order moment of the continuous function is completely ignored, and only the second-order moment of the discrete data is used; when a=1, it completely relies on the second-order moment of the continuous function, and does not consider the second-order moment of the discrete data.

16. The method according to claim 1, characterized in that The hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, achieve defect area segmentation and positioning, and then track the position of the image defect area, including: The hybrid modeling image defects are applied to the improved U-Net model to obtain defect detection results, and the defect area is segmented and located based on the defect detection results; The image defect position is tracked based on the segmented and located defect area.

17. The method according to claim 1, wherein The semantic segmentation model of the improved U-Net model includes an encoder, a decoder and a skip connection, the encoder includes a residual connection, and the decoder includes an attention mechanism; The hybrid modeling image defects are subjected to the improved U-Net model to obtain defect detection results, achieving defect area segmentation and positioning, including: After performing convolution processing on the image defect area in the encoder, an output feature map and an input feature map are obtained, and the output feature map and the input feature map are added together through the residual connection to obtain a new output feature map; Input the new output feature map into the attention mechanism of the decoder to obtain a weighted feature map and then perform a convolution operation to obtain a TFT ray map; The TFT ray map is input into the improved U-Net model to perform image region segmentation to obtain a binary image of the image defect area.

18. The method according to claim 17, characterized in that The method of performing convolution processing on the image defect area in the encoder to obtain an output feature map and an input feature map, and adding the output feature map and the input feature map through the residual connection to obtain a new output feature map includes: The residual connection is introduced in the encoder, and the weight matrix of the residual connection is The output feature map after two convolutional layers is With the input feature map X i (x, y, z), and add the output feature map and the input feature map through the residual connection to obtain a new output feature map The formula is as follows: Where (x, y, z) is the image coordinate.

19. The method according to claim 17, wherein The new output feature map is input into the attention mechanism of the decoder to obtain a weighted feature map, and then a convolution operation is performed to obtain a TFT ray map, including: The attention mechanism is introduced in the decoder, and the attention weight matrix is Feature map after upsampling and concatenation by the encoder The weighted features are obtained through the attention mechanism That is the TFT ray diagram, the formula is as follows: Among them, (x, y, z) are image coordinates, a represents the relative position offset involved when the feature map performs a weighted operation using the attention mechanism, c is the output channel index, and d and e are both input channel indices.

20. The method according to claim 1, wherein The above mentioned decision making and optimization of production process parameters based on defect tracking and fault tracing results include: A defect evolution dynamics model is constructed based on discrete data points of defect area and perimeter and a continuous function second-order moment model; Based on the defect position image, a preset tracking function is used to analyze the morphological change characteristics of the crack propagation angle and wear depth increment of the defect in the defect area position of the image at different times, and the defect evolution process is visualized and quantitatively analyzed; Obtain the physical defect locations in the 3D structural information and use a spatial mapping algorithm to match the physical defect locations in the 3D structural information with the electrical anomaly data and microstructure images, respectively, to establish a correlation map between physical defects and electrical faults. Based on the correlation map between physical defects and electrical faults and the morphological change characteristics, combined with the abnormal current distribution analysis in the wear area of ​​the resistor surface, the fault is traced to the location of the defective area in the image; Based on defect tracking and fault tracing, the welding process, material replacement process and online monitoring process of the decision-making products are improved.

21. The method according to claim 20, characterized in that Improve the welding process of the products to be decided, including: Finite element simulation is used to analyze the effects of different welding parameters such as laser power, welding speed, and spot diameter on the residual stress, fatigue life, and resistance value of the welded joint for the product to be determined. The simulation results are verified by combining multimodal data from ultrasonic scanning results to determine the optimal process parameters for laser welding and resistance brazing. The effects of multiple power sources such as photovoltaic, green electricity, energy storage, mains electricity, combined electricity, and nuclear power on the welding parameters are also considered. The defect location images of weld cracks and cold welds detected by convolutional neural networks are combined with multimodal data from ultrasonic scanning to quantify the welding quality. Improve the material replacement process for the products to be decided, including: Establish a material performance database to record the resistance change rate and capacitance stability performance parameters of different materials such as metal film resistors, carbon film resistors, and ceramic dielectric capacitors under various environments. Through accelerated aging tests simulating extreme working conditions, use convolutional neural networks to detect defects in images of aged materials, and evaluate material durability based on performance data. Improve the online monitoring process of decision-making products, including: Deploy a real-time image recognition system based on deep learning to analyze X-ray or OBIRCH inspection images in milliseconds to automatically label and classify defects; Use deep learning algorithms to train X-ray or OBIRCH inspection images to establish a defect recognition model.

22. A multimodal CNN / Transformer image defect diagnosis, tracking and decision-making device, characterized by: The device comprises: A multimodal data acquisition module is configured to acquire defect location images of multiple multimodal data of the product to be diagnosed, wherein the multiple defect location images are defect location images containing the multimodal data captured by different production batches, different process stages, and different equipment; the multimodal data includes image data, electrical data, and operating condition data; A data preprocessing and data alignment module, configured to perform data preprocessing and data alignment on the defect position images of the plurality of multimodal data; a feature extraction module for extracting features from the defect location images of the plurality of multimodal data using a position encoding method based on the defect location image of each of the image data, thereby obtaining image features after position encoding of each defect location image; and simultaneously extracting features from the electrical data and the operating condition data, respectively, to obtain electrical features and operating condition features; the image features include contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features; A multimodal feature fusion module is used to perform feature weight fusion on each of the contrast features, correlation features, energy features, entropy features, defect entropy value features, and position texture parameter features based on a preset feature weight distribution scheme as an image feature, and to splice the image features, electrical features, and operating condition features into a unified vector to obtain multiple multimodal fusion feature vectors; A multimodal detection model construction and detection module is used to sequentially perform independent sub-model training and model evaluation on a convolutional neural network model based on multiple multimodal fusion feature vectors to obtain a single-process image defect detection model, and sequentially perform model training and model evaluation on a Transformer network model based on the multiple multimodal fusion feature vectors to obtain a multi-process image defect detection model; obtain multiple actually captured product images to be diagnosed and input them into the single-process image defect detection model and / or the multi-process image defect detection model for image defect diagnosis, and obtain a detection result indicating whether the corresponding area of ​​the product image has defects or not; A hybrid modeling and analysis module, configured to perform hybrid modeling and analysis on the defective product image output by the image defect detection model and the image to be diagnosed, to obtain hybrid modeled image defects, wherein the hybrid modeling and analysis includes shape model construction, defect quantification, and severity diagnosis; The tracking decision module is used to obtain defect detection results from the hybrid modeling image defects through the improved U-Net model, realize defect area segmentation and positioning, and then track the position of the image defect area. Decisions are made based on the defect tracking and fault tracing results to optimize production process parameters.

Images