A digital twin prior enhanced defect prediction method, device and storage medium

Abstract

The present application belongs to the field of defect detection, and particularly relates to a kind of digital twin priori enhanced defect prediction method, equipment and storage medium.Method includes: obtaining the multi-modal detection data of current synchronization sampling period and evaluating modal confidence;When at least one mode is high confidence mode, repair low confidence mode and form each mode final weighted feature, the final weighted feature is constructed as mode-space grid node graph, graph topology interaction feature is extracted, node confidence and physical priori reliability are gated graph features and priori features respectively, and after fusion, it is restored as spatially maintained enhanced feature map, and the defect category, position or size is output;When obtaining fusion feature, the next period digital twin physical priori is generated based on current period fusion feature, and initialization physical priori is used in the first synchronization sampling period.The present application can reduce false alarm, false negative and unnecessary recheck caused by local sensor fluctuation.

Description

Technical Field

[0001] This invention belongs to the field of defect detection, specifically relating to a defect prediction method, device, and storage medium with digital twin prior enhancement. Background Technology

[0002] With the development of intelligent manufacturing and industrial online inspection technologies, product quality inspection in the production process is gradually shifting from traditional manual sampling and single-sensor detection to multi-source, multi-modal online inspection. In the intermediate stages of industrial production, optical images can characterize the surface texture, scratches, pits, and color differences of products; thermal imaging data can reflect the local temperature distribution, cooling status, and thermal anomalies; and ultrasonic data can characterize internal defects such as bubbles, cracks, delamination, and abnormal echo structures. Because different detection modalities have different sensing mechanisms and defect characterization capabilities, and are highly complementary, the joint analysis of multiple detection data to achieve comprehensive perception of product surface defects, internal defects, and their related physical states has become an important technological direction in the field of industrial production quality inspection.

[0003] In existing technologies, to improve the accuracy of industrial defect detection, multimodal fusion models are typically used to jointly process optical images, thermal imaging data, and ultrasonic data. Common methods include directly stitching together features extracted from different modalities, using pre-set fixed weights for weighted fusion, or obtaining the detection results of each modality separately and then determining the final detection result through a backend voting method. Some technologies also utilize convolutional neural networks, attention mechanisms, or other deep learning networks to extract deep features from multimodal data to further complete defect classification, defect localization, or defect size prediction. Furthermore, with the development of digital twin technology, some industrial inspection systems also use digital twin models to simulate the physical states of products, such as temperature, stress, and deformation, and use the digital twin results for production status display, process monitoring, or auxiliary analysis.

[0004] However, existing technologies still have the following shortcomings: On the one hand, current multimodal fusion methods typically do not fully consider the dynamic reliability of different detection modes in actual production environments. When optical data is affected by strong light, vibration, or local overexposure, thermal imaging data is affected by environmental thermal noise or changes in emissivity, or ultrasonic data is affected by factors such as probe coupling instability, low-quality modes may still participate in subsequent fusion, thereby contaminating the fusion features and increasing the risk of false detection and missed detection. On the other hand, existing methods usually lack effective feature repair mechanisms for locally damaged or low-confidence modes. Directly discarding them can easily lead to the loss of effective information, while retaining them may introduce abnormal features. At the same time, traditional feature stitching or flattening processing can easily weaken the correspondence between different modal features and the actual spatial location of the product, making it difficult to effectively express the topological association between adjacent areas and different modes at the same spatial location. Furthermore, existing defect prediction methods are mainly data-driven and do not make sufficient use of physical states related to defect formation and evolution, such as product temperature, stress, and residual deformation. Existing digital twin models are often used as independent simulation or visualization tools and do not form continuous state feedback and recursive correlation with multimodal detection features. Therefore, in scenarios such as complex industrial interference, small sample size, and fluctuations in detection data quality, there are still problems with insufficient prediction stability, robustness, and physical consistency. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a defect prediction method, device and storage medium with digital twin prior enhancement, which reduces false alarms, false negatives and unnecessary whole-piece re-inspection caused by local sensor fluctuations, and helps to reduce the production cost caused by on-site manual verification and erroneous rejection.

[0006] A defect prediction method using digital twin prior enhancement includes: Multimodal detection data of the same detection object in the current synchronous sampling period is acquired. Spatial alignment and feature extraction are performed on the multimodal detection data to obtain modal features. Confidence evaluation is performed on each modal feature to determine high-confidence modal and low-confidence modal. The detection object is an injection-molded part with a preset CAD geometric model and a unified detection mesh that can be established. The modal features corresponding to the high-confidence modal are high-confidence modal features, and the modal features corresponding to the low-confidence modal are low-confidence modal features. When the current synchronous sampling period is the first synchronous sampling period, the initial digital twin physical prior is obtained; when the current synchronous sampling period is the second synchronous sampling period or a subsequent synchronous sampling period, the digital twin virtual physical features of the detection object in the previous synchronous sampling period are obtained as the digital twin physical prior. The digital twin physical prior is obtained based on the corrected surface temperature field, equivalent stress projection field, and surface residual deformation field of the detection object. When at least one mode is a high-confidence mode, the low-confidence mode features are repaired using high-confidence mode features and digital twin physical priors. The repair reconstruction error and the repaired confidence are determined based on the difference between the repaired features and the reference features. The repaired features are fused with the corresponding low-confidence mode features to obtain the repaired mode features. The high-confidence mode features and the repaired mode features are weighted to obtain the final weighted features of each mode. When all modalities are low-confidence modalities, a preset defect category is output, the detected object is determined to be defective, the subsequent defect prediction of the current synchronous sampling period is terminated, and the data of the current synchronous sampling period is not used to update the digital twin recursive state; wherein, the preset defect category is used to characterize that the detection data of each modality does not meet the confidence requirements; When the final weighted features are obtained, they are mapped to a unified feature dimension and organized along the node dimension according to modality and spatial location to obtain a modality-spatial grid node graph; based on the spatial adjacency and cross-modal association relationships in the modality-spatial grid node graph, node features are aggregated to obtain graph topology interaction features; The digital twin physical prior is mapped to the modal-spatial grid node graph according to its spatial location to obtain digital twin prior features; a node confidence mask is formed based on the node reliability and the graph topology interaction features are gated; a physical prior mask is formed based on the reliability of the digital twin physical prior and the digital twin prior features are gated; the two types of gated features are concatenated to obtain fused features. The fused features are aggregated according to their spatial location and restored to the spatial grid corresponding to the detected object to obtain a digital twin enhanced feature map; the digital twin enhanced feature map is input into the defect prediction model to obtain the defect prediction result; when the fused features are obtained, a digital twin virtual physical feature for the current synchronous sampling period is generated based on the fused features, and the digital twin virtual physical feature is used as the digital twin physical prior for the next synchronous sampling period of the same detected object.

[0007] Optionally, the multimodal detection data includes optical data, thermal imaging data, and ultrasonic data. The optical data, thermal imaging data, and ultrasonic data are acquired synchronously within the same synchronous sampling period. The spatial alignment is performed based on the product CAD unfolded coordinates of the detected object or a unified detection grid, so that each modal feature corresponds to the same spatial grid position.

[0008] Optionally, the step of evaluating the confidence level of each modal feature to determine the high-confidence mode and the low-confidence mode includes: The probability corresponding to each feature value interval is determined based on the feature value distribution of each modality feature, and the information entropy corresponding to each modality feature is determined based on the probability. The signal-to-noise ratio corresponding to each modal feature is determined based on the effective signal power and noise power in each modal feature; The spatial continuity score for each modal feature is determined based on the differences between features corresponding to spatially adjacent spatial grids in each modal feature. Based on the information entropy reference center and allowable deviation corresponding to normal samples, the information entropy is converted into an information entropy quality score. Based on the upper and lower bounds of the signal-to-noise ratio and the upper and lower bounds of the spatial continuity score corresponding to the training set or stable production batch, the signal-to-noise ratio and the spatial continuity score are normalized respectively to obtain the signal-to-noise ratio quality score and the spatial continuity quality score. The information entropy quality score, signal-to-noise ratio quality score, and spatial continuity quality score are weighted to obtain the modal confidence level corresponding to each modal feature; The modality confidence level is compared with a preset confidence threshold. Modalities with a confidence level not lower than the preset confidence threshold are identified as high-confidence modes, and modalities with a confidence level lower than the preset confidence threshold are identified as low-confidence modes.

[0009] Optionally, the step of repairing the low-confidence modal features using the high-confidence modal features of the current synchronization sampling period and the digital twin physical prior includes: The low-confidence modal features are used as query features, and the high-confidence modal features and the digital twin physical prior are aligned and used as the repair information source. The query features and the repair information source are processed through cross-modal attention to obtain the repair features corresponding to the low-confidence modality. Specifically, when one modality is a low-confidence modality, the other two high-confidence modal features and the digital twin physical prior are used for repair. When two modalities are low-confidence modalities, the remaining high-confidence modal features and the digital twin physical prior are used to repair the two low-confidence modal features respectively.

[0010] Optionally, determining the repair reconstruction error and post-repair confidence based on the difference between the repaired features and the reference features includes: The repair and reconstruction error is determined based on the difference between the repaired feature and the reference feature. The reference feature is obtained from at least one of the modal features of the same detection object in adjacent stable synchronous sampling periods before the current synchronous sampling period, the cross-modal mapping result of the high-confidence modal features in the current synchronous sampling period, or the undamaged modal features corresponding to the training phase. The repaired confidence level of the low-confidence mode is obtained by weighting the repair and reconstruction error and the mode confidence level corresponding to the low-confidence mode.

[0011] Optionally, mapping the final weighted features to a unified feature dimension and organizing them along the node dimension according to modality and spatial location to obtain a modality-space grid node graph includes: The final weighted features corresponding to optical data, thermal imaging data and ultrasound data are respectively mapped to the same d-dimensional feature space to obtain the unified dimensional features corresponding to the three modalities; Assuming each mode corresponds to N spatial grid positions, the unified dimensional features corresponding to the three modes are stacked along the node dimension to form the mode-spatial grid node graph containing 3N nodes; wherein, spatial adjacency is established between nodes with adjacent spatial positions in the same mode, and cross-modal association is established between nodes corresponding to different modes at the same spatial position.

[0012] Optionally, the step of aggregating node features based on spatial adjacency relationships and cross-modal association relationships in the modal-spatial grid node graph to obtain graph topology interaction features includes: Establish spatial adjacency relationships based on the adjacency relationships between different spatial grid locations within the same mode; Establish cross-modal association relationships based on the feature similarity between nodes corresponding to different modes at the same spatial grid location; Based on the spatial adjacency relationship and cross-modal association relationship, the node features in the modal-spatial grid node graph are aggregated, so that each node aggregates the corresponding spatial adjacent node features and other modal node features at the same spatial grid position, to obtain the graph topology interaction features of the current synchronous sampling period.

[0013] Optionally, generating digital twin virtual physical features for the current synchronization sampling period based on the fused features, and using the digital twin virtual physical features as the digital twin physical prior for the next synchronization sampling period of the same detection object, includes: Based on the fusion features of the current synchronous sampling period, a corrected surface temperature field, an equivalent stress projection field, and a surface residual deformation field corresponding to the two-dimensional spatial grid of the detected object are generated. The corrected surface temperature field is subject to transient heat conduction physical constraints, the equivalent stress projection field is subject to displacement-strain relations, constitutive relations, and stress balance physical constraints, and the surface residual deformation field is subject to residual deformation supervision constraints and geometric compatibility constraints. The corrected surface temperature field, the equivalent stress projection field, and the surface residual deformation field are feature-mapped to obtain the digital twin virtual physical features of the current synchronous sampling period, and these features are used as the digital twin physical priors for the next synchronous sampling period of the same detection object.

[0014] A terminal device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs a digital twin prior enhancement defect prediction method.

[0015] A computer-readable storage medium storing a computer program that, when loaded and executed by a processor, employs a digital twin prior-enhanced defect prediction method.

[0016] The beneficial effects of this invention are: 1. This invention enables differentiated processing for low-confidence local and all modal states, reducing the risk of erroneous quality assessments due to sensor interference in the production process. In actual production, optical inspection is susceptible to strong light, vibration, and localized overexposure; thermal imaging is affected by environmental thermal noise; and ultrasonic inspection may generate abnormal data due to changes in probe coupling. When only some modal confidence is reduced, this invention uses the remaining high-confidence modalities and the digital twin physical state of the previous synchronous sampling cycle for the same product to repair the damaged information. This allows currently usable data to continue participating in the inspection, avoiding the entire product being judged as abnormal or requiring re-inspection due to a short-term sensor anomaly. When all modal confidence is low, a preset defect category is output, and the product is conservatively judged as defective. Subsequent defect predictions for the current synchronous sampling cycle are terminated, and the digital twin recursive state is not updated, preventing all low-confidence data from entering subsequent prediction and inspection cycles. This reduces false alarms, missed alarms, and unnecessary re-inspections of entire parts caused by fluctuations in local sensors, and generates a definite rejection output when all modes are unreliable, maintaining the continuity of the online inspection process in the intermediate production stages.

[0017] 2. This invention can map defect detection results to their actual spatial location on the product and provide defect dimensions, providing direct evidence for on-site spot checks, partial rework, and product sorting, reducing the workload of re-inspecting the entire product after a defect is discovered. The invention spatially aligns optical, thermal imaging, and ultrasonic detection data according to the product's CAD coordinates or a unified detection grid. It maintains the correspondence between detection information and the actual product location through topological interaction between modal-spatial grid nodes. Simultaneously, it incorporates digital twin physical states such as surface temperature, equivalent stress, and residual deformation corresponding to the same spatial location into the fusion of current cycle detection information, ultimately reconstructing a spatial feature map corresponding to the product's CAD grid, and outputting the defect category, defect location, and defect size accordingly. Therefore, when the system detects scratches, cracks, bubbles, delamination, or local abnormal areas, it can not only determine whether a defect exists, but also map the defect area and its size to the specific inspection location on the product. This allows on-site personnel to conduct focused re-inspection, partial rework, or sorting of the corresponding area, without having to re-inspect the entire product based solely on a general pass or fail result. This shortens the defect handling path, reduces ineffective inspections and rework, and improves the targeted nature of quality handling in intermediate production stages. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a defect prediction method using digital twin prior enhancement according to the present invention. Detailed Implementation

[0019] A defect prediction method using digital twin prior enhancement includes: S1. Obtain multimodal detection data of the same detection object in the current synchronous sampling period, perform spatial alignment and feature extraction on the multimodal detection data to obtain each modal feature; evaluate the confidence of each modal feature to determine the high-confidence mode and the low-confidence mode. The detection object is an injection molded part with a preset CAD geometric model and a unified detection mesh that can be established. The modal features corresponding to the high-confidence mode are high-confidence modal features, and the modal features corresponding to the low-confidence mode are low-confidence modal features. Specifically, the inspection object can be an injection-molded part with a preset CAD geometric model and a unified inspection grid. This embodiment takes a PC+ABS mobile phone shell after injection molding as an example. The mobile phone shell has a fixed posture, complete CAD geometry and thickness distribution that match the inspection station, and is associated with each synchronous sampling cycle through a unique product identifier. Optical data is used to characterize surface defects such as scratches, dents, color differences and weld lines; thermal imaging data is used to characterize uneven cooling and local thermal anomalies; ultrasonic data is used to characterize internal bubbles, voids, cracks and echo structure anomalies. Before the start of the first synchronous sampling cycle, the CAD geometry of the mobile phone shell, PC+ABS material parameters, inspection station environment and boundary conditions, reference temperature field, initial equivalent stress field and initial residual deformation field are used to obtain the initial digital twin physical features through spatial alignment, normalization and channel mapping. The reference physical field is derived from product calibration tests, material parameter libraries, or finite element models calibrated using measured data. Starting from the second synchronous sampling period, the virtual physical features of the digital twin actually generated in the previous synchronous sampling period are used as the prior physical features of the digital twin. .

[0020] Multimodal detection data includes optical data, thermal imaging data, and ultrasonic data. Optical data, thermal imaging data, and ultrasonic data are acquired synchronously within the same synchronous sampling period. Spatial alignment is performed based on the product CAD unfolded coordinates of the object being detected or a unified detection grid, so that each modal feature corresponds to the same spatial grid position.

[0021] Optical data is used to characterize surface defects such as scratches, dents, color differences, and weld lines; thermal imaging data is used to characterize uneven cooling and local thermal anomalies; ultrasonic data is used to characterize internal bubbles, voids, cracks, and echo anomalies; and the CAD unfolded mesh of the mobile phone shell is used for three-modal spatial alignment.

[0022] Specifically, the synchronous sampling cycle refers to the processing cycle in which the same object being tested is simultaneously acquired by the same hardware trigger signal or a correctable hardware timestamp within the same testing station, and the corresponding preprocessing, feature extraction, and status update are completed.

[0023] Multimodal detection data: refers to optical data, thermal imaging data, and ultrasonic data acquired for the same detection object.

[0024] Spatial alignment: refers to mapping data from different modalities to the product's CAD unfolded coordinates or a unified inspection grid, so that different modalities have a corresponding relationship at the same spatial grid position.

[0025] Product CAD unfolded coordinates: These refer to the two-dimensional unfolded spatial coordinates determined by the product's CAD geometric model, used to establish the correspondence between inspection data and the actual spatial position of the product.

[0026] Unified detection grid: refers to an H×W two-dimensional grid obtained by discretizing the detection area according to a unified spatial order.

[0027] Modal features: These refer to the shallow features obtained by preprocessing and feature extraction of spatially aligned optical, thermal imaging, and ultrasonic data.

[0028] Three-modal data are synchronously acquired using the same hardware trigger signal or a calibrable hardware timestamp. Optical images are filtered, denoised, and contrast-enhanced to extract surface texture and contour features; thermal imaging data are combined with material emissivity, ambient temperature, and sensor calibration curves to convert them into a two-dimensional absolute temperature matrix; and ultrasonic data undergoes time-frequency analysis to extract echo structure features and suppress noise.

[0029] Let the height and width of the uniform detection grid be H and W, respectively. Then the number of spatial grid locations N is:

[0030] in, This indicates the number of grid cells in the height direction of the unified detection grid; This indicates the number of grid cells in the width direction of the uniform detection grid; This represents the total number of spatial grid locations in the unified detection grid. and The coordinates are determined by the product's CAD unfolded coordinates or by the discrete settings of a unified detection grid.

[0031] The shallow features of each modality can be normalized using Z-score to reduce the dimensional differences between different modalities and different feature channels:

[0032] in, Indicates modal index, ,in 、 、 These represent optical, thermal imaging, and ultrasonic modes, respectively. This represents the unique identifier of the object being tested; Indicates the current synchronous sampling period; Indicates the spatial grid location index; Indicates the modal feature channel index; Represents the eigenvalues ​​before standardization; This represents the standardized feature value. and These represent the first and second elements in the training set or stable production batch, respectively. The modality of the first The statistical mean and statistical standard deviation of each channel are derived from the training set or stable production batch statistics. >0 A smoothing constant is used to prevent the denominator from being zero.

[0033] Standardized Each modal shallow layer feature can be represented as ,in Indicates the first The number of shallow feature channels for each modality. Optical, thermal imaging, and ultrasound shallow features are denoted as follows: , and The three modes have the same number of spatial grid positions N and the same spatial arrangement order in this stage, but their feature channel dimensions... , and They can be different.

[0034] In one specific implementation, the optical acquisition unit uses a 12-megapixel industrial camera with a focal length of 50 mm, a reference exposure time of 1 / 1000 s, and a reference frame rate of 60 fps; the thermal imaging acquisition unit has a temperature measurement range of -20℃ to 150℃; and the ultrasonic acquisition unit uses a 20MHz tunable ultrasonic probe with a detection depth range of 0 to 50 mm. The three modes are synchronized based on the product's unique identifier p and a hardware timestamp, with a synchronization error not exceeding 1 ms.

[0035] To verify the spatial sampling capability for minute defects, the calculation of the physical resolution of a single pixel is also given:

[0036] in, Indicates the physical resolution of a single pixel; This indicates the actual horizontal field of view width of the optical camera, which is obtained by measurement using a calibration plate after installation. This represents the horizontal pixel count of an image, obtained from the output image size of the optical camera. =4096 and When it is no more than 60mm, Approximately 0.0146 mm / pixel.

[0037] Confidence assessments were performed on each modal feature to identify high-confidence and low-confidence modes, including: The probability corresponding to each feature value interval is determined based on the feature value distribution of each modality feature, and the information entropy corresponding to each modality feature is determined based on the probability. The signal-to-noise ratio corresponding to each modal feature is determined based on the effective signal power and noise power in each modal feature; The spatial continuity score for each modal feature is determined based on the differences between features corresponding to spatially adjacent spatial grids in each modal feature. Based on the information entropy reference center and allowable deviation corresponding to normal samples, the information entropy is converted into an information entropy quality score. Based on the upper and lower bounds of the signal-to-noise ratio and the upper and lower bounds of the spatial continuity score corresponding to the training set or stable production batch, the signal-to-noise ratio and spatial continuity scores are normalized respectively to obtain the signal-to-noise ratio quality score and the spatial continuity quality score. The modal confidence scores corresponding to each modal feature are obtained by weighting the information entropy quality score, signal-to-noise ratio quality score, and spatial continuity quality score. The modality confidence level is compared with a preset confidence threshold. Modalities with a confidence level not lower than the preset confidence threshold are identified as high-confidence modes, and modalities with a confidence level lower than the preset confidence threshold are identified as low-confidence modes.

[0038] Information entropy: A statistical measure used to describe the uncertainty of the distribution of modal characteristic values. This scheme does not directly adopt the monotonic relationship that higher information entropy equates to higher quality, but instead compares the deviation between the current information entropy and the reference center of normal samples.

[0039] Signal-to-noise ratio (SNR): Used to reflect the relative strength of the effective signal to the noise in each modal characteristic.

[0040] Spatial continuity score: used to reflect the degree of spatial structural continuity between corresponding features of adjacent spatial grid locations.

[0041] Modal confidence: a reliability index in the [0,1] interval obtained by weighting information entropy quality score, signal-to-noise ratio quality score and spatial continuity quality score.

[0042] High confidence modalities: Modal confidence levels are not lower than the preset confidence threshold.

[0043] Low confidence modes: Modes whose confidence level is lower than a preset confidence threshold.

[0044] Regarding information entropy, the first step is to establish it based on normal samples. m Information Entropy Reference Center for Each Modality and allowable deviation Calculate the information entropy quality score:

[0045] in, Indicates the first The mode in the th ... Information entropy quality score for each synchronous sampling period; This represents the information entropy obtained from the distribution of eigenvalues ​​of this mode; Indicates the first m Information entropy reference center for each modality normal sample; This indicates the allowable deviation of information entropy for normal samples; and All were established from normal samples; It is a smoothing constant; (·,0,1) means restricting the result to the range [0,1].

[0046] For the signal-to-noise ratio, normalization is performed based on the upper and lower bounds of the training set or stable production batches:

[0047] in, This represents the normalized signal-to-noise ratio quality score. Indicates the current synchronous sampling period number. Signal-to-noise ratio of each mode; and These represent the first and second elements in the training set or stable production batch, respectively. The lower and upper bounds of the modal signal-to-noise ratio are derived from the training set or stable production batch statistics; ε is a smoothing constant.

[0048] For spatial continuity scores, normalization is also performed based on the upper and lower bounds of the training set or stable production batches:

[0049] in, This represents the normalized spatial continuity quality score. Indicates the current synchronous sampling period number. Spatial continuity score for each modality; and ε represents the lower and upper bounds of the spatial continuity score in the training set or stable production batch, respectively, which are derived from the statistics of the training set or stable production batch; ε is the smoothing constant.

[0050] The normalized modal confidence is:

[0051] in, No. The mode in the th ... Modal confidence for each synchronous sampling period; , , Let α, β, and γ represent the non-negative weights of the information entropy quality score, signal-to-noise ratio quality score, and spatial continuity quality score, respectively, satisfying α + β + γ = 1.

[0052] Will Compared with the preset confidence threshold In comparison, when < When, the mode is identified as a low-confidence mode; when ≥ When this condition is met, the mode is identified as a high-confidence mode. In this embodiment, α=0.4, β=0.4, and γ=0.2 are preset confidence thresholds. =0.75.

[0053] S2. When the current synchronous sampling period is the first synchronous sampling period, the initial digital twin physical prior is obtained; when the current synchronous sampling period is the second synchronous sampling period or a subsequent synchronous sampling period, the digital twin virtual physical features of the detection object in the previous synchronous sampling period are obtained as the digital twin physical prior. The digital twin physical prior is obtained based on the corrected surface temperature field, equivalent stress projection field, and surface residual deformation field of the detection object. S3. When at least one mode is a high-confidence mode, the low-confidence mode features are repaired using the high-confidence mode features and the digital twin physical prior. The repair reconstruction error and the repair confidence are determined based on the difference between the repaired features and the reference features. The repaired features are fused with the corresponding low-confidence mode features to obtain the repaired mode features. The high-confidence mode features and the repaired mode features are weighted to obtain the final weighted features of each mode. S4. When all modes are low-confidence modes, output a preset defect category, determine the detected object as defective, terminate the subsequent defect prediction of the current synchronous sampling period, and do not use the data of the current synchronous sampling period to update the digital twin recursive state; the preset defect category is used to characterize that the detection data of each mode in the current synchronous sampling period does not meet the confidence requirements, and is not used to characterize the specific defect type, location or size.

[0054] By utilizing the high-confidence modal features of the current synchronous sampling period and the physical prior of the digital twin, the low-confidence modal features are repaired, including: Low-confidence modal features are used as query features. High-confidence modal features and the digital twin physical prior used in the current synchronization sampling period are aligned and used as the repair information source. Cross-modal attention processing is used to process the query features and the repair information source to obtain the repair features corresponding to the low-confidence modality. Specifically, when one modality is a low-confidence modality, the other two high-confidence modal features and the digital twin physical prior used in the current synchronization sampling period are used for repair. When two modalities are low-confidence modalities, the remaining high-confidence modal features and the digital twin physical prior used in the current synchronization sampling period are used to repair the two low-confidence modal features respectively. The digital twin physical prior used in the current synchronization sampling period is the initialized digital twin physical prior in the first synchronization sampling period, and the digital twin virtual physical features generated in the previous synchronization sampling period in the second synchronization sampling period and subsequent synchronization sampling periods.

[0055] Specifically, the digital twin physical prior used in the current synchronous sampling cycle is not the abstract hidden layer of a regular neural network, but rather the physical state features corresponding point-by-point to the CAD spatial grid of the actual object being detected. The first synchronous sampling cycle uses the initialized digital twin physical prior, while the second and subsequent synchronous sampling cycles use the virtual physical features of the digital twin generated in the previous synchronous sampling cycle. Each row corresponds to a CAD spatial grid position on the phone casing, used to characterize the thermal-mechanical-deformation state at the corresponding position.

[0056] Query features: In cross-modal attention, the Query is the low-confidence modal feature to be patched.

[0057] Repair information source: The source of key and value information in cross-modal attention is composed of high-confidence modal features of the current synchronization sampling period and digital twin physical priors used in the current synchronization sampling period.

[0058] Cross-modal attention: Scaling dot product attention is performed based on Query, Key, and Value, enabling low-confidence modalities to obtain patching information from high-confidence modalities and historical physical states.

[0059] Repair features: Features from cross-modal attention outputs used to recover effective information from low-confidence modalities.

[0060] Specifically, for cases where at least one high-confidence modality exists, the low-confidence modality features serve as the Query, while the high-confidence modality features, along with the digital twin physical prior used in the current synchronization sampling period, are aligned across domains and used together as the Key and Value.

[0061]

[0062]

[0063]

[0064] in, , and These represent the parts used to repair the first... The query features corresponding to each low-confidence modality ( ), key features ( ) and value characteristics ( The vectors are represented in matrix form, with each row corresponding to a query vector, key vector, or value vector. This indicates a low-confidence modal feature that needs to be repaired; This represents the set of high-confidence modes in the current synchronous sampling period; Represents high-confidence modes Features; This indicates the digital twin physical prior used in the current synchronous sampling period; Indicates the use of virtual physical features of digital twins to connect with the first Mapping of cross-domain features for each modality; Indicates feature splicing, for Each query vector and The dimension of each key vector in the array. This is the scaling factor.

[0065] , and They represent , and The linear mapping matrix; This represents the normalized attention weights; This represents the repair feature corresponding to the low-confidence mode.

[0066] In a specific example, when the optical mode is a low-confidence mode, the optical features can be... As High-confidence thermal imaging features Ultrasound characteristics The digital twin physical prior used in the current synchronous sampling period After cross-domain alignment, they jointly serve as and The repair features corresponding to the optical modes are obtained. .

[0067] When the optical, thermal imaging, and ultrasonic modalities are all below the confidence threshold, a preset defect category is output, the current mobile phone casing is judged to be defective, the subsequent defect prediction of the current synchronous sampling cycle is terminated, and the digital twin recursive state is not updated with the data of the synchronous sampling cycle. The preset defect category indicates that the detection data of the three modalities do not meet the confidence requirements, and can be used as a conservative rejection result of the production line. It does not correspond to specific physical defect types such as scratches, cracks, or bubbles, and does not output the defect location or size.

[0068] The repair reconstruction error and post-repair confidence are determined based on the difference between the repaired features and the reference features, including: The repair and reconstruction error is determined based on the difference between the repaired features and the reference features. The reference features are obtained from at least one of the following: modal features of the same detection object in adjacent stable synchronous sampling periods before the current synchronous sampling period, cross-modal mapping results of high-confidence modal features in the current synchronous sampling period, or undamaged modal features corresponding to the training phase. The repaired confidence level of the low-confidence mode is obtained by weighting the repair reconstruction error and the mode confidence level corresponding to the low-confidence mode.

[0069] Specifically, reference features are control features used to evaluate the quality of feature restoration after patching. During online inference, reference features can be obtained from the cross-modal mapping results of modal features of the same detected object in adjacent stable synchronous sampling periods before the current synchronous sampling period, or from high-confidence modal features in the current synchronous sampling period; during training, undamaged modal features corresponding to the training phase can also be used. This avoids using future data after the current period for online inference.

[0070] Repair and Reconstruction Error: An error metric used to measure the relative difference between the repaired feature and the reference feature.

[0071] Post-repair confidence: The reliability index of the repair result obtained by combining the confidence of the original mode and the repair reconstruction error is used to determine the participation weight of the repaired mode in the subsequent process.

[0072]

[0073] in, Indicates the repair and reconstruction error; Indicates repair features; Indicates a reference feature; Denotes the Frobenius norm; To prevent smoothing constants with zero denominators, online reference feature sources are limited to historical stable features prior to the current synchronization sampling period or high-confidence cross-modal mapping results for the current period; undamaged original features can be used during the training phase.

[0074]

[0075] in, Indicates the confidence level after repair; This indicates the modal confidence level before repair; Indicates the repair and reconstruction error; ∈[0,1] represents the weighting coefficient between the original mode confidence and the repair quality.

[0076]

[0077] in, Indicates the gated residual fusion ratio; Indicates the confidence level after repair; Indicates the modal confidence level before repair; This is the smoothing constant.

[0078]

[0079] in, Indicates the modal characteristics after repair; Indicates repair features; This represents the corresponding original low-confidence modal feature.

[0080] Specifically, participation weights are continuous participation strengths obtained through nonlinear mapping based on modal confidence or repaired confidence, used to control the strength of each modal feature entering the subsequent graph structure.

[0081] Final weighted features: The features obtained by multiplying the original features of the unpatched mode or the patched features of the patched mode by the corresponding participation weights.

[0082]

[0083]

[0084] The first formula is used for modes that do not trigger patching, and the second formula is used for modes that trigger patching. This indicates that the original mode participates in the weighting. This indicates that the repaired version will be included in the weighting. Let m be the weight magnitude parameter for the m-th mode. Let the slope of the Logistic mapping be the slope of the m-th mode. To pre-set the reliability threshold, both equations use the same set of values ​​corresponding to mode m. and The parameters are determined by the training or validation process.

[0085] For modalities that have not been repaired, their participation weights are used to weight the original modal features. For modalities that have been repaired, the updated participation weights are used to weight the repaired modal features, resulting in the final weighted features for each modality. This process ensures consistency in repair quality, original confidence, and dynamic participation strength.

[0086] S5. Map the final weighted features to a unified feature dimension and organize them along the node dimension according to modality and spatial location to obtain a modality-spatial grid node graph; aggregate node features based on the spatial adjacency and cross-modal association relationships in the modality-spatial grid node graph to obtain graph topology interaction features; Specifically, unifying feature dimensions refers to mapping the originally different number of feature channels of different modalities to the same d-dimensional feature space through independent learnable projection layers.

[0087] Unified dimensional features: The final weighted features of each modality are obtained after being mapped through the corresponding learnable projection layer. Feature matrix.

[0088]

[0089] in, This represents the final weighted feature obtained after confidence assessment, necessary repair, gated residual fusion, and dynamic weighting of the m-th mode; and Let represent the weight matrix and bias of the learnable projection layer for the m-th modality, respectively; Let N represent the feature of the m-th mode after projection onto a unified d-dimensional space, and let N represent the number of spatial grid positions.

[0090] The final weighted features of each modality are mapped to a unified feature dimension to obtain the unified dimension features corresponding to each modality. These unified dimension features are then organized along the node dimension according to modality and spatial location, including: The final weighted features corresponding to optical data, thermal imaging data and ultrasound data are respectively mapped to the same d-dimensional feature space to obtain the unified dimensional features corresponding to the three modalities; Assuming each mode corresponds to N spatial grid locations, the unified dimensional features corresponding to the three modes are stacked along the node dimension to form a mode-spatial grid node graph containing 3N nodes. Among them, spatial adjacency is established between nodes with adjacent spatial locations in the same mode, and cross-modal association is established between nodes corresponding to different modes at the same spatial location.

[0091] Specifically, node dimension: refers to the dimension in which spatial grid features of different modalities are arranged as independent nodes, which is different from splicing three modalities along the channel dimension.

[0092] Modal-spatial grid node: An independent graph node corresponding to one mode and one spatial grid position.

[0093] Modal-spatial grid node graph: A graph structure with 3N modal-spatial grid nodes as the node set, and edge sets consisting of spatial adjacent edges, cross-modal edges, and self-connected edges.

[0094] Spatial adjacency: The connection between two nodes that are spatially adjacent in the same modality.

[0095] Cross-modal association: The association between nodes of different modes at the same spatial grid location.

[0096]

[0097] in, This represents the input features of a shallow graph formed by stacking nodes along their dimensions; This indicates stacking along the node dimension; , and These represent the unified dimensional features of optical, thermal imaging, and ultrasonic modes, respectively. Indicates each of the three modes The total number of nodes corresponding to each spatial grid location.

[0098]

[0099] in, A modal-space grid node diagram representing the current synchronous sampling period; Represents a set of nodes, containing One node; Represents the set of edges.

[0100]

[0101] in, Indicates the graph node index; Indicates the first Node characteristics of a modal-space grid node; This represents the first feature in the input features of the shallow graph. OK Dimensional features.

[0102] Based on the spatial adjacency and cross-modal association relationships in the modal-spatial grid node graph, node features are aggregated to obtain the graph topological interaction features of the current synchronous sampling period, including: Establish spatial adjacency relationships based on the adjacency relationships between different spatial grid locations within the same mode; Establish cross-modal association relationships based on the feature similarity between nodes corresponding to different modes at the same spatial grid location; Based on spatial adjacency and cross-modal association, the node features in the modal-spatial grid node graph are aggregated, so that each node aggregates the corresponding spatial adjacent node features and other modal node features at the same spatial grid location, thus obtaining the graph topology interaction features of the current synchronous sampling period.

[0103] Specifically, spatial adjacency matrix : A modal spatial connection matrix established based on the shared edge relationships of eight neighborhoods of a regular two-dimensional mesh or an irregular CAD mesh.

[0104] Cross-modal adjacency matrix: An association matrix established based on the non-negative cosine similarity between nodes of different modalities at the same spatial location.

[0105] Self-connection matrix: Represents the identity matrix of each node and its own connections, used to preserve the node's own characteristics when aggregating neighborhood information.

[0106] Graph topology interaction features: Mid-level node features obtained by aggregating information about itself, adjacent spatial locations of the same modality, and other modalities at the same spatial location.

[0107]

[0108] in, Represents the set of spatially adjacent edges; Represents the set of cross-modal edges; Represents the set of self-connected edges; Represents the set of edges in a modal-space grid node graph.

[0109] For regular 2D detection meshes, spatial adjacency relationships are determined using eight-neighborhoods; for irregular CAD meshes, spatial adjacency relationships are determined based on whether two meshes share an edge. Spatial Adjacency Matrix When the spatial edge weight is set to 1:

[0110] in, and These represent the modes to which node i and node j belong, respectively. and Representing nodes respectively and nodes The corresponding spatial grid position; This indicates that the positions of two spatial grids satisfy an eight-neighbor relationship or a shared edge relationship between CAD grids.

[0111] Cross-modal edges are established between nodes belonging to different modalities at the same spatial grid location. The weights of the cross-modal edges are based on the non-negative cosine similarity of the node features.

[0112] in, Represents a node With nodes Cross-modal edge weights between; and Indicates the characteristics of the corresponding node; Denotes the L2 norm; ε is the smoothing constant; Used to truncate negative similarity to 0. This is a transpose.

[0113] Self-connecting edges represent each node connected to itself, and their matrix is:

[0114] in, Represents a self-connected matrix; express 3N An identity matrix of order 1. Self-joining of the identity matrix is ​​a common self-loop processing method in GCN, used to allow nodes to retain their own characteristics while aggregating neighborhood information.

[0115] Right now:

[0116] Spatial adjacency matrix, cross-modal adjacency matrix, and self-connection matrix are combined according to the following formula:

[0117] in, Represents the combined adjacency matrix; This represents the cross-modal edge weight coefficient, used to control the propagation strength of cross-modal information relative to spatial adjacency information.

[0118] The linear combination of adjacency matrices is a common graph modeling method, but the specific combination of the three types of relations and the cross-modal propagation coefficients are topological organization methods designed for multimodal spatial graphs in this invention; they are used to uniformly control the relative roles of spatial information and cross-modal information in graph propagation.

[0119] Calculate the degree matrix based on the combined adjacency matrix:

[0120]

[0121] in, Represents a node i The connection strength; Represents the i-th element in the combined adjacency matrix Line 1 j Column elements, This represents a symmetric normalized adjacency matrix, used to reduce the impact of differences in degree between different nodes on the strength of neighborhood aggregation. For degree matrix, The position located in the degree matrix is ​​the th Line 1 The elements of the column.

[0122] Graph convolution is calculated as follows:

[0123] in, The graph topology interaction features representing the current synchronization sampling period; Represents a nonlinear activation function; This indicates that graph convolution can learn the weight matrix; Channel dimension representing the topological interaction features of a graph; Represents a symmetric normalized adjacency matrix; express Shallow graph input features.

[0124] In this scheme, each modal-space grid node is used to simultaneously absorb information from itself, neighboring locations of the same modality, and other modalities at the same location.

[0125] S6. Map the physical prior of the digital twin to the modal-spatial grid node graph according to the spatial location to obtain the prior features of the digital twin; form a node confidence mask based on the node reliability and gate the graph topology interaction features; form a physical prior mask based on the reliability of the physical prior of the digital twin and gate the prior features of the digital twin; and concatenate the two types of gated features to obtain the fused features. Specifically, the prior features of the digital twin are obtained by mapping the initial digital twin physical features of the first synchronous sampling period or the digital twin virtual physical features of the second synchronous sampling period and the previous synchronous sampling period through channels and copying them to different modal nodes corresponding to the same spatial grid position, thereby obtaining features corresponding to 3N modal-spatial grid nodes.

[0126] Channel mapping: N× The 3D digital twin virtual physical feature mapping is suitable for the fusion of current graph nodes. N× Dimensional features.

[0127] For the second synchronous sampling period and subsequent synchronous sampling periods, let the digital twin virtual physical characteristics of the previous synchronous sampling period be... for N× 3D physical mesh features, each row corresponds to a 2D CAD spatial mesh; through cross-domain mapping Map it to N× Then, it is copied to the three modal nodes. The first synchronous sampling period is... Use the same channel mapping and node replication operations:

[0128] in, Indicates and Digital twin prior features corresponding to each modal-space grid node; This represents a vector of length 3 consisting entirely of 1s. Represents the Kronecker product; Indicates cross-domain / channel mapping; This indicates the digital twin physical prior used in the current synchronous sampling period; This represents the channel dimension of the original digital twin virtual physical features; This represents the channel dimension of the mapped digital twin prior features.

[0129] Effective confidence of a node: The reliability value assigned to each node before entering the graph structure; nodes that have not triggered repair use the corresponding modal confidence, and nodes that have triggered repair use the corresponding post-repair confidence.

[0130] Graph propagation confidence: The confidence level obtained by propagating the effective confidence of neighboring nodes using the normalized adjacency matrix employed in graph topological interactions.

[0131] Update node confidence: The confidence level is obtained by weighting the node's own effective confidence level with the graph propagation confidence level, and is used for reliability control after graph topology interaction.

[0132] Fusion features: The current periodic graph topological interaction features and historical digital twin prior features are obtained by splicing along the channel dimension after being gated by their own reliability masks.

[0133] After the graph topology interaction is completed, the effective confidence scores of nodes can be aggregated using a symmetric normalized adjacency matrix. Nodes that have not triggered repair are assigned the corresponding modal confidence scores, while nodes that have triggered repair are assigned the corresponding repaired confidence scores.

[0134]

[0135] in, Represents a node Effective confidence level before entering the graph structure; The symmetric normalized adjacency matrix used in graph convolution represents the first... Line 1 Column elements; Represents the target node The graph propagation confidence is obtained by aggregating the reliability of neighboring nodes. Since the combined adjacency matrix includes self-connections, the aggregation simultaneously considers the confidence of the node itself, adjacent positions of the same modality, and other modalities of the same position.

[0136] The node's own effective confidence is weighted by the graph propagation confidence to obtain the updated node confidence, and a broadcast mask is constructed based on this.

[0137]

[0138] in, ∈[0,1] represents the retention coefficient of the node's own effective confidence. For nodes The confidence level of the updated node; This indicates that the confidence scores of 3N scalars will be updated along the current graph features in node order. Each channel is replicated to form ∈ .

[0139] Node confidence mask Graph topology interaction features used only for gating the current synchronization sampling period:

[0140] in, The graph topology interaction features representing the current synchronization sampling period; ⊙ represents the current graph feature after node confidence gating; ⊙ represents element-wise multiplication.

[0141] The physical prior used in the current synchronous sampling period is not reused. The effective confidence level of the physical prior is calculated based on its comprehensive physical residual and the actual time interval between adjacent synchronous sampling periods:

[0142] in, The comprehensive physical residual is formed after normalizing and aligning the residuals of heat conduction, stress balance and residual deformation constraints to 3N nodes; This represents the actual time interval between adjacent synchronous sampling periods; This is a pre-defined or learnable mapping that monotonically does not increase with respect to the integrated physical residuals and the actual time interval. The first synchronous sampling period uses the calibration residuals of the initial physical priors and the time interval from initialization to the first sampling.

[0143]

[0144]

[0145] After the two types of features are independently gated, they are spliced ​​along the channel dimension to obtain the fused features of the current synchronous sampling period:

[0146] in, Only reflects the current reliability of sensor / graph nodes and gates the current state. ; Only reflects the physical consistency and timeliness of historical physical priors and is gated. The two do not share the same mask, thus avoiding the need for reliable historical physical priors for synchronous suppression even when the current sensor quality deteriorates.

[0147] S7. Aggregate the fused features according to their spatial location and restore them to the spatial grid corresponding to the detected object to obtain a digital twin enhanced feature map; input the digital twin enhanced feature map into the defect prediction model to obtain the defect prediction result; when the fused features are obtained, generate a digital twin virtual physical feature for the current synchronous sampling period based on the fused features, and use the digital twin virtual physical feature as the digital twin physical prior for the next synchronous sampling period of the same detected object; Specifically, spatial location aggregation: recombine the optical, thermal imaging, and ultrasonic modal nodes at the same spatial grid location into a single spatial grid feature.

[0148] Digital twin enhanced feature map: obtained by aggregating and rearranging node-level fused features according to spatial location. The feature map maintains a one-to-one correspondence with the product's CAD unfolded coordinates or a unified detection grid.

[0149] Specifically, to generate digital twin augmented feature maps that simultaneously support classification, localization, and size regression, the system does not perform global pooling on the localization branch. For the nth spatial grid, modality-aware aggregation is performed on the three modal nodes corresponding to that location:

[0150] in, Indicates the spatial grid location index; Indicates the first After aggregating spatial grid locations C 3D grid features; , and Ψ represents the optical, thermal imaging, and ultrasonic node fusion features corresponding to the spatial location, respectively; Ψ represents the feature mapping operation on the splicing result of the three modal nodes; C represents the number of feature channels of the aggregated mesh.

[0151]

[0152] in, A digital twin enhanced feature map representing the current synchronous sampling period; This indicates that tensors are rearranged according to a uniform spatial grid order; and The height and width of the uniform detection grid are represented by C; C represents the number of channels in the enhanced feature map. Indicates the number of spatial grid locations.

[0153] Classification prediction heads can be used Then global pooling is performed; the defect localization prediction header is retained. Spatial grid; the defect size regression head performs regression based on the local features corresponding to the localized region. Thus, the spatial structure of the enhanced feature map and the relationship between the three tasks are clearly defined.

[0154] Specifically, the defect prediction model is a prediction network that takes digital twin enhanced feature maps as input and may include a classification head, a localization head, and a size regression head.

[0155] Defect prediction results include at least one of defect category, defect spatial location, or defect size.

[0156] Classification Head: A network branch that performs classification prediction on the augmented feature maps of the digital twin. It can perform global pooling after the augmented feature maps are augmented.

[0157] Positioning head: Preserves the complete H×W spatial mesh and outputs the predicted branch of the defect spatial location.

[0158] Size regression head: outputs a prediction branch for defect size based on the local features corresponding to the localized region.

[0159] The digital twin enhanced feature map maintains the spatial correspondence with the CAD unfolded coordinates of the detected object or a unified detection grid. For classification tasks, global pooling can be performed after F_aug,p(k); for localization tasks, the complete H×W spatial grid is preserved; and for size prediction tasks, regression is performed based on the local features corresponding to the localized region.

[0160] The defect prediction model can be trained in stages. The first stage pre-trains a physically constrained digital twin generator network, with supervision signals including real or simulated temperature fields, equivalent stress fields, residual deformation fields, and optional real visual images. The second stage generates damaged features through artificial occlusion, noise addition, overexposure, thermal noise, and ultrasonic coupling attenuation, and pre-trains low-confidence modal repair, dynamic weights, and graph topology interactions. The third stage uses defect category, defect location, and defect size labels for joint fine-tuning.

[0161]

[0162] in, This indicates the total loss from joint fine-tuning; The defect prediction loss consists of classification loss, location loss, and size regression loss. Indicates the loss of the digital twin generative network; Represents the feature reconstruction loss; and These represent the weights of the generator network loss and the feature reconstruction loss in the joint total loss, respectively.

[0163] The methods for obtaining the virtual physical features and prior features of the digital twin during the current synchronous sampling period include: Based on the fusion features of the current synchronous sampling period, physical state features corresponding to the spatial location of the detected object are generated, and the digital twin virtual physical features of the current synchronous sampling period are obtained. The digital twin virtual physical features of the current synchronous sampling period are used as the digital twin physical prior of the same detected object in the next synchronous sampling period, so that the next synchronous sampling period can be mapped to the corresponding modal-spatial grid node according to the spatial location, and the digital twin prior features corresponding to the next synchronous sampling period are obtained.

[0164] Based on the fused features, a digital twin virtual physical feature for the current synchronous sampling period is generated, and the digital twin virtual physical feature is used as the digital twin physical prior for the next synchronous sampling period of the same detection object, including: Based on the fusion features of the current synchronous sampling period, the corrected surface temperature field, equivalent stress projection field, and surface residual deformation field corresponding to the two-dimensional spatial mesh of the detection object are generated. Among them, the corrected surface temperature field is subject to transient heat conduction physical constraints, the equivalent stress projection field is subject to displacement-strain relationship, constitutive relationship and stress balance physical constraints, and the surface residual deformation field is subject to residual deformation supervision constraints and geometric compatibility constraints. The corrected surface temperature field, equivalent stress projection field, and surface residual deformation field are feature-mapped to obtain the digital twin virtual physical features of the current synchronous sampling period, which are then used as the digital twin physical prior for the next synchronous sampling period of the same detection object.

[0165] Specifically, physical state characteristics: temperature, stress, and residual deformation related characteristics corresponding to the spatial location of the detected object, generated by fusing features from the current synchronous sampling period under physical constraints.

[0166] Correcting the surface temperature field The surface temperature field, which is physically constrained and corrected and corresponds to the two-dimensional CAD mesh, is used to complete and correct thermal imaging measurements, calculate thermal conduction residuals, and serve as a thermal expansion load.

[0167] Equivalent stress projection field The two-dimensional CAD mesh field is obtained by projecting the von Mises equivalent stress in the three-dimensional physical domain along the product thickness direction.

[0168] Surface residual deformation field : A two-dimensional CAD mesh field formed by the permanent offset of the product surface displacement relative to the reference geometric position.

[0169] Two-dimensional equivalent physical field diagram: an H×W two-dimensional mesh field obtained by extracting temperature, stress and displacement information in the three-dimensional physical domain through surface extraction, thickness direction projection or equivalent mapping.

[0170] Digital twin physical prior: The name of the application of the digital twin virtual physical feature generated in the current synchronous sampling period in the next synchronous sampling period; the digital twin physical prior of the first synchronous sampling period is the initial digital twin physical feature F_twin,p^(0). The two are mapped spatially and through channels to form the digital twin prior features of the corresponding synchronous sampling period.

[0171] Physical constraint digital twin modeling can be implemented using PI-GAN. The physical constraint equations and finite element weakly supervised data can be defined in a three-dimensional physical domain Ω that includes the thickness direction z, but the direct output used for feature derivation and defect prediction in this method is a two-dimensional equivalent physical field map corresponding to the two-dimensional CAD unfolded mesh H×W of the product.

[0172] The test object in this embodiment is a PC+ABS mobile phone casing, and the material parameters include density. Specific heat capacity thermal conductivity Elasticity matrix C, Poisson's ratio Coefficient of thermal expansion and reference temperature The above material parameters were obtained from the PC+ABS material parameter library, supplier test data, or calibration tests using materials from the same batch as the actual mobile phone casing.

[0173] For thin-walled products, the three-dimensional physical field can be projected into a two-dimensional equivalent physical field:

[0174]

[0175]

[0176] in, Representing 2D CAD grid coordinates Corrected surface temperature at the location; Represents the position of the surface to be detected in the three-dimensional physical domain. The temperature at that location. The location of the surface to be inspected is determined by the geometric position of the surface of the object being inspected.

[0177] Represents a two-dimensional equivalent stress projection field; This represents the von Mises equivalent stress in a three-dimensional physical domain; This represents the product thickness, which is derived from the product dimensions / CAD geometry information. For products that are not suitable for maximum value projection, thickness-weighted averaging, multi-channel mapping, or equivalent projection determined by the CAD mesh can be used.

[0178] Represents the residual deformation field on the surface; This represents the three-dimensional displacement at the location on the surface to be detected. This represents the reference displacement corresponding to the baseline geometric position. Displacement and residual deformation reference data can be obtained through digital image correlation, laser profilometry, or 3D scanning.

[0179] Transient heat conduction physical constraints: No. Each synchronous sampling period corresponds to an actual physical time τ(p,k), and the actual time interval between adjacent periods Δτ(p,k) = τ(p,k) - τ(p,k-1) is obtained from the hardware timestamp. When the sampling interval is not constant, the time derivative of heat conduction is approximated according to the actual time interval. The transient heat conduction residual is:

[0180] in, This represents the residual of the heat conduction equation; Indicates the density of the material; Indicates the specific heat capacity of the material; Indicates the thermal conductivity of the material; , and The source of this information is a material parameter library, supplier testing data, or calibration tests; T represents the temperature field. Represents continuous physical time; Represents the spatial differential operator; This indicates the heat source item.

[0181]

[0182] in, This represents the physical constraint loss of the temperature field; MSE represents the mean square error. Represents the physical domain; Represents the physical domain boundary; Describes boundary operators for temperature or heat flux; This indicates a boundary measurement value, the source of which is a boundary measurement. This represents the initial temperature field, which can be provided by the reference temperature field applied during the initialization of the object being detected. and These represent the weights of the boundary condition loss and the initial condition loss, respectively.

[0183] Stress field physical constraints:

[0184]

[0185]

[0186] in, Represents the displacement field; This represents the strain tensor determined by the displacement field; It represents the displacement gradient; the superscript T indicates transpose.

[0187] Represents the stress tensor; The material elasticity matrix is ​​represented by material parameter libraries, supplier test data, or calibration tests. Indicates the coefficient of thermal expansion; Represents the temperature field; Indicates reference temperature; Represents a unit tensor; This represents the residual strain term.

[0188] 'b' represents the stress balance residual; 'b' represents the body force term.

[0189] This represents the loss due to physical constraints in the stress field; Represents boundary operators for displacement or surface force; Indicates the corresponding mechanical boundary value; This represents the weight of the mechanical boundary loss.

[0190] Physical constraints on residual deformation:

[0191] in, This represents the predicted residual deformation data; This represents the displacement field under the current physical state; This represents the reference displacement corresponding to the baseline geometry. Displacement and residual deformation reference data can be obtained through digital image correlation, laser profilometry, or 3D scanning.

[0192]

[0193] in, This represents the residual deformation constraint loss; The residual deformation reference data can be derived from digital image correlation, laser profilometry, or 3D scanning. This represents the predicted residual deformation data; Indicates geometric compatibility residual; Indicates the weight of the geometric compatibility residual loss. For reference deformation data.

[0194] The physical constraint loss and the generation of virtual physical features in digital twins are represented as follows:

[0195] in, This represents the overall physical constraint loss; , and These represent the physical constraint losses due to temperature, stress, and residual deformation, respectively. , and These represent the weights of the losses for the three types of physical constraints. The specific weights are determined based on the physical residual scale and the validation set.

[0196]

[0197] in, This represents the total loss of the generator network; Indicates resistance to loss; Represents structural similarity loss; This represents the overall physical constraint loss; and These represent the weights of the structural similarity loss and the physical constraint loss, respectively. The virtual visual image output branch can be used to calculate the adversarial loss and structural similarity loss during the training phase; this branch and the discriminator are not run during the inference phase.

[0198] Temperature field reference data can be obtained from thermal imaging calibration data; displacement and residual deformation reference data can be obtained from digital image correlation, laser profilometry, or 3D scanning; stress field reference data can be obtained from strain measurements combined with constitutive relations, residual stress testing, or validated finite element simulations. When some physical quantities lack measured labels, finite element data calibrated with measured temperature or deformation data can be used as weak supervision.

[0199] Correcting the surface temperature field Used to correct and complete thermal imaging measurements, form a global temperature distribution consistent with the CAD mesh of the mobile phone casing, calculate thermal conduction residuals, and serve as thermal loads for inferring thermal expansion strain, stress, and residual deformation. , and After normalization and channel mapping, the common This refers to the digital twin virtual physical characteristics of the current synchronization sampling period, which can only be used in the next synchronization sampling period for the same mobile phone casing.

[0200] This application also discloses a terminal device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs a digital twin prior enhancement defect prediction method.

[0201] The terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server. The terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0202] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0203] The memory can be an internal storage unit of the terminal device, such as a hard disk or RAM of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the terminal device. Furthermore, the memory can be a combination of internal storage units and external storage devices of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0204] In this terminal device, a digital twin prior enhancement defect prediction method from the above embodiments is stored in the terminal device's memory and loaded and executed on the terminal device's processor for convenient use.

[0205] This application also discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it employs a digital twin prior enhancement defect prediction method as described in the above embodiments.

[0206] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0207] The above-described digital twin prior enhancement defect prediction method is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the method.

[0208] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0209] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A defect prediction method using digital twin prior enhancement, characterized in that, include: Acquire multimodal detection data of the same detection object during the current synchronous sampling period, perform spatial alignment and feature extraction on the multimodal detection data, and obtain the features of each modality; The confidence level of each modal feature is evaluated to determine the high-confidence mode and the low-confidence mode. The detection object is an injection molded part with a preset CAD geometric model and a unified detection grid. The modal features corresponding to the high-confidence mode are high-confidence modal features, and the modal features corresponding to the low-confidence mode are low-confidence modal features. When the current synchronous sampling period is the first synchronous sampling period, the initial digital twin physical prior is obtained; when the current synchronous sampling period is the second synchronous sampling period or a subsequent synchronous sampling period, the digital twin virtual physical features of the detection object in the previous synchronous sampling period are obtained as the digital twin physical prior. The digital twin physical prior is obtained based on the corrected surface temperature field, equivalent stress projection field, and surface residual deformation field of the detection object. When at least one mode is a high-confidence mode, the low-confidence mode features are repaired using high-confidence mode features and digital twin physical priors. The repair reconstruction error and the repaired confidence are determined based on the difference between the repaired features and the reference features. The repaired features are fused with the corresponding low-confidence mode features to obtain the repaired mode features. The high-confidence mode features and the repaired mode features are weighted to obtain the final weighted features of each mode. When all modalities are low-confidence modalities, a preset defect category is output, the detected object is determined to be defective, the subsequent defect prediction of the current synchronous sampling period is terminated, and the data of the current synchronous sampling period is not used to update the digital twin recursive state; wherein, the preset defect category is used to characterize that the detection data of each modality does not meet the confidence requirements; When the final weighted features are obtained, they are mapped to a unified feature dimension and organized along the node dimension according to modality and spatial location to obtain a modality-spatial grid node graph; based on the spatial adjacency and cross-modal association relationships in the modality-spatial grid node graph, node features are aggregated to obtain graph topology interaction features; The digital twin physical prior is mapped to the modal-spatial grid node graph according to its spatial location to obtain digital twin prior features; a node confidence mask is formed based on the node reliability and the graph topology interaction features are gated; a physical prior mask is formed based on the reliability of the digital twin physical prior and the digital twin prior features are gated; the two types of gated features are concatenated to obtain fused features. The fused features are aggregated according to their spatial location and restored to the spatial grid corresponding to the detected object to obtain a digital twin enhanced feature map; the digital twin enhanced feature map is input into the defect prediction model to obtain the defect prediction result; when the fused features are obtained, a digital twin virtual physical feature for the current synchronous sampling period is generated based on the fused features, and the digital twin virtual physical feature is used as the digital twin physical prior for the next synchronous sampling period of the same detected object.

2. The defect prediction method with digital twin prior enhancement according to claim 1, characterized in that, The multimodal detection data includes optical data, thermal imaging data, and ultrasonic data. The optical data, thermal imaging data, and ultrasonic data are collected synchronously within the same synchronous sampling period. The spatial alignment is performed based on the product CAD unfolded coordinates of the detected object or a unified detection grid, so that each modal feature corresponds to the same spatial grid position.

3. The defect prediction method with digital twin prior enhancement according to claim 1, characterized in that, The step of evaluating the confidence level of each modal feature to determine the high-confidence mode and the low-confidence mode includes: The probability corresponding to each feature value interval is determined based on the feature value distribution of each modality feature, and the information entropy corresponding to each modality feature is determined based on the probability. The signal-to-noise ratio corresponding to each modal feature is determined based on the effective signal power and noise power in each modal feature; The spatial continuity score for each modal feature is determined based on the differences between features corresponding to spatially adjacent spatial grids in each modal feature. Based on the information entropy reference center and allowable deviation corresponding to normal samples, the information entropy is converted into an information entropy quality score. Based on the upper and lower bounds of the signal-to-noise ratio and the upper and lower bounds of the spatial continuity score corresponding to the training set or stable production batch, the signal-to-noise ratio and the spatial continuity score are normalized respectively to obtain the signal-to-noise ratio quality score and the spatial continuity quality score. The information entropy quality score, signal-to-noise ratio quality score, and spatial continuity quality score are weighted to obtain the modal confidence level corresponding to each modal feature; The modality confidence level is compared with a preset confidence threshold. Modalities with a confidence level not lower than the preset confidence threshold are identified as high-confidence modes, and modalities with a confidence level lower than the preset confidence threshold are identified as low-confidence modes.

4. The defect prediction method with digital twin prior enhancement according to claim 1, characterized in that, The step of repairing the low-confidence modal features using the high-confidence modal features of the current synchronous sampling period and the digital twin physical prior includes: The low-confidence modal features are used as query features, and the high-confidence modal features and the digital twin physical prior are aligned and used as the repair information source. The query features and the repair information source are processed through cross-modal attention to obtain the repair features corresponding to the low-confidence modality. Specifically, when one modality is a low-confidence modality, the other two high-confidence modal features and the digital twin physical prior are used for repair. When two modalities are low-confidence modalities, the remaining high-confidence modal features and the digital twin physical prior are used to repair the two low-confidence modal features respectively.

5. The defect prediction method with digital twin prior enhancement according to claim 1, characterized in that, The process of determining the repair reconstruction error and post-repair confidence based on the difference between the repaired features and the reference features includes: The repair and reconstruction error is determined based on the difference between the repaired feature and the reference feature. The reference feature is obtained from at least one of the modal features of the same detection object in adjacent stable synchronous sampling periods before the current synchronous sampling period, the cross-modal mapping result of the high-confidence modal features in the current synchronous sampling period, or the undamaged modal features corresponding to the training phase. The repaired confidence level of the low-confidence mode is obtained by weighting the repair and reconstruction error and the mode confidence level corresponding to the low-confidence mode.

6. The defect prediction method with digital twin prior enhancement according to claim 1, characterized in that, The process of mapping the final weighted features to a unified feature dimension and organizing them along the node dimension according to modality and spatial location to obtain a modality-space grid node graph includes: The final weighted features corresponding to optical data, thermal imaging data and ultrasound data are respectively mapped to the same d-dimensional feature space to obtain the unified dimensional features corresponding to the three modalities; Assuming each mode corresponds to N spatial grid positions, the unified dimensional features corresponding to the three modes are stacked along the node dimension to form the mode-spatial grid node graph containing 3N nodes; wherein, spatial adjacency is established between nodes with adjacent spatial positions in the same mode, and cross-modal association is established between nodes corresponding to different modes at the same spatial position.

7. The defect prediction method with digital twin prior enhancement according to claim 1, characterized in that, The aggregation of node features based on spatial adjacency and cross-modal association relationships in the modal-spatial grid node graph yields graph topological interaction features, including: Establish spatial adjacency relationships based on the adjacency relationships between different spatial grid locations within the same mode; Establish cross-modal association relationships based on the feature similarity between nodes corresponding to different modes at the same spatial grid location; Based on the spatial adjacency relationship and cross-modal association relationship, the node features in the modal-spatial grid node graph are aggregated, so that each node aggregates the corresponding spatial adjacent node features and other modal node features at the same spatial grid position, to obtain the graph topology interaction features of the current synchronous sampling period.

8. The defect prediction method with digital twin prior enhancement according to claim 1, characterized in that, Based on the fused features, a digital twin virtual physical feature for the current synchronous sampling period is generated, and the digital twin virtual physical feature is used as the digital twin physical prior for the next synchronous sampling period of the same detection object, including: Based on the fusion features of the current synchronous sampling period, a corrected surface temperature field, an equivalent stress projection field, and a surface residual deformation field corresponding to the two-dimensional spatial grid of the detected object are generated. The corrected surface temperature field is subject to transient heat conduction physical constraints, the equivalent stress projection field is subject to displacement-strain relations, constitutive relations, and stress balance physical constraints, and the surface residual deformation field is subject to residual deformation supervision constraints and geometric compatibility constraints. The corrected surface temperature field, the equivalent stress projection field, and the surface residual deformation field are feature-mapped to obtain the digital twin virtual physical features of the current synchronous sampling period, and these features are used as the digital twin physical priors for the next synchronous sampling period of the same detection object.

9. A terminal device, comprising a memory and a processor, characterized in that, The memory stores a computer program that can run on a processor, and when the processor loads and executes the computer program, it employs the method described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs the method described in any one of claims 1 to 8.

Images