An actuarial AOI detection method and system based on a multi-modal large model

Abstract

The application discloses an actuary AOI detection method and system based on a multi-modal large model, which comprises the following steps: constructing a multi-modal defect sample set, performing multi-modal feature extraction and fusion, and constructing a defect knowledge pedigree graph of a node-edge graph structure; performing semantic layering, causal propagation and feature aggregation on the multi-modal fusion features to generate a defect gene expression vector; establishing a process mechanism model corresponding to the defect type, performing reverse deduction and process-detection joint optimization based on the defect gene expression vector to obtain an optimal process-detection strategy; fusing and packaging the defect gene, the process parameter optimization result and the detection strategy to generate an actuary, mapping the actuary to generate a detection program code, and deploying the detection program code to an AOI device. The application breaks through the limitation of traditional manual programming, improves the AOI detection efficiency and precision, and realizes the adaptive generation and optimization of the detection program by breaking through the whole link of data-knowledge-process-program.

Description

Technical Field

[0001] This invention relates to the field of automated inspection technology, and in particular to an actuarial AOI inspection method and system based on a multimodal large model. Background Technology

[0002] Automated Optical Inspection (AOI) is a crucial quality control method in industries such as electronics manufacturing, semiconductor packaging, and automotive electronics. It primarily utilizes optical imaging and image recognition technologies to automatically detect issues such as welding defects, foreign object contamination, and structural misalignment. However, with product miniaturization and process complexity, traditional AOI programming methods are facing core bottlenecks including low efficiency, poor adaptability, and insufficient intelligence.

[0003] Currently, the development of AOI inspection programs is still primarily driven by human experience. Engineers need to manually set optical parameters, inspection logic, and defect thresholds, and the debugging process involves repeated trial and error. This results in long new product introduction cycles, quality dependence on individual capabilities, and insufficient consistency and reusability of defect detection. In practical engineering applications, the lack of systematic modeling of the "defect-parameter-process" relationship means that engineers must configure inspection strategies from scratch every time they encounter a new defect or process, severely limiting the efficiency and accuracy of AOI systems.

[0004] While current mainstream optimization methods achieve partial automation through template matching and machine learning algorithms, they still lack intelligent generalization capabilities across products and scenarios, failing to effectively support rapid response and dynamic optimization of complex defects. Furthermore, the fragmented storage of defect knowledge within enterprises makes it difficult to form a structured, reusable knowledge base, thus limiting experience transfer and large-scale application. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an actuarial AOI detection method and system based on a multimodal large model. By deeply integrating defect knowledge graphs with industrial mechanism models, it achieves adaptive generation and optimization of the detection program, breaking through the limitations of traditional manual programming and realizing a dual leap in AOI detection efficiency and accuracy.

[0006] This invention provides an actuarial AOI detection method based on a multimodal large model, the specific technical solution of which is as follows: S1: Collect multimodal data of the tested object at the same defect location, perform preprocessing, and construct a multimodal defect sample set; Based on the multimodal defect sample set, features are extracted from the multimodal data respectively, and multimodal feature fusion is performed to obtain multimodal fused features; Using defects, parameters, processes, and strategies as nodes, and the logical relationships between nodes as edges, a defect knowledge spectrum graph with a node-edge graph structure is constructed to generate an embedded defect knowledge spectrum base map. Among them, defect nodes represent defect types and multimodal characteristics, parameter nodes describe process control parameters, process nodes represent key process links and their causal mechanisms, strategy nodes define detection methods and optimization strategies, and edges define the logical relationships between nodes, such as causality, influence, optimization suggestions, etc.

[0007] Furthermore, the multimodal data includes AOI detection images, 3D contour data, spectral reflectance data, and / or X-ray image data; The preprocessing includes labeling the multimodal data with defect types, defect locations, morphological features, process parameters, and causes.

[0008] Furthermore, the Vision Transformer model, PointNet++ network model, and 1D-CNN network model are used respectively to extract the modal features of the AOI detection image, 3D contour data, and spectral reflectance data to obtain image modal features, 3D modal features, and spectral modal features. The extracted features are then concatenated to obtain multimodal fusion features.

[0009] Furthermore, the GraphSAGE framework is used to implement iterative propagation and aggregation of node features, generating an embedded defect knowledge hierarchy base map. The update formula is expressed as follows:

[0010] in, For the first k Layer nodes v Feature representation, For neighborhood feature aggregation function, σ For activation function, For the first Layer nodes Feature representation, Represents the set of node features. Let be the learnable weight matrix of the k-th layer. Let be the bias vector of the k-th layer.

[0011] S2: Decouple and map the multimodal fusion features according to three semantic layers: morphology layer, material layer, and process layer, to obtain the feature representation of each modality in each semantic layer, calculate the modality importance weight based on the defect type, and perform weighted fusion of multimodal features in the same semantic layer to obtain three-layer semantic fusion features; On the base map of the defect knowledge spectrum, a gated graph convolutional network is used to backpropagate along the causal edges to obtain causal chain inversion enhancement features. The three-layer semantic fusion features and the causal chain inversion enhancement features are then concatenated and dimensionality is reduced through an attention mapping network to generate a 128-dimensional defect gene expression vector that is both discriminative and causal.

[0012] Furthermore, modal importance weights are calculated using an attention mechanism, and multimodal features within the same semantic layer are weighted and fused, including: Global average pooling is used to obtain global description vectors for each modality in the same semantic layer. These vectors are then input into an attention mapping network (MLP) to calculate modality importance weights. Finally, the global description vectors and modality importance weights are fused to obtain the enhanced features of that semantic layer.

[0013] S3: Establish a process mechanism model corresponding to the defect type, take the defect gene expression vector as the target expression quantity, and solve the optimal process parameter combination by using the physical information neural network PINN and Bayesian inverse estimation based on the process mechanism model and physical constraint residuals. Within the adjustable process window, with the goals of minimizing defect risk, maximizing detection score, and taking manufacturing cost into account, the optimal combination of process parameters and detection parameters are jointly optimized by multiple objectives to obtain the final process-detection strategy.

[0014] Furthermore, the process mechanism model is used to describe the mapping relationship between defect manifestation and process parameters, including at least one of thermodynamic equations, fluid dynamics equations, mechanical equations and optical equations.

[0015] S4: Encapsulate the defect ID, defect gene expression vector, final process-detection strategy, confidence score, and interpretable link into a standardized actuarial algorithm; Based on the programming design map, the actuary is automatically mapped into the inspection program code that can be executed by the AOI equipment, and the inspection program is pushed to the AOI equipment on the production line through a one-click release interface to complete the inspection deployment.

[0016] This invention also provides an actuarial AOI detection system based on a multimodal large model, characterized in that, based on the above-described actuarial AOI detection method based on a multimodal large model, the system includes: The multimodal data acquisition and annotation module is used to acquire multimodal data of the tested object at the same defect location, perform multidimensional annotation, and construct a multimodal defect sample set; The multimodal feature extraction and fusion module is used to extract features from multimodal data and fuse multimodal features to obtain multimodal fused features. The Defect Knowledge Spectrum Base Graph Construction Module is used to construct a defect knowledge spectrum graph with defects, parameters, processes and strategies as nodes and logical relationships between nodes as edges, generating an embedded defect knowledge spectrum base graph. The defect gene identification module is used to perform semantic hierarchical analysis, causal propagation, and feature aggregation on multimodal fusion features to generate defect gene expression vectors. The process mechanism inversion and optimization module is used to establish a process mechanism model corresponding to the defect type. Based on the defect gene expression vector, it performs inverse deduction and process-detection joint optimization to obtain the optimal process-detection strategy. The actuary encapsulation and one-click programming push module is used to integrate and encapsulate defect genes, process parameter optimization results and detection strategies to generate actuaries, map the actuaries to generate detection program code, and deploy them to AOI equipment.

[0017] The beneficial effects of this invention are as follows: This invention performs multimodal feature encoding on AOI inspection images, 3D contours, and spectral data. Combined with process parameters, it establishes a defect knowledge hierarchy map with a four-layer causal relationship of "defect-parameter-process-strategy," forming a unified and structured knowledge representation. Then, using semantic hierarchical decoupling and modal selectivity enhancement, it extracts discriminative and causal features and generates defect gene expression vectors with process traceability capabilities through causal chain collaborative inversion. Subsequently, by constructing defect gene and process mechanism models, multi-type mechanism analysis, and inverse inversion optimization modules, it achieves the inverse deduction of defect manifestations to process parameters and completes the joint optimization of process and detection strategies under process constraints, outputting the optimal parameter combination to improve defect identification accuracy and detection strategy effectiveness. Finally, the optimization results are encapsulated into standardized actuaries, and detection programs are automatically generated through programming design maps and pushed to front-end devices with one click, significantly improving programming efficiency and detection quality, and comprehensively reducing manual intervention and reliance on experience. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0019] Figure 2 This is a schematic diagram of the multimodal feature fusion and defect gene identification process. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention are clearly and completely described in the following description. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is conventionally placed during use, or the orientation or positional relationship in which those skilled in the art conventionally understand it during use. This is only for the convenience of describing the present invention and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of the present invention. Furthermore, the terms "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0023] Example 1 Embodiment 1 of the present invention discloses an actuarial AOI detection method based on a multimodal large model, such as... Figure 1 As shown, the specific process is as follows: S101: Collect multimodal data of the tested object at the same defect location, perform preprocessing, and construct a multimodal defect sample set; Specifically, the data collected for constructing the defect knowledge spectrum base map covers typical application scenarios such as PCB (printed circuit board), SMT (surface mount technology), and semiconductor packaging. In this embodiment, the multimodal data includes AOI detection images, 3D contour data, spectral reflectance data and / or X-ray image data, specifically defined as shown in Table 1 below; Table 1: Data Collection Instructions

[0024] In this embodiment, the preprocessing includes multi-dimensional annotation of the multimodal data, including defect type, defect location, morphological features, process parameters, and causes, to ensure the accuracy and consistency of the data. The annotation strategy combines semi-automated annotation tools with expert review.

[0025] This embodiment also includes using a Generative Adversarial Network (GAN) to enhance defect morphology and simulate defect variations under different process parameters; by simulating multi-angle projection and illumination changes, the sample space is expanded, and the generalization ability of the model is improved. It not only performs data augmentation but also simulates data generation along the process-defect causal chain, achieving full-chain data expansion from appearance to mechanism.

[0026] S102: Based on the multimodal defect sample set, feature extraction is performed on the multimodal data, including the extraction of image modal features, three-dimensional morphology modal features and spectral modal features, and multimodal feature fusion is performed to obtain multimodal fused features; Specifically, image modal feature extraction can employ the Vision Transformer (ViT) model, which segments the detected image into a patch sequence and achieves global feature modeling through a self-attention mechanism. ViT possesses powerful long-distance dependency capture capabilities, enabling accurate extraction of complex morphological features such as crack direction, foreign object distribution, and contour offset. The feature extraction expression is as follows: , For image modal features, Let N be the i-th image patch, and N be the total number of patches. This is the patch number.

[0027] 3D morphological modal feature extraction can be based on the PointNet++ network to extract local geometric and global contour features from point cloud data. Through neighborhood sampling and feature aggregation, it captures minute morphological changes and macroscopic structural deviations, achieving high-precision modeling of defects such as weld point collapse and warping. Feature extraction expression: , It is a 3D morphological feature. For a neighboring point within the neighborhood of the center point, With the center point, Features of neighboring points, such as height, normal, curvature, etc.

[0028] Spectral modal feature extraction utilizes a 1D convolutional neural network (1D-CNN) combined with a self-attention mechanism to extract key features from the spectral reflectance curve, such as reflectance peak position, bandwidth variation, and absorption characteristics. This captures the material response characteristics of latent defects such as contamination and delamination. The feature extraction expression is as follows: , For spectral modal characteristics, For attention module, For activation function, The weighting coefficients for the attention enhancement term. The input is the spectral reflectance curve data.

[0029] The final fused representation of multimodal features: In this expression, " " indicates feature splicing.

[0030] S103: Using defects, parameters, processes, and strategies as nodes and the logical relationships between nodes as edges, construct a defect knowledge spectrum graph with a node-edge graph structure to generate an embedded defect knowledge spectrum base graph. Among them, Defect Nodes represent defect types and multimodal characteristics; Parameter Nodes describe process control parameters, such as exposure intensity, temperature, and pressure; Process Nodes represent key process steps and their causal mechanisms; Strategy Nodes define detection methods and optimization strategies; and edges define the logical relationships between nodes, such as causality, influence, and optimization suggestions.

[0031] In this embodiment, the Graph Neural Network (GNN) modeling uses the GraphSAGE framework to achieve iterative propagation and aggregation of node features, generating an embedded defect knowledge hierarchy base map. The update formula is expressed as follows:

[0032] in, For the first k Layer nodes v Feature representation, For neighborhood feature aggregation function, σ For activation function, For the first Layer nodes Feature representation, Represents the set of node features. Let be the learnable weight matrix of the k-th layer. Let be the bias vector of the k-th layer.

[0033] The iterative propagation mechanism of GNN enables each defect node to not only integrate its own multimodal features, but also to perceive the information of process parameters and strategy nodes associated with it, thus realizing a high-level abstraction from data to knowledge.

[0034] In this embodiment, the knowledge embedding and genealogical base map generation also include the introduction of Knowledge Embedding technology, which vectorizes the nodes and edges in the genealogical graph and maps them to a unified embedding space:

[0035] in, Embedding vectors into nodes involves converting nodes such as defects, parameters, processes, and strategies into computable vectors. For multimodal fusion embedding functions, The characteristics of process nodes, parameter nodes, and strategy nodes are represented respectively. The spectral base map is stored in the form of an embedded matrix, which supports efficient defect query, parameter inversion, and strategy recommendation.

[0036] Steps S201-S203, as follows Figure 2 As shown.

[0037] S201: To address the multimodal feature heterogeneity problem in AOI detection, this invention designs a feature decoupling mechanism based on semantic level. For input image modal features... 3D morphological features Spectral modal characteristics The three semantic levels of morphology (M), material (R), and process (P) are decoupled to extract a high-value feature subspace for defect identification. Specifically, a learnable decoupling mapping matrix can be constructed for each modality feature. Perform a projection transformation on the semantic vector space: , mod

[0038] in, This represents the feature representation of the modality mod at semantic layer l. Represents the primitive characteristics of a certain modality. Represents image modality, Represents three-dimensional morphological modes. Represents spectral modes.

[0039] For example, for solder joint defects, the collapse depth and curvature information in the 3D morphology mode are preferentially mapped to the morphology layer, the spectral reflection peak position shift is focused on the material layer, and the temperature cooling rate influence characteristics are mainly attributed to the process layer.

[0040] This decoupling mechanism enables the method or the system executing the method to highlight the contributions of each modality at different semantic levels while maintaining the integrity of modal information, thus laying a clear structural foundation for subsequent fusion and causal analysis.

[0041] S202: Calculate modal importance weights based on defect type, and perform weighted fusion of multimodal features within the same semantic layer to obtain three-layer semantic fusion features; After completing the semantic layering, based on the defect type and task requirements, it is necessary to dynamically adjust the weight contribution of each modality in a specific semantic layer, strengthen the expressive power of key modalities, and suppress irrelevant interference information.

[0042] In this embodiment, selective enhancement is achieved through weighted fusion, specifically through a modal attention mechanism. Taking semantic layer l as an example, the process is as follows: First, global average pooling (GAP) is performed on the features of each modality to extract the global description vector:

[0043] Then, the input is fed into an attention mapping network (MLP) to calculate the modality importance weights:

[0044] The semantic layer is obtained after fusion. Enhanced features:

[0045] in This represents the attention weight matrix on the l-th semantic layer, used to determine the importance of different modalities. This represents the normalized exponential function.

[0046] Taking a real-world example, in the scenario of detecting poor welds, 3D morphology and spectral reflectance have a stronger ability to distinguish between collapse and wettability defects. Their weights in the morphology layer and material layer are increased to 0.45 and 0.40, respectively, while the image modality is reduced to 0.15 to ensure optimal discriminability of the fused features.

[0047] S203: On the base map of the defective knowledge spectrum, a gated graph convolutional network is used to propagate backward along the causal edges to obtain causal chain inversion enhancement features; Defects originate from the coupling effects of multiple factors such as process, materials, and environment. Relying solely on feature fusion cannot accurately trace the cause. Therefore, in this embodiment, a causal chain collaborative inversion mechanism is introduced, which combines the process causal chain relationship in the defect knowledge spectrum to achieve reverse tracing modeling of defect genes to process parameters.

[0048] Specifically, during feature propagation, the information flow intensity is dynamically adjusted based on edge weights and node importance:

[0049] in, For nodes v In the k Layer feature representation; This is the adjacency matrix of the phylogenetic graph; for Hadamard The product, combined with gating weights, enables the regulation of causal chain information flow; A learnable causal weight matrix. This represents the activation function. Represents the feature transformation weights of the k-th layer. This represents the features of the node at the previous level. Through multi-level causal chain inversion, the system can effectively capture the nonlinear correlation between defective gene features and their causal parameters, providing highly reliable support for subsequent optimization and strategy formulation.

[0050] Then, the three-layer semantic fusion features are... , , Enhanced features of causal chain inversion By splicing the data, a defective gene expression vector is formed: =

[0051] in, As a high-dimensional feature reduction and nonlinear mapping module, it projects the spliced ​​fused features onto a 128-dimensional defective gene expression space. This indicates morphological layer fusion characteristics. Indicates the fusion characteristics of material layers. Indicates the fusion characteristics of the process layer; This indicates an enhanced feature of causal chain inversion; This vector not only has high discriminative power (supporting defect classification and detection judgment), but also contains process origin (supporting inversion optimization). In the subsequent generation of actuarial agents and program push, it serves as the core feature carrier throughout the entire chain.

[0052] Taking a real-world example, in the defect gene expression vector of a cold solder joint, the first 50 dimensions are mainly contributed by morphological features, dimension 6190 reflects the reflectivity characteristics of the material layer, and dimension 91128 mainly reflects the influence of the process flow. Using Attention Rollout technology, a saliency heatmap of the gene vector can also be output, enhancing the interpretability and engineering applicability of the results.

[0053] S301: Establish a process mechanism model corresponding to the defect type; In this embodiment, for common defect types in AOI inspection such as cold solder joints, cracks, foreign objects, and misalignment, a causal relationship between defect manifestations and process parameters is described by establishing a mechanism model covering four categories: thermodynamics, mechanics, fluid dynamics, and electromagnetic optics. Taking the defect of incomplete soldering as an example, its mechanism equation can be described as follows:

[0054] in, It is the effect of the reflow soldering temperature profile on the probability of cold solder joints (temperature gradient, heating time). The influence of solder paste printing thickness and wetting angle on soldering quality; This relates to the effect of flux activators on solder joint wetting and void formation. It is an indicator of the quantity of defects in cold solder joints or a risk indicator of cold solder joints. It is the reflow soldering temperature profile. It refers to the thickness of the solder paste printing. It is a moistening corner. It is a flux activator.

[0055] For different products and defect scenarios, the mechanism model combination is adaptively selected based on the causal graph to achieve scenario-based modeling and coupled analysis of complex factors.

[0056] S302: Using the defective gene expression vector as the target expression quantity, based on the process mechanism model and physical constraint residuals, the optimal combination of process parameters is obtained by jointly solving the problem using the Physical Information Neural Network (PINN) and Bayesian inverse estimation (BIE). Specifically as follows: Defective gene expression vector Mapping to the process parameter space completes the inverse inversion from defect manifestations to process causes. The inversion optimization objective is defined as:

[0057] in, It is a mechanism model function, with input process parameters. P Output quantitative indicators of defect performance; It is the target expression obtained by mapping defective gene vectors; It is a physical mechanism that constrains the residual term, ensuring the physical rationality of the inversion results; It is the mechanism constraint weight (empirical value 0.1~0.5, dynamically adjusted according to sample size and reliability).

[0058] By using the PINN and BIE methods, the interpretability and process feasibility of the inversion can be ensured: PINN incorporates the mechanistic equation as a soft constraint of the neural network into the loss function; BIE estimates the posterior distribution of process parameters based on Bayesian inference, reflecting the uncertainty of the inversion results.

[0059] S303: After completing the process parameter inversion, within the process adjustable window, perform multi-objective joint optimization on the optimal process parameter combination and detection parameters to obtain the final process-detection strategy.

[0060] In this embodiment, the objective function is designed as follows:

[0061] in, It is a risk assessment model for cold solder joint defects, which predicts the defect rate based on process parameters; It is a detection strategy (A balancing indicator between detection rate and false alarm rate); The additional costs are due to adjustments in processes and testing strategies; These are weighting coefficients, set according to the company's needs and quality objectives. This indicates a combination of process parameters, such as temperature, pressure, and solder paste thickness. This represents a combination of detection parameters, such as light source brightness, camera exposure, detection threshold, and algorithm parameters.

[0062] Specifically, the optimization algorithm can combine genetic algorithm (GA) with Bayesian optimization (BO) to balance global search capability and local convergence speed.

[0063] In AOI intelligent programming, the multi-type mechanism analysis and reverse inversion optimization module undertakes the core task of inferring process parameters from defect genes. Its goal is to accurately invert the key process parameters leading to defects based on defect gene expression vectors and process mechanism models, and to achieve globally optimal detection strategies and process adjustment suggestions under process constraints. Unlike the data layer fusion and causal extraction of the defect gene identification module, this invention focuses on physical layer inversion solutions and process optimization decisions, achieving a two-way closed loop from performance characteristics to process causality.

[0064] S401: The defect ID, defect gene expression vector, final process-detection strategy, confidence score, and interpretable link are encapsulated into a standardized actuarial function, with the following structure:

[0065] The fields are defined as follows: The defect node number corresponds to the base map of the genealogy. It is a defect gene expression vector (dimension 128), serving as a comprehensive representation of defect type and causal characteristics; It is the recommended combination of process parameters (such as soldering temperature zone, solder paste thickness, etc.) after reverse inversion and optimization. It is the result of the detection strategy optimization, including optical parameters, algorithm thresholds, feature weighting, etc. The confidence score of the inversion and optimization results quantifies the reliability of the model output; By tracing causal links from gene identification to mechanism inversion, we can improve the transparency of our strategies.

[0066] The actuary is essentially a decision mapping from defect type to process optimization to detection strategy, and it is the core data asset that supports subsequent automatic programming and intelligent push.

[0067] S402: In order to convert the actuarial algorithm into an executable detection program for the front-end device, mapping and adaptation are performed based on the Programming Design Graph, automatically mapping the actuarial algorithm into executable detection program code for the AOI device; Specifically, the programming design atlas defines the program structure and parameter mapping rules for different equipment models, testing stations, and task requirements; The mapping relationship is represented as follows:

[0068] in, It is a mapping function that dynamically adjusts the actuarial sub-parameters and program template based on equipment characteristics, detection targets, and strategy priorities; It is a programming design graph, which includes modular nodes such as detection algorithms, optical configurations, and process control.

[0069] S403: After the mapping is completed, the detection program code is automatically generated based on the combination of the actuarial algorithm and the programming design map, and pushed to the production line AOI equipment through the one-click release interface to complete the automatic deployment and activation of the program; Program generation illustration:

[0070] in, The code generation module converts the mapped detection strategy into device-executable instructions (such as parameter configuration files, process scripts, etc.).

[0071] The push mechanism supports: Automated batch deployment: Supports one-click synchronous push across multiple workstations and devices; Version management and rollback: Assigns a unique version number to each generated program, supporting effect evaluation and version rollback; Real-time verification and simulation: Before pushing, the system automatically performs program logic verification and effect simulation to ensure the quality of the online deployment.

[0072] In a preferred embodiment, during program execution, the following is also included: It receives real-time feedback data from the production line (such as detection rate, false alarm rate, and yield deviation), and performs adaptive fine-tuning based on the deviation between the actuarial calculation and the actual results, realizing a linkage mechanism of running and optimizing simultaneously. The formula for dynamically updating parameters is expressed as follows:

[0073] in, To adaptively update the step size, it is dynamically adjusted based on the quality of the feedback data; To optimize the loss function, yield rate, false positive rate, and policy execution performance are taken into account. This represents the gradient or adjustment direction of the parameters relative to yield deviation. This refers to errors related to false detections / missed detections.

[0074] This mechanism ensures the actuarial system's continuous optimization capabilities in practical applications, adapting to process fluctuations and product iteration needs.

[0075] Example 2 Embodiment 2 of the present invention discloses an actuarial AOI detection system based on a multimodal large model. Based on the actuarial AOI detection method based on a multimodal large model described in Embodiment 1 above, the system includes: The multimodal data acquisition and annotation module is used to execute step S101, which involves multimodal data acquisition and annotation to construct a multimodal defect sample set. The multimodal feature extraction and fusion module is used to execute step S102, extract features from multimodal data, and fuse multimodal features to obtain multimodal fused features; The defect knowledge spectrum base map construction module is used to execute step S103, using defects, parameters, processes and strategies as nodes and the logical relationships between nodes as edges, to construct a defect knowledge spectrum graph with a node-edge graph structure, and generate an embedded defect knowledge spectrum base map. The defect gene identification module is used to execute steps S201-S203, which perform semantic hierarchical processing, causal propagation and feature aggregation on multimodal fusion features to generate defect gene expression vectors. The process mechanism inversion and optimization module is used to execute steps S301-S303, establish a process mechanism model corresponding to the defect type, and perform inverse deduction and process-detection joint optimization based on the defect gene expression vector to obtain the optimal process-detection strategy. The actuary encapsulation and one-click programming push module is used to execute steps S401-S403, which integrates and encapsulates the defect genes, process parameter optimization results and detection strategies to generate an actuary, maps the actuary to generate detection program code, and deploys it to the AOI equipment. The online feedback and adaptive update module is used to collect detection rate, false alarm rate and yield feedback in real time during equipment operation, and dynamically update the actuator based on gradient descent to achieve optimization while running; The system is implemented through a hardware processor and storage medium, and communicates with the AOI equipment on the production line to automatically generate, distribute, and optimize the inspection program online.

[0076] Based on the above system, the following explanation is given using high-density PCB cold solder joint defect detection as an application scenario. For the task of detecting cold solder joints, the system first collects multimodal data, including AOI images, 3D contour scans and spectral reflectance data, and combines process information such as reflow soldering temperature profiles, solder paste printing thickness and flux parameters to complete the annotation and enhancement of defect samples.

[0077] After multimodal feature encoding, the system constructs a four-layer hierarchical relationship graph of "virtual solder joint - parameters - process - strategy" using a graph neural network (GNN), forming a structured defect knowledge base map; in this graph, node features are defined by the formula:

[0078] By performing layer-by-layer aggregation, a causal relationship model of defects, processes, and strategies is achieved, providing a unified expression for subsequent identification and inversion. Represents the set of node features. Let be the learnable weight matrix of the k-th layer. Let be the bias vector of the k-th layer.

[0079] Based on the spectral base map, the system performs semantic hierarchical decoupling on the input modal features, projecting the image, 3D morphology, and spectral features onto the vector spaces of the morphology layer, material layer, and process layer, respectively. l∈{M,R,P} By combining modal selective enhancement mechanisms, modal weights are dynamically adjusted based on defect type: Softmax ( GAP )) The fused output features three layers: =

[0080] Simultaneously, the system combines the process causal chain in the lineage and enhances the causal inversion of defective genes through gated graph convolution:

[0081] Ultimately, a defective gene expression vector is generated: =

[0082] This vector possesses both discriminative and causal characteristics, serving as input for mechanism inversion.

[0083] Based on the physical mechanism of solder joint defects, a composite influence model of temperature, wetting, and materials is established:

[0084] Based on defective gene expression vectors The system uses an inversion optimization method combining PINN and Bayesian inverse estimation to solve for the most likely combination of process parameters leading to the defect.

[0085] Results after inversion: The reflow soldering temperature difference was too large, the solder paste thickness was too thin, and the flux activity decreased.

[0086] Under process constraints, the system further conducts joint optimization of detection strategies and process parameters:

[0087] Optimized outputs: fine-tuning of process temperature range, solder paste printing thickness, AOI detection reflection sensitivity, and 3D morphology discrimination threshold.

[0088] Finally, the system encapsulates defect genes, process optimization results, and detection strategies into a standardized actuarial algorithm:

[0089] in, This indicates a combination of process parameters, such as temperature, pressure, and solder paste thickness. This represents a combination of detection parameters, such as light source brightness, camera exposure, detection threshold, and algorithm parameters.

[0090] By designing the graph through programming, the system automatically maps the actuaries to the AOI device program structure:

[0091] The system generates corresponding testing programs and pushes them to production line equipment via "one-click release," enabling automatic programming without human intervention. The system also combines real-time testing feedback to dynamically fine-tune testing strategies, achieving adaptive iteration of "testing and optimizing simultaneously."

[0092] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A method for actuarial AOI detection based on a multimodal large model, characterized in that, include: S1: Construct a multimodal defect sample set, extract and fuse multimodal features, and construct a defect knowledge genealogy graph with a node-edge graph structure; S2: On the constructed defect knowledge genealogy graph, semantic hierarchical, causal propagation and feature aggregation are performed on the multimodal fusion features to generate defect gene expression vectors; S3: Establish a process mechanism model corresponding to the defect type, and based on the defect gene expression vector, perform reverse deduction and process-detection joint optimization to obtain the optimal process-detection strategy; S4: Integrate and encapsulate the defect genes, process parameter optimization results and detection strategies to generate a precision calculator, map the precision calculator to generate detection program code, and deploy it to the AOI equipment.

2. The actuarial AOI detection method based on a multimodal large model according to claim 1, characterized in that, Step S1 is as follows: Collect multimodal data of the tested object at the same defect location, perform preprocessing, and construct a multimodal defect sample set; Based on the multimodal defect sample set, features are extracted from the multimodal data respectively, and multimodal feature fusion is performed to obtain multimodal fused features; Using defects, parameters, processes, and strategies as nodes, and the logical relationships between nodes as edges, a defect knowledge spectrum graph with a node-edge graph structure is constructed to generate an embedded defect knowledge spectrum base graph.

3. The actuarial AOI detection method based on a multimodal large model according to claim 2, characterized in that, The multimodal data includes AOI detection images, 3D contour data, spectral reflectance data, and / or X-ray image data; The preprocessing includes labeling the multimodal data with defect types, defect locations, morphological features, process parameters, and causes.

4. The actuarial AOI detection method based on a multimodal large model according to claim 3, characterized in that, The Vision Transformer model, PointNet++ network model, and 1D-CNN network model were used to extract modal features from the AOI detection image, 3D contour data, and spectral reflectance data, respectively, to obtain image modal features, 3D modal features, and spectral modal features. The extracted features were then concatenated to obtain multimodal fusion features.

5. The actuarial AOI detection method based on a multimodal large model according to claim 4, characterized in that, The GraphSAGE framework is used to implement iterative propagation and aggregation of node features, generating an embedded defect knowledge hierarchy base map. The update formula is expressed as follows: in, For the first k Layer nodes v Feature representation, For neighborhood feature aggregation function, σ For activation function, For the first Layer nodes Feature representation, Represents the set of node features. Let be the learnable weight matrix of the k-th layer. Let be the bias vector of the k-th layer.

6. The actuarial AOI detection method based on a multimodal large model according to claim 5, characterized in that, Step S2 is as follows: The multimodal fusion features are semantically decoupled by layer to obtain the feature representation of each modality in each semantic layer. The modality importance weight is calculated based on the defect type. The multimodal features in the same semantic layer are weighted and fused to obtain three-layer semantic fusion features. On the base map of the defective knowledge spectrum, a gated graph convolutional network is used to backpropagate along the causal edges to obtain causal chain inversion enhancement features. The three-layer semantic fusion features and the causal chain inversion enhancement features are then concatenated and dimensionality is reduced through an attention mapping network to generate a 128-dimensional defective gene expression vector.

7. The actuarial AOI detection method based on a multimodal large model according to claim 6, characterized in that, Modal importance weights are calculated using an attention mechanism, and multimodal features within the same semantic layer are weighted and fused, including: Global average pooling is used to obtain global description vectors for each modality in the same semantic layer. These vectors are then input into an attention mapping network (MLP) to calculate modality importance weights. Finally, the global description vectors and modality importance weights are fused to obtain the enhanced features of that semantic layer.

8. The actuarial AOI detection method based on a multimodal large model according to claim 1, characterized in that, Step S3 is as follows: Establish a process mechanism model corresponding to the defect type, use the defect gene expression vector as the target expression quantity, and solve the inversion to obtain the optimal combination of process parameters based on the process mechanism model and physical constraint residuals; Within the adjustable process window, the optimal combination of process parameters and detection parameters are jointly optimized by multiple objectives to obtain the final process-detection strategy.

9. The actuarial AOI detection method based on a multimodal large model according to claim 1, characterized in that, Step S4 is as follows: Defect ID, defect gene expression vector, final process-detection strategy, confidence score, and interpretable link are encapsulated into a standardized actuarial algorithm; Based on the programming design map, the actuary is automatically mapped into the inspection program code that can be executed by the AOI equipment, and the inspection program is pushed to the AOI equipment on the production line through a one-click release interface to complete the inspection deployment.

10. An actuarial AOI detection system based on a multimodal large model, characterized in that, The actuarial AOI detection method based on a multimodal large model according to any one of claims 1-9, the system includes: The multimodal data acquisition and annotation module is used to acquire multimodal data of the tested object at the same defect location, perform multidimensional annotation, and construct a multimodal defect sample set; The multimodal feature extraction and fusion module is used to extract features from multimodal data and fuse multimodal features to obtain multimodal fused features. The Defect Knowledge Spectrum Base Graph Construction Module is used to construct a defect knowledge spectrum graph with defects, parameters, processes and strategies as nodes and logical relationships between nodes as edges, generating an embedded defect knowledge spectrum base graph. The defect gene identification module is used to perform semantic hierarchical analysis, causal propagation, and feature aggregation on multimodal fusion features to generate defect gene expression vectors. The process mechanism inversion and optimization module is used to establish a process mechanism model corresponding to the defect type. Based on the defect gene expression vector, it performs inverse deduction and process-detection joint optimization to obtain the optimal process-detection strategy. The actuary encapsulation and one-click programming push module is used to integrate and encapsulate defect genes, process parameter optimization results and detection strategies to generate actuaries, map the actuaries to generate detection program code, and deploy them to AOI equipment.

Images