A method and system for evaluating the quality of HTCC products based on a multi-modal multi-layer heterogeneous graph attention network

Abstract

The application discloses a kind of based on multi-modal multi-layer heterogeneous graph attention network's HTCC product quality evaluation method and system;By defining process parameter node, quality defect node, process state node and time slice node four kinds of nodes and five kinds of edge relations, quality defect node characteristics and process parameters, time sequence information are uniformly encoded into graph structure;Build three layers MMHGAT network, process internal parameter-defect attention aggregation layer adopts multi-head attention to calculate the influence weight of each process internal process parameter on defect and output influence intensity ranking, defect conduction attention aggregation layer between processes calculates attention weight between process state nodes along process sequence and identifies cross-process conduction path, time evolution attention aggregation layer along time axis aggregation each time instant production line state output final quality characterization vector;Based on attribution information, a multi-objective optimization model is constructed, and an evolutionary algorithm is used to solve the optimal process parameter combination and issue execution.The present application is aimed at the technical problems of HTCC production line multi-source heterogeneous data fusion difficulty, process internal parameter-defect coupling relationship is not clear, cross-process defect conduction path is difficult to trace, by constructing multi-modal heterogeneous graph and designing three layers MMHGAT network, realize the integrated processing from data fusion, causal reasoning to quality evaluation, and the evaluation result has explainability.

Description

Technical Field

[0001] This invention belongs to the field of online quality assessment and process optimization technology in the manufacturing process of high-temperature co-fired ceramics (HTCC), specifically involving a method and system for HTCC product quality assessment that integrates multimodal data and multilayer heterogeneous graph attention networks. Background Technology

[0002] Multilayer high-temperature co-fired ceramic technology is one of the core pathways to achieve high-density, high-performance electronic packaging. This technology uses unsintered green ceramic sheets as the substrate. After tape casting, through-holes are sequentially fabricated, conductive paste is filled, and circuit patterns are printed. Several layers of green ceramic sheets are then stacked sequentially and finally sintered in a single firing process within a temperature range of 1350℃ to 1600℃, simultaneously achieving ceramic densification and metallization co-firing, forming a packaging substrate or shell with multilayer wiring capabilities. Since IBM first used it for commercial computer circuit boards in the early 1980s, this technology has been widely deployed in electronic packaging fields such as radio frequency, digital, power, and optical communications.

[0003] The entire manufacturing process of HTCC involves multiple technological steps, each involving several process variables and potentially leading to corresponding quality anomalies. These variables interact with each other, and a final defect is often the result of the combined effects of multiple upstream parameters, rather than being determined by a single factor. Therefore, the HTCC manufacturing process is characterized by numerous procedures, high parameter dimensionality, and complex interactions. There is no simple linear correspondence between product quality and process conditions; instead, it constitutes a highly coupled and interconnected system.

[0004] Due to the aforementioned complexities, domestic HTCC production lines currently face two main bottlenecks in quality control. First, process parameters, equipment status, and visual inspection records collected independently from each process are scattered across different databases, exhibiting heterogeneous formats and inconsistent semantics. The lack of a unified, standardized expression and retrieval mechanism hinders effective cross-process data correlation analysis and in-depth data mining. Second, because the coupling relationship between defects and parameters is difficult to clarify using traditional statistical methods, when quality anomalies occur in finished products, production line engineers often rely solely on experience for retrospective analysis, unable to quickly pinpoint the key causative processes and parameters.

[0005] Currently, publicly available technologies related to HTCC product manufacturing mainly focus on material formulation modification, physical sintering process control, and individual equipment structure optimization. For example, existing patents reduce the thermal expansion coefficient of the substrate by adjusting the ratio of alumina and additives. Other solutions propose process improvement measures for specific physical aspects such as green blank debinding sintering curves, large-size substrate deformation suppression, and tube shell sidewall forming collapse. However, these traditional manufacturing methods are generally based on a "trial and error" or "experience-based table lookup" model. Their technical focus is limited to adjusting the physical parameters of a single process or strengthening local structures, failing to consider long-process steps such as drilling and filling as a holistic system with nonlinear coupling characteristics. When more than twenty types of final inspection defects such as gold contamination, pin deformation, or ceramic body breakage occur in the finished product, existing patent technologies cannot trace which parameter drift in the preceding process induced the defect, let alone quantitatively characterize the transmission path of the interactive disturbances between various process parameters on the final yield. The aforementioned traditional solutions have inherent technical blind spots in cross-process data association, defect root cause tracing, and parameter collaborative optimization, making it difficult to meet the intelligent control needs of modern HTCC production lines for proactive early warning and accurate attribution of quality fluctuations.

[0006] At the algorithmic level for quality assessment and defect detection, existing solutions include a chip packaging defect detection method based on a multimodal heterogeneous attention network. While this solution incorporates multimodal visual data and a heterogeneous network architecture, its graph structure design is entirely geared towards high-density chip packaging scenarios. The node types and edge relationships cannot express the unique concepts of "process state transitions" and "timing slice accumulation" specific to HTCC production lines. Furthermore, its attention mechanism only reaches the level of image feature fusion, lacking explicit modeling of the physical causal links between process parameters and defects—a core requirement for achieving interpretable defect attribution. Therefore, these existing solutions cannot be directly migrated to the long-process manufacturing environment of HTCC due to mismatched application scenarios, overly simplified graph topology, and the lack of a timing attribution dimension.

[0007] In summary, existing technologies, when dealing with the HTCC manufacturing process, share common technical shortcomings, including insufficient cross-process causal traceability, limited depth of multimodal heterogeneous data fusion, and a lack of interpretability in evaluation results. Summary of the Invention

[0008] To address the technical challenges of multimodal data fusion, unclear parameter-defect coupling relationships within processes, and difficulty in tracing cross-process defect propagation paths in HTCC product manufacturing, this invention proposes a Multi-modal Multi-layer Heterogeneous Graph Attention Network (MMHGAT) specifically designed for HTCC manufacturing processes. By constructing a multimodal heterogeneous graph covering visual, process, defect, and temporal information, a three-layer graph attention network architecture is designed to perform representation learning on the micro-correlation of parameters-defects within processes, the macro-path of defect propagation between processes, and the temporal evolution law of the entire production chain. Furthermore, the defect-parameter influence intensity ranking and cross-process propagation links are directly extracted from the attention weights, enabling accurate assessment and interpretable attribution of product quality.

[0009] To achieve the objectives of this invention, the technical solution adopted is as follows:

[0010] A product quality assessment method for HTCC based on multimodal multilayer heterogeneous graph attention networks includes the following steps:

[0011] Step S1: Based on the multi-source heterogeneous data of the HTCC production line, by defining four types of nodes and five types of edge relationships, the defect images, defect content descriptions, process parameters and production time sequence information collected from the production line are uniformly encoded into node features and edge attributes in the graph structure to construct a multimodal heterogeneous graph.

[0012] Step S2: Construct a three-layer heterogeneous graph attention network. Sequentially, extract and aggregate features from the multimodal heterogeneous graph through an intra-process parameter-defect attention aggregation layer, an inter-process defect propagation attention aggregation layer, and a temporal evolution attention aggregation layer to obtain the final product quality characterization vector. Use this quality characterization vector to output a quality assessment score and generate interpretable attribution information based on the attention weights of each layer, including the defect-parameter influence intensity ranking and cross-process defect propagation path.

[0013] Step S3: Based on the attribution information output in Step S2, construct a multi-objective process parameter optimization model with pass rate, number of defects, and defect severity as objectives. Use a multi-objective evolutionary algorithm to solve for the optimal combination of process parameters under physical feasibility constraints, empirical range constraints, and equipment performance constraints. Then, send the model to the programmable logic controller of the corresponding process through the production line control network for execution. The quality data of the new batch is fed back to the data base to update the graph node features or adjust the network, forming a continuous improvement closed loop.

[0014] Furthermore, the process of constructing the multimodal heterogeneous graph in step S1 includes:

[0015] Step S1.1: Define four types of nodes, namely process parameter nodes, quality defect nodes, process status nodes, and time slice nodes, and design multimodal feature encoding methods for each type of node;

[0016] Step S1.2: Define five types of edge relationships: (a) weighted directed edges between process parameter nodes and quality defect nodes, used to learn the influence strength of parameters on defects; (b) belonging edges between quality defect nodes and process status nodes, used to aggregate defect information to the process level; (c) predecessor-successor edges of process status nodes along the process sequence, used to express the physical order of product flow; (d) temporal association edges between process parameter nodes and the corresponding process parameter nodes of the next process, used to capture the progressive constraints of parameter settings; (e) timestamp edges between time slice nodes and the corresponding process status nodes at each moment, used to establish temporal context relationships.

[0017] Step S1.3: Instantiate the multi-source heterogeneous data into a heterogeneous graph according to the above node and edge definitions. After concatenating the visual features and semantic features, assign them to quality defect nodes. Assign process parameters to corresponding nodes. Assign time-series information to time nodes according to slices. Store the edge relationships in the form of an adjacency matrix, which constitutes the only input for subsequent network processing.

[0018] Furthermore, the process parameter node stores the real-time collected values ​​or set values ​​of the equipment parameters for each process.

[0019] Furthermore, the feature vector of the quality defect node is composed of two parts. The visual feature vector is extracted from the defect image by the YOLOv11 depth detection model, and includes the bounding box coordinates, category confidence, pixel-level semantic mask, and geometric morphological parameters such as area, perimeter, and roundness obtained through connected component analysis. The semantic feature vector includes the defect category label, quantity, spatial distribution location, and process information.

[0020] Furthermore, the process status node represents the overall operating status of each process, and its initial characteristics are generated by mean pooling of all process parameter nodes and quality defect nodes within that process.

[0021] Furthermore, time slice nodes are established at fixed intervals along the production batch time axis to store the real-time progress and cumulative defect statistics of each process at the corresponding moment.

[0022] Furthermore, the construction of a three-layer heterogeneous graph attention network in step S2 specifically includes the following steps:

[0023] Step S2.1: Construct an intra-process parameter-defect attention aggregation layer. This layer executes independently within each process, using a multi-head graph attention mechanism to calculate the attention weights of process parameter nodes on quality defect nodes. For defect node i and process parameter node j in process k, first calculate the attention score: ;in, and W represents the feature vectors of defect node i and process parameter node j, respectively. q W k Let be the learnable query and key projection matrix, 'a' be the attention weight vector, and || denote the concatenation operation. The attention weights α are then obtained through softmax normalization. ij , denoted as:

[0024] ;in, Quality defect node The set of neighboring process parameter nodes, The aforementioned attention score;

[0025] And using M attention heads for parallel computation, the aggregation result of the m-th head is: Among them, W v (m) W is the projection matrix of the m-th attention head; q (m) W k (m) and W v (m) These correspond to query, key, and value projection, respectively.

[0026] Finally, the aggregation results of the M attention heads are concatenated and linearly mapped: Among them, W o This is the output parameter matrix;

[0027] After aggregation, each quality defect node obtains updated features that incorporate surrounding process parameter information;

[0028] At the same time, for each attention head corresponding to α ij (m) Perform cross-head averaging to obtain the average attention weight. ij ;willᾱ ij Higher than the preset threshold θ intra The edges are extracted to form the "parameter-defect" relationship for this process, and then processed according to ᾱ ij Generate a list sorted by influence strength from largest to smallest;

[0029] After the defect node update is completed, based on the belonging edges between the quality defect nodes and the process status nodes defined in step S1.2, the updated defect node features within each process are subjected to mean pooling or attention pooling to obtain the first layer of updated process status features h. s,k (1) ;

[0030] Step S2.2: Construct an inter-process defect propagation attention aggregation layer; using the updated process state node features h from the first layer... s,k (1) Using this as input, calculate the attention score between adjacent process state nodes on the process sequence graph: Where k represents the current process, and l represents a candidate process that has a predecessor-successor relationship or is allowed to cross processes with the k-th process; a s (m) W is the inter-process attention vector of the m-th attention head. s,q (m) W s,k (m) W s,v (m) These are the query, key, and value projection matrices, respectively.

[0031] For e s,kl In the process neighborhood N s(k) β is obtained by performing softmax normalization within the inner layer. kl ;

[0032] The process status nodes are updated by multiple heads of attention. The updated process state matrix H of the second layer is obtained. s (2) ;

[0033] Average weight of the cross Above the threshold θ inter Directed edges are identified as candidate critical paths for cross-process defect propagation;

[0034] Step S2.3: Construct a temporal evolution attention aggregation layer. The input of the third layer is obtained by fusing the updated state features of each process in the second layer with the original temporal features of the corresponding time slices, forming a time slice feature matrix H. t (0) The self-attention mechanism is used to aggregate the global state of the production line at different times along the time axis, and the time attention score of the m-th attention head is calculated.

[0035] ;

[0036] The temporal attention score in the temporal neighborhood N t(τ)perform softmax normalization to obtain the temporal attention weight of the m-th attention head , and update the features of each time slice using a multi-head mechanism ; take the updated feature of the last time slice T as the final product quality representation vector z;

[0037] Step S2.4: Feed the final quality representation vector z into a decoder composed of a multi-layer perceptron, and output a quality evaluation score s; perform grade division according to the preset qualification threshold θ q and the severity division thresholds θ1, θ2, where 0≤θ q <θ1<θ2≤10; set s≤θ q as qualified, θ q <s≤θ1 as minor defect, θ1<s≤θ2 as moderate defect, and s>θ2 as severe defect;

[0038] Meanwhile, extract attention edges higher than θ from the first layer average attention matrix intra to obtain the key influencing parameters and their weights corresponding to each defect; extract attention edges higher than θ from the second layer average attention matrix inter , and combine with the process sequence to form a multi-hop propagation link from the source process to the end defect.

[0039] A system for implementing the above-mentioned HTCC product quality evaluation method based on a multi-modal multi-layer heterogeneous graph attention network, comprising a data base module, a graph construction and reasoning module, a quality evaluation module, and a process optimization module; the data base module is responsible for access, cleaning and semantic alignment of multi-source heterogeneous data, and provides a data cube for graph construction; the graph construction and reasoning module converts production line data into a multi-modal heterogeneous graph according to steps S1 and S2, runs the three-layer attention network of MMHGAT, and outputs quality evaluation scores and interpretable attribution information; the quality evaluation module performs defect grade division and qualification judgment according to the scores, and generates a visual attribution report; the process optimization module calls a multi-objective evolutionary algorithm to solve the optimal parameter combination based on the attribution information, issues and executes the combination through the production line interface, and receives quality feedback to continuously optimize the model at the same time.

[0040] Beneficial effects of the present invention:

[0041] 1. According to the present invention, through the multi-modal heterogeneous graph representation method, visual images, process parameters, defect records and time series information scattered in different systems on the HTCC production line are fused into a single computation graph, which breaks through the limitation that existing multi-source heterogeneous data cannot be modeled uniformly, and provides a unified data base for the application of deep learning methods in HTCC quality evaluation.

[0042] 2. This invention utilizes an in-process parameter-defect attention aggregation layer to automatically calculate the influence intensity of each process parameter on a specific defect with learnable attention weights and outputs a sorted list. This solves the problem that traditional statistical methods are difficult to capture multidimensional nonlinear coupling relationships and provides a quantifiable data-driven basis for defect attribution.

[0043] 3. This invention uses an inter-process defect transmission attention aggregation layer to identify multi-hop indirect transmission links formed through intermediate processes along the process sequence, enabling managers to trace the attribution chain from the end defect to the upstream process, thus making up for the lack of cross-process causal reasoning in existing solutions.

[0044] 4. This invention incorporates the time evolution information of the entire production cycle into the final quality characterization through a temporal evolution attention aggregation layer, enabling the evaluation results to reflect the dynamic cumulative effect of quality status and improving the accuracy and robustness of the evaluation.

[0045] 5. The present invention provides structured and interpretable attribution information directly from attention weights while completing the quality assessment of the entire MMHGAT network, without the need for additional post-analysis methods, thus achieving the integration of assessment and interpretation.

[0046] 6. The system of the present invention takes conformity assessment as its core, and meets the inspection requirements in quality control by making post-production judgments on finished products, thereby helping to improve the yield and stability of HTCC production.

[0047] Note that the description of these effects does not preclude the existence of other effects. One aspect of the invention does not necessarily have all the aforementioned effects. Effects other than those described above can be readily observed and extracted from the description, drawings, claims, etc. Attached Figure Description

[0048] Figure 1 This is a diagram illustrating the overall architecture of the method proposed in this invention, showing the entire process from data access to process optimization and the four core modules.

[0049] Figure 2 This diagram illustrates the process of constructing a multimodal heterogeneous graph representation model, including node definition, edge relationship establishment, and multimodal feature splicing.

[0050] Figure 3 This is a network structure for the process parameter-defect attention aggregation layer, demonstrating the multi-head attention calculation between process parameters and defect nodes.

[0051] Figure 4 The attention aggregation layer network structure for inter-process defect transmission demonstrates the attention propagation of each workstation's status node along the process sequence.

[0052] Figure 5The network structure of the temporal evolution attention aggregation layer is used to demonstrate the aggregation of full production line state features along time slices.

[0053] Figure 6 The overall computation process of the MMHGAT network is shown, demonstrating the cascading relationship of the three attention layers and the final output.

[0054] Figure 7 Link diagram. Detailed Implementation

[0055] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the scope of protection of the present invention is not limited thereto.

[0056] Example 1: A method for evaluating the quality of HTCC products based on multimodal multilayer heterogeneous graph attention networks, comprising the following steps:

[0057] Step S1: Construction of multimodal heterogeneous graph.

[0058] The steps S1.1 to S1.3 describe in detail how to convert the multi-source heterogeneous data of the HTCC production line into a heterogeneous graph structure that can be processed by the MMHGAT network.

[0059] Step S1.1: Define node types and encode multimodal features.

[0060] In each process of the HTCC production line (taking drilling as an example), four types of nodes are defined as follows:

[0061] (1) The process parameter node stores the collected values ​​of the equipment parameters for this process, such as drilling pressure and punch specifications. The feature vector of each parameter node contains the parameter value and normalized statistical features, denoted as dimension d. p .

[0062] (2) The quality defect node stores defect information captured by the visual inspection system. The feature vector of this node is composed of two parts: a visual feature vector and a semantic feature vector. The visual feature vector is extracted from the defect image by the YOLOv11 (You Only Look Once version 11) depth detection model, covering bounding box coordinates, class confidence, instance segmentation mask, and geometric parameters such as area, perimeter, and roundness. It is then mapped to d through a fully connected layer. v Dimensional embedding; the semantic feature vector contains defect category labels, quantity, spatial distribution heatmap, and process code, which are mapped to d after word embedding. s Dimensional embedding. The two are concatenated and then subjected to a linear transformation. Unified to a d-dimensional hidden layer space.

[0063] (3) The process status node serves as the global representation of the process. Its initial features are generated by mean pooling of all process parameter nodes and quality defect nodes within the process, with a dimension of d. This node is updated in subsequent attention aggregation.

[0064] (4) Time slice nodes are established along the production batch time axis at specific time intervals (the specific time is set according to the actual situation), and there are T slices in total. The initial feature of each slice node is the spliced ​​vector of the state of each process at that moment (dimension N×d, where N is the number of processes), which is then compressed to d dimensions by linear transformation.

[0065] Symbolic Unification: Throughout the text, K is used to represent the total number of processes. Therefore, the "number of processes N" in the above time slice nodes is uniformly denoted as K; the corresponding splicing dimension is uniformly written as K×d.

[0066] Step S1.2: Define edge relationships.

[0067] Establish five types of directed or bidirectional edges in the graph:

[0068] (a) Weighted directed edges connect process parameter nodes and quality defect nodes within the same process. The initial weights are set to zero, and after training, they represent the influence intensity.

[0069] (b) Belonging edge, connecting the quality defect node to the process status node to which it belongs;

[0070] (c) Predecessor-successor edge, which connects adjacent process state nodes based on the process sequence to express the direction of production flow;

[0071] (d) Temporal associated edges, connecting the process parameter nodes of the previous process with the corresponding process parameter nodes of the next process, such as the progressive constraint between drilling pressure and filling pressure.

[0072] (e) Timestamp edge, connecting each time slice node to all process status nodes at the corresponding time.

[0073] Step S1.3: Encode into graph structure data.

[0074] Based on the above definitions of nodes and edges, multi-source data is instantiated into a heterogeneous graph G in batches. The node set V of graph G contains process parameter nodes, defect nodes, process status nodes, and global time slice nodes for all processes; the edge set E contains instances of all five types of edges. The initial feature matrix of each node and the adjacency matrix of each type of edge are stored together to form the input of the subsequent network.

[0075] Step S2: Construction and core formula of the three-layer heterogeneous graph attention network. Steps S2.1 to S2.4 detail the three-layer network structure of MMHGAT and its mathematical expression.

[0076] Let the total number of processes be K, and the number of process parameter nodes in the k-th process be P. k The number of quality defect nodes is D k All node features are uniformly embedded into the d-dimensional space after processing in step S1.

[0077] Step S2.1: In-process parameters - Defect attention aggregation layer.

[0078] For process k, let the feature matrix of the process parameter nodes be... The feature matrix of quality defect nodes is .in , Each is a process Number of internal process parameter nodes and number of quality defect nodes To unify feature dimensions, this layer centers on the quality defect node and uses a multi-head attention mechanism to learn the strength of the influence of each process parameter node on it.

[0079] To avoid overlapping meanings in subsequent indexes, the first layer uses k to represent the process number, i to represent the quality defect node number, j to represent the process parameter node number, and m to represent the attention head number. The second layer uses k and l to represent the process index, and the third layer uses τ and ν to represent the time slice index.

[0080] For each quality defect node i and each associated process parameter node j, calculate the attention score:

[0081]

[0082] in, For defective nodes eigenvectors, For process parameter nodes eigenvectors; W q W k ∈ R d'×d , a ∈ R 2d' For learnable query and key projection matrices, d' represents the attention weight vector; || denotes the concatenation operation, and d' is the attention dimension.

[0083] Among them, W q and W k These are the query projection matrix and the key projection matrix, respectively, with 'a' being the attention weight vector. Each attention head has an independent set of projection parameters. The node feature matrix is ​​stacked row by row, while individual node features are used as d-dimensional vectors during attention calculation. In the specific implementation, the transpose can be used according to the row / column vector storage method without changing the above mapping relationship.

[0084] Attention score using the softmax function Normalization is performed to obtain the attention weights. The calculation formula is:

[0085]

[0086] in, Quality defect node The set of neighboring process parameter nodes, For the aforementioned attention score, This represents the normalized attention weights. subscript Indicates the search on this neighbor set Normalization does not change its meaning as a variable index.

[0087] Using M attention heads for parallel computation, each head has independent projection parameters. The quality defect node update feature of the m-th head is:

[0088]

[0089] Among them, W v (m) W is the projection matrix of the value of the m-th attention head; q (m) W k (m) and W v (m) These correspond to the projections of the query, key, and value, respectively. Finally, the aggregated results of the M attention heads are concatenated and linearly mapped:

[0090]

[0091] W o This is the output parameter matrix.

[0092] After aggregation, each quality defect node obtains updated features that incorporate surrounding process parameter information.

[0093] The cross-head averaging object is the weight α of the same "defect node i - process parameter node j" edge across M attention heads. ij (m) First, calculate the average weight of the span. ij , denoted as:

[0094] Then ᾱ ij Higher than the preset threshold θ intra Write the edges into the attribution list as shown in Table 1 below, and follow the ᾱij Sort the parameters from largest to smallest to obtain the ranking of the "parameter-defect" impact intensity of the process.

[0095] Table 1 Attribution List

[0096] Serial Number Process parameters Cross-head average attention weight Influence intensity ranking 1 Drilling pressure 0.58 1 2 Punch specifications 0.42 2

[0097] After completing the first layer of defect node updates, based on the belonging edges between quality defect nodes and process status nodes, the updated defect node features and corresponding process parameter node features within the k-th process are subjected to mean pooling or attention pooling to obtain the first layer of updated process status features h. s,k (1) .

[0098]

[0099] This feature integrates information on the impact of process parameters within a process on defects, serving as input for attention aggregation between the next layer of processes.

[0100] Step S2.2: Inter-process defect transmission attention aggregation layer.

[0101] Based on the updated process status node features h of the first layer s,k (1) As input, on the process sequence diagram, for the current process k and the candidate associated processes l (l∈N)... s(k) Calculate the inter-process attention score, where N s(k) This is the set of processes that have a predecessor-successor relationship or a pre-allowed cross-process association with the k-th process. Using k and l as process indices for this layer avoids confusion with defect node i and process parameter node j in the first layer.

[0102]

[0103] The inter-process attention score of the m-th attention head in the process neighborhood N s(k) Perform softmax normalization internally to obtain the inter-process attention weight β. kl (m) :

[0104]

[0105] And update the process status nodes using a multi-head mechanism:

[0106]

[0107] Among them, a s (m) W is the inter-process attention vector of the m-th attention head. s,q (m) Ws,k (m) W s,v (m) These are the query, key, and value projection matrices, respectively, W. s,o For multi-head output mapping matrix, h represents the process state characteristics of the k-th process after the first layer of parameter-defect aggregation and pooling. s,l (1) The state features of candidate process l after the first layer update are given. This leads to the process state matrix H after the second layer update. s (2) ∈R (K×d) For M attention heads, β kl (m) Calculate the average to get ,Will Exceeding the threshold θ inter Furthermore, edges whose directions conform to the process flow constraints are extracted as candidate paths for cross-process defect propagation, and multi-hop propagation links are formed according to their weights.

[0108] Step S2.3: Temporal Evolution Attention Aggregation Layer.

[0109] For the τth time slice, the K process state features h obtained from the second layer at that moment s,k,τ (2) Compared with the original temporal features h of the time slice obtained in step S1 t,τ init The input h of the third layer is obtained by fusion and linear mapping. t,τ (0) :

[0110]

[0111] The input features of T time slices are stacked in chronological order to obtain H. t (0) ∈R (T×d) This layer uses multi-head self-attention to aggregate information along the time axis. Let τ represent the current time slice and ν represent the time slice being followed; the τ-th time slice can focus on slice nodes ν∈N within the preset time windows before and after it. t(τ) Calculate the time attention score for the m-th attention head:

[0112]

[0113] The temporal attention score in the temporal neighborhood N t(τ) Perform softmax normalization within the function to obtain the temporal attention weights of the m-th attention head. :

[0114]

[0115] Updated features:

[0116]

[0117]

[0118] where a t (m) is the attention vector of the m-th temporal attention head, W t,q (m) , W t,k (m) , W t,v (m) are respectively the query, key and value projection matrices of the temporal layer, W t,o is the multi-head output mapping matrix. After processing T time slices, the updated feature h of the last time slice T is taken t,T (1) as the final product quality characterization vector z, that is, z=h t,T (1) , so that the final characterization simultaneously inherits the cross-process state information of the second layer and the full time-domain context information of the third layer.

[0119] Step S2.4: Quality assessment output and interpretable attribution.

[0120] The final quality characterization vector z is fed into a decoder composed of a multi-layer perceptron (MLP). The MLP output layer adopts Sigmoid activation and maps proportionally to a quality assessment interval from 0 to 10:

[0121]

[0122]

[0123]

[0124] where is the raw output of the MLP decoder, is the Sigmoid function. Through a 10-fold proportional mapping, the quality assessment score s falls into the evaluation interval from 0 to 10, and a higher score indicates a greater defect severity. Set 0≤θ q <θ1<θ2≤10, and s≤θ q , θ q <s≤θ1, θ1<s≤θ2, and s>θ2 are respectively determined as qualified, minor defect, medium defect and severe defect; each threshold can be set according to the product acceptance criteria and the labeling result of the training set.

[0125] The attribution information consists of two parts:

[0126] (1) In-process influence ranking: Extract all elements in the first layer that are higher than θ intra By focusing on the attention edge, we can obtain the key impact parameters and their weights corresponding to each defect;

[0127] (2) Cross-process transmission path: Extract the path above θ in the second layer inter By focusing on the edge and combining it with the process sequence, a multi-hop transmission link is formed from the source process to the end defect.

[0128] Both types of information are output in a structured format, serving as a basis for process engineers to trace and analyze data.

[0129] Step S3: After obtaining the attribution information from Step S2.4, the process optimization process is triggered. The optimization model uses the defect-parameter influence relationship and transmission path as constraints to establish a multi-objective optimization problem that includes indicators such as pass rate, defect quantity, and defect severity. The decision variables cover the key process parameters of each process. A multi-objective evolutionary algorithm is used to search for the Pareto optimal parameter combination under multiple constraints such as physical feasibility, empirical range, and equipment performance. The obtained optimal solution is sent to the PLC of the corresponding process through the industrial control network to achieve adaptive parameter adjustment. The quality data of the new batch is fed back to the data base, forming a continuous improvement closed loop of "data acquisition - graph construction - quality assessment - attribution analysis - parameter optimization - feedback", which is used to update the graph node features or fine-tune the network to form a continuous improvement closed loop.

[0130] Example 2: System Operation Mode

[0131] A system for implementing the above-mentioned HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network includes a data base module, a graph construction and inference module, a quality assessment module, and a process optimization module. The data base module is responsible for the access, cleaning, and semantic alignment of multi-source heterogeneous data, and provides data cubes for graph construction. The graph construction and inference module transforms production line data into a multimodal heterogeneous graph according to steps S1 and S2, runs the MMHGAT three-layer attention network, and outputs quality assessment scores and interpretable attribution information. The quality assessment module classifies defects and determines acceptance based on the scores, and generates a visual attribution report. The process optimization module calls a multi-objective evolutionary algorithm based on the attribution information to solve for the optimal parameter combination, executes it through the production line interface, and receives quality feedback to continuously optimize the model.

[0132] This system reads real-time process parameters and visual inspection results for each step of the current batch, constructs a heterogeneity graph, performs MMHGAT forward inference, and outputs a product quality assessment score to directly determine whether the product meets the factory standards. This mode effectively supports post-production quality assessment and helps improve the yield and stability of HTCC production.

[0133] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, equivalent changes, and alterations made to the embodiments based on the technical essence of the present invention fall within the protection scope of the present invention.

Claims

1. A method for evaluating the quality of HTCC products based on a multimodal, multilayer heterogeneous graph attention network, characterized in that, Includes the following steps: Step S1: Based on the multi-source heterogeneous data of the HTCC production line, by defining four types of nodes and five types of edge relationships, the defect images, defect content descriptions, process parameters and production time sequence information collected by the production line are uniformly encoded into node features and edge attributes in the graph structure to construct a multimodal heterogeneous graph. Step S2: Construct a three-layer heterogeneous graph attention network. Sequentially, extract and aggregate features from the multimodal heterogeneous graph through an intra-process parameter-defect attention aggregation layer, an inter-process defect propagation attention aggregation layer, and a temporal evolution attention aggregation layer to obtain the final product quality characterization vector. Use this quality characterization vector to output a quality assessment score and generate interpretable attribution information based on the attention weights of each layer, including the defect-parameter influence intensity ranking and cross-process defect propagation path. Step S3: Based on the attribution information output in Step S2, construct a multi-objective process parameter optimization model with pass rate, number of defects, and defect severity as objectives. Use a multi-objective evolutionary algorithm to solve for the optimal combination of process parameters under physical feasibility constraints, empirical range constraints, and equipment performance constraints. Then, send the model to the programmable logic controller of the corresponding process through the production line control network for execution. The quality data of the new batch is fed back to the data base to update the graph node features or adjust the network, forming a continuous improvement closed loop.

2. The HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network according to claim 1, characterized in that, The process of constructing the multimodal heterogeneous graph in step S1 includes: Step S1.1: Define four types of nodes, namely process parameter nodes, quality defect nodes, process status nodes, and time slice nodes, and design multimodal feature encoding methods for each type of node; Step S1.2: Define five types of edge relationships: (a) weighted directed edges between process parameter nodes and quality defect nodes, used to learn the influence strength of parameters on defects; (b) belonging edges between quality defect nodes and process status nodes, used to aggregate defect information to the process level; (c) predecessor-successor edges of process status nodes along the process sequence, used to express the physical order of product flow; (d) temporal association edges between process parameter nodes and the corresponding process parameter nodes of the next process, used to capture the progressive constraints of parameter settings; (e) timestamp edges between time slice nodes and the corresponding process status nodes at each moment, used to establish temporal context relationships. Step S1.3: Instantiate the multi-source heterogeneous data into a heterogeneous graph according to the above node and edge definitions. After concatenating the visual features and semantic features, assign them to quality defect nodes. Assign process parameters to corresponding nodes. Assign time-series information to time nodes according to slices. Store the edge relationships in the form of an adjacency matrix, which constitutes the only input for subsequent network processing.

3. The HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network according to claim 2, characterized in that, The process parameter node stores the real-time collected or set values ​​of equipment parameters for each process.

4. The HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network according to claim 2, characterized in that, The feature vector of the quality defect node is composed of two parts. The visual feature vector is extracted from the defect image by the YOLOv11 depth detection model and includes the bounding box coordinates, category confidence, pixel-level semantic mask, and geometric parameters of area, perimeter, and roundness obtained through connected component analysis. The semantic feature vector includes the defect category label, quantity, spatial distribution location, and process information.

5. The HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network according to claim 2, characterized in that, The process status node represents the overall operating status of each process. Its initial characteristics are generated by pooling the average values ​​of all process parameter nodes and quality defect nodes within the process.

6. The HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network according to claim 2, characterized in that, Time slice nodes are established at fixed intervals along the production batch time axis, storing the real-time progress and cumulative defect statistics of each process at the corresponding moment.

7. The HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network according to claim 1, characterized in that, The construction of a three-layer heterogeneous graph attention network in step S2 specifically includes the following steps: Step S2.1: Construct an intra-process parameter-defect attention aggregation layer. This layer executes independently within each process, using a multi-head graph attention mechanism to calculate the attention weights of process parameter nodes on quality defect nodes. For defect node i and process parameter node j in process k, first calculate the attention score: ;in, and W represents the feature vectors of defect node i and process parameter node j, respectively. q W k Let be the learnable query and key projection matrix, 'a' be the attention weight vector, and || denote the concatenation operation. The attention weights α are then obtained through softmax normalization. ij , denoted as: ;in, Quality defect node The set of neighboring process parameter nodes, The aforementioned attention score is used; and M attention heads are used for parallel computation, with the aggregated result of the m-th head being: Among them, W v (m) W is the projection matrix of the m-th attention head; q (m) W k (m) and W v (m) These correspond to query, key, and value projection, respectively; finally, the aggregated results of the M attention heads are concatenated and linearly mapped: Among them, W o The output parameter matrix is ​​used; after aggregation, each quality defect node obtains updated features that incorporate surrounding process parameter information; simultaneously, the α corresponding to each attention head is... ij (m) Perform cross-head averaging to obtain the average attention weight ᾱ ij ;willᾱ ij Higher than the preset threshold θ intra The edges are extracted to form the "parameter-defect" relationship for this process, and then processed according to ᾱ ij Generate a list of influence strengths from largest to smallest; after updating the defect nodes, based on the attribution edges between the quality defect nodes and process status nodes defined in step S1.2, perform mean pooling or attention pooling on all updated defect node features within each process to obtain the first layer of updated process status features h. s,k (1) ; Step S2.2: Construct an inter-process defect propagation attention aggregation layer; using the updated process state node features h from the first layer... s,k (1) Using this as input, calculate the attention score between adjacent process state nodes on the process sequence diagram: Where k represents the current process, and l represents a candidate process that has a predecessor-successor relationship or is allowed to cross processes with the k-th process; a s (m) Let W be the inter-process attention vector of the m-th attention head. s,q (m) W s,k (m) W s,v (m) These are the query, key, and value projection matrices, respectively; for e s,kl In the process neighborhood N s(k) β is obtained by performing softmax normalization within the inner layer. kl The process status nodes are updated by multiple heads of attention. The updated process state matrix H of the second layer is obtained. s (2) ; average weight of the cross Above the threshold θ inter The directed edges are identified as candidate critical paths for cross-process defect propagation; Step S2.3: Construct a temporal evolution attention aggregation layer. The input of the third layer is obtained by fusing the updated state features of each process in the second layer with the original temporal features of the corresponding time slices, forming a time slice feature matrix H. t (0) The self-attention mechanism is used to aggregate the global state of the production line at different times along the time axis, and the time attention score of the m-th attention head is calculated. ; The temporal attention score in the temporal neighborhood N t(τ) The temporal attention weights of the m-th attention head are obtained by performing softmax normalization within the inner area. A multi-head mechanism is used to update the features of each time slice. Take the updated features of the last time slice T as the final quality representation vector z of the product; Step S2.4: Input the final quality characterization vector z into a decoder composed of a multi-layer perceptron, and output a quality assessment score s; according to the preset qualification threshold θ q and severity demarcation thresholds θ1 and θ2 to perform grade division, where 0≤θ q <θ1<θ2≤10; set s≤θ q as qualified, θ q <s≤θ1 as minor defect, θ1<s≤θ2 as medium defect, and s>θ2 as severe defect; at the same time, extract attention edges higher than θ from the first-layer average attention matrix, to obtain key influence parameters corresponding to each defect and weights thereof; extract attention edges higher than θ from the second-layer average attention matrix, and combine with process sequences to form a multi-hop propagation link from the source process to the end defect. intra inter ​​ 8. A system for implementing the HTCC product quality assessment method based on a multimodal multilayer heterogeneous graph attention network as described in claim 1, characterized in that, The system includes a data foundation module, a graph construction and inference module, a quality assessment module, and a process optimization module. The data foundation module is responsible for the access, cleaning, and semantic alignment of multi-source heterogeneous data, and provides data cubes for graph construction. The graph construction and inference module transforms production line data into a multimodal heterogeneous graph according to steps S1 and S2, runs the MMHGAT three-layer attention network, and outputs quality assessment scores and interpretable attribution information. The quality assessment module classifies defects and determines acceptance based on the scores, and generates a visual attribution report. The process optimization module calls a multi-objective evolutionary algorithm based on the attribution information to solve for the optimal parameter combination, executes it through the production line interface, and receives quality feedback to continuously optimize the model.

Images