A multimodal data quality traceability collaborative regulation system for optoelectronic display products

Abstract

The application discloses a kind of photoelectric display product multimodal data quality traceability collaborative regulation systems, including multimodal data acquisition module, pre-processing alignment module, feature extraction fusion module, quality traceability analysis module, collaborative regulation decision module and feedback learning module.The application is collected by production line optics, electricity, film thickness and so on Multimodal data and carries out space-time alignment, utilizes the heterogenous feature of multi-head cross attention mechanism fusion, carries out root cause tracing in combination with multimodal causal graph and graph neural network, locks key process node;Again, a set of collaborative regulation parameters is generated by multi-objective particle swarm optimization, and multiple adjustable parameters are collaboratively adjusted under process constraints, and finally the causal model and proxy model are updated according to the feedback of regulation effect.The application realizes the accurate positioning of defect causes and the collaborative optimization of multiple process parameters, can continuously adapt to production line changes, and significantly improves the yield and regulation efficiency of photoelectric display products.

Description

Technical Field

[0001] This invention relates to the field of display product quality control, and more specifically to a multimodal data quality traceability and collaborative control system for optoelectronic display products. Background Technology

[0002] The manufacturing process of optoelectronic display products such as organic light-emitting diode (OLED) displays and liquid crystal displays involves multiple steps, including evaporation, photolithography, and encapsulation. Numerous process parameters are involved, and extremely high quality requirements are placed on film thickness uniformity, electrical properties, and optical performance. When defects such as uneven brightness or color deviation occur, the root cause often stems from the interaction of multiple process steps and equipment parameters.

[0003] Existing quality analysis methods often rely on single-modal inspection data, such as analyzing only optical inspection images or focusing only on a single process parameter curve. This single-modal analysis approach has significant limitations: on the one hand, a single data source cannot fully reflect the comprehensive state of a product across multiple dimensions, including optical, electrical, physical morphology, and process environment, and is prone to missing anomalies that are correlated across modalities; on the other hand, when defects are caused by parameter deviations in multiple processes, it is difficult to clarify the coupling relationship between various factors by observing only a single dimension, and the accuracy of defect root cause localization is severely limited.

[0004] In the traceability stage, current practices heavily rely on the personal experience of process engineers, requiring manual inspection of potential abnormal processes one by one. This is time-consuming, inefficient, and difficult to standardize and reuse. In the control stage, a single-parameter adjustment strategy based on a fixed threshold is typically used, such as adjusting only a specific temperature setpoint or a specific gas flow rate. This single-parameter adjustment method ignores the objectively existing mutually exclusive or synergistic relationships between process parameters. Adjusting a single parameter may alleviate the current defect, but it can easily lead to the deterioration of other quality indicators, resulting in repeated production line optimization and yield fluctuations. Furthermore, existing quality control systems generally lack effective feedback learning mechanisms. After system deployment, their analytical logic and decision-making models remain static, unable to adaptively update to keep pace with production line status drifts such as equipment aging, raw material batch differences, and seasonal environmental changes. Over the long term, their effectiveness gradually diminishes. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal data quality traceability and collaborative control system for optoelectronic display products, in order to solve the technical problems in the prior art where single-modal detection is difficult to reveal multi-process coupling defects, manual parameter adjustment is prone to overlooking some aspects and lacks continuous optimization capabilities.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multimodal data quality traceability and collaborative control system for optoelectronic display products includes:

[0008] The multimodal data acquisition module is used to acquire multimodal data during the production process of optoelectronic display products.

[0009] The preprocessing alignment module, whose input is connected to the output of the multimodal data acquisition module, is used to perform spatiotemporal alignment on the multimodal data to obtain an aligned multimodal sample sequence.

[0010] The feature extraction and fusion module, whose input is connected to the output of the preprocessing and alignment module, is used to extract the features of each modality data after alignment and fuse them to generate a fused feature vector.

[0011] The quality traceability analysis module, whose input is connected to the output of the feature extraction and fusion module, is used to trace the root cause of the fused feature vector based on the pre-built multimodal causal graph and anomaly detection results, and outputs the set of key process nodes that cause quality anomalies and the confidence level of each node.

[0012] The collaborative control decision module, whose input is connected to the output of the quality traceability analysis module, is used to perform multi-objective collaborative optimization with the adjustable process parameters corresponding to key process nodes as decision variables, and generate a collaborative control parameter set. The objectives of the multi-objective collaborative optimization include minimizing quality defect losses and minimizing control costs, while satisfying preset process constraints.

[0013] The feedback learning module connects to the multimodal data acquisition module at its input end to obtain the multimodal data re-acquired after regulation, and connects to the quality traceability analysis module and the collaborative regulation decision module at its output end. It is used to update the multimodal causal graph and the collaborative regulation decision model based on the multimodal quality data re-acquired after regulation.

[0014] As a further aspect of the present invention, the multimodal data includes at least three types: optical detection data, electrical test data, film thickness distribution data, process environment parameter data, and equipment status data; the optical detection data includes brightness uniformity images and chromaticity coordinates, the process environment parameter data includes temperature and cleanliness, and the film thickness distribution data includes thickness measurements of each functional layer.

[0015] As a further aspect of the present invention, the preprocessing alignment module performs spatiotemporal alignment of multimodal data based on product identification and location coordinates, wherein spatial alignment adopts coordinate grid mapping and temporal alignment adopts a sliding time window mechanism with a window width ranging from 0.5 seconds to 5 seconds.

[0016] As a preferred embodiment of the present invention, the feature extraction and fusion module includes:

[0017] The image feature extractor employs a convolutional neural network structure to process image-type modal data;

[0018] The sequence feature extractor, employing a Transformer encoder structure, is used to process time series modal data.

[0019] The multimodal fusion unit has its input connected to the output of the image feature extractor and the output of the sequence feature extractor, respectively. It uses a multi-head cross-attention mechanism to interactively fuse various modal features and generate a fused feature vector. The dimensional range of the fused feature vector is 256 to 1024.

[0020] As a preferred embodiment of the present invention, the quality traceability analysis module includes:

[0021] An anomaly detection unit receives a fused feature vector at its input, uses the isolated forest algorithm to identify anomalies in the fused feature vector, and outputs an anomaly score.

[0022] The causal reasoning unit has its input connected to the output of the anomaly detection unit and the output of the feature extraction and fusion module, respectively. It has a built-in multimodal causal graph, which is a directed acyclic graph. Its nodes represent process parameters, environmental parameters, equipment status, intermediate quality characteristics and final quality indicators, and the directed edges represent causal relationships and their strength.

[0023] The causal reasoning unit uses a graph neural network to calculate the abnormal contribution of each candidate root cause node based on the fused feature vector and abnormal score at abnormal moments, and selects nodes with abnormal contribution exceeding a preset threshold as key process nodes. The preset threshold ranges from 0.5 to 0.95.

[0024] As a further aspect of the present invention, the collaborative regulation decision-making module includes:

[0025] The constraint definition unit, whose input is connected to the output of the quality traceability analysis module, is used to obtain the allowable adjustment range, maximum adjustment step size, and mutual or cooperative constraint relationship between adjustable process parameters corresponding to key process nodes.

[0026] The proxy model unit uses a gradient boosting tree model to predict the product quality defect rate based on historical multimodal fusion features and corresponding process parameters.

[0027] A multi-objective optimization solver, whose input is connected to the output of the constraint definition unit and the output of the surrogate model unit respectively, aims to minimize the quality defect rate and minimize the control energy consumption cost, with the allowable adjustment range and the maximum adjustment step size as constraints, and uses a multi-objective particle swarm optimization algorithm to generate the Pareto front solution set;

[0028] The solution selection unit has its input connected to the output of the multi-objective optimization solver. It selects a set of coordinated control parameters from the Pareto front solution set according to preset preference weights. The quality weight in the preset preference weights ranges from 0.6 to 0.95.

[0029] As a further aspect of the present invention, the feedback learning module includes:

[0030] The effect evaluation unit obtains the quality indicators of the same batch of products before and after the control at its input end, and uses them to compare and calculate the quality improvement rate.

[0031] The model update unit, whose input is connected to the output of the effect evaluation unit, uses a Bayesian structure learning algorithm to incrementally update the edge weights of the multimodal causal graph when the quality improvement rate is lower than the preset improvement rate threshold, and retrains the gradient boosting tree surrogate model using the newly accumulated data.

[0032] The output of the model update unit is connected to the multimodal causal graph in the quality traceability analysis module and the surrogate model unit in the collaborative regulation decision module to achieve the update.

[0033] As a preferred embodiment of the present invention, the preprocessing alignment module is also used to perform missing value imputation and standardization processing on multimodal data.

[0034] This invention also provides a method for traceability and collaborative control of multimodal data quality in optoelectronic display products, applied to the aforementioned system for traceability and collaborative control of multimodal data quality in optoelectronic display products, comprising the following steps:

[0035] Step S1: Collect multimodal data from the optoelectronic display product production line through the multimodal data acquisition module and transmit it to the preprocessing and alignment module;

[0036] Step S2: The preprocessing alignment module performs spatiotemporal alignment on the multimodal data to obtain an aligned multimodal sample sequence, which is then output to the feature extraction and fusion module; wherein, the temporal alignment adopts a sliding time window with a window width ranging from 0.5 seconds to 5 seconds;

[0037] Step S3: The feature extraction and fusion module extracts the features of each modality data, and fuses them through a multi-head cross-attention mechanism to generate a fused feature vector, which is then output to the quality traceability analysis module.

[0038] Step S4: The quality traceability analysis module performs quality traceability on the fused feature vector based on the pre-built multimodal cause-effect graph and anomaly detection, locates the key process nodes that cause quality anomalies, and transmits the key process node information to the collaborative control decision module.

[0039] Step S5: The collaborative control decision module uses the adjustable process parameters corresponding to the key process nodes as decision variables to perform multi-objective collaborative optimization and generate a collaborative control parameter set. The objectives of the multi-objective collaborative optimization include minimizing quality defect losses and minimizing control costs, and the constraints include the allowable adjustment range of parameters and the maximum adjustment step size.

[0040] Step S6: Adjust the production line equipment according to the collaborative control parameter set, collect the adjusted multimodal data again through the multimodal data acquisition module, evaluate the control effect by the feedback learning module, and update the multimodal causal graph and collaborative control decision model using the evaluation results.

[0041] The aforementioned method for tracing and coordinating the quality of multimodal data in optoelectronic display products, wherein step S4 specifically includes:

[0042] The anomaly score is calculated by the anomaly detection unit using the isolated forest algorithm to fuse feature vectors;

[0043] The anomaly score and the fused feature vector are input into the causal inference unit. The causal inference unit injects the anomaly score into the multimodal causal graph and calculates the anomaly contribution of each candidate root cause node through graph neural network message passing.

[0044] Nodes whose abnormal contribution exceeds a preset threshold are selected as critical process nodes, with the preset threshold ranging from 0.5 to 0.95.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] In this invention, the spatiotemporal alignment and cross-attention fusion of multimodal data construct a comprehensive characterization covering optics, electricity, physical morphology, and process environment, making the identification of abnormal states no longer dependent on partial information. The tracing method based on causal graphs and graph neural networks removes the confusing factors associated with changes, enabling the separation of the true triggering nodes from the interaction of multiple processes, allowing for direct targeting of the root cause during regulation. Multi-objective collaborative optimization simultaneously solves the adjustment amounts of multiple related parameters under engineering constraints, reducing defect rates while controlling energy consumption costs, avoiding quality fluctuations or the introduction of new defects caused by isolated adjustment of single parameters. The feedback learning path allows the system's causal knowledge and quality prediction model to automatically adapt to equipment aging and changes in raw material batches, continuously improving regulation accuracy and yield during long-term operation, and significantly reducing manual analysis and parameter tuning work. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only examples of embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the connection relationship in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0050] Example 1:

[0051] See Figure 1 As shown in the figure, the optoelectronic display product multimodal data quality traceability and collaborative control system provided in this embodiment of the invention includes a multimodal data acquisition module, a preprocessing alignment module, a feature extraction and fusion module, a quality traceability analysis module, a collaborative control decision module, and a feedback learning module.

[0052] The multimodal data acquisition module is used to acquire multimodal data during the production process of optoelectronic display products.

[0053] The input of the preprocessing alignment module is connected to the output of the multimodal data acquisition module, and is used to perform spatiotemporal alignment on the multimodal data to obtain an aligned multimodal sample sequence.

[0054] The input of the feature extraction and fusion module is connected to the output of the preprocessing and alignment module. It is used to extract the features of each modality data after alignment and fuse them to generate a fused feature vector.

[0055] The input of the quality traceability analysis module is connected to the output of the feature extraction and fusion module. It is used to trace the root cause of the fused feature vector based on the pre-built multimodal causal graph and anomaly detection results, and output the set of key process nodes that cause quality anomalies and the confidence level of each node.

[0056] The input end of the collaborative control decision module is connected to the output end of the quality traceability analysis module. It is used to perform multi-objective collaborative optimization with the adjustable process parameters corresponding to key process nodes as decision variables, and generate a collaborative control parameter set. The objectives of multi-objective collaborative optimization include minimizing quality defect losses and minimizing control costs, while satisfying preset process constraints.

[0057] The input end of the feedback learning module is connected to the multimodal data acquisition module to obtain the multimodal data re-acquired after regulation. Its output end is connected to the quality traceability analysis module and the collaborative regulation decision module, respectively, to update the multimodal causal graph and the collaborative regulation decision model based on the multimodal quality data re-acquired after regulation.

[0058] This solution integrates multimodal data acquisition, spatiotemporal alignment, feature fusion, causal tracing, multi-objective collaborative optimization, and closed-loop feedback updates into a unified system. This allows heterogeneous and asynchronous production data to be comprehensively utilized under a unified temporal and spatial benchmark. By using causal graphs for root cause tracing rather than relying on surface correlations, it can accurately separate the sources of defects under the coupling effects of multiple process nodes. Multi-objective collaborative optimization simultaneously provides adjustment amounts for multiple process parameters while taking into account quality loss and control costs, avoiding the chain of negative effects caused by single-parameter adjustment. The feedback learning path enables the causal model and decision model to be continuously corrected according to changes in production line status. The system has adaptive capabilities to process drift, thereby steadily improving product yield and reducing the frequency of manual intervention in the long term.

[0059] Multimodal data includes at least three types: optical inspection data, electrical test data, film thickness distribution data, process environment parameter data, and equipment status data; optical inspection data includes brightness uniformity images and chromaticity coordinates, process environment parameter data includes temperature and cleanliness, and film thickness distribution data includes thickness measurements of each functional layer.

[0060] This approach further limits the collected multimodal data to encompass multiple dimensions, including optical performance, electrical properties, physical film thickness, environmental conditions, and equipment status, covering the main physical fields and process aspects affecting display quality. Since different modal data provide complementary information from the perspectives of luminescence uniformity, carrier transport capability, thin film geometry, and manufacturing environment disturbances, subsequent feature fusion and source tracing analysis can be based on more comprehensive evidence, avoiding misjudgments or omissions due to limited data. This enhances the reliability and robustness of root cause localization.

[0061] The preprocessing alignment module performs spatiotemporal alignment of multimodal data based on product identifiers and location coordinates. Spatial alignment uses coordinate grid mapping, while temporal alignment uses a sliding time window mechanism with a window width ranging from 0.5 seconds to 5 seconds.

[0062] This scheme defines a specific mechanism for spatiotemporal alignment. Spatial alignment is achieved by mapping product identifiers to a coordinate grid, ensuring a one-to-one spatial correspondence between measurements taken by different testing equipment for the same substrate area. Temporal alignment is achieved by using a sliding time window, which aggregates all modal records of the same product unit into a single sample within a reasonable time span. This approach eliminates data mismatches caused by sampling time deviations or detection position offsets, enabling the sample sequences used for subsequent feature extraction and fusion to accurately reflect the multidimensional attributes of the same physical object under approximately the same state. This improves the consistency of fusion characterization and the accuracy of quality analysis.

[0063] The feature extraction and fusion module includes: an image feature extractor, a sequence feature extractor, and a multimodal fusion unit, wherein:

[0064] The image feature extractor employs a convolutional neural network structure to process image-like modal data.

[0065] The sequence feature extractor uses a Transformer encoder structure to process time series modal data.

[0066] The input of the multimodal fusion unit is connected to the output of the image feature extractor and the output of the sequence feature extractor, respectively. It uses a multi-head cross-attention mechanism to interactively fuse various modal features and generate a fused feature vector. The dimensional range of the fused feature vector is 256 to 1024.

[0067] This scheme refines the internal structure of the feature extraction and fusion module. Convolutional neural networks excel at extracting local spatial anomaly patterns and texture information from data such as brightness uniformity images, while the Transformer encoder effectively captures long-range dependencies and dynamic trends in process parameter sequences. Both types of features retain the structural characteristics of their respective modalities. The subsequent multi-head cross-attention mechanism allows each modality's features to actively query relevant content from other modalities when generating the fused representation, thereby explicitly modeling the mutual influence between modalities. The resulting fused feature vector not only contains the unique information of each modality but also highlights cross-modal correlation features, enabling subsequent anomaly detection and causal inference to be performed in a higher-quality representation space.

[0068] The quality traceability analysis module includes an anomaly detection unit and a causal reasoning unit.

[0069] The anomaly detection unit receives the fused feature vector at its input, uses the isolated forest algorithm to identify anomalies in the fused feature vector, and outputs anomaly scores.

[0070] The input of the causal reasoning unit is connected to the output of the anomaly detection unit and the output of the feature extraction and fusion module, respectively. It has a built-in multimodal causal graph, which is a directed acyclic graph. Its nodes represent process parameters, environmental parameters, equipment status, intermediate quality characteristics and final quality indicators, and the directed edges represent causal relationships and their strength.

[0071] The causal reasoning unit uses a graph neural network to calculate the abnormal contribution of each candidate root cause node based on the fused feature vector and abnormal score at abnormal moments, and selects nodes with abnormal contribution exceeding a preset threshold as key process nodes. The preset threshold ranges from 0.5 to 0.95.

[0072] This solution outlines the working principle of the quality traceability analysis module. Anomaly detection using isolated forests does not rely on prior labels for specific defect types, enabling the discovery of anomalies with unknown patterns. The directed acyclic graph constructed in the causal reasoning unit explicitly defines the causal propagation path between equipment parameters, process conditions, intermediate characteristics, and final indicators. The graph neural network propagates anomaly scores along this structure and calculates their contribution, distinguishing between direct causes and passively following factors. The key process nodes identified in this way possess causal explanatory power, guiding engineers to understand the defect occurrence mechanism, reducing trial-and-error troubleshooting time. Furthermore, the contribution threshold can be adjusted according to the actual production line conditions to balance traceability recall and accuracy.

[0073] The collaborative control decision-making module includes: constraint definition unit, surrogate model unit, multi-objective optimization solver, and solution selection unit.

[0074] The input of the constraint definition unit is connected to the output of the quality traceability analysis module to obtain the allowable adjustment range, maximum adjustment step size, and mutual or cooperative constraint relationships between adjustable process parameters corresponding to key process nodes.

[0075] The proxy model unit adopts a gradient boosting tree model to predict the product quality defect rate based on historical multimodal fusion features and corresponding process parameters.

[0076] The input of the multi-objective optimization solver is connected to the output of the constraint definition unit and the output of the surrogate model unit, respectively. With the objectives of minimizing the quality defect rate and minimizing the control energy consumption cost, and with the allowable adjustment range and the maximum adjustment step size as constraints, the Pareto front solution set is generated by the multi-objective particle swarm optimization algorithm.

[0077] The input of the solution selection unit is connected to the output of the multi-objective optimization solver. A set of coordinated control parameters is selected from the Pareto front solution set according to the preset preference weights. The quality weight in the preset preference weights ranges from 0.6 to 0.95.

[0078] This scheme establishes a complete decision-making chain within the collaborative control decision-making module, encompassing constraint acquisition, quality prediction, multi-objective optimization, and solution selection. The constraint definition unit transforms the operational boundaries of the actual process and the dependencies between parameters into mathematical constraints, ensuring that the optimization results are feasible and safe in engineering practice. The surrogate model can quickly assess the potential defect rate of any parameter combination, avoiding repeated trial and error. Multi-objective particle swarm optimization simultaneously considers quality loss and energy cost as optimization objectives, and the output Pareto front intuitively demonstrates the trade-off between quality and cost. The solution selection unit automatically selects a set of parameters that balances or favors quality based on configured preference weights, enabling the production line to flexibly adjust its optimization bias according to current operational needs. Furthermore, simultaneous adjustment of multiple parameters prevents single-point adjustments from causing deterioration in other indicators.

[0079] The feedback learning module includes an effectiveness evaluation unit and a model update unit.

[0080] The input of the effect evaluation unit obtains the quality indicators of the same batch of products before and after the control, which are used for comparison and calculation of the quality improvement rate.

[0081] The input of the model update unit is connected to the output of the effect evaluation unit. When the quality improvement rate is lower than the preset improvement rate threshold, the Bayesian structure learning algorithm is used to incrementally update the edge weights of the multimodal causal graph, and the gradient boosting tree surrogate model is retrained using the newly accumulated data.

[0082] The output of the model update unit is connected to the multimodal causal graph in the quality traceability analysis module and the surrogate model unit in the collaborative regulation decision module to achieve the update.

[0083] This scheme endows the system with self-correcting capabilities during operation through effect evaluation and condition-triggered model updates. The effect evaluation unit continuously monitors the actual quality changes after each adjustment and quantifies the degree of improvement. When the improvement effect fails to meet expectations, it indicates that the current causal strength estimation or quality prediction model has deviated from the actual production line state. At this point, Bayesian structural learning is triggered to incrementally correct the edge weights of the causal graph, while the surrogate model is retrained, enabling the knowledge model and decision model to synchronously track slow changes such as equipment aging and raw material batch differences. This mechanism reduces the risk of system performance degradation over time and maintains long-term adjustment effectiveness.

[0084] The preprocessing alignment module is also used for imputing missing values ​​and standardizing multimodal data.

[0085] This scheme incorporates missing value imputation and standardization in the data preprocessing stage, which can eliminate data quality problems caused by differences in measurement range, units, or occasional signal interruptions from different sensors. Missing value imputation avoids the situation where the entire sample is invalidated due to the absence of individual modal data, thus improving sample utilization. Standardization unifies the dimensions of features across modalities, ensuring that subsequent neural network training and distance measurement are not affected by differences in the original scale, which is beneficial for improving the stability and convergence speed of feature extraction and anomaly detection.

[0086] Example 2:

[0087] This embodiment provides a method for tracing and coordinating the quality of multimodal data in optoelectronic display products, applied to the system of Embodiment 1, and includes the following steps:

[0088] Step S1: Collect multimodal data from the optoelectronic display product production line through the multimodal data acquisition module and transmit it to the preprocessing and alignment module;

[0089] Step S2: The preprocessing alignment module performs spatiotemporal alignment on the multimodal data to obtain an aligned multimodal sample sequence, which is then output to the feature extraction and fusion module; wherein, the temporal alignment adopts a sliding time window with a window width ranging from 0.5 seconds to 5 seconds;

[0090] Step S3: The feature extraction and fusion module extracts the features of each modality data, and fuses them through a multi-head cross-attention mechanism to generate a fused feature vector, which is then output to the quality traceability analysis module.

[0091] Step S4: The quality traceability analysis module performs quality traceability on the fused feature vector based on the pre-built multimodal cause-effect graph and anomaly detection, locates the key process nodes that cause quality anomalies, and transmits the key process node information to the collaborative control decision module.

[0092] Step S5: The collaborative control decision module uses the adjustable process parameters corresponding to the key process nodes as decision variables to perform multi-objective collaborative optimization and generate a collaborative control parameter set. The objectives of the multi-objective collaborative optimization include minimizing quality defect losses and minimizing control costs, and the constraints include the allowable adjustment range of parameters and the maximum adjustment step size.

[0093] Step S6: Adjust the production line equipment according to the collaborative control parameter set, collect the adjusted multimodal data again through the multimodal data acquisition module, evaluate the control effect by the feedback learning module, and update the multimodal causal graph and collaborative control decision model using the evaluation results.

[0094] This scheme embodies a closed-loop process from data acquisition, alignment and fusion, causal tracing, collaborative optimization to regulation execution and feedback updates. In the spatiotemporal alignment stage, a sliding time window ensures the temporal consistency of samples; multi-head cross-attention fusion enables full interaction of heterogeneous information; joint tracing of causal graphs and anomaly detection makes the discovered root causes interpretable; multi-objective collaborative optimization automatically generates the optimal adjustment scheme to coordinate various parameters under process constraints; and the feedback learning step re-injects the regulation results into the model update stage, giving the entire method an autonomous loop capability from perception to decision-making to adaptation, enabling continuous optimization of the product manufacturing process.

[0095] Step S4 specifically includes:

[0096] The anomaly score is calculated by the anomaly detection unit using the isolated forest algorithm to fuse feature vectors.

[0097] The anomaly score and the fused feature vector are input into the causal inference unit. The causal inference unit then injects the anomaly score into the multimodal causal graph and calculates the anomaly contribution of each candidate root cause node through graph neural network message passing.

[0098] Nodes whose abnormal contribution exceeds a preset threshold are selected as critical process nodes, with the preset threshold ranging from 0.5 to 0.95.

[0099] This scheme refines the execution logic of the source tracing steps. First, it uses an isolation forest to quantify anomalies in the fused features, enabling online labeling of anomaly levels without relying on an anomaly sample library. Then, by injecting anomaly scores into the causal graph and performing message passing within the graph neural network, the anomaly signal propagates along the causal edges and is distributed to various candidate root cause nodes. The resulting anomaly contribution reflects the likelihood of each node being a cause of the defect. Nodes exceeding a preset threshold are selected as output, maintaining reasonable sensitivity while suppressing noisy nodes. This ensures that the final set of key process nodes focuses on the truly necessary interventions, improving the targeting and efficiency of subsequent adjustments.

[0100] Example 3:

[0101] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the multimodal data quality traceability and collaborative control method for optoelectronic display products in Embodiment 2.

[0102] This solution embeds the control method into a computer program on a readable storage medium, enabling its convenient deployment on production line industrial control computers, edge servers, or cloud platforms. By executing the program through a processor, it can automatically complete alignment, fusion, traceability, and collaborative optimization decisions after multimodal data acquisition, reducing manual analysis and parameter tuning steps and improving the automation level and response speed of quality control in the optoelectronic display product manufacturing process.

[0103] In summary, the working principle of this invention is as follows:

[0104] The multimodal data acquisition module acquires multidimensional data such as optical detection, electrical testing, film thickness distribution, environmental parameters, and equipment status. The preprocessing and alignment module performs spatiotemporal alignment through spatial coordinate grid mapping and sliding time windows, and performs missing value imputation and standardization to unify heterogeneous data to the same benchmark. The feature extraction and fusion module uses convolutional neural networks and Transformers to extract image and temporal features respectively, and then uses a multi-head cross-attention mechanism to interact and fuse them to generate a complete fused feature vector. The anomaly detection unit in the quality traceability analysis module calculates anomaly scores through an isolated forest. The causal inference unit injects the scores into a pre-constructed multimodal causal graph. This graph is a directed acyclic graph that describes the causal chain from process parameters, environment, intermediate characteristics to the final index. The graph neural network transmits anomaly signals along the causal edges, calculates the anomaly contribution of each candidate root cause node, and selects nodes with contribution exceeding a set threshold as key process nodes. The collaborative control decision-making module receives information from key process nodes. The constraint definition unit extracts the allowable range, maximum step size, and mutually exclusive collaborative constraints of the corresponding adjustable parameters. The surrogate model predicts the quality defect rate based on a gradient boosting tree. The multi-objective optimization solver aims to minimize quality defect losses and control energy consumption costs, and uses a particle swarm optimization algorithm to generate a Pareto front. The solution selection unit selects a set of collaborative control parameters from this front based on preset preference weights. After control is implemented, the effect evaluation unit of the feedback learning module compares the batch quality improvement rate before and after control. When the improvement rate is lower than expected, the model update unit incrementally corrects the edge weights of the causal graph through Bayesian structure learning and retrains the surrogate model with newly accumulated data, ensuring that the causal model and the decision model are updated synchronously.

[0105] In use, the system is connected to the production line data acquisition link, and automatically and continuously receives multimodal data streams from various detection devices and sensors. After spatiotemporal alignment and feature fusion, it performs real-time anomaly detection and causal tracing. Once a quality anomaly is detected, it immediately locates key process nodes and generates a set of collaborative control parameters, which are then sent to the production equipment for intervention. Subsequently, the system automatically collects the multimodal data after control, evaluates the control effect, and triggers model updates, forming a closed loop from perception, tracing, decision-making to self-evolution, all without interrupting production.

[0106] The above description illustrates the basic principles of the present invention. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. The above embodiments and descriptions in the specification are only for illustrating the principles of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present invention without departing from the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal data quality traceability and collaborative control system for optoelectronic display products, characterized in that... include: The multimodal data acquisition module is used to acquire multimodal data during the production process of optoelectronic display products. The preprocessing alignment module, whose input is connected to the output of the multimodal data acquisition module, is used to perform spatiotemporal alignment on the multimodal data to obtain an aligned multimodal sample sequence. The feature extraction and fusion module has its input end connected to the output end of the preprocessing and alignment module. It is used to extract the features of each modality data after alignment and fuse them to generate a fused feature vector. The quality traceability analysis module, whose input is connected to the output of the feature extraction and fusion module, is used to perform root cause analysis on the fused feature vector based on the pre-constructed multimodal causal graph and anomaly detection results, and output the set of key process nodes that cause quality anomalies and the confidence level of each node. The collaborative control decision module, whose input is connected to the output of the quality traceability analysis module, is used to perform multi-objective collaborative optimization with the adjustable process parameters corresponding to the key process nodes as decision variables, and generate a collaborative control parameter set; the objectives of the multi-objective collaborative optimization include minimizing quality defect losses and minimizing control costs, while satisfying preset process constraints; The feedback learning module has its input end connected to the multimodal data acquisition module to obtain multimodal data re-acquired after regulation, and its output end connected to the quality traceability analysis module and the collaborative regulation decision module, respectively, for updating the multimodal causal graph and the collaborative regulation decision model based on the multimodal quality data re-acquired after regulation.

2. The multimodal data quality traceability and collaborative control system for optoelectronic display products according to claim 1, characterized in that: The multimodal data includes at least three types: optical detection data, electrical test data, film thickness distribution data, process environment parameter data, and equipment status data; the optical detection data includes brightness uniformity images and chromaticity coordinates, the process environment parameter data includes temperature and cleanliness, and the film thickness distribution data includes thickness measurements of each functional layer.

3. The multimodal data quality traceability and collaborative control system for optoelectronic display products according to claim 2, characterized in that: The preprocessing alignment module performs spatiotemporal alignment of the multimodal data based on product identification and location coordinates. Spatial alignment uses coordinate grid mapping, and temporal alignment uses a sliding time window mechanism with a window width ranging from 0.5 seconds to 5 seconds.

4. The multimodal data quality traceability and collaborative control system for optoelectronic display products according to claim 1, characterized in that: The feature extraction and fusion module includes: The image feature extractor employs a convolutional neural network structure to process image-type modal data; The sequence feature extractor, employing a Transformer encoder structure, is used to process time series modal data. The multimodal fusion unit has its input connected to the output of the image feature extractor and the output of the sequence feature extractor, respectively. It uses a multi-head cross-attention mechanism to interactively fuse various modal features to generate a fused feature vector. The dimensional range of the fused feature vector is 256 to 1024.

5. The multimodal data quality traceability and collaborative control system for optoelectronic display products according to claim 1, characterized in that: The quality traceability analysis module includes: An anomaly detection unit receives the fused feature vector at its input end, uses the isolated forest algorithm to identify anomalies in the fused feature vector, and outputs an anomaly score. The causal reasoning unit has its input terminals connected to the output terminals of the anomaly detection unit and the feature extraction and fusion module, respectively. It has the built-in multimodal causal graph, which is a directed acyclic graph. Its nodes represent process parameters, environmental parameters, equipment status, intermediate quality characteristics and final quality indicators, and the directed edges represent causal relationships and their strengths. The causal reasoning unit calculates the abnormal contribution of each candidate root cause node using a graph neural network based on the fused feature vector and abnormal score at abnormal moments, and selects nodes with abnormal contribution exceeding a preset threshold as key process nodes. The preset threshold ranges from 0.5 to 0.

95.

6. The multimodal data quality traceability and collaborative control system for optoelectronic display products according to claim 5, characterized in that: The coordinated regulation and decision-making module includes: The constraint definition unit, whose input is connected to the output of the quality traceability analysis module, is used to obtain the allowable adjustment range, maximum adjustment step size, and mutual exclusion or cooperative constraint relationship between the adjustable process parameters corresponding to the key process node. The proxy model unit uses a gradient boosting tree model to predict the product quality defect rate based on historical multimodal fusion features and corresponding process parameters. A multi-objective optimization solver, whose input is connected to the output of the constraint definition unit and the output of the surrogate model unit respectively, aims to minimize the quality defect rate and minimize the control energy consumption cost, and uses the allowable adjustment range and the maximum adjustment step size as constraints. It generates a Pareto front solution set using a multi-objective particle swarm optimization algorithm. The solution selection unit has its input end connected to the output end of the multi-objective optimization solver. It selects a set of coordinated control parameters from the Pareto front solution set according to preset preference weights. The quality weight in the preset preference weights ranges from 0.6 to 0.

95.

7. The multimodal data quality traceability and collaborative control system for optoelectronic display products according to claim 6, characterized in that: The feedback learning module includes: The effect evaluation unit obtains the quality indicators of the same batch of products before and after the control at its input end, and uses them to compare and calculate the quality improvement rate. The model update unit, whose input is connected to the output of the effect evaluation unit, when the quality improvement rate is lower than the preset improvement rate threshold, uses the Bayesian structure learning algorithm to incrementally update the edge weights of the multimodal causal graph, and retrains the gradient boosting tree surrogate model using the newly accumulated data. The output of the model update unit is connected to the multimodal causal graph in the quality traceability analysis module and the surrogate model unit in the collaborative regulation decision module, respectively, to achieve the update.

8. The multimodal data quality traceability and collaborative control system for optoelectronic display products according to claim 1, characterized in that: The preprocessing alignment module is also used to perform missing value imputation and standardization on the multimodal data.

9. A method for traceability and collaborative control of multimodal data quality in optoelectronic display products, applied to the multimodal data quality traceability and collaborative control system for optoelectronic display products as described in any one of claims 1 to 8, characterized in that... Includes the following steps: Step S1: Collect multimodal data from the optoelectronic display product production line through the multimodal data acquisition module and transmit it to the preprocessing and alignment module; Step S2: The preprocessing alignment module performs spatiotemporal alignment on the multimodal data to obtain an aligned multimodal sample sequence, which is then output to the feature extraction and fusion module; wherein, the temporal alignment adopts a sliding time window with a window width ranging from 0.5 seconds to 5 seconds; Step S3: The feature extraction and fusion module extracts the features of each modality data, and fuses them through a multi-head cross-attention mechanism to generate a fused feature vector, and outputs the fused feature vector to the quality traceability analysis module; Step S4: The quality traceability analysis module performs quality traceability on the fused feature vector based on the pre-built multimodal causal graph and anomaly detection, locates the key process nodes that cause quality anomalies, and transmits the key process node information to the collaborative control decision module. Step S5: The collaborative control decision module uses the adjustable process parameters corresponding to the key process nodes as decision variables to perform multi-objective collaborative optimization and generate a collaborative control parameter set; the objectives of the multi-objective collaborative optimization include minimizing quality defect losses and minimizing control costs, and the constraints include the allowable adjustment range of parameters and the maximum adjustment step size; Step S6: Adjust the production line equipment according to the set of coordinated control parameters, collect the adjusted multimodal data again through the multimodal data acquisition module, evaluate the control effect by the feedback learning module, and update the multimodal causal graph and coordinated control decision model using the evaluation results.

10. The method for multimodal data quality traceability and collaborative control of optoelectronic display products according to claim 9, characterized in that: Step S4 specifically includes: The anomaly score of the fused feature vector is calculated by the anomaly detection unit using the isolated forest algorithm; The anomaly score and the fused feature vector are input into the causal inference unit. The causal inference unit injects the anomaly score into the multimodal causal graph and calculates the anomaly contribution of each candidate root cause node through graph neural network message passing. Nodes whose abnormal contribution exceeds a preset threshold are selected as critical process nodes, and the preset threshold ranges from 0.5 to 0.95.

Images