Method and system for identifying and predicting production line risk of multi-mode optical device

Abstract

The invention relates to the technical field of optical communication, and discloses a multi-modal optical device production line risk identification and prediction method and system, and the method comprises the steps: collecting multi-modal data in real time from each process of an optical device production line, carrying out the preprocessing of the multi-modal data, carrying out the feature extraction, constructing a unified feature matrix, respectively extracting depth time sequence features and depth space features, and carrying out the recognition and prediction of the risk of the multi-modal optical device production line. The depth time sequence features and the spatial features are fused, a risk identification prediction decision model carries out risk prediction through the fused features, process nodes are obtained, target features of the process nodes are screened through an SHAP method, abnormal areas in the target features are positioned through gradient mapping, and an early warning threshold value is dynamically set. According to the method and the device, the technical problems that risk early warning and identification cannot be carried out in time and decisions cannot be taken in time during production of the optical device production line are solved.

Description

Technical Field

[0001] This invention relates to the field of optical communication technology, and in particular to a method and system for risk identification and prediction in multimodal optical device production lines. Background Technology

[0002] Optical devices undergo multiple production processes during mass production on a production line, including chip mounting, wire bonding, optical coupling, and sealing testing. These processes operate sequentially in an assembly line fashion. Any station can be affected by factors such as equipment operation, precision variations, operator skill levels, and mechanical vibrations, leading to issues like solder misalignment or bubbles, wire bonding collapse, and chip solder ball peeling. The combined effects of these factors can severely impact product yield. When problems occur simultaneously, they can even trigger shutdowns, halting the entire production line and significantly impacting production efficiency and schedule.

[0003] Existing technical solutions typically employ human-machine collaboration as the primary strategy. This involves manually conducting regular spot checks on product yield and performing preventative maintenance on corresponding production machines. Simultaneously, technical training is provided to operators in the corresponding positions, and standard operating procedures (SOPs) are updated regularly. This is combined with some machine vision AOI (Automated Optical Inspection) and the setting of threshold alarms for corresponding process machines to jointly monitor yield.

[0004] Existing technical solutions suffer from several drawbacks: First, there is a lag in defect transmission, with the average execution cycle from defect occurrence to sampling detection and traceability isolation taking tens of minutes. Second, manual sampling is inefficient and lacks coverage, while machine vision sampling also has insufficient coverage. Third, there is a disconnect between processes; for example, when an anomaly is detected in a previous process, some defects have already been passed on to the next process, and the later the discovery, the greater the loss, resulting in a large time dimension for data coordination between upstream and downstream processes. Fourth, these solutions rely on fixed maintenance strategies and human experience, placing high demands on machinery and operators. Once a defect occurs, process adjustments are required, leading to long average waiting times and production delays. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problem that risks cannot be warned and identified in a timely manner during the production of optical devices, and that timely decision-making cannot be made. This invention provides a risk identification and prediction method and system based on multimodal optical device production lines.

[0006] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A method for identifying and predicting risks in a multimodal optical device production line includes the following steps:

[0008] Multimodal data is collected in real time from each process of the optical device production line, and the multimodal data is preprocessed.

[0009] Feature extraction is performed on the preprocessed multimodal data to form a temporal feature matrix, a spatial feature matrix, and an environmental feature matrix. The three matrices are then fused to construct a unified feature matrix.

[0010] The risk identification and prediction decision model constructs a directed graph through a unified feature matrix. The directed graph extracts deep temporal features and deep spatial features through a spatiotemporal feature encoder. The deep temporal features and spatial features are then fused, and risk prediction is performed through the fused features.

[0011] The risk identification, prediction, and decision-making model obtains process nodes through risk prediction results and inter-process dependencies, filters target features of process nodes using the SHAP method, and locates abnormal regions in target features through gradient mapping.

[0012] The risk identification, prediction, and decision-making model dynamically sets early warning thresholds based on risk prediction results and contribution mapping, issues current risk warnings based on these thresholds, and generates actions to minimize risk.

[0013] To address the problem of early risk underreporting caused by the difficulty in fusing multi-source heterogeneous data, this invention constructs a multimodal unified feature matrix that integrates temporal, spatial, and environmental features, and performs spatiotemporal feature encoding and fusion prediction based on a directed graph constructed from process topology. This solves the problem of traditional single-dimensional monitoring models failing to fully represent composite risk factors and drowning out weak abnormal signals.

[0014] To address the difficulty in quantifying and locating the propagation of risks along complex process chains, this invention combines predicted risks with process dependencies through an analysis module. It utilizes an attention mechanism to locate key risk source processes and then uses the SHAP method and gradient mapping to achieve precise tracing from process to specific characteristics and abnormal areas, thus solving the problems of difficulty in tracing the root causes of risks and lack of transparency in decision-making.

[0015] To address the issues of delayed early warnings and poor adaptability of static models in dynamic production lines, this invention implements adaptive threshold early warning based on real-time risk distribution through a dynamic decision module. It also combines multi-task loss and online learning mechanisms to continuously optimize the model, thus solving the problems of high false alarm and false negative rates and performance degradation of the fixed threshold early warning mechanism during continuous operation of the production line.

[0016] Compared with existing technologies, the beneficial effects of this application are as follows: Through a closed-loop design of the entire chain, including multimodal perception, feature fusion, network prediction, explainable tracing, dynamic decision-making, and online evolution, it achieves comprehensive perception and deep integration of multi-source risk factors in optical device production lines at the data level; at the model level, it constructs a risk propagation quantification model that fits the production topology, significantly improving the early identification rate and predictive foresight of risks, especially hidden coupling risks; and at the application level, it provides an explainable integrated solution from risk warning and root cause localization to optimization decision-making, realizing the transformation from passive response to proactive intervention and effectively ensuring the efficient and stable operation of optical device production lines.

[0017] Furthermore, a method for identifying and predicting risks in a multimodal optical device production line is provided, wherein the collected multimodal data includes equipment operating parameters, sensor monitoring data, visual inspection data, and environmental parameters;

[0018] The equipment operating parameters include welding, bonding, and mounting processes;

[0019] The sensor monitoring data includes optical device vibration parameters, optical device temperature parameters, optical device pressure deformation parameters, and optical device displacement parameters.

[0020] The visual inspection data includes images of the optical device surface;

[0021] The environmental parameters include the cleanliness of the production environment, the humidity of the production environment, the temperature of the production environment, and the electromagnetic interference of the production environment.

[0022] In the above solution, by clearly defining the specific dimensions (equipment operating parameters, sensor monitoring data, visual inspection data, and environmental parameters) of the multimodal data collected from the optical device production line and the processes and physical quantities they cover, the problem of fragmented risk feature representation caused by isolated data sources and incomplete modalities, and the inability to comprehensively depict the risk coupling relationship under complex production conditions, which are present in existing production line monitoring methods, is solved. This invention constructs a collaborative monitoring data system integrating equipment, sensors, vision, and environment, enabling a unified feature matrix to fully capture multidimensional features strongly correlated with various risk mechanisms (such as deformation caused by process parameter drift and abnormal displacement caused by mechanical vibration) from multiple perspectives, including process settings, physical responses, appearance, and environmental disturbances. This invention provides a complete and structured input for subsequent directed graph-based spatiotemporal feature encoding and fusion prediction, achieving end-to-end accurate mapping and early warning from multi-source heterogeneous data to the comprehensive risk status of the production line.

[0023] Furthermore, a method for identifying and predicting risks in multimodal optical device production lines, wherein fusing the three matrices to construct a unified feature matrix includes the following sub-steps:

[0024] The preprocessed equipment operating parameters and sensor monitoring data are integrated into a time series feature set, and time series features are extracted to form a time series feature matrix.

[0025] The preprocessed visual detection data is integrated into an image feature set, spatial features are extracted, and a spatial feature matrix is ​​formed.

[0026] The preprocessed environmental parameters are integrated into an environmental feature set, and statistical features are extracted to form an environmental feature matrix.

[0027] A unified feature matrix is ​​constructed by integrating the temporal feature matrix, spatial feature matrix, and environmental feature matrix.

[0028] In the above-described scheme, this invention solves the problems of feature alignment difficulties, increased information redundancy and noise interference, and the inability to form a unified and efficient digital representation of production line status caused by directly mixing multi-source heterogeneous data in existing methods. This is achieved by extracting and constructing feature matrices from temporal, spatial, and environmental features separately and then fusing them. Through a strategy of "modal extraction followed by fusion and integration," this invention, while preserving the inherent physical and statistical characteristics of each modality of data (such as equipment temporal parameters, visual spatial images, and environmental steady-state parameters), first transforms them into structured matrices of the same dimension, and then performs ordered splicing. This not only achieves lossless integration and noise reduction of multi-dimensional risk information but also constructs a unified, regular, and highly condensed risk feature foundation. This invention provides high-quality input that can be directly and efficiently processed for subsequent deep spatiotemporal coding and risk prediction models based on directed graphs, significantly improving the model's ability to capture complex risk patterns and its computational efficiency.

[0029] Furthermore, a method for identifying and predicting risks in a multimodal optical device production line, wherein the risk prediction through feature fusion includes the following sub-steps:

[0030] The directed graph construction module constructs a directed graph based on a unified feature matrix;

[0031] The directed graph is input into an LSTM network with a multi-head attention mechanism to learn the temporal features of each process node, and the deep temporal features are output to the multimodal feature fusion module.

[0032] The directed graph is input into the graph attention network to learn the spatial features of each process node, and the deep spatial features are output to the multimodal feature fusion module.

[0033] The multimodal feature fusion module uses a gated fusion mechanism to fuse deep spatial features and deep temporal features, and outputs them to the risk prediction output layer.

[0034] The risk prediction output layer calculates the current risk prediction and the future risk prediction, and outputs them to the analysis module.

[0035] In the above-described scheme, this invention models the production line process topology using a directed graph, extracts the deep temporal and spatial features of nodes via a multi-head attention LSTM network and a graph attention network, and finally fuses them through a gating mechanism for risk prediction. This solves the problems of traditional risk prediction models that treat temporal and spatial features separately, struggle to quantify complex dependencies between processes, and suffer from insufficient information fusion due to simple feature splicing. This invention uses a directed graph structure to uniformly represent the physical connections and data associations of processes, enabling the model to simultaneously learn the evolution of risks in the temporal dimension and their spatial propagation paths within the process network. The gating fusion mechanism adaptively calibrates and integrates spatiotemporal dual-stream features according to the prediction task, thereby achieving accurate modeling and forward-looking prediction of complex risk patterns with spatiotemporal coupling characteristics in optical device production lines, significantly improving the accuracy and timeliness of risk warnings.

[0036] Furthermore, a method for identifying and predicting risks in a multimodal optical device production line, wherein learning the spatial features of each process node includes the following sub-steps:

[0037] The attention coefficient module multiplies the weight matrix with the spatial features respectively and concatenates them. The concatenated matrix is ​​then processed by the attention vector, activated by the LeakyReLU activation function, and output to the attention weight module.

[0038] The normalization module normalizes the attention vector and outputs it to the feature aggregation module;

[0039] Feature aggregation is performed using normalized attention vectors, and the output is sent to the multimodal feature fusion module.

[0040] In the above scheme, this invention extracts the spatial features of process nodes by explicitly designing three sub-steps: attention coefficient calculation, normalization, and feature aggregation. This solves the key problem of traditional graph neural networks treating all adjacency relationships equally and failing to adaptively distinguish and quantify the influence intensity between processes when processing production process graphs. This invention introduces a learnable attention mechanism, enabling the model to dynamically evaluate and focus on key process connections that contribute more to the current risk state, rather than treating all topological relationships equally. Specifically, the original attention coefficients between nodes are first calculated, then normalized to a stable weight distribution, ultimately guiding the efficient aggregation of features. This invention endows the model with the ability to simulate the selective nature of risk transmission in real production lines, significantly improving the accuracy of deep spatial features in characterizing potential risk paths, and laying the foundation for subsequent multimodal fusion and accurate prediction.

[0041] Furthermore, a method for identifying and predicting risks in a multimodal optical device production line, wherein calculating the current risk prediction and future risk prediction includes the following sub-steps:

[0042] The current risk prediction module will fuse features multiplied by a weight matrix, add a bias vector, and then map the data through a Sigmoid activation function before outputting it to the future risk prediction and analysis modules.

[0043] The future risk prediction module concatenates the historical feature set and the current predicted risk, multiplies it by the prediction weight matrix, and adds a bias vector before outputting the future risk prediction to the dynamic decision-making module.

[0044] In the above-mentioned scheme, this invention solves the problems of traditional risk warning models that only output a single instantaneous risk value, cannot depict the risk evolution trend, and suffer from inaccurate trend prediction and insufficient foresight due to ignoring historical state sequences. It constructs a cascaded prediction architecture consisting of a current risk prediction module and a future risk prediction module, and clarifies their respective calculation processes and data flows. This invention maps fused features to current risk probabilities using the Sigmoid function, achieving stable and interpretable risk quantification of the production line status. Then, it concatenates this current risk with historical fused feature sequences, and uses linear mapping to prospectively extrapolate future multi-step risk trends. This not only accurately diagnoses whether there is current risk but also predicts how the risk will develop, forming a complete risk evolution cognitive chain from history, present to future. This provides the dynamic decision-making module with multi-level, cross-time-dimensional key decision-making basis, from immediate alerts to pre-scheduling.

[0045] Furthermore, a method for identifying and predicting risks in a multimodal optical device production line, wherein locating abnormal regions in target features through gradient mapping includes the following sub-steps:

[0046] The process impact module calculates the importance of process impact using a normalized attention vector and the current predicted risk, obtains the main risk sources, and outputs them to the feature contribution module.

[0047] The feature contribution module calculates the feature contribution based on the main risk sources using the SHAP method, generates a parameter contribution ranking list, obtains the target features, and outputs them to the root cause tracing module.

[0048] The root cause tracing module is based on gradient-weighted class activation mapping. It locates the data / image region that causes the anomaly based on the target features and outputs the result to the dynamic decision module.

[0049] In the above-described solution, this invention addresses the critical bottleneck of existing risk warning methods, which can only provide general risk signals and cannot penetrate to the specific physical level for problem tracing, leading to a lack of precise operational basis for maintenance decisions. This is achieved by constructing a three-tiered interpretable analysis chain of "process impact - feature contribution - root cause tracing." First, this invention locates the main risk sources at the process network level using an attention mechanism. Then, within that process, it uses the SHAP method to filter out key abnormal parameters from a large number of features. Finally, it uses gradient mapping technology to accurately pinpoint the specific segments or regions causing the anomalies within the original data or image space. This invention achieves a progressive focus and visualization of risk attribution from the macro-topology layer to the meso-feature layer and then to the micro-data / pixel layer. It transforms abstract model predictions into diagnostic reports pointing to specific processes, parameters, and locations, providing direct and actionable decision-making basis for precise intervention, maintenance, or process adjustments on the production line, greatly improving the efficiency of risk management.

[0050] Furthermore, a method for identifying and predicting risks in a multimodal optical device production line, wherein dynamically setting a warning threshold based on risk prediction results and contribution mapping, and issuing a current risk warning based on the warning threshold includes the following sub-steps:

[0051] The current risk warning threshold is dynamically adjusted based on historical risks, using the following formula:

[0052] ;

[0053] in, The warning threshold is set at time step t. The historical risk probability mean at time step t-1. This is the risk tolerance coefficient. The standard deviation of the historical risk probability at time step t-1;

[0054] Based on the current risk forecast, the corresponding warning level is triggered:

[0055] like This triggers a Level 1 warning, and continuous monitoring is initiated.

[0056] like This triggers a Level 2 warning; an immediate inspection is required.

[0057] like The system has triggered a Level 3 warning and will immediately shut down for maintenance.

[0058] in, This is a current risk forecast.

[0059] In the above-mentioned solution, this invention addresses the key problems of high false alarm and missed alarm rates under fluctuating operating conditions and the inability of a single alarm signal to match differentiated response procedures caused by the use of fixed early warning thresholds in traditional production line risk monitoring systems. This is achieved by introducing an early warning threshold formula dynamically adjusted based on historical risk data and a clearly tiered multi-level early warning mechanism. This invention enables the early warning threshold to adaptively adjust with the mean and volatility of the production line's historical risk level. The risk tolerance coefficient provides a configurable interface for adjusting early warning sensitivity, effectively overcoming the shortcomings of static thresholds in adapting to equipment performance degradation or changes in production rhythm. Furthermore, by setting three-level early warning intervals (monitoring, inspection, and shutdown maintenance) bound to specific handling suggestions, it maps continuous model risk prediction values ​​into discrete, immediately executable differentiated operation and maintenance instructions. This invention constructs a complete closed loop from risk quantification and dynamic assessment to tiered response, significantly improving the accuracy of early warnings and the timeliness and effectiveness of risk response actions.

[0060] Furthermore, a method for identifying and predicting risks in a multimodal optical device production line is provided. The risk identification and prediction decision model is iteratively trained using a multi-task loss function, as shown in the following formula:

[0061] ;

[0062] in, For multi-task loss function, The current risk prediction loss function is... For future risk prediction loss function, To analyze the loss function, For dynamic decision loss function, For training parameters, The number of training samples. , Let i be the current true risk vector of training sample i. Let i be the current risk prediction vector for training sample i. Let i be the future predicted risk vector for training sample i. Let i be the future true risk vector of the training sample i. For the input data of training sample i, The feature contribution vector of training sample i. The contribution vector of the target features for training sample i. The decision action corresponding to the original input of training sample i. Add the decision action corresponding to the perturbation input to the training sample i. The input features are small perturbation values ​​(hyperparameters). It is the square of the L2 norm of the vector.

[0063] Furthermore, a risk identification and prediction system for multimodal optical device production lines includes a data acquisition module, a preprocessing module, a feature extraction and fusion module, and a risk identification and prediction decision model; the risk identification and prediction decision model includes a risk identification and prediction model, an analysis module, and a dynamic decision module.

[0064] The data acquisition module is used to collect multimodal data in real time from each process of the optical device production line.

[0065] The preprocessing module is used to preprocess multimodal data;

[0066] The feature extraction and fusion module is used to extract features from the preprocessed multimodal data to form a temporal feature matrix, a spatial feature matrix, and an environmental feature matrix, and then fuse the three matrices to construct a unified feature matrix.

[0067] The risk prediction model is used to construct a directed graph through a unified feature matrix. The directed graph extracts deep temporal features and deep spatial features through a spatiotemporal feature encoder, and then fuses the deep temporal features and spatial features to perform risk prediction.

[0068] The analysis module is used to obtain process nodes through risk prediction results and inter-process dependencies, filter target features of process nodes through the SHAP method, and locate abnormal regions in target features through gradient mapping.

[0069] The dynamic decision-making module is used to dynamically set early warning thresholds based on risk prediction results and contribution mapping, issue current risk warnings based on the early warning thresholds, and generate actions to minimize risks.

[0070] In the above solution, by systematically dividing the end-to-end risk identification and prediction system into seven core functional modules and clarifying their collaborative logic, the engineering and technical problem of the disconnect between algorithms, models and business processes in existing industrial risk management solutions, which makes it difficult to form a closed loop from perception, analysis, decision-making to self-evolution, is solved. This invention, through a modular architecture, clarifies the standardized information flow and responsibility boundaries from data acquisition, preprocessing, feature fusion, model prediction, analysis, dynamic decision-making, and iterative optimization, transforming the methodology into a stable and reliable system engineering implementation. Furthermore, through the loose coupling and collaborative work of each module, a complete production line risk management intelligent agent with continuous perception, intelligent diagnosis, proactive early warning, and autonomous evolution capabilities is constructed. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a flowchart of the method of the present invention.

[0073] Figure 2 This is a structural diagram of a risk prediction model.

[0074] Figure 3 This is a structural diagram of the system of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0076] Example 1: A method for identifying and predicting risks in multimodal optical device production lines.

[0077] like Figure 1 As shown, a method for identifying and predicting risks in a multimodal optical device production line includes the following steps:

[0078] S1: Real-time acquisition of multimodal data from each process of the optical device production line;

[0079] The multimodal data includes equipment operating parameters, sensor monitoring data, visual inspection data, and environmental parameters;

[0080] The equipment operating parameters include welding, bonding, and mounting processes.

[0081] The welding process includes welding temperature, welding time, and welding pressure;

[0082] The bonding process includes bonding force, ultrasonic power, and bonding time;

[0083] The mounting process includes mounting accuracy, mounting angle, and mounting pressure.

[0084] The sensor monitoring data includes optical device vibration parameters, optical device temperature parameters, optical device pressure deformation parameters, and optical device displacement parameters.

[0085] In this embodiment, the sensors include a vibration sensor, a temperature sensor, a pressure sensor, and a displacement sensor. The vibration sensor has a sampling frequency range of 0-1000Hz and a sampling rate of 2kHz. The temperature sensor has a measurement range of 0-300℃ and an accuracy of ±0.1℃. The pressure sensor has a measurement range of 0-1000Pa and an accuracy of ±1Pa. The displacement sensor has a measurement range of 0-100μm and a resolution of 0.1μm.

[0086] The visual inspection data includes images of the optical device surface;

[0087] In the real-time example, a 20-megapixel industrial camera was used. Under coaxial white light illumination, the image resolution was set to 5472×3648 pixels, and images of the optical device surface were acquired at a frequency of 5 frames per minute.

[0088] The environmental parameters include the cleanliness of the production environment, the humidity of the production environment, the temperature of the production environment, and the electromagnetic interference of the production environment.

[0089] In this embodiment, the cleanliness of the production environment is defined as the measured particle concentration (unit: To prevent contaminants from adhering to the surface of optical devices and affecting performance or yield; the humidity measurement range of the production environment is 0-100%RH, with an accuracy of ±2%, to prevent oxidation and condensation caused by excessive humidity, or electrostatic damage caused by excessively low humidity; the temperature measurement range of the production environment is -10~50℃. Accuracy ±0.5 It provides a stable thermal environment to ensure the consistency of production processes; the measurement frequency range of electromagnetic interference in the production environment is 10Hz-1GHz to assess the intensity of electromagnetic noise in the environment and prevent interference or impact on the operation of precision instruments (such as optical devices).

[0090] S2: Preprocess the multimodal data;

[0091] S21: Outliers in multimodal data are removed based on the 3σ criterion, using the following formula:

[0092] ;

[0093] in, For the q-th multimodal data, The mean of the multimodal data points. represents the standard deviation of the multimodal data points.

[0094] S22: After removing outliers, missing data in the multimodal data is completed using linear interpolation. The formula is as follows:

[0095] ;

[0096] in, for The interpolation value at time, for The valid data corresponding to the time. for The valid data corresponding to the time. For the valid data after missing data time, For the valid data after missing data time, For missing data time.

[0097] S23: Standardize the completed multimodal data using the Min-Max standard, with the following formula:

[0098] ;

[0099] in, For the q-th standardized multimodal data, The minimum value of the multimodal data. This represents the maximum value of the multimodal data.

[0100] S3: Extract features from the preprocessed multimodal data to form a temporal feature matrix, a spatial feature matrix, and an environmental feature matrix. Merge the three matrices to construct a unified feature matrix.

[0101] S31: Integrate the preprocessed equipment operating parameters and sensor monitoring data into a time series feature set, extract time series features, and form a time series feature matrix;

[0102] S311: Statistical feature extraction of time series feature set;

[0103] S3111: Calculate the mean of each element in the time series feature set, using the following formula:

[0104] ;

[0105] in, The mean of the elements in the time series feature set is calculated (e.g., the mean of welding temperature), where n is the total number of elements in the time series feature set. Calculate the j-th sampled value under the element in the time series feature set, where j=1,2,...n.

[0106] The elements in the time-series feature set include welding temperature, welding time, welding pressure, bonding force, ultrasonic power, bonding time, mounting accuracy, mounting angle, mounting pressure, optical device vibration parameters, optical device temperature parameters, optical device pressure deformation parameters, and optical device displacement parameters.

[0107] S3112: Calculate the variance of each element in the time series feature set, using the following formula:

[0108] ;

[0109] in, Calculate the variance of elements in the time series feature set.

[0110] S3113: Calculate the skewness of each element in the time series feature set, using the following formula:

[0111] ;

[0112] in, To calculate the skewness of elements in the time series feature set, Calculate the standard deviation of elements in the time series feature set;

[0113] S3114: Calculate the kurtosis of each element in the time series feature set, using the following formula:

[0114] ;

[0115] in, Calculate the kurtosis of elements in the time series feature set.

[0116] S3115: Use the mean, variance, skewness, and kurtosis of each element in the time series feature set as statistical feature vectors.

[0117] S312: Extract temporal features from the temporal feature set;

[0118] S3121: Calculate the root mean square value of each element in the time series feature set, using the following formula:

[0119] ;

[0120] in, The root mean square value of the elements in the time series feature set is calculated.

[0121] S3122: Calculate the peak factor of each element in the time series feature set, using the following formula:

[0122] ;

[0123] in, Calculate the peak factor of elements in the time series feature set. for The maximum absolute value.

[0124] S3123: Calculate the impulse factor of each element in the time series feature set, using the following formula:

[0125] ;

[0126] in, The impulse factor of the elements in the time series feature set is calculated.

[0127] S3124: Use the root mean square value, peak factor, and impulse factor of each element in the time series feature set as the time domain feature vector.

[0128] S313: Extract the frequency components of each element in the time-series feature set using Fast Fourier Transform (FFT) to form a frequency domain feature vector. The formula is as follows:

[0129] ;

[0130] in, The k-th frequency component in the frequency domain The imaginary unit, The dominant frequency in the signal. The argmax function is used to extract the maximum value of the k-th frequency component.

[0131] S314: Integrate statistical feature vectors, time-domain feature vectors, and frequency-domain feature vectors to form a time-series feature matrix;

[0132] S32: Integrate the preprocessed visual detection data into an image feature set, extract spatial features, and form a spatial feature matrix;

[0133] S321: The LBP algorithm is used to extract local texture features from the surface image of the optical device, and the various local texture features are integrated into a texture feature vector. The formula is as follows:

[0134] ;

[0135] in, The grayscale value of the center pixel. Let be the grayscale value of the neighboring pixels, and p be the index of the p-th pixel in the neighborhood surrounding the center pixel. Let these be the coordinates of the center pixel in the image of the optical device surface. For local texture features, It can be 0 or 1.

[0136] It is important to note that Based on the center pixel The local texture features are encoded by comparing the binarized gray values ​​of the pixels and their neighboring pixels.

[0137] S322: Integrate the area, perimeter, circularity, and convexity of the surface images of each optical device as a morphological feature vector;

[0138] S323: The number of pixels in each channel of the image on the surface of the optical device is counted by analyzing the histogram of each RGB channel. The number of pixels in each channel is then integrated into a color feature vector. The formula is as follows:

[0139] ;

[0140] in, For the R channel in grayscale The number of pixels, For the G channel in grayscale The number of pixels, For the B channel in grayscale The number of pixels, For the R channel in pixels The intensity value, For the G channel in pixels The intensity value, For the B channel in pixels The intensity value, Let x be the Dirac function, and when x=0, =1; otherwise =0, m=1,2,...M, .

[0141] S325: Integrate texture feature vectors, morphological feature vectors, and color feature vectors to form a spatial feature matrix.

[0142] S33: Integrate the preprocessed environmental parameters into an environmental feature set, extract statistical features, and form an environmental feature matrix.

[0143] Specifically, the mean, variance, skewness, and kurtosis of each element in the environmental feature set are used as environmental feature vectors, and these environmental feature vectors are integrated to form an environmental feature matrix.

[0144] The elements in the environmental feature set include the cleanliness of the production environment, the humidity of the production environment, the temperature of the production environment, and the electromagnetic interference of the production environment.

[0145] S34: Merge the temporal feature matrix, spatial feature matrix, and environmental feature matrix to construct a unified feature matrix, using the following formula:

[0146] ;

[0147] in, To unify the feature matrix, The time series feature matrix, , For image feature matrix, , For the environmental feature matrix, , For the number of processes, For time-series feature dimensions, For spatial feature dimensions, Let R be the environmental feature dimension, and R be a real number.

[0148] The risk identification, prediction, and decision-making model includes a risk prediction model, an analysis module, and a dynamic decision-making module.

[0149] like Figure 2 As shown, the risk prediction model includes a directed graph construction module, a spatiotemporal feature encoder, a multimodal feature fusion module, and a risk prediction output layer; the spatiotemporal feature encoder includes an LSTM network and a graph attention network.

[0150] S4: The risk prediction model constructs a directed graph through a unified feature matrix. The directed graph extracts deep temporal features and deep spatial features through a spatiotemporal feature encoder. The deep temporal features and spatial features are then fused, and risk prediction is performed through the fused features.

[0151] S41: The directed graph construction module constructs a directed graph based on the unified feature matrix. The input formula is:

[0152] ;

[0153] Where G is a directed graph and V is the set of production process nodes. For the first There are several production process nodes, where E represents the dependencies between production processes. W is the adjacency weight matrix. W, the first Line number The column is the first The generation process node for the first The influence strength of the generation process node, the first Each generation process node corresponds to a unified feature matrix. OK.

[0154] S42: Input the directed graph into an LSTM network with a multi-head attention mechanism, learn the temporal features of each process node, and output the deep temporal features to the multimodal feature fusion module;

[0155] The LSTM network includes a forget gate, an input gate, a cell state module, an output gate, and a hidden state output layer; the cell state module includes a candidate cell module and a cell update module.

[0156] S421: The forget gate concatenates the hidden state at time step t-1 with the input data at time step t, multiplies the vector by the weight matrix, adds the bias vector, and then maps the data through the Sigmoid activation function before outputting it to the cell state module. The formula is:

[0157] ;

[0158] in, This is the output data of the forget gate at time step t. , Here is the weight matrix for the forget gate. This represents the hidden state at time step t-1. Let be the bias vector of the forget gate. It is the Sigmoid activation function. For the input data of the forget gate at time step t, further, That is, the comprehensive feature vector of a single production process node at time step t.

[0159] S422: The input gate concatenates the hidden state at time step t-1 with the input data at time step t, multiplies the vector by the weight matrix, adds the bias vector, and then maps the data through the Sigmoid activation function before outputting it to the cell state module. The formula is:

[0160] ;

[0161] in, The output data of the input gate at time step t. , Here is the weight matrix of the input gate. The bias vector for the input gate;

[0162] S423: The candidate cell state module concatenates the hidden state at time step t-1 with the input data at time step t, multiplies it by the weight matrix, adds the bias vector, normalizes the data using the hyperbolic tangent activation function, and then outputs it to the cell state update module. The formula is:

[0163] ;

[0164] in, Let t represent the candidate cell state at time step t. , This is the weight matrix for the candidate cell state module. This is the bias vector for the candidate cell state module. It is the hyperbolic tangent activation function.

[0165] S424: The cell state update module performs element-wise product of the output data of the forget gate at time step t and the updated cell state at time step (t-1), and performs element-wise product of the output data of the input gate at time step t and the candidate cell state at time step t. The element-wise products are then summed and output to the hidden state output layer. The formula is:

[0166] ;

[0167] in, This represents the cell state after the update at time step t. , This represents the cell state after the update at time step t-1.

[0168] S425: The output gate concatenates the hidden state at time step t-1 with the input data at time step t, multiplies the concatenated vector by the weight matrix, adds the bias vector, and then maps the data through the Sigmoid activation function before outputting it to the hidden state output layer. The formula is as follows:

[0169] ;

[0170] in, To output the gate's output data at time step t, , This is the weight matrix of the output gate. This is the bias vector for the output gate.

[0171] S426: The hidden state output layer normalizes the cell state updated at time step t using the hyperbolic tangent activation function. The normalized state is then element-wise multiplied with the output data from the output gate at time step t before being output to the multimodal feature fusion module. The formula is:

[0172] ;

[0173] in, The output depth temporal feature of the hidden state output layer at time step t is given.

[0174] It is important to note that , , .

[0175] S43: Input the directed graph into the Graph Attention Network (GAT), learn the spatial features of each process node, and output the deep spatial features to the multimodal feature fusion module;

[0176] The graph attention network includes an attention coefficient module, a normalization module, and a feature aggregation module.

[0177] S431: The attention coefficient module multiplies the weight matrix by the spatial features and concatenates them. The concatenated matrix is ​​then processed through the attention vector, activated using the LeakyReLU activation function, and output to the attention weight module. The formula is as follows:

[0178] ;

[0179] in, This is the attention coefficient, which represents the importance of spatial feature b to spatial feature c. LeakyReLU is the LeakyReLU activation function, and a is the attention vector. T is the transpose. This is the attention weight matrix. , For the b-th spatial feature, For the c-th spatial feature, , To transform the feature dimensions.

[0180] It is important to note that the comprehensive feature vectors of individual production process nodes in the directed graph are input into the LSTM network and the graph attention network respectively to learn deep temporal and spatial features. That is, spatial features pass through The new feature dimension after linear transformation.

[0181] S432: The normalization module normalizes the attention vector and outputs it to the feature aggregation module. The formula is as follows:

[0182] ;

[0183] in, For the normalized attention vector, , Let b be the set of neighboring nodes of spatial feature b. That is, the importance of spatial feature b to spatial feature z.

[0184] S433: Feature aggregation is performed using normalized attention vectors, and the result is output to the multimodal feature fusion module. The formula is as follows:

[0185] ;

[0186] in, For the updated b-th depth space feature, This is the ELU activation function.

[0187] S44: The multimodal feature fusion module uses a gated fusion mechanism to fuse deep spatial features and deep temporal features, outputting them to the risk prediction output layer. The formula is as follows:

[0188] ;

[0189] in, For gating signals, , For gating weights, It is the Sigmoid activation function. For deep temporal features, For depth space features, This is the gated bias vector. As a feature of fusion, , .

[0190] This can be understood as a gating signal. The value is not fixed between 0 and 1, but is determined by the current depth temporal features. and depth spatial features Together, the multimodal feature fusion module dynamically calculates a specific feature for each set of input data. value:

[0191] when When the value is close to 1, the fused features are more biased towards deep temporal features;

[0192] when When the value is close to 0, the fused features are more biased towards deep spatial features;

[0193] Therefore, the risk prediction model can adaptively determine the proportion of temporal and spatial information in the final feature representation based on the specific circumstances of the current data, rather than simply splicing them together or adding them in a fixed proportion.

[0194] S45: The risk prediction output layer calculates the current risk prediction and future risk prediction, and outputs them to the analysis module.

[0195] The risk prediction output layer includes a current risk prediction module and a future risk prediction module;

[0196] S451: The current risk prediction module will fuse features, multiply them by a weight matrix, add a bias vector, and then map the data using a Sigmoid activation function before outputting it to the future risk prediction and analysis modules. The formula is as follows:

[0197] ;

[0198] in, For current risk forecasting, This is the current weight matrix for the risk prediction module. This is the bias vector for the risk prediction module. .

[0199] S452: The future risk prediction module concatenates the historical fusion feature set and the current predicted risk, multiplies it by the prediction weight matrix, adds the bias vector, and outputs the future risk prediction to the dynamic decision-making module. The formula is as follows:

[0200] ;

[0201] in, For the future Risk prediction for each time step (e.g., the next 5 batches). for A set of historical fusion features at each time step , For the prediction weight matrix, , This is the bias vector for the future risk prediction module.

[0202] It is important to note that obtaining The historical fusion feature set of each time step is designed to enable the future risk prediction module to capture the long-term temporal dependence of production line risks, filter random fluctuations, and improve the accuracy and stability of future predictions. It is the core data support for the time series prediction task.

[0203] S5: The analysis module obtains process nodes through risk prediction results and inter-process dependencies, filters target features of process nodes through the SHAP method, and locates abnormal areas in target features through gradient mapping.

[0204] The analysis module includes a process impact module, a feature contribution module, and a root cause tracing module.

[0205] S51: The process impact module calculates the importance of process impact using a normalized attention vector and the current predicted risk, identifies the main risk sources, and outputs this information to the feature contribution module. The formula is as follows:

[0206] ;

[0207] in, For process nodes The overall importance, Process node The current predicted risks.

[0208] This can be understood as the process impact module indicating which process node is the main source of risk by assessing the importance of the process impact.

[0209] S52: The feature contribution module calculates the feature contribution based on the main risk sources using the SHAP method, generates a parameter contribution ranking list, obtains the target features, and outputs them to the root cause tracing module. The formula is:

[0210] ;

[0211] in, Target features in process nodes The SHAP value (i.e., feature contribution, which quantifies the degree of influence of a feature on risk). For all feature sets, This is a feature subset where process nodes are the main sources of risk. For inclusion and eigenvectors, For input Current forecast risks, Input is Current forecast risks, For only containing The eigenvectors of , where ! is the factorial.

[0212] This can be understood as knowing which process node is the main source of risk, assessing the contribution of features such as welding temperature to the prediction results, generating a ranking of parameter contributions, and the top-ranked feature indicates that, for example, the contribution of welding temperature variance is large, i.e., welding temperature variance is the target feature.

[0213] S53: The root cause tracing module, based on gradient-weighted class activation mapping, locates the data / image region that triggers the anomaly based on the target features, and outputs it to the dynamic decision module. The formula is as follows:

[0214] ;

[0215] in, Here, the contribution mapping matrix is ​​used, and ReLU is the ReLU activation function. To output the gradient with respect to the attention weights, This is the attention weight matrix for layer k2.

[0216] This can be understood as knowing which feature is the target feature, and using gradient mapping of the target feature to locate the specific abnormal data / image region, that is, locating the specific location of the risk in the target feature.

[0217] S6: The dynamic decision-making module dynamically sets early warning thresholds based on current risk prediction, future risk prediction, and contribution mapping matrix. It then issues early warnings for current risks based on these thresholds and generates actions that minimize risk.

[0218] S61: Dynamically adjust the current risk warning threshold based on historical risks, using the following formula:

[0219] ;

[0220] in, The warning threshold is set at time step t. The historical risk probability mean at time step t-1. This is the risk tolerance coefficient. The standard deviation of the historical risk probability at time step t-1.

[0221] It is important to note that It can be adjusted according to the production line process.

[0222] S62: Based on the current risk forecast, the corresponding warning level is triggered:

[0223] like This triggers a Level 1 warning, and continuous monitoring is initiated.

[0224] like This triggers a Level 2 warning; an immediate inspection is required.

[0225] like The system triggered a Level 3 warning and immediately shut down for maintenance.

[0226] S63: Based on the warning level, the execution decision action is based on the principle of minimizing risk, and the formula is:

[0227] ;

[0228] Where argmin is the argmin function. Let A be the set of available actions, representing the optimal decision action. To perform the action The probability of subsequent risks For action Execution costs The process node belongs to the set of production process nodes.

[0229] The risk identification and prediction decision model is trained iteratively using a multi-task loss function, and its parameters are updated using an online learning mechanism.

[0230] Specifically, the risk identification and prediction decision model is trained and optimized using a multi-task loss function, as shown in the formula:

[0231] ;

[0232] in, For multi-task loss function, The current risk prediction loss function is... For future risk prediction loss function, To analyze the loss function, For dynamic decision loss function, For training parameters, The number of training samples. , Let i be the current true risk vector of training sample i. Let i be the current risk prediction vector for training sample i. Let i be the future predicted risk vector for training sample i. Let i be the future true risk vector of the training sample i. For the input data of training sample i, The feature contribution vector of training sample i. The contribution vector of the target features for training sample i. The decision action corresponding to the original input of training sample i. Add the decision action corresponding to the perturbation input to the training sample i. The input features are small perturbation values ​​(hyperparameters). It is the square of the L2 norm of the vector.

[0233] The online learning mechanism updates the risk identification and prediction decision model through a sliding window, using the following formula:

[0234] ;

[0235] in, For the updated risk identification and prediction decision model parameters, These are the parameters for the current risk identification, prediction, and decision-making model. For learning rate, The gradient of the loss function. This is the most recent full-link data set from time step t−w to time step t.

[0236] In this embodiment, a three-stage training model for risk identification and prediction decision-making is used, namely:

[0237] Pre-training phase: Using historical data, learning rate 0.001, batch size 32, and multi-task loss function. To optimize the objective, the model is trained using a large-scale historical dataset, with α, β, and γ set as initial values;

[0238] Fine-tuning phase: Using online data with a learning rate of 0.0001 and a batch size of 16, this phase switches to online real-time data stream for training based on the pre-trained model. To adapt to the dynamic characteristics of the production line, this phase focuses on optimizing the accuracy and stability of real-time risk prediction in the loss function. Specifically, the weights of the loss terms can be adjusted so that the model can quickly adapt to the specific working conditions of the current production line and remain robust to normal fluctuations.

[0239] Reinforcement learning phase: In this phase, decision feedback is incorporated, the learning rate is 0.00001, and training no longer directly minimizes... The goal is not to achieve this, but rather to convert the model's predicted output (by...) The constraints are mapped to production line actions, and feedback on the actual benefits brought by these actions is received. In this case, the indirect optimization is... The model learns through reinforcement learning how to make predictions and decisions that maximize long-term cumulative rewards, thereby ultimately achieving the fundamental goal of minimizing risk, by considering the comprehensive value of each defined task from a long-term decision-making perspective.

[0240] Example 2: A risk identification and prediction system for multimodal optical device production lines.

[0241] like Figure 3 As shown, a risk identification and prediction system for multimodal optical device production lines includes a data acquisition module, a preprocessing module, a feature extraction and fusion module, and a risk identification and prediction decision model. The risk identification and prediction decision model includes a risk identification and prediction model, an analysis module, and a dynamic decision module.

[0242] The data acquisition module is used to collect multimodal data in real time from each process of the optical device production line.

[0243] The preprocessing module is used to preprocess multimodal data;

[0244] The feature extraction and fusion module is used to extract features from the preprocessed multimodal data to form a temporal feature matrix, a spatial feature matrix, and an environmental feature matrix, and then fuse the three matrices to construct a unified feature matrix.

[0245] The risk prediction model is used to construct a directed graph through a unified feature matrix. The directed graph extracts deep temporal features and deep spatial features through a spatiotemporal feature encoder, and then fuses the deep temporal features and spatial features to perform risk prediction.

[0246] The analysis module is used to obtain process nodes through risk prediction results and inter-process dependencies, filter target features of process nodes through the SHAP method, and locate abnormal regions in target features through gradient mapping.

[0247] The dynamic decision-making module is used to dynamically set early warning thresholds based on risk prediction results and contribution mapping, issue current risk warnings based on the early warning thresholds, and generate actions to minimize risks.

[0248] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying and predicting risks in a multimodal optical device production line, characterized in that, Includes the following steps: Multimodal data is collected in real time from each process of the optical device production line, and the multimodal data is preprocessed. Feature extraction is performed on the preprocessed multimodal data to form a temporal feature matrix, a spatial feature matrix, and an environmental feature matrix. The three matrices are then fused to construct a unified feature matrix. The risk identification and prediction decision model constructs a directed graph through a unified feature matrix. The directed graph extracts deep temporal features and deep spatial features through a spatiotemporal feature encoder. The deep temporal features and spatial features are then fused, and risk prediction is performed through the fused features. The risk identification, prediction, and decision-making model obtains process nodes through risk prediction results and inter-process dependencies, filters target features of process nodes using the SHAP method, and locates abnormal regions in target features through gradient mapping. The risk identification, prediction, and decision-making model dynamically sets early warning thresholds based on risk prediction results and contribution mapping, issues current risk warnings based on these thresholds, and generates actions to minimize risk.

2. The method for identifying and predicting risks in multimodal optical device production lines according to claim 1, characterized in that, The collected multimodal data includes equipment operating parameters, sensor monitoring data, visual inspection data, and environmental parameters; The equipment operating parameters include welding, bonding, and mounting processes; The sensor monitoring data includes optical device vibration parameters, optical device temperature parameters, optical device pressure deformation parameters, and optical device displacement parameters. The visual inspection data includes images of the optical device surface; The environmental parameters include the cleanliness of the production environment, the humidity of the production environment, the temperature of the production environment, and the electromagnetic interference of the production environment.

3. The method for identifying and predicting risks in multimodal optical device production lines according to claim 1, characterized in that, The process of fusing the three matrices to construct a unified feature matrix includes the following sub-steps: The preprocessed equipment operating parameters and sensor monitoring data are integrated into a time series feature set, and time series features are extracted to form a time series feature matrix. The preprocessed visual detection data is integrated into an image feature set, spatial features are extracted, and a spatial feature matrix is ​​formed. The preprocessed environmental parameters are integrated into an environmental feature set, and statistical features are extracted to form an environmental feature matrix. A unified feature matrix is ​​constructed by integrating the temporal feature matrix, spatial feature matrix, and environmental feature matrix.

4. The method for identifying and predicting risks in multimodal optical device production lines according to claim 1, characterized in that, The risk prediction through feature fusion includes the following sub-steps: The directed graph construction module constructs a directed graph based on a unified feature matrix; The directed graph is input into an LSTM network with a multi-head attention mechanism to learn the temporal features of each process node, and the deep temporal features are output to the multimodal feature fusion module. The directed graph is input into the graph attention network to learn the spatial features of each process node, and the deep spatial features are output to the multimodal feature fusion module. The multimodal feature fusion module uses a gated fusion mechanism to fuse deep spatial features and deep temporal features, and outputs them to the risk prediction output layer. The risk prediction output layer calculates the current risk prediction and the future risk prediction, and outputs them to the analysis module.

5. The method for identifying and predicting risks in multimodal optical device production lines according to claim 4, characterized in that, The learning of the spatial features of each process node includes the following sub-steps: The attention coefficient module multiplies the weight matrix with the spatial features respectively and concatenates them. The concatenated matrix is ​​then processed by the attention vector, activated by the LeakyReLU activation function, and output to the attention weight module. The normalization module normalizes the attention vector and outputs it to the feature aggregation module; Feature aggregation is performed using normalized attention vectors, and the output is sent to the multimodal feature fusion module.

6. The method for identifying and predicting risks in multimodal optical device production lines according to claim 4, characterized in that, The calculation of current risk forecast and future risk forecast includes the following sub-steps: The current risk prediction module will fuse features multiplied by a weight matrix, add a bias vector, and then map the data through a Sigmoid activation function before outputting it to the future risk prediction and analysis modules. The future risk prediction module concatenates the historical feature set and the current predicted risk, multiplies it by the prediction weight matrix, and adds a bias vector before outputting the future risk prediction to the dynamic decision-making module.

7. The method for identifying and predicting risks in multimodal optical device production lines according to claim 1, characterized in that, The step of locating abnormal regions in target features through gradient mapping includes the following sub-steps: The process impact module calculates the importance of process impact using a normalized attention vector and the current predicted risk, obtains the main risk sources, and outputs them to the feature contribution module. The feature contribution module calculates the feature contribution based on the main risk sources using the SHAP method, generates a parameter contribution ranking list, obtains the target features, and outputs them to the root cause tracing module. The root cause tracing module is based on gradient-weighted class activation mapping. It locates the data / image region that causes the anomaly based on the target features and outputs the result to the dynamic decision module.

8. The method for identifying and predicting risks in multimodal optical device production lines according to claim 1, characterized in that, The step of dynamically setting the early warning threshold based on the risk prediction results and contribution mapping, and issuing a current risk warning based on the early warning threshold, includes the following sub-steps: The current risk warning threshold is dynamically adjusted based on historical risks, using the following formula: ； in, The warning threshold is set at time step t. The historical risk probability mean at time step t-1. This is the risk tolerance coefficient. The standard deviation of the historical risk probability at time step t-1; Based on the current risk forecast, the corresponding warning level is triggered: like This triggers a Level 1 warning, and continuous monitoring is initiated. like This triggers a Level 2 warning; an immediate inspection is required. like The system has triggered a Level 3 warning and will immediately shut down for maintenance. in, This is a current risk forecast.

9. The method for identifying and predicting risks in multimodal optical device production lines according to claim 1, characterized in that, The risk identification and prediction decision model is iteratively trained using a multi-task loss function, as shown in the formula: ； in, For multi-task loss function, The current risk prediction loss function is... For future risk prediction loss function, To analyze the loss function, For dynamic decision loss function, For training parameters, The number of training samples. , Let i be the current true risk vector of the training sample i. Let i be the current risk prediction vector for training sample i. Let i be the future predicted risk vector for training sample i. Let i be the future true risk vector of the training sample i. For the input data of training sample i, The feature contribution vector of training sample i. The contribution vector of the target features for training sample i. The decision action corresponding to the original input of training sample i. Add the decision action corresponding to the perturbation input to the training sample i. The input features are small perturbation values ​​(hyperparameters). It is the square of the L2 norm of the vector.

10. A risk identification and prediction system for multimodal optical device production lines, used to implement the risk identification and prediction method for multimodal optical device production lines as described in any one of claims 1-9, characterized in that, It includes a data acquisition module, a preprocessing module, a feature extraction and fusion module, and a risk identification, prediction, and decision-making model; the risk identification, prediction, and decision-making model includes a risk identification and prediction model, an analysis module, and a dynamic decision-making module. The data acquisition module is used to collect multimodal data in real time from each process of the optical device production line. The preprocessing module is used to preprocess multimodal data; The feature extraction and fusion module is used to extract features from the preprocessed multimodal data to form a temporal feature matrix, a spatial feature matrix, and an environmental feature matrix, and then fuse the three matrices to construct a unified feature matrix. The risk prediction model is used to construct a directed graph through a unified feature matrix. The directed graph extracts deep temporal features and deep spatial features through a spatiotemporal feature encoder, and then fuses the deep temporal features and spatial features to perform risk prediction. The analysis module is used to obtain process nodes through risk prediction results and inter-process dependencies, filter target features of process nodes through the SHAP method, and locate abnormal regions in target features through gradient mapping. The dynamic decision-making module is used to dynamically set early warning thresholds based on risk prediction results and contribution mapping, issue current risk warnings based on the early warning thresholds, and generate actions to minimize risks.

Images