Product quality prediction method and system based on time series analysis and visual features

Abstract

The application discloses a product quality prediction method and system based on time sequence analysis and visual features, and belongs to the technical field of machine vision, and comprises the following steps: S1, multi-modal sample collection; S2, visual feature time sequence evolution extraction; S3, sensor data space-time feature coding; S4, fusion prediction and early warning based on a dynamic state space; and S5, prediction and early warning implementation. Through the above manner, the application significantly enhances the forward-looking nature of quality prediction and the timeliness of early warning, and provides an interpretable early warning traceability capability. By using a trained dynamic state space model, the product quality of multiple production cycles in the future can be multi-step forward-looking predicted based on the current system state estimation, early warning is given when the current product appearance and quality have not yet obviously degraded, valuable time windows are obtained for process adjustment and equipment maintenance intervention measures, and batch quality accidents are avoided to the greatest extent.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and more specifically to a product quality prediction method and system based on time series analysis and visual features. Background Technology

[0002] In the field of modern intelligent manufacturing, ensuring the smooth operation of production lines and the quality of final products is crucial. With increasing industrial automation, faster production pace, and more complex processes, even minor fluctuations in the production line can be amplified during continuous production, leading to batch-wide product quality defects. Currently, traditional production line monitoring mainly relies on time-series data reflecting the physical state of equipment collected by industrial sensors, as well as discrete product quality sampling or automated inspections at key process nodes or the end of the production line. Simultaneously, machine vision technology has also been introduced into product appearance inspection, analyzing individual product images to determine their compliance.

[0003] Existing technologies are mainly based on quality prediction and early warning methods using a single data source. These include using sensor data in conjunction with control chart theory or multivariate statistical methods for anomaly monitoring after dimensionality reduction, and using industrial cameras to acquire single-frame images and classifying and confirming quality labels based on image processing or deep learning classification networks.

[0004] However, existing quality prediction and early warning methods lack the ability to model spatiotemporal dynamic correlations. They typically separate the time-continuous sensor sequences from single-frame product images, failing to effectively utilize the dynamic laws of the process that change continuously with the production cycle contained in the multi-frame image sequences. Secondly, the data fusion methods are too superficial and unrelated. The simple feature splicing assumes that the modal data are independent of each other, ignoring the nonlinear intrinsic temporal causal dependence and cross-modal interaction information between sensor data changes and product appearance defects. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a product quality prediction method and system based on time series analysis and visual features.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A product quality prediction method based on time series analysis and visual features includes the following steps:

[0008] S1. Collect sensor data vectors, continuous multi-frame image sequences, and actual quality indicators from the product unit;

[0009] S2. Based on a continuous multi-frame image sequence, extract single-frame spatial visual features and obtain a single-frame spatial visual feature sequence. Input the single-frame spatial visual feature sequence into the time evolution modeler and output the real temporal visual features of the product.

[0010] S3. Map the sensor data vector to instantaneous sensor coding features, input the sensor coding features into the sensor timing modeler, calculate and output the sensor timing driving features;

[0011] S4. Calculate the hidden state of the production line through sensor time-driven features, and obtain the optimal model parameter set by combining consistency constraints. This includes the following steps:

[0012] S401. Establish state transition equations based on sensor timing-driven characteristics and calculate the hidden state of the production line;

[0013] S402. Input the hidden state of the production line into the visual observation equation to obtain the reconstructed visual temporal features. Then, by constraining the consistency between the reconstructed visual temporal features and the real temporal visual features, the conditional probability of visual observation is obtained.

[0014] S403. Input the hidden state of the production line into the quality prediction head function to obtain the predicted quality index, and establish the quality observation conditional probability by constraining the consistency between the predicted quality index and the actual quality index.

[0015] S404. Calculate the optimal set of model parameters based on the conditional probability of visual observation and the conditional probability of quality observation.

[0016] S5. Obtain feature prediction values ​​through the time series prediction model, combine them with the optimal model parameter set to obtain quality prediction values, and obtain early warning signals, early warning steps, and future multi-step quality prediction sequences as the final output based on the quality prediction values.

[0017] Furthermore, in step S2, a spatial feature encoder is used to extract single-frame spatial visual features;

[0018] The spatial visual features of each frame are calculated sequentially for all image frames to obtain the spatial visual feature sequence of each frame.

[0019] Furthermore, in step S3, the sensor data vector is mapped to low-dimensional latent features by the sensor encoder to obtain instantaneous sensor encoded features.

[0020] Furthermore, in step S401, the formula for calculating the hidden state of the production line is:

[0021]

[0022] in, This represents the hidden state of the production line during the k-th production cycle. This represents the hidden state of the production line during the previous production cycle. For the sensor timing driving characteristics corresponding to the k-th product, It is a nonlinear state transition function. These are the learnable parameters of the state transition function. This refers to the uncertainty noise in the state evolution process.

[0023] Furthermore, in step S402, the formula for calculating the reconstructed visual temporal features is as follows:

[0024]

[0025] in, For the reconstructed visual temporal features corresponding to the k-th product, For visual observation decoder, For the learnable parameters of the visual observation decoder, This is visual observation noise;

[0026] The conditional probability expression for visual observation is established as follows:

[0027]

[0028] in, For the reconstructed visual temporal features corresponding to the k-th product, For the k-th product, the actual temporal visual features are... This represents the conditional probability of visual observation under the condition of reconstructing visual temporal features based on real temporal visual features.

[0029] Furthermore, in step S403, the formula for calculating the conditional probability of quality observation is:

[0030]

[0031] in, Let be the predicted quality index for the k-th product. For quality prediction head function, These are the learnable parameters for the quality prediction head. For quality observation noise;

[0032] The conditional probability expression for quality observation is:

[0033]

[0034] in, Let be the predicted quality index for the k-th product. For the true quality index of the k-th product, This represents the conditional probability of quality observation under the conditions of the predicted quality indicators, representing the true quality indicators.

[0035] Furthermore, in S404, the formula for calculating the optimal model parameter set is:

[0036]

[0037]

[0038] in, This is the set of optimal model parameters obtained after training. For the set of all parameters to be learned, This represents the conditional probability of visual observation based on real temporal visual features under the condition of reconstructing visual temporal features. This represents the conditional probability of quality observation under the condition of predicting quality indicators, representing the true quality indicators. These are the optimal state transition parameters. For optimal visual observation decoding parameters, These are the optimal quality prediction parameters.

[0039] Furthermore, in step S5, based on the hidden state of the production line, the production line state prediction value is obtained by combining the optimal state transition parameters and feature prediction values.

[0040] Furthermore, in step S5, based on the production line status prediction value and combined with the quality prediction head and the optimal quality prediction parameters, the quality prediction value of the future product is calculated.

[0041] A product quality prediction system based on time series analysis and visual features, the system comprising:

[0042] Sample acquisition module: Acquires sensor data vectors, continuous multi-frame image sequences, and real quality indicators from the product unit, and constructs multimodal samples;

[0043] Visual feature evolution extraction module: Based on a continuous multi-frame image sequence, a spatial feature encoder is used to extract single-frame spatial visual features to obtain a single-frame spatial visual feature sequence. The single-frame spatial visual feature sequence is then input into a temporal evolution modeler to output the real temporal visual features of the product.

[0044] Spatiotemporal feature encoding module: Through the sensor encoder, the sensor data vector is mapped into instantaneous sensor encoded features, and the sensor encoded features are input into the sensor time series modeler to calculate and output the sensor time series driving features;

[0045] The fusion prediction and early warning module establishes a state transition equation based on the sensor time-series driven features, calculates the hidden state of the production line, inputs the hidden state of the production line into the visual observation equation to obtain the reconstructed visual time-series features, and establishes the visual observation conditional probability based on the consistency constraints between the reconstructed visual time-series features and the real time-series visual features. At the same time, the hidden state of the production line is input into the quality prediction head function to obtain the predicted quality index, and establishes the quality observation conditional probability based on the consistency constraints between the predicted quality index and the real quality index. Based on the visual observation conditional probability and the quality observation conditional probability, the optimal model parameter set is calculated.

[0046] Prediction and early warning implementation module: It obtains feature prediction values ​​through time series prediction model, and combines them with the optimal model parameter set to obtain production line status prediction values ​​and quality prediction values. Based on the quality prediction values, it obtains early warning signals, early warning steps, and future multi-step quality prediction sequences as the final output.

[0047] Compared with the prior art, the beneficial effects of this invention are as follows:

[0048] 1. This invention significantly enhances the foresight and timeliness of quality prediction and provides interpretable early warning and source tracing capabilities. Utilizing a trained dynamic state-space model, it can make multi-step forward-looking predictions of product quality for multiple production cycles based on the current system state estimate. It issues early warnings before the current product appearance and quality have significantly deteriorated, providing valuable time windows for intervention measures such as process adjustments and equipment maintenance, and minimizing the occurrence of batch quality accidents. At the same time, when the system triggers an early warning, it can quantify the contribution of different input variables to state deterioration by analyzing the sensitivity of the system state to input variables or calculating gradients, accurately locating key sensor parameters that lead to state deterioration, and providing operators with clear and operable root cause analysis, effectively overcoming the shortcomings of traditional early warnings that are lagging and unable to trace the source.

[0049] 2. This invention constructs a state-space-based intelligent agent model, using device sensor data as the driving force for state transitions and the temporal evolution of visual features from multiple consecutive frames of images as the external manifestation of the state. Under a unified framework, joint learning and recursive estimation are performed to accurately capture the spatiotemporal causal relationship between device state fluctuations, changes in product appearance features, and final quality. This deep collaborative mechanism of multimodal data enables sensors to provide dynamic priors and visual features to provide feedback calibration. Through tight coupling of common latent states, even when noise or missing data occurs in one modality, a relatively stable state estimation can still be maintained by relying on information from another modality. This effectively solves the problems of shallow fusion and lack of dynamic causal modeling in existing technologies. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0051] Figure 1 This is a system flow diagram of the present invention;

[0052] Figure 2 This is a flowchart illustrating the integrated prediction and early warning module of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0054] Example 1: As Figure 1-2 As shown, the product quality prediction method based on time series analysis and visual features includes the following steps:

[0055] S1: Multimodal Sample Acquisition

[0056] The system collects sensor data vectors from the product unit (including process parameters reflecting the internal physical state of the production line, such as temperature, pressure, vibration, and current), continuous multi-frame image sequences (to reflect changes in product appearance characteristics during the production process), and actual quality indicators, forming a multimodal sample, specifically:

[0057] Assume the production line operates at a constant cycle time, with each cycle time corresponding to one product unit. For the first For each product, the sensor data vector is:

[0058]

[0059] in, This is the sensor data vector corresponding to the k-th product. Let i be the sensor variable corresponding to the k-th product. For the number of sensor variables, Transpose of a vector;

[0060] Simultaneously acquire a continuous multi-frame image sequence of the k-th product:

[0061]

[0062] in, For the k-th product, a sequence of consecutive multi-frame images. For the j-th frame image of the k-th product, The number of image frames captured for the same product;

[0063] By using timestamp hard synchronization, sensor data and image sequences are aligned at the product level to form a multimodal sample for the k-th product. The multimodal sample expression is as follows:

[0064]

[0065] in, For the multimodal sample of the k-th product, This is the sensor data vector corresponding to the k-th product. For the k-th product, a sequence of consecutive multi-frame images. This represents the true quality index of the k-th product;

[0066] By synchronizing product-level time, the internal process status of the production line is strictly bound to the external visual performance of the product, avoiding time misalignment between sensor data and image data, and providing an accurate, complete and traceable data foundation for subsequent dynamic coupling modeling;

[0067] S2: Temporal Evolution Extraction of Visual Features

[0068] Based on a continuous multi-frame image sequence, a spatial feature encoder is used to extract single-frame spatial visual features, resulting in a single-frame spatial visual feature sequence. This single-frame spatial visual feature sequence is then input into a temporal evolution modeler to output the true temporal visual features of the product, specifically:

[0069] Based on continuous multi-frame image sequences in multimodal samples For the first The first product Frame Image A spatial feature encoder is used to extract single-frame spatial visual features. The expression for a single-frame spatial visual feature is as follows:

[0070]

[0071] in, For each product The Single-frame spatial visual features of a frame image For spatial feature encoders, For the j-th frame image of the k-th product, These are the learnable parameters for a spatial feature encoder (such as ResNet-50).

[0072] Single-frame spatial visual features are used to express local or overall appearance information such as texture, edges, shape, scratches, shrinkage, and burrs in each frame of an image;

[0073] After calculating all image frames of the same product sequentially, a single-frame spatial visual feature sequence is obtained;

[0074]

[0075] in, For the single-frame spatial visual feature sequence of the k-th product;

[0076] Inputting a single-frame spatial visual feature sequence into a temporal evolution modeler (typically a gated recurrent unit or a Transformer encoder) outputs the product's true temporal visual features:

[0077]

[0078]

[0079] in, For the k-th product, the actual temporal visual features are... To handle the visual temporal hidden state at frame j, For time evolution modelers, Learnable parameters for the time evolution modeler For each product The Single-frame spatial visual features of a frame image The number of image frames captured for the same product;

[0080] Real-time visual feature representation not only includes static appearance information in a single frame image, but also dynamic process information such as defects from non-existence to presence, from small to large, texture gradually becoming abnormal, scratches gradually deepening, or size features continuously shifting.

[0081] By upgrading from "single-frame defect judgment" to "multi-frame visual evolution analysis", this step can capture the dynamic changes in the product defect formation process, enabling the system not only to see whether the defect exists, but also to identify how the defect gradually arises and expands, thereby enhancing the ability to perceive the external manifestations of process fluctuations.

[0082] S3: Spatiotemporal Feature Encoding of Sensor Data

[0083] The sensor encoder maps sensor data vectors from multimodal samples to instantaneous sensor encoded features, and inputs these features into a sensor time-series modeler to calculate and output the sensor time-series driving features. Specifically:

[0084] Based on sensor data vectors in multimodal samples Since sensor data can be highly dimensional and there are complex nonlinear interactions between variables, the sensor data vector is first mapped to low-dimensional latent features using a sensor encoder to obtain instantaneous sensor encoded features.

[0085]

[0086] in, The instantaneous sensor coding features corresponding to the k-th product, For sensor encoders, For sensor data vectors, These are the learnable parameters of the sensor encoder;

[0087] Because production line status has temporal inertia—meaning the current product's process status is typically influenced by the production status of several previous products—the sequence corresponding to the instantaneous sensor encoding features is further input into the sensor timing modeler to calculate the sensor timing driving features.

[0088]

[0089] in, For the sensor timing driving characteristics corresponding to the k-th product, For sensor timing modelers, The instantaneous sensor coding features corresponding to the k-th product, This is the sequence corresponding to the instantaneous sensor encoded features;

[0090] If a separate sensor timing modeler is not set up in the simplified implementation, then At this point, the instantaneous sensor coding features are directly used as the driving input for subsequent state transitions;

[0091] By performing deep encoding and time-dependent modeling on sensor data, high-dimensional, noisy, and coupled equipment data is compressed into stable production line physical state driving features, enabling subsequent models to more accurately understand the impact of production line fluctuations such as temperature drift, pressure pulsation, and vibration anomalies on the quality formation process.

[0092] S4: Fusion prediction and early warning based on dynamic state space, specifically including the following steps:

[0093] S401. Establish state transition equations based on sensor timing-driven characteristics and calculate the hidden state of the production line;

[0094] The formula for calculating the hidden state of the production line is:

[0095]

[0096] in, This represents the hidden state of the production line during the k-th production cycle. This represents the hidden state of the production line during the previous production cycle. For the sensor timing driving characteristics corresponding to the k-th product, It is a nonlinear state transition function. These are the learnable parameters of the state transition function. This refers to the uncertainty noise in the state evolution process;

[0097] S402. Input the hidden state of the production line into the visual observation equation to obtain the reconstructed visual temporal features. Then, by constraining the consistency between the reconstructed visual temporal features and the real temporal visual features, the conditional probability of visual observation is obtained.

[0098] Establish a visual observation equation, reconstruct visual temporal features using the current hidden state of the production line, and obtain the reconstructed visual temporal features.

[0099]

[0100] in, For the reconstructed visual temporal features corresponding to the k-th product, For visual observation decoder, For the learnable parameters of the visual observation decoder, This is visual observation noise;

[0101] Furthermore, consistency constraints are applied between the reconstructed visual temporal features and the true temporal visual features to establish visual observation conditional probabilities:

[0102]

[0103] in, For the reconstructed visual temporal features corresponding to the k-th product, For the k-th product, the actual temporal visual features are... It represents the visual observation conditional probability of real temporal visual features under the condition of reconstructed visual temporal features, and is used to measure the degree of consistency between reconstructed visual temporal features and real temporal visual features.

[0104] S403. Input the hidden state of the production line into the quality prediction header function to obtain the predicted quality index, and establish the conditional probability of quality observation by constraining the consistency between the predicted quality index and the actual quality index:

[0105]

[0106] in, Let be the predicted quality index for the k-th product. For quality prediction head function, These are the learnable parameters for the quality prediction head. For quality observation noise;

[0107] Furthermore, consistency constraints are imposed between predicted and actual quality indicators to establish conditional probabilities for quality observation:

[0108]

[0109] in, Let be the predicted quality index for the k-th product. For the true quality index of the k-th product, The conditional probability of quality observation under the condition of predicting quality indicators, representing the true quality indicators.

[0110] S404. Calculate the optimal set of model parameters based on the conditional probability of visual observation and the conditional probability of quality observation.

[0111] The formula for calculating the optimal model parameter set is:

[0112]

[0113]

[0114] in, This is the set of optimal model parameters obtained after training. For the set of all parameters to be learned, This represents the conditional probability of visual observation based on real temporal visual features under the condition of reconstructing visual temporal features. This represents the conditional probability of quality observation under the condition of predicting quality indicators, representing the true quality indicators. These are the optimal state transition parameters. For optimal visual observation decoding parameters, These are the optimal quality prediction parameters;

[0115] In practice, extended Kalman filtering, particle filtering, deep Kalman filtering, or neural network recursive estimation methods can be combined to perform online estimation and updating of the hidden state of the production line.

[0116] Through a unified dynamic state space model, this step achieves deep coupling between the "intrinsic causes" of sensor data and the "extrinsic results" of visual features, overcoming the problem of insufficient synergy in traditional feature stitching methods, and enabling the model to more accurately capture the spatiotemporal causal relationship between production line fluctuations and quality formation.

[0117] S5: Prediction and Early Warning Implementation

[0118] Feature prediction values ​​are obtained through a time series forecasting model and combined with the optimal model parameter set to obtain production line status prediction values ​​and quality prediction values. Based on the quality prediction values, early warning signals, early warning steps, and future multi-step quality prediction sequences are obtained as the final output, specifically:

[0119] Since the features of future sensors have not yet been actually generated, it is necessary to obtain the predicted values ​​of the time-series driven features of future sensors based on time series prediction models (e.g., ARIMA, GRU, Transformer, etc.). The sensor timing-driven characteristics of the step are denoted as:

[0120]

[0121] in, For the k-th product in the future relative to the current product The predicted feature value driven by the timing of the production cycle sensor;

[0122] Based on the hidden state of the production line, and combining the optimal state transition parameters with the feature prediction values, the predicted values ​​of the production line state are obtained:

[0123]

[0124] in, For the future Predicted production line status for each production cycle (when) When =1, directly take the current hidden state of the production line. This is the predicted value of the production line status from the previous step. For the future The predicted value of the sensor time-driven features in the step. It is a nonlinear state transition function. These are the optimal state transition parameters;

[0125] Based on the production line status prediction values, and combined with the quality prediction head and optimal quality prediction parameters, the predicted quality values ​​for future products are calculated:

[0126]

[0127] in, For the future Predicted quality values ​​for each product. For the future Predicted production line status for each production cycle. For quality prediction head function, These are the optimal quality prediction parameters;

[0128] The early warning mechanism compares future quality predictions with a preset quality threshold. If the quality indicator is the probability of defects, the early warning judgment formula is:

[0129]

[0130] If the quality index is a continuous quality score, and the normal quality range is [ , The formula for early warning judgment is:

[0131]

[0132] in, For early warning signals ( =1 triggers an alert. =0 means no warning is triggered). For the future Predicted quality values ​​for each product. For a preset defect probability threshold (e.g., when When it is the defect probability, (Can be set to 0.8).

[0133] When an alert is triggered, the advance warning time is defined as the number of future steps before the earliest anomaly is triggered:

[0134]

[0135] in, This refers to the number of advance steps from the current k-th production cycle to the predicted occurrence of an anomaly;

[0136] At the same time, it is also possible to analyze the status. Sensor characteristics gradient This allows us to pinpoint the key sensor variables that cause the condition to deteriorate, providing operators with interpretable root cause analysis.

[0137] The final output is the future H-step quality prediction sequence:

[0138]

[0139] in, For future H-step quality prediction sequences;

[0140] The final output is a multi-step quality prediction sequence for the future. Warning signals and advance warning steps It plays a core role in collaborative management and control;

[0141] Furthermore, the quality prediction sequence comprehensively quantifies and tracks the dynamic evolution trend of production line quality indicators, enabling operators to intuitively grasp future quality trends and changes.

[0142] The early warning signal compares the predicted trend with the safety boundary in real time, so that risks can be detected and intercepted in advance before appearance defects actually appear or production line abnormalities cause actual losses to finished products.

[0143] The early warning system accurately pinpoints the safe buffer time between the current node and the occurrence of the anomaly.

[0144] These three key data points are interconnected, successfully upgrading the traditional reactive quality inspection or immediate fault alarm to a proactive, pre-emptive defense, providing high-value and actionable decision support for proactive adjustment of process parameters, preventive maintenance of equipment, and adaptive optimization of the production line.

[0145] Example 2: Figures 1-2 As shown, the product quality prediction system based on time series analysis and visual features includes:

[0146] Sample acquisition module: Acquires sensor data vectors, continuous multi-frame image sequences, and real quality indicators from the product unit, and constructs multimodal samples;

[0147] Visual feature evolution extraction module: Based on a continuous multi-frame image sequence, a spatial feature encoder is used to extract single-frame spatial visual features to obtain a single-frame spatial visual feature sequence. The single-frame spatial visual feature sequence is then input into a temporal evolution modeler to output the real temporal visual features of the product.

[0148] Spatiotemporal feature encoding module: Through the sensor encoder, the sensor data vector is mapped into instantaneous sensor encoded features, and the sensor encoded features are input into the sensor time series modeler to calculate and output the sensor time series driving features;

[0149] The fusion prediction and early warning module establishes a state transition equation based on the sensor time-series driven features, calculates the hidden state of the production line, inputs the hidden state of the production line into the visual observation equation to obtain the reconstructed visual time-series features, and establishes the visual observation conditional probability based on the consistency constraints between the reconstructed visual time-series features and the real time-series visual features. At the same time, the hidden state of the production line is input into the quality prediction head function to obtain the predicted quality index, and establishes the quality observation conditional probability based on the consistency constraints between the predicted quality index and the real quality index. Based on the visual observation conditional probability and the quality observation conditional probability, the optimal model parameter set is calculated.

[0150] Prediction and early warning implementation module: It obtains feature prediction values ​​through time series prediction model, and combines them with the optimal model parameter set to obtain production line status prediction values ​​and quality prediction values. Based on the quality prediction values, it obtains early warning signals, early warning steps, and future multi-step quality prediction sequences as the final output.

[0151] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A product quality prediction method based on time series analysis and visual features, characterized in that, Includes the following steps: S1. Collect sensor data vectors, continuous multi-frame image sequences, and actual quality indicators from the product unit; S2. Based on a continuous multi-frame image sequence, extract single-frame spatial visual features and obtain a single-frame spatial visual feature sequence. Input the single-frame spatial visual feature sequence into the time evolution modeler and output the real temporal visual features of the product. S3. Map the sensor data vector to instantaneous sensor coding features, input the sensor coding features into the sensor timing modeler, calculate and output the sensor timing driving features; S4. Calculate the hidden state of the production line through sensor time-driven features, and obtain the optimal model parameter set by combining consistency constraints. This includes the following steps: S401. Establish state transition equations based on sensor time-driven characteristics and calculate hidden states of the production line. S402. Input the hidden state of the production line into the visual observation equation to obtain the reconstructed visual temporal features. Then, by constraining the consistency between the reconstructed visual temporal features and the real temporal visual features, the conditional probability of visual observation is obtained. S403. Input the hidden state of the production line into the quality prediction head function to obtain the predicted quality index, and establish the quality observation conditional probability by constraining the consistency between the predicted quality index and the actual quality index. S404. Calculate the optimal set of model parameters based on the conditional probability of visual observation and the conditional probability of quality observation. S5. Obtain feature prediction values ​​through the time series prediction model, combine them with the optimal model parameter set to obtain quality prediction values, and obtain early warning signals, early warning steps, and future multi-step quality prediction sequences as the final output based on the quality prediction values.

2. The product quality prediction method based on time series analysis and visual features according to claim 1, characterized in that, In step S2, a spatial feature encoder is used to extract single-frame spatial visual features; The spatial visual features of each frame are calculated sequentially for all image frames to obtain the spatial visual feature sequence of each frame.

3. The product quality prediction method based on time series analysis and visual features according to claim 1, characterized in that, In step S3, the sensor data vector is mapped to low-dimensional latent features by the sensor encoder to obtain instantaneous sensor encoded features.

4. The product quality prediction method based on time series analysis and visual features according to claim 1, characterized in that, In step S401, the formula for calculating the hidden state of the production line is: ； in, This represents the hidden state of the production line during the k-th production cycle. This represents the hidden state of the production line during the previous production cycle. For the sensor timing driving characteristics corresponding to the k-th product, It is a nonlinear state transition function. These are the learnable parameters of the state transition function. This refers to the uncertainty noise in the state evolution process.

5. The product quality prediction method based on time series analysis and visual features according to claim 4, characterized in that, In step S402, the formula for calculating the reconstructed visual temporal features is as follows: ； in, For the reconstructed visual temporal features corresponding to the k-th product, For visual observation decoder, For the learnable parameters of the visual observation decoder, This is visual observation noise; The conditional probability expression for visual observation is established as follows: ； in, For the reconstructed visual temporal features corresponding to the k-th product, For the k-th product, the actual temporal visual features are... This represents the conditional probability of visual observation under the condition of reconstructing visual temporal features based on real temporal visual features.

6. The product quality prediction method based on time series analysis and visual features according to claim 5, characterized in that, In step S403, the formula for calculating the conditional probability of quality observation is: ； in, Let be the predicted quality index for the k-th product. For quality prediction head function, These are the learnable parameters for the quality prediction head. For quality observation noise; The conditional probability expression for quality observation is: ； in, Let be the predicted quality index for the k-th product. For the true quality index of the k-th product, This represents the conditional probability of quality observation under the conditions of the predicted quality indicators, representing the true quality indicators.

7. The product quality prediction method based on time series analysis and visual features according to claim 6, characterized in that, In S404, the formula for calculating the optimal model parameter set is: ； ； in, This is the set of optimal model parameters obtained after training. For the set of all parameters to be learned, This represents the conditional probability of visual observation based on real temporal visual features under the condition of reconstructing visual temporal features. This represents the conditional probability of quality observation under the condition of predicting quality indicators, representing the true quality indicators. These are the optimal state transition parameters. For optimal visual observation decoding parameters, These are the optimal quality prediction parameters.

8. The product quality prediction method based on time series analysis and visual features according to claim 1, characterized in that, In step S5, based on the hidden state of the production line, the predicted state value of the production line is obtained by combining the optimal state transition parameters and the feature prediction value.

9. The product quality prediction method based on time series analysis and visual features according to claim 8, characterized in that, In step S5, based on the production line status prediction value and combined with the quality prediction head and the optimal quality prediction parameters, the quality prediction value of the future product is calculated.

10. A product quality prediction system based on time series analysis and visual features, characterized in that, The system is used in the product quality prediction method and system based on time series analysis and visual features as described in any one of claims 1-9, the system comprising: Sample acquisition module: Acquires sensor data vectors, continuous multi-frame image sequences, and real quality indicators from the product unit, and constructs multimodal samples; Visual feature evolution extraction module: Based on a continuous multi-frame image sequence, a spatial feature encoder is used to extract single-frame spatial visual features to obtain a single-frame spatial visual feature sequence. The single-frame spatial visual feature sequence is then input into a temporal evolution modeler to output the real temporal visual features of the product. Spatiotemporal feature encoding module: Through the sensor encoder, the sensor data vector is mapped into instantaneous sensor encoded features, and the sensor encoded features are input into the sensor time series modeler to calculate and output the sensor time series driving features; The fusion prediction and early warning module establishes a state transition equation based on the sensor time-series driven features, calculates the hidden state of the production line, inputs the hidden state of the production line into the visual observation equation to obtain the reconstructed visual time-series features, and establishes the visual observation conditional probability based on the consistency constraints between the reconstructed visual time-series features and the real time-series visual features. At the same time, the hidden state of the production line is input into the quality prediction head function to obtain the predicted quality index, and establishes the quality observation conditional probability based on the consistency constraints between the predicted quality index and the real quality index. Based on the visual observation conditional probability and the quality observation conditional probability, the optimal model parameter set is calculated. Prediction and early warning implementation module: It obtains feature prediction values ​​through time series prediction model, and combines them with the optimal model parameter set to obtain production line status prediction values ​​and quality prediction values. Based on the quality prediction values, it obtains early warning signals, early warning steps, and future multi-step quality prediction sequences as the final output.

Images