Visual system reliability early warning method, system and device and storage medium

Abstract

The invention relates to the technical field of computer vision and system reliability engineering, in particular to a visual system reliability early warning method, a visual system reliability early warning system, visual system reliability early warning equipment and a storage medium. The multi-modal time series data comprises input data features, output data features, environmental parameters and performance degradation indexes; carrying out fusion processing on the multi-modal time sequence data based on a multi-modal fusion model to generate a system state representation vector; inputting the system state representation vector into a time sequence probability model, updating parameters of the time sequence probability model through an online learning mechanism, and calculating a fault probability of the visual system in a preset future time window; dynamically calculating an early warning threshold according to the environmental parameters and the performance degradation index at the current moment; and when the predicted fault probability exceeds an early warning threshold, triggering an early warning signal. By adopting the scheme, the early warning accuracy and the system reliability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and system reliability engineering, and in particular to a visual system reliability early warning method, system, device and storage medium. BACKGROUND

[0002] The deep application of computer vision systems in key scenes such as industrial detection and automatic driving puts high requirements on the reliability of long-term operation. In actual operation, the visual system faces double challenges: on the one hand, external environmental factors such as light mutation, weather influence and shielding interference will directly affect the image quality and recognition effect; on the other hand, the internal performance of the system gradually decays over time, including hardware component aging, model parameter drift and other hardware and software collaborative degradation problems. The two types of factors are coupled and dynamically intertwined, resulting in significant uncertainty risk in system output, which seriously restricts its credibility in key tasks.

[0003] The current reliability guarantee technology has obvious limitations. First, the traditional method based on environmental correction usually only relies on limited environmental parameters to passively enhance the input image, and lacks the ability to cope with reliability problems caused by internal performance degradation. Second, although the offline evaluation method can detect the system state regularly, it needs to interrupt the system operation and cannot realize continuous online monitoring and real-time early warning, which has a monitoring blind area and is difficult to capture the precursors of sudden or gradual failures. In addition, the existing early warning mechanism relies on fixed thresholds or simple rules, lacks dynamic perception ability of system running situation, and cannot adaptively adjust the alarm threshold according to environmental changes and performance degradation, resulting in frequent false alarms and missed alarms, which seriously affects the actual availability of the early warning system.

[0004] In summary, the deficiencies of the existing technology in multi-modal data fusion, real-time state evaluation and dynamic early warning limit the reliability guarantee capability of the visual system in complex scenes. Therefore, it is urgent to provide a visual system reliability early warning method that can realize dynamic prediction of future failure probability of the visual system and adaptive adjustment of early warning threshold, and improve the early warning accuracy and system reliability. SUMMARY

[0005] The present application provides a visual system reliability early warning method, system, device and storage medium, which realizes dynamic prediction of future failure probability of the visual system and adaptive adjustment of early warning threshold, and improves the early warning accuracy and system reliability.

[0006] The present application provides a basic scheme one:

[0007] The visual system reliability early warning method comprises the following steps:

[0008] Real-time acquisition of multi-modal time series data of a vision system, the multi-modal time series data comprising input data features, output data features, environmental parameters and performance degradation indicators;

[0009] Fusion processing of the multi-modal time series data based on a multi-modal fusion model to generate a system state representation vector;

[0010] Inputting the system state representation vector into a time series probability model, updating parameters of the time series probability model through an online learning mechanism, and calculating a failure probability of the vision system within a preset future time window;

[0011] Dynamically calculating a warning threshold according to the environmental parameters and performance degradation indicators at the current time;

[0012] Analyzing whether the failure probability of the vision system exceeds the warning threshold, and if so, triggering a warning signal.

[0013] The principle and advantages of the present application are that the external environmental parameters and internal performance degradation indicators are placed in a unified time series probability model for collaborative analysis, which not only solves the one-sidedness problem of relying only on limited environmental parameters in the prior art, but also solves the lag problem of only monitoring the real-time state in the prior art through the calculation of the failure probability within the future time window, realizes timely prediction of risks, and through the dynamicization of the warning threshold, the system has the self-adaptability to complex operating conditions, and improves the warning accuracy and system reliability.

[0014] Further, the input data features include one or more of brightness, contrast, noise level and target size distribution of image data received by the vision system.

[0015] Further, the output data features include one or more of confidence, consistency and error rate of the prediction results output by the vision system.

[0016] Further, the environmental parameters include one or more of illumination intensity, temperature, humidity and external occlusion state.

[0017] Further, the performance degradation indicators include one or more of model parameter drift, hardware aging coefficient and historical failure record.

[0018] Further, the multi-modal fusion model is a multi-modal fusion model based on a Transformer architecture.

[0019] Further, the fusion processing of the multi-modal time series data based on the multi-modal fusion model to generate the system state representation vector comprises:

[0020] Spatial-temporal encoding of the input data features to generate an image feature vector;

[0021] performing dynamic confidence analysis on the output data features to generate a model stability feature vector;

[0022] performing normalization on the environment parameters and cross-modal fusion with the image feature vector and the model stability feature vector to generate an environment and model interaction feature vector;

[0023] performing historical trend modeling on the performance degradation indicators to generate a degradation dynamic feature vector;

[0024] fusing the image feature vector, the model stability feature vector, the interaction feature vector and the degradation dynamic feature vector in the time sequence dimension to generate a system state representation vector at the current time.

[0025] Further, the time sequence probability model is a dynamic Bayesian network.

[0026] Further, the parameters of the time sequence probability model are updated through an online learning mechanism, including:

[0027] obtaining a system state representation vector at a historical time;

[0028] comparing the system state representation vector at the current time with the system state representation vector at the historical time to generate a system state comparison result;

[0029] According to the system state comparison result, adjusting the parameters of the conditional probability table of the dynamic Bayesian network.

[0030] Further, before inputting the system state representation vector into the time sequence probability model, the system state representation vector is further subjected to dimension reduction processing to generate a low-dimensional feature vector.

[0031] Further, principal component analysis or autoencoder is used to reduce the dimension of the system state representation vector.

[0032] Beneficial effects: the contradiction between high-dimensional features and the calculation complexity of the time sequence probability model is solved through dimension reduction processing; through PCA or autoencoder, the data is mapped to a low-dimensional essential space under the premise of retaining most of the information, which not only guarantees real-time performance, but also improves the robustness and generalization ability of the probability prediction due to the filtering of noise.

[0033] Further, a numerical integration method is used to calculate the failure probability of the vision system within a predetermined future time window.

[0034] Further, according to the environment parameters and performance degradation indicators at the current time, a warning threshold is dynamically calculated, including:

[0035] obtaining reference environment parameters and reference performance degradation indicators;

[0036] generating an environment parameter difference amount according to the environment parameter of the current moment and the reference environment parameter;

[0037] generating a performance degradation difference amount according to the performance degradation index of the current moment and the reference performance degradation index;

[0038] generating a warning threshold according to the environment parameter difference amount and the performance degradation difference amount.

[0039] Beneficial effects: The adaptive adjustment of the warning threshold with the environment parameter and the performance degradation index is realized, and the false positive rate and the false negative rate are reduced.

[0040] Further, the reference environment parameter is a pre-stored reference environment parameter or an environment parameter of a historical moment; and the reference performance degradation index is a pre-stored reference performance degradation index or a performance degradation index of a historical moment.

[0041] Further, the calculation formula of the warning threshold is:

[0042]

[0043] In the formula, is the warning threshold, is a reference threshold, is an environment weight coefficient, is the environment parameter difference amount, is a performance weight coefficient, is the performance degradation difference amount.

[0044] Further, when the reference environment parameter is an environment parameter of a historical moment, generating an environment parameter difference amount according to the environment parameter of the current moment and the reference environment parameter includes:

[0045] calculating a temperature space-time gradient according to the temperature of the current moment and the temperature of the historical moment;

[0046] calculating a humidity space-time gradient according to the humidity of the current moment and the humidity of the historical moment;

[0047] calculating an illumination intensity space-time gradient according to the illumination intensity of the current moment and the illumination intensity of the historical moment;

[0048] performing normalization processing on the temperature space-time gradient, the humidity space-time gradient, and the illumination intensity space-time gradient;

[0049] performing weighted summation on the normalized temperature space-time gradient, the normalized humidity space-time gradient, and the normalized illumination intensity space-time gradient to generate the environment parameter difference amount.

[0050] Beneficial effects: By calculating the spatiotemporal gradient of environmental variables, the dynamic trend of environmental changes can be captured, improving the timeliness of environmental difference perception.

[0051] Furthermore, the calculation formulas for the temperature spatiotemporal gradient, humidity spatiotemporal gradient, and light intensity spatiotemporal gradient are as follows:

[0052]

[0053]

[0054]

[0055] In the formula, For a moment Temperature spatiotemporal gradient, Normalized time temperature, The preset sampling time interval, Normalized time temperature, For a moment The spatiotemporal gradient of humidity, Normalized time humidity, Normalized time humidity, For a moment The spatiotemporal gradient of light intensity, Normalized time Light intensity, Normalized time Light intensity.

[0056] Furthermore, the formula for calculating the difference in environmental parameters is as follows:

[0057]

[0058] In the formula, This represents the difference in environmental parameters. Temperature weighting coefficient, This represents the normalized temperature spatiotemporal gradient. Humidity weighting coefficient This represents the normalized spatiotemporal gradient of humidity. This is the illumination weighting coefficient. This represents the normalized spatiotemporal gradient of illumination intensity.

[0059] Furthermore, when the reference performance degradation index is a historical performance degradation index, the formula for calculating the performance degradation difference is:

[0060]

[0061]

[0062] In the formula, This represents the difference in performance degradation. For the first The weight of each indicator For the first The standardized decay of a performance degradation index. For the first The attenuation coefficient of the item index For the first Performance degradation index, For the normalized first The performance degradation index at time The original value, This represents the drift of the model parameters. This refers to the hardware aging factor. This represents the historical frequency of failures.

[0063] Furthermore, based on the differences in environmental parameters and performance degradation, a warning threshold is generated, including:

[0064] Obtain historical false positive rate and historical false negative rate;

[0065] Based on the historical false positive rate and historical false negative rate, an objective function is constructed, and environmental weight coefficients and performance weight coefficients are generated through the objective function.

[0066] Early warning thresholds are generated based on differences in environmental parameters, differences in performance degradation, environmental weighting coefficients, and performance weighting coefficients.

[0067] Furthermore, the objective function is:

[0068]

[0069] In the formula, Let be the objective function. This is the false alarm rate weighting coefficient. For false alarm rate, This is the weighting coefficient for the false negative rate. This represents the underreporting rate.

[0070] Furthermore, generating an early warning threshold based on the differences in environmental parameters and performance degradation also includes:

[0071] The environmental weight coefficient and the performance weight coefficient are iteratively optimized using Bayesian optimization or gradient descent methods.

[0072] The application provides a basic scheme two: a visual system reliability early warning system, comprising a data acquisition module, a fusion analysis module, a probability prediction module, a threshold generation module and an early warning module.

[0073] The data acquisition module is used for collecting multi-modal time series data of the visual system in real time, wherein the multi-modal time series data comprises input data features, output data features, environmental parameters and performance degradation indexes.

[0074] The fusion analysis module is used for performing fusion processing on the multi-modal time series data based on a multi-modal fusion model to generate a system state representation vector.

[0075] The probability prediction module is used for inputting the system state representation vector into a time series probability model, updating parameters of the time series probability model through an online learning mechanism, and calculating a failure probability of the visual system within a preset future time window.

[0076] The threshold generation module is used for dynamically calculating an early warning threshold according to the environmental parameters and the performance degradation indexes at the current moment.

[0077] The early warning module is used for analyzing whether the failure probability of the visual system exceeds the early warning threshold, and if yes, triggering an early warning signal.

[0078] Further, the above-mentioned visual system reliability early warning method is used.

[0079] The application provides a basic scheme three: a visual system reliability early warning device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the above-mentioned visual system reliability early warning method.

[0080] The application provides a basic scheme four: a visual system reliability early warning storage medium, used for storing computer executable instructions, wherein the computer executable instructions are used for implementing the above-mentioned visual system reliability early warning method when executed. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 It is a flow chart of the visual system reliability early warning method embodiment of the application.

[0082] Figure 2 It is a logic block diagram of the visual system reliability early warning system embodiment of the application.

[0083] Figure 3 It is a logic block diagram of the visual system reliability early warning device embodiment of the application. DETAILED DESCRIPTION

[0084] The application will be further described in detail through specific embodiments.

[0085] Embodiment 1:

[0086] The visual system reliability early warning method, as shown in the figure, comprises the following steps: Figure 1

[0087] S100, real-time acquisition of multi-modal time series data of the visual system, the multi-modal time series data comprising input data features, output data features, environmental parameters and performance degradation indicators. The input data features comprise one or more of brightness, contrast, noise level and target size distribution of image data received by the visual system, reflecting statistical changes in image quality over time, which can be used to capture image acquisition deviations caused by environmental interference or hardware aging. The output data features comprise one or more of confidence, consistency and error rate of the prediction results output by the visual system, the change rate of the confidence can be used to measure the stability of the model output, and the fluctuation coefficient of the error rate can reflect the degradation trend of the model analysis performance. The environmental parameters comprise one or more of illumination intensity, temperature, humidity and external occlusion state, and the changes in the external conditions where the visual system is located are recorded in real time, which are key external factors affecting system reliability. The performance degradation indicators comprise one or more of model parameter drift, hardware aging coefficient and historical failure records. Specifically, the model parameter drift can be calculated by comparing the difference between the current model parameter and the initial parameter, and the hardware aging coefficient can be modeled based on the device running time and the failure frequency.

[0088] In this embodiment, the input data features, output data features, environmental parameters and performance degradation indicators comprise all the above corresponding parameters. By real-time acquisition of multi-modal time series data, the comprehensive influence of external environmental interference and system performance degradation can be captured, providing a multi-dimensional data basis for subsequent feature fusion and failure prediction, and significantly improving the comprehensiveness and real-time performance of reliability analysis.

[0089] S200, fusion processing of the multi-modal time series data based on a multi-modal fusion model to generate a system state representation vector; the multi-modal fusion model is a multi-modal fusion model based on a Transformer architecture. S200 comprises:

[0090] S201, spatio-temporal encoding of the input data features to generate an image feature vector. Specifically, the spatio-temporal feature encoding of the input data features at each time point is mapped to a high-dimensional vector, and the global correlation between the image features at different time points is captured through a self-attention mechanism. For example, the self-attention mechanism can identify the long-term dependence of image brightness and contrast over time, enhancing the time series perception ability of the state representation.

[0091] ​S202, perform dynamic confidence analysis on the output data features to generate a model stability feature vector. Specifically, calculate the confidence rate change and error rate fluctuation coefficient of the prediction results at adjacent time points as model stability indicators. For example, the confidence rate change reflects the consistency of the model output, and the error rate fluctuation coefficient reveals the fluctuation of the model analysis performance, and the two together constitute the model stability feature vector.

[0092] S203, normalize the environmental parameters and cross-modal fuse the image feature vector and the model stability feature vector to generate an interaction feature vector between the environment and the model. Specifically, normalize the environmental parameters to eliminate dimensional differences, and then fuse them with the image feature vector and the model stability feature vector through a cross-modal encoder. This process can capture cross-modal correlation features such as "how does the change of light intensity affect the image quality and the model output confidence".

[0093] S204, model the historical trend of the performance degradation indicator to generate a degradation dynamic feature vector. Specifically, calculate the change rate of the performance degradation indicator using a sliding window, and combine the performance degradation indicator at the current time to generate a degradation dynamic feature vector. For example, sliding window average and trend fitting are performed on indicators such as model parameter drift amount and hardware aging coefficient to construct a feature representation reflecting the dynamic of system performance degradation.

[0094] S205, fuse the image feature vector, model stability feature vector, interaction feature vector and degradation dynamic feature vector in the time sequence dimension to generate a system state representation vector at the current time. Specifically, the above four types of feature vectors are spliced or weighted fused in the time sequence dimension, input into the decoding layer of the Transformer architecture, and a system state representation vector at the current time is generated. The vector comprehensively reflects the multi-dimensional state information of external environmental interference, model output stability and system degradation.

[0095] Through the multi-modal fusion of the Transformer architecture, the system can extract and strengthen the cross-modal temporal correlation features, and construct a system state representation with high discriminability, providing robust and interpretable feature input for subsequent fault probability prediction.

[0096] S300, input the system state representation vector into a time series probability model, update the parameters of the time series probability model through an online learning mechanism, and calculate the fault probability of the visual system within a predetermined future time window; the time series probability model is a dynamic Bayesian network. S300 includes:

[0097] S301, dimension reduction processing is performed on the system state representation vector to generate a low-dimensional feature vector; specifically, principal component analysis or an autoencoder is used to perform dimension reduction processing on the system state representation vector. The system state representation vector is a high-dimensional vector, denoted as , which includes image features, model stability features, interaction features, and degradation dynamic features. Through dimension reduction processing, a corresponding low-dimensional feature vector is obtained, denoted as: This process preserves key time sequence correlation features while significantly reducing the computational complexity of the dynamic Bayesian network.

[0098] S302, the system state representation vector after dimension reduction processing is input to a time sequence probability model.

[0099] The dynamic Bayesian network is initialized. Based on historical training data, an initial Bayesian network structure is constructed, the probability dependency relationship between each feature node is determined, and an initial conditional probability table is trained.

[0100] S303, the system state representation vector at the historical time is obtained.

[0101] S304, the system state representation vector at the current time is compared with the system state representation vector at the historical time to generate a system state comparison result. Specifically, after receiving a new system state representation vector each time, a difference matrix between the system state representation vector at the current time and the system state representation vector at the historical time is calculated.

[0102] S305, the parameters of the dynamic Bayesian network are updated in real time. According to the system state comparison result, the parameters of the conditional probability table of the dynamic Bayesian network are adjusted. Specifically, based on the difference matrix, the conditional probability table in the dynamic Bayesian network is dynamically adjusted. In this embodiment, a recursive Bayesian formula is used for incremental updating:

[0103]

[0104] In the formula, is the parent node set of node , represents the state of the i-th feature node in the dynamic Bayesian network at the future time , represents the high-dimensional system state representation vector sequence from time 1 to time , represents the state of the i-th feature node at time in the dynamic Bayesian network. represents the state of the i-th feature node at time in the dynamic Bayesian network. represents the state of the i-th feature node at time in the dynamic Bayesian network. represents the state of the i-th feature node at time in the dynamic Bayesian network. represents the state of the i-th feature node at time conditional probability of the current time step; denotes its approximation, which is calculated by only relying on the node state of the latest time step and its parent node relationship. This incremental updating strategy decomposes the global CPT updating into local updating, only considering the feature increment of the latest time step, thereby avoiding the high overhead of global joint probability calculation.

[0105] When the parameter updating amplitude of multiple consecutive time points exceeds a preset threshold, a model parameter resetting mechanism is triggered to avoid model overfitting or performance degradation caused by abnormal fluctuations.

[0106] S306, a numerical integration method is used to calculate the failure probability of the visual system within a preset future time window. Specifically, based on the updated dynamic Bayesian network, an approximate inference strategy is used to calculate the failure probability within a preset future time window, and the high-dimensional time series joint probability is approximated as a conditional probability in a low-dimensional space:

[0107]

[0108] wherein, and are the low-dimensional feature vectors of the current time step and the next time step of the current time step, respectively, denotes the high-dimensional system state representation vector sequence from time step 1 to time step . denotes the probability of the future high-dimensional state under the condition of all high-dimensional historical data . denotes its approximation, i.e. the probability of the future low-dimensional feature under the condition of the current low-dimensional feature .

[0109] The Monte Carlo sampling or Gaussian approximate integration method is used to numerically integrate the failure probability density function to obtain the specific probability value of system failure within a preset future time window:

[0110]

[0111] wherein, is the final probability value of system failure at future time step obtained by numerical approximate integration, i.e. the predicted failure probability; is the conditional probability density function, which represents the probability distribution of system failure at future time step under the condition of the current low-dimensional feature . is the number of Monte Carlo sampling times; denotes the i-th sample obtained from the low-dimensional feature distribution by Monte Carlo sampling a sample.

[0112] The above scheme is used to predict the failure probability, which significantly reduces the calculation cost under high-dimensional data on the premise of ensuring the prediction accuracy, and meets the dual needs of efficient calculation and accurate early warning of the visual system.

[0113] S400, dynamically calculating the early warning threshold according to the environmental parameter and the performance degradation index at the current time; comprising:

[0114] S401, obtaining a reference environmental parameter and a reference performance degradation index; the reference environmental parameter is a pre-stored benchmark environmental parameter or an environmental parameter at a historical time; the reference performance degradation index is a pre-stored benchmark performance degradation index or a performance degradation index at a historical time. When the reference environmental parameter is a pre-stored benchmark environmental parameter, and the reference performance degradation index is a pre-stored benchmark performance degradation index, jump to execute S404; when the reference environmental parameter is a pre-stored benchmark environmental parameter, and the reference performance degradation index is a performance degradation index at a historical time, jump to execute S403; when the reference environmental parameter is an environmental parameter at a historical time, and the reference performance degradation index is a pre-stored benchmark performance degradation index, execute S402, and jump to execute S404 after ending S402. In the embodiment, the reference environmental parameter is an environmental parameter at a historical time, the reference performance degradation index is a performance degradation index at a historical time, and S402 is executed.

[0115] S402, generating an environmental parameter difference amount according to the environmental parameter at the current time and the reference environmental parameter; when the reference environmental parameter is an environmental parameter at a historical time, S402 comprises:

[0116] S4021, calculating a temperature space-time gradient according to the temperature at the current time and the temperature at the historical time.

[0117] S4022, calculating a humidity space-time gradient according to the humidity at the current time and the humidity at the historical time.

[0118] S4023, calculating an illumination intensity space-time gradient according to the illumination intensity at the current time and the illumination intensity at the historical time.

[0119] The calculation formulas of the temperature space-time gradient, the humidity space-time gradient and the illumination intensity space-time gradient are respectively:

[0120]

[0121]

[0122]

[0123] In the formula, t is the current time, t0 is the historical time, T is the temperature, H is the humidity, and I is the illumination intensity. is the time Temperature spatiotemporal gradient, Normalized time temperature, The preset sampling time interval, Normalized time temperature, For a moment The spatiotemporal gradient of humidity, Normalized time humidity, Normalized time humidity, For a moment The spatiotemporal gradient of light intensity, Normalized time Light intensity, Normalized time Light intensity.

[0124] S4024 normalizes the spatiotemporal gradients of temperature, humidity, and light intensity.

[0125] S4025, the normalized spatiotemporal gradients of temperature, humidity, and light intensity are weighted and summed to generate the environmental parameter difference. The calculation formula for the environmental parameter difference is as follows:

[0126]

[0127] In the formula, This represents the difference in environmental parameters. Temperature weighting coefficient, This represents the normalized temperature spatiotemporal gradient. Humidity weighting coefficient This represents the normalized spatiotemporal gradient of humidity. This is the illumination weighting coefficient. This represents the normalized spatiotemporal gradient of light intensity. In this embodiment, the temperature weighting coefficient, humidity weighting coefficient, and light intensity weighting coefficient are determined through historical fault sensitivity analysis. In other embodiments of this application, they can also be determined using the entropy weighting method.

[0128] S403, generate a performance degradation difference based on the current performance degradation index and a reference performance degradation index; when the reference performance degradation index is a historical performance degradation index, the calculation formula for the performance degradation difference is:

[0129]

[0130]

[0131] In the formula, This represents the difference in performance degradation. For the first The weight of each indicator For the first The standardized decay of a performance degradation index. For the first The attenuation coefficient of the item index For the first Performance degradation index, For the normalized first The performance degradation index at time The original value, This represents the drift of the model parameters. This refers to the hardware aging factor. This represents the historical failure frequency. In this embodiment, the first... Weight of each indicator It can be calculated using principal component analysis. In other embodiments of this application, it can also be calculated using historical fault correlation.

[0132] S404, based on the difference in environmental parameters and the difference in performance degradation, generates a warning threshold. S404 includes:

[0133] S4041, obtain historical false alarm rate and historical false negative rate.

[0134] S4042, Based on the historical false positive rate and historical false negative rate, construct an objective function, and generate environmental weight coefficients and performance weight coefficients through the objective function; the objective function is:

[0135]

[0136] In the formula, Let be the objective function. This is the false alarm rate weighting coefficient. For false alarm rate, This is the weighting coefficient for the false negative rate. The false negative rate is used. The objective function is minimized using Bayesian optimization or gradient descent methods, with iterative optimization of the environmental and performance weight coefficients.

[0137] In actual operation, an incremental update method using a sliding window can be adopted. When environmental parameters or performance degradation indicators change significantly, the optimal value can be recalculated. , This ensures that the warning threshold is adaptively adjusted according to the system status.

[0138]

[0139] In the formula, and are updated environment weight coefficients and performance weight coefficients, is a learning rate.

[0140] S4043, generating a warning threshold according to the environment parameter difference, the performance degradation difference, the environment weight coefficient and the performance weight coefficient. The calculation formula of the warning threshold is:

[0141]

[0142] In the formula, is a warning threshold, is a reference threshold, is an environment weight coefficient, is an environment parameter difference, is a performance weight coefficient, is a performance degradation difference. When the reference environment parameter is a pre-stored reference environment parameter, the environment parameter difference is the difference between the current environment parameter and the pre-stored reference environment parameter; when the reference performance degradation indicator is a pre-stored reference performance degradation indicator, the performance degradation difference is the difference between the current performance degradation indicator and the pre-stored reference performance degradation indicator. Through the above method, The environment disturbance, performance degradation and historical failure law can be comprehensively considered to realize a dynamic and adjustable warning threshold, and the accuracy and real-time performance of system reliability monitoring are significantly improved.

[0143] S500, analyzing whether the failure probability of the visual system exceeds the warning threshold, and if so, triggering a warning signal. The warning signal is sent to the user end or the control system through a visual interface, an API interface or a message queue, supporting multi-terminal collaborative response.

[0144] Embodiment 2:

[0145] Embodiment 2 is basically the same as embodiment 1, and the difference is that in S300 of embodiment 2, the system state representation vector sequence is input into an online learning time sequence neural network model to directly predict the failure probability of the visual system in a preset future time window in an end-to-end manner. Let the system state representation vector at each time be which contains the reduced image features, model output features, environment features and degradation features. The time sequence neural network model is a long short-term memory network or a Transformer time sequence model.

[0146] S300 includes the following steps:

[0147] S301, online standardization is performed on the input system state representation vector sequence to maintain the numerical stability of model training. Specifically, the mean and variance are updated in an incremental calculation manner:

[0148]

[0149]

[0150] where, and are the mean and variance of the feature at time , respectively, is a smoothing coefficient. The normalized feature is .

[0151] S302, add the time position information to the normalized feature sequence. If the time series neural network model is a long short-term memory network, directly input the time difference between adjacent time points as an additional feature. If the Transformer time series model is used, calculate and add the absolute position encoding or relative position encoding for each position in the sequence.

[0152] S303, in order to distinguish different sources of features, assign a learnable modal embedding vector to the image features, model output features, environment features, and degradation features, respectively, and add or concatenate them with the corresponding normalized features, and then map and fuse them through a linear layer.

[0153] S304, input the feature sequence processed in the above steps into the time series neural network model. The output of the model is a probability vector , , , which represents the probability of the system failing at future time predicted by the model. The future time window is a preset set of discrete time points.

[0154] S305, the time series neural network model is trained and the parameters are updated by minimizing the weighted binary cross-entropy loss function:

[0155]

[0156] where, is the true label (such as the failure event detected by the sensor) from the actual observation or historical record, indicating whether the system actually fails at future time (1 for failure, 0 for normal); is the failure probability predicted by the model; is the loss weight of the th future prediction point, which is used to guide the model to pay more attention to the key minority class (failure samples) and important time periods in the case of class imbalance and multi-time point prediction; This represents the number of discrete time points within the future prediction time window.

[0157] This embodiment provides an alternative implementation approach, distinct from Embodiment 1, by directly modeling future failure probabilities using a temporal neural network with online learning capabilities. This approach better captures the complex nonlinearities and long-term dependencies in multimodal time-series data. It also achieves end-to-end real-time failure prediction and, combined with a dynamic early warning threshold mechanism, provides a more flexible and powerful model selection for visual system reliability early warning while maintaining high accuracy.

[0158] Example 3:

[0159] This application provides a vision system reliability early warning system, such as Figure 2 As shown, it includes a data acquisition module, a fusion analysis module, a probability prediction module, a threshold generation module, and an early warning module;

[0160] The data acquisition module is used to acquire multimodal temporal data of the vision system in real time. The multimodal temporal data includes input data features, output data features, environmental parameters, and performance degradation indicators.

[0161] The fusion analysis module is used to perform fusion processing on the multimodal time series data based on the multimodal fusion model to generate a system state representation vector;

[0162] The probability prediction module is used to input the system state representation vector into the time series probability model, update the parameters of the time series probability model through an online learning mechanism, and calculate the failure probability of the vision system within a preset future time window.

[0163] The threshold generation module is used to dynamically calculate the warning threshold based on the environmental parameters and performance degradation indicators at the current moment.

[0164] The early warning module is used to analyze whether the failure probability of the vision system exceeds the early warning threshold. If so, an early warning signal is triggered.

[0165] The visual system reliability early warning system provided in this application uses the above-mentioned visual system reliability early warning method. Other technical features in the system are the same as those disclosed in the above-mentioned method embodiments, and will not be repeated here.

[0166] Example 4:

[0167] This application provides a vision system reliability early warning device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the aforementioned vision system reliability early warning method. In this embodiment, as... Figure 3As shown, the device includes at least one processor, at least one storage medium and at least one bus, and in this embodiment, also includes at least one communication interface. Among them, the bus is used to realize the direct connection communication of these components, the communication interface is used for signaling or data communication with other node devices, and the storage medium stores machine readable instructions executable by the processor. When the device is running, the processor communicates with the storage medium through the bus, the machine readable instructions are executed by the processor, and the steps of the above visual system reliability early warning method are realized.

[0168] Embodiment 5:

[0169] The present application provides a visual system reliability early warning storage medium for storing computer executable instructions, which when executed, implement the above visual system reliability early warning method. That is, if the above method is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above embodiment method, which can also be completed by instructing related hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The readable storage medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0170] The above is only an embodiment of the present application, and the common knowledge of specific structures and characteristics in the scheme is not described in detail here. The ordinary skilled person in the art knows all the ordinary technical knowledge in the field of the present application before the application date or the priority date, can know all the prior art in this field, and has the ability to apply conventional experimental means before that date. The ordinary skilled person in the art can improve and implement the present scheme based on their own ability under the guidance of the present application. Some typical known structures or known methods should not be an obstacle for the ordinary skilled person in the art to implement the present application. It should be noted that for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should be considered as the protection scope of the present application. These will not affect the effect and practicality of the present application. The scope of protection claimed in the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.

Claims

1. A method for visual system reliability warning, characterized in that: The method comprises the following steps: Real-time acquisition of multi-modal time series data of a vision system, the multi-modal time series data comprising input data features, output data features, environmental parameters and performance degradation indicators; Fusion processing of the multi-modal time series data based on a multi-modal fusion model to generate a system state representation vector; Inputting the system state representation vector into a time series probability model, updating parameters of the time series probability model through an online learning mechanism, and calculating a failure probability of the vision system within a preset future time window; Dynamically calculating a warning threshold according to the environmental parameters and performance degradation indicators at the current time; Analyzing whether the failure probability of the vision system exceeds the warning threshold, and if so, triggering a warning signal.

2. The method of claim 1, wherein: The input data features comprise one or more of brightness, contrast, noise level and target size distribution of image data received by the vision system.

3. The method of claim 1, wherein: The output data features comprise one or more of confidence, consistency and error rate of prediction results output by the vision system.

4. The method of claim 1, wherein: The environmental parameters comprise one or more of illumination intensity, temperature, humidity and external occlusion state.

5. The method of claim 1, wherein: The performance degradation indicators comprise one or more of model parameter drift, hardware aging coefficient and historical failure record.

6. The method of claim 1, wherein: The multi-modal fusion model is a multi-modal fusion model based on a Transformer architecture.

7. The method of claim 1, wherein: The fusion processing of the multi-modal time series data based on the multi-modal fusion model to generate the system state representation vector comprises: Temporal and spatial encoding of the input data features to generate an image feature vector; Dynamic confidence analysis of the output data features to generate a model stability feature vector; Normalization processing of the environmental parameters and cross-modal fusion with the image feature vector and the model stability feature vector to generate an interaction feature vector of the environment and the model; Historical trend modeling of the performance degradation indicators to generate a degradation dynamic feature vector; Fusion of the image feature vector, the model stability feature vector, the interaction feature vector and the degradation dynamic feature vector in the time series dimension to generate the system state representation vector at the current time.

8. The method of claim 7, wherein: The time series probability model is a dynamic Bayesian network.

9. The method of claim 8, wherein: The updating of the parameters of the time series probability model through the online learning mechanism comprises: Obtaining system state representation vectors at historical times; Comparing the system state representation vector at the current time with the system state representation vectors at the historical times to generate a system state comparison result; Adjusting parameters of a conditional probability table of the dynamic Bayesian network according to the system state comparison result.

10. The method of claim 1, wherein: Before the inputting of the system state representation vector into the time series probability model, the method further comprises: dimension reduction processing of the system state representation vector to generate a low-dimensional feature vector.

11. The method of claim 10, wherein: The dimension reduction processing of the system state representation vector is performed by principal component analysis or an autoencoder.

12. The method of claim 1, wherein: The failure probability of the vision system within the preset future time window is calculated by a numerical integration method.

13. The method of claim 1, wherein: The warning threshold is dynamically calculated according to the environmental parameter and the performance degradation index at the current time, including: obtaining a reference environmental parameter and a reference performance degradation index; generating an environmental parameter difference based on the environmental parameter at the current time and the reference environmental parameter; generating a performance degradation difference based on the performance degradation index at the current time and the reference performance degradation index; and generating the warning threshold based on the environmental parameter difference and the performance degradation difference.

14. The method of claim 13, wherein: The reference environmental parameter is a pre-stored benchmark environmental parameter or an environmental parameter at a historical time. The reference performance degradation index is a pre-stored benchmark performance degradation index or a performance degradation index at a historical time.

15. The method of claim 13, wherein: The calculation formula of the early warning threshold is: In the formula, is the early warning threshold, is the reference threshold, is the environmental weight coefficient, is the environmental parameter difference, is the performance weight coefficient, is the performance degradation difference.

16. The method of claim 14, wherein: When the reference environmental parameter is an environmental parameter at a historical time, the environmental parameter difference is generated based on the environmental parameter at the current time and the reference environmental parameter, including: calculating a temperature space-time gradient based on the temperature at the current time and the temperature at the historical time; calculating a humidity space-time gradient based on the humidity at the current time and the humidity at the historical time; calculating an illumination intensity space-time gradient based on the illumination intensity at the current time and the illumination intensity at the historical time; normalizing the temperature space-time gradient, the humidity space-time gradient, and the illumination intensity space-time gradient; and generating the environmental parameter difference by weighted summation of the normalized temperature space-time gradient, the normalized humidity space-time gradient, and the normalized illumination intensity space-time gradient.

17. The method of claim 16, wherein: The calculation formulas for the spatiotemporal gradients of temperature, humidity, and light intensity are as follows: In the formula, For a moment Temperature spatiotemporal gradient, Normalized time temperature, The preset sampling time interval, Normalized time temperature, For a moment The spatiotemporal gradient of humidity, Normalized time humidity, Normalized time humidity, For a moment The spatiotemporal gradient of light intensity, Normalized time Light intensity, Normalized time Light intensity.

18. The method of claim 17, wherein: The calculation formula of the environmental parameter difference amount is: In the formula, is the environmental parameter difference amount, is the temperature weight coefficient, represents the normalized temperature spatiotemporal gradient, is the humidity weight coefficient, represents the normalized humidity spatiotemporal gradient, is the illumination weight coefficient, represents the normalized illumination intensity spatiotemporal gradient.

19. The method of claim 14, wherein: When the reference performance degradation index is a historical performance degradation index, the formula for calculating the performance degradation difference is: In the formula, This represents the difference in performance degradation. For the first The weight of each indicator For the first The standardized decay of a performance degradation index. For the first The attenuation coefficient of the item index For the first Performance degradation index, For the normalized first The performance degradation index at time The original value, This represents the drift of the model parameters. This refers to the hardware aging factor. This represents the historical frequency of failures.

20. The method of claim 15, wherein: The warning threshold is generated based on the environmental parameter difference and the performance degradation difference, including: obtaining a historical false alarm rate and a historical missed alarm rate; constructing an objective function based on the historical false alarm rate and the historical missed alarm rate, and generating an environmental weight coefficient and a performance weight coefficient through the objective function; and generating the warning threshold based on the environmental parameter difference, the performance degradation difference, the environmental weight coefficient, and the performance weight coefficient.

21. The method of claim 20, wherein: The objective function is: wherein, is the objective function, is the false positive rate weight coefficient, is the false positive rate, is the false negative rate weight coefficient, is the false negative rate.

22. The method of claim 20, wherein: The warning threshold is generated based on the environmental parameter difference and the performance degradation difference, and further includes: iteratively optimizing the environmental weight coefficient and the performance weight coefficient using a Bayesian optimization method or a gradient descent method.

23. A vision system reliability alert system characterized by: The method includes a data acquisition module, a fusion analysis module, a probability prediction module, a threshold generation module, and a warning module; the data acquisition module is configured to acquire multi-modal time series data of a visual system in real time, the multi-modal time series data including input data features, output data features, environmental parameters, and performance degradation indexes; The fusion analysis module is configured to perform fusion processing on the multi-modal time series data based on a multi-modal fusion model to generate a system state representation vector; the probability prediction module is configured to input the system state representation vector into a time series probability model, update parameters of the time series probability model through an online learning mechanism, and calculate a failure probability of the visual system within a preset future time window; and the threshold generation module is configured to dynamically calculate a warning threshold based on environmental parameters and performance degradation indexes at a current time. The warning module is configured to analyze whether the failure probability of the visual system exceeds the warning threshold, and if so, trigger a warning signal.

24. The vision system reliability alerting system of claim 23, wherein: The visual system reliability warning method of any one of the preceding claims 1 to 22 is used.

25. A vision system reliability alerting device characterized by: A computer program product comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program configured to implement the visual system reliability warning method of any of claims 1 to 22.

26. Visual system reliability alerting storage media for storing computer executable instructions, characterized in that: The computer executable instructions, when executed, implement the visual system reliability warning method of any of claims 1 to 22.

Images