Device operation and maintenance method and system based on multi-modal large model

Abstract

The application relates to the technical field of data processing, in particular to a device operation and maintenance method and system based on a multi-modal large model. The method comprises the following steps: time aligning historical data of each mode of a target device; for each time point, determining a multi-modal conflict degree of the target device and a modal abnormal deviation degree of each mode according to the difference between the historical data of each mode at the time point and the corresponding standard value; determining a modal credibility of each mode at the time point according to the multi-modal conflict degree and the modal abnormal deviation degree of each mode; mapping feature vectors of the historical data of each mode at the time point to a unified semantic space, weighting and summing the mapped feature vectors of each mode according to the modal credibility of each mode to obtain a multi-modal fusion feature vector at the time point; and performing abnormality judgment on the target device according to the multi-modal fusion feature vector of the target device at each time point. The method improves the accuracy of device abnormality judgment.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a device operation and maintenance method and system based on a multimodal large model. Background Technology

[0002] Equipment maintenance requires monitoring equipment status data, identifying anomalies based on this data, and then performing maintenance on equipment exhibiting abnormalities. Currently, equipment maintenance typically relies on single-type data, such as sensor time-series data, log text, or single-modality data like images and videos, lacking in-depth analysis of the correlations and consistency between different modalities of data.

[0003] While some methods are currently attempting to combine multimodal data for equipment status monitoring, they generally involve simple weighted fusion of data from multiple modalities using fixed weights, and anomaly detection is based on the weighted fused data. This approach fails to fully consider the differences in contributions from different modalities, and anomalies in data from one modality can easily lead to misjudgments of the overall equipment status. This results in inaccurate anomaly detection and a high false alarm rate, thus limiting its applicability and reliability in complex industrial scenarios. Summary of the Invention

[0004] To address the technical problem of inaccurate equipment anomaly detection, the present invention aims to provide an equipment operation and maintenance method and system based on a multimodal large model. The specific technical solution adopted is as follows:

[0005] This invention provides a device operation and maintenance method based on a multimodal large model, the method comprising:

[0006] Time-align the historical data of each mode of the target device;

[0007] After time alignment, for each time point, based on the difference between the historical data of each mode and the corresponding standard value at that time point, the degree of multimodal conflict of the target device and the degree of modal anomaly deviation of each mode at that time point are determined;

[0008] Based on the degree of multimodal conflict and the degree of modal anomaly deviation of each mode, the modal credibility of each mode at the given time is determined;

[0009] The feature vectors of historical data of each modality at the specified time are mapped to a unified semantic space. Based on the modality credibility of each modality, the mapped feature vectors of each modality are weighted and summed to obtain the multimodal fusion feature vector at the specified time.

[0010] Based on the multimodal fusion feature vector of the target device at each time point, anomaly detection is performed on the target device.

[0011] According to the device operation and maintenance method based on a multimodal large model provided by the present invention, the step of time-aligning the historical data of each modality of the target device includes:

[0012] The time sensitivity of each mode is determined based on the changes in historical data of each mode of the target device;

[0013] The historical data for each modality is divided according to the length of the basic time window to obtain several time windows;

[0014] For each historical data point, the time alignment weight of the historical data within the time window is determined based on its relative position within the time window and the time sensitivity of the modality to which the historical data belongs.

[0015] Based on the time alignment weights of the historical data, the historical data of each modality of the target device are time aligned.

[0016] According to the equipment operation and maintenance method based on a multimodal large model provided by the present invention, the step of determining the time sensitivity of each modality based on the changes in historical data of each modality of the target equipment includes:

[0017] For each mode, calculate the slope of the historical data of the target device at each acquisition time in the mode;

[0018] The time sensitivity of the mode is determined based on the average of the absolute values ​​of the slopes at each acquisition time.

[0019] According to the equipment operation and maintenance method based on a multimodal large model provided by the present invention, the relative position includes the degree of distance from the center of the time window;

[0020] For each historical data point, determining the time alignment weight within the time window based on its relative position within the time window and the time sensitivity of the modality to which it belongs includes:

[0021] For each historical data point, the degree of distance from the center of the time window of the historical data is determined based on the difference between the original timestamp of the historical data and the timestamp of the middle moment of the time window to which the historical data belongs.

[0022] The time alignment weight of the historical data within the time window is determined based on the degree of distance from the center of the time window and the time sensitivity of the modality to which the historical data belongs.

[0023] According to the equipment operation and maintenance method based on a multimodal large model provided by the present invention, the step of determining the multimodal conflict degree and the modal anomaly deviation degree of each mode of the target equipment at each time point based on the difference between the historical data of each mode and the corresponding standard value at that time point includes:

[0024] For each time point, calculate the difference between the historical data of each modality and the corresponding standard value at that time point, and record it as the target difference;

[0025] The number of modes whose target differences exceed the anomaly threshold is determined as the number of consensus modes;

[0026] The degree of multimodal conflict of the target device at the given time is determined based on the degree of dispersion of the target differences of each modality and the number of consensus modalities.

[0027] For each mode, the modality anomaly deviation at the given time is determined based on the difference between the target difference of the mode and the mean of the target difference.

[0028] According to the device operation and maintenance method based on a multimodal large model provided by the present invention, determining the modal reliability of each mode at a given time based on the degree of multimodal conflict and the modal anomaly deviation of each mode includes:

[0029] Based on the degree of multimodal conflict and the degree of modal anomaly deviation of each modality, the modal dispute degree corresponding to each modality at the given time is determined.

[0030] Based on the modality controversy degree of each modality at the given time, the modality credibility corresponding to each modality at the given time is determined.

[0031] According to the equipment operation and maintenance method based on a multimodal large model provided by the present invention, determining the modal dispute degree corresponding to each mode at the given time based on the degree of multimodal conflict and the modal anomaly deviation degree of each mode includes:

[0032] For each mode, the outlier degree of the mode is determined based on the difference between the mode outlier deviation of the mode and the mean of the mode outlier deviation of the other modes.

[0033] Based on the degree of multimodal conflict and the degree of outlier status of the mode, the modal controversy degree of the mode at the given time is determined.

[0034] According to the equipment operation and maintenance method based on a multimodal large model provided by the present invention, determining the outlier degree of the mode based on the difference between the mode anomaly deviation degree of the mode and the mean of the mode anomaly deviation degrees of the other modes excluding the mode includes:

[0035] The outlier degree of a mode is determined by the ratio between the mode outlier deviation of the stated mode and the mean of the mode outlier deviations of all other modes.

[0036] According to the equipment operation and maintenance method based on a multimodal large model provided by the present invention, the method further includes:

[0037] For each modality at each time point, the contribution of the modality to the anomalous behavior at that time point is determined based on the difference between the mapped feature vector of the modality at that time point and the reference feature vector of the modality, as well as the modality confidence at that time point.

[0038] Output the ranking of the contribution of each modality to the abnormal performance at each time point; the ranking of the contribution of each modality to the abnormal performance at each time point is used to analyze the cause of the abnormality.

[0039] This invention provides an equipment operation and maintenance system based on a multimodal large model. The system includes a memory and a processor. The memory is used to store executable program code. The processor is used to call and run the executable program code from the memory to implement the equipment operation and maintenance method based on a multimodal large model provided by this invention.

[0040] This invention has the following beneficial effects: For each time point, based on the difference between the historical data of each modality of the target device at that time point and the corresponding standard value, the multimodal conflict degree and the modal anomaly deviation degree of each modality of the target device at that time point are determined. Based on the multimodal conflict degree and the modal anomaly deviation degree of each modality, the modal credibility of each modality at that time point is determined. Then, the feature vectors of the historical data of each modality at that time point are mapped to a unified semantic space. Based on the modal credibility of each modality, the mapped feature vectors of each modality are weighted and summed to obtain the multimodal fusion feature vector at that time point. This takes into account the difference in contribution of different modal data, so that the high credibility modality contributes more to the state representation, and the low credibility modality is suppressed, thereby improving the robustness of the multimodal fusion result, avoiding misjudgment of the overall state of the device due to the data anomaly of a certain modality, improving the fault tolerance of noisy data and abnormal data, and enabling more accurate judgment of device anomalies. Attached Figure Description

[0041] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a device operation and maintenance method based on a multimodal large model, provided in one embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of a time alignment process provided in one embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of a process for determining the degree of multimodal conflict and the degree of modal anomaly deviation provided in one embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of a process for determining modal confidence level according to an embodiment of the present invention;

[0046] Figure 5 This is a structural block diagram of a device operation and maintenance system based on a multimodal large model, provided as an embodiment of the present invention. Detailed Implementation

[0047] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a device operation and maintenance method and system based on a multimodal large model proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] The following description, in conjunction with the accompanying drawings, details a specific solution for a device operation and maintenance method and system based on a multimodal large model provided by this invention.

[0050] Please see Figure 1 The diagram illustrates a flowchart of a device operation and maintenance method based on a multimodal large model according to an embodiment of the present invention, including the following steps:

[0051] Step 101: Time-align the historical data of each mode of the target device.

[0052] Historical data can be data from a preset time period in the past. For example, the preset time period can be one week.

[0053] As shown in Table 1, a modality can include at least two of the following: text log data, image / video data, and network topology data. Table 1 also provides examples of the corresponding operation and maintenance objects (target devices), data content, data structure, time-series attributes, and collection sources for each modality. It should be understood that the content in Table 1 is only a general example of each modality; there may be many more specific data types.

[0054] Table 1

[0055]

[0056] In one embodiment, historical data for each modality can be preprocessed, and then step 101 and subsequent steps can be performed based on the preprocessed historical data for each modality. Different preprocessing methods can be used for data from different modalities. For example, preprocessing may include moving average filtering to remove high-frequency noise and avoid interference from instantaneous fluctuations, or cleaning text log data to remove redundant characters, such as special symbols and duplicate logs, or extracting key information, such as fault codes and component names, or preprocessing image data, such as performing target detection on visible light images to locate device components, such as CPU fans, or performing temperature calibration on infrared thermal imaging data to convert pixel values ​​into actual temperature values.

[0057] It is understandable that before conducting correlation analysis of different modal data, it is necessary to integrate them in a unified time dimension, that is, time alignment, so as to ensure the data homogeneity of subsequent fusion analysis, that is, multimodal data of the same time and the same equipment component can be correlated and analyzed.

[0058] In one embodiment, the cumulative cost between historical data of each modality can be calculated using the Dynamic Time Warping (DTW) algorithm, and the historical data of each modality of the target device can be time-aligned based on the cumulative cost.

[0059] In another embodiment, the time alignment weight of each historical data can be determined based on the changes and time location distribution of the historical data of each modality. The cost between the historical data of each modality is weighted according to the time alignment weight. The cumulative cost between the historical data of each modality is calculated based on the weighted cost using a dynamic time warping algorithm. The historical data of each modality of the target device is then time-aligned based on the cumulative cost.

[0060] Step 102: After time alignment, for each time point, based on the difference between the historical data of each mode and the corresponding standard value at that time point, determine the degree of multimodal conflict of the target device and the degree of modal anomaly deviation of each mode at that time point.

[0061] Here, "time" refers to the moment corresponding to the historical data of each modality after time alignment. "Standard value" refers to the data for the corresponding modality under conditions where no abnormal faults occur. "Multimodal conflict degree" measures the degree of contradiction or conflict between the anomalies reflected in the historical data of each modality of the target device. A higher value indicates greater discrepancies and contradictions in the anomalies reflected in the historical data of each modality of the target device; a lower value indicates that the anomalies reflected in the historical data of each modality of the target device are more mutually corroborative and reach a consensus. "Modal anomaly deviation degree" measures the degree of deviation of the anomaly manifestation of any modality from the overall anomaly performance of the multimodal approach.

[0062] Step 103: Determine the modal credibility of each mode at a given time based on the degree of multimodal conflict and the modal anomaly deviation of each mode.

[0063] Modal credibility is used to measure the credibility of historical data for any given modality.

[0064] Step 104: Map the feature vectors of historical data of each modality at time step to a unified semantic space. Based on the modality credibility of each modality, perform a weighted summation of the mapped feature vectors of each modality to obtain the multimodal fusion feature vector at time step.

[0065] The feature vectors of the historical data for each modality are obtained by extracting features from the preprocessed historical data for each modality.

[0066] In one embodiment, for each modality, the feature vectors of the historical data of that modality are mapped to a unified semantic space using the feature mapping function corresponding to that modality. The feature mapping function can be implemented using a pre-trained model or linear transformation, etc.

[0067] In one embodiment, the multimodal fusion feature vector can be determined according to the following formula:

[0068]

[0069] in, Indicates time The multimodal fusion feature vector is obtained from the following. Indicates time Lower mode Modal reliability. This represents the total number of modes. Indicates time Lower mode The feature vector of historical data. Indicates modality The feature mapping function is used to map modes... The feature vectors of the data are mapped to a unified semantic space.

[0070] Step 105: Based on the multimodal fusion feature vectors of the target device at various times, perform anomaly judgment on the target device.

[0071] In one embodiment, a baseline feature vector of the target device under normal conditions is constructed based on the normal data (data during operation without abnormal faults) from the historical data of each modality of the target device. The multimodal fused feature vector of the target device at each time point is compared with the baseline feature vector, and the anomaly judgment of the target device is made based on the comparison result. The normal data must ensure that it covers the normal state under different load and environmental conditions.

[0072] In one embodiment, the step of constructing a baseline feature vector of the target device in a normal state based on the normal data in the historical data of each modality of the target device includes: performing time alignment on the normal data of each modality, then calculating the modal confidence of the normal data of each modality based on the time-aligned normal data, and performing a weighted summation on the mapped feature vectors of the normal data of each modality based on the modal confidence of the normal data of each modality to obtain the baseline feature vector of the target device in a normal state.

[0073] In one embodiment, the step of comparing the multimodal fusion feature vector of the target device at each time step with the reference feature vector, and determining anomalies in the target device based on the comparison results, includes: performing statistical analysis on the reference feature vector of the target device at each time step, establishing a Gaussian distribution model of the reference feature vector, and obtaining the mean μ and variance σ of the Gaussian distribution of the reference feature vector; and calculating the distance between the mean μ of the Gaussian distribution of the multimodal fusion feature vector of the target device at each time step and the reference feature vector. If there is a distance If the variance σ of the Gaussian distribution of the baseline eigenvector is greater than or equal to the variance of the baseline eigenvector, the target device is determined to be in an abnormal state. In one embodiment, when an abnormal state is determined, distance can be used as a criterion. Determine the anomaly level. If the distance... If the distance is greater than or equal to σ and less than 2σ, it is judged as the first level of anomaly; if the distance is... If the distance is greater than or equal to 2σ and less than 3σ, it is classified as the second level of anomaly; If the anomaly is greater than or equal to 3σ, it is classified as the third level of abnormality. The degree of abnormality in the third level of abnormality is higher than that in the second level of abnormality, and the degree of abnormality in the second level of abnormality is higher than that in the first level of abnormality.

[0074] In the aforementioned equipment operation and maintenance method based on a multimodal large model, for each time point, the degree of multimodal conflict and the degree of modal anomaly deviation of each modality of the target equipment at that time point are determined based on the difference between the historical data of each modality and the corresponding standard value at that time point. Based on the degree of multimodal conflict and the degree of modal anomaly deviation of each modality, the modal credibility of each modality at that time point is determined. Then, the feature vectors of the historical data of each modality at that time point are mapped to a unified semantic space. Based on the modal credibility of each modality, the mapped feature vectors of each modality are weighted and summed to obtain the multimodal fusion feature vector at that time point. This method takes into account the differences in the contribution of different modal data, so that the high-credibility modality contributes more to the state representation, while the low-credibility modality is suppressed, thereby improving the robustness of the multimodal fusion result, avoiding misjudgment of the overall state of the equipment due to the data anomaly of a certain modality, improving the fault tolerance of noisy data and abnormal data, and enabling more accurate judgment of equipment anomalies. Furthermore, this method has strong versatility and scalability, providing effective technical support for the in-depth application of multimodal data fusion in the industrial field.

[0075] In one embodiment, see Figure 2 The historical data of each mode of the target device is time-aligned, including the following steps:

[0076] Step 201: Determine the time sensitivity of each mode based on the changes in historical data of each mode of the target device.

[0077] Among them, time sensitivity is used to measure how sensitive modal data is to changes over time.

[0078] It is understandable that, due to the significant differences in sampling frequencies between different modal data, directly aggregating data according to a fixed time window would result in the loss of key information. For example, if the sampling frequency of sensor data is 1Hz and the sampling frequency of logs is 1 minute / log, the sensor data would often change abruptly due to the timing of log generation. Therefore, before aligning the time windows, it is necessary to analyze the importance of each modal data within the time window in order to achieve accurate alignment.

[0079] In one embodiment, for each mode, the slope of the historical data of the target device in that mode at each acquisition time is calculated, and the time sensitivity of each mode is determined based on the slope at each acquisition time. Here, acquisition time refers to the moment when data is acquired.

[0080] Step 202: Divide the historical data of each modality into several time windows according to the basic time window length.

[0081] The base time window length is a preset time window length that can cover at least one historical data point for each modality. The length of each segmented time window conforms to the base time window length.

[0082] In one embodiment, the base time window length can be taken as the least common multiple of the acquisition frequencies of each mode.

[0083] Step 203: For each historical data point, determine the time alignment weight of the historical data within the time window based on its relative position within the time window and the time sensitivity of the modality to which the historical data belongs.

[0084] In one embodiment, the closer historical data is to the edge of its time window, the smaller its corresponding time alignment weight; conversely, the closer historical data is to the center of its time window, the larger its corresponding time alignment weight. The higher the time sensitivity of the modality to which the historical data belongs, the greater its time alignment weight within the time window; conversely, the lower the time sensitivity, the smaller its time alignment weight within the time window.

[0085] In one embodiment, for each historical data point, the distance from the center of the time window is determined based on the difference between the original timestamp of the historical data and the timestamp of the midpoint of the time window to which the historical data belongs, thus measuring the relative position of the historical data within its respective time window. The distance from the center of the time window measures how far the historical data is from the center of the window within its respective time window.

[0086] Step 204: Based on the time alignment weight of each historical data, perform time alignment on the historical data of each modality of the target device.

[0087] In one embodiment, the costs between historical data of each modality are weighted according to the time alignment weight of each historical data, and the cumulative cost between historical data of each modality is calculated based on the weighted cost using a dynamic time warping algorithm. The historical data of each modality of the target device are then time-aligned based on the cumulative cost.

[0088] In the above embodiments, based on the relative position of historical data within its time window and the time sensitivity of the modality to which the historical data belongs, the time alignment weight of historical data within the time window can be accurately determined. Based on the time alignment weight of each historical data, the historical data of each modality of the target device can be time aligned, thereby achieving high-precision time alignment.

[0089] In one embodiment, determining the time sensitivity of each mode based on the changes in historical data of each mode of the target device includes: calculating the slope of the historical data of the target device in each mode at each acquisition time for each mode; and determining the time sensitivity of the mode based on the mean of the absolute values ​​of the slopes at each acquisition time.

[0090] In one embodiment, the mean of the absolute values ​​of the slopes corresponding to each mode can be normalized, and the normalization result can be used as the time sensitivity of each mode. The value range of the normalization result is [0,1]. For example, the normalization process can be performed using a linear mapping.

[0091] In the above embodiments, since the larger the absolute value of the slope at a certain acquisition time, the greater the change in historical data at that acquisition time, the average of the absolute values ​​of the slope at each acquisition time can be used to measure the sensitivity of historical data to time changes in that mode, and the time sensitivity of that mode can be accurately determined.

[0092] In one embodiment, the relative position includes the degree of distance from the center of the time window. For each historical data point, the time alignment weight within the time window is determined based on the relative position of the historical data within its time window and the time sensitivity of the modality to which the historical data belongs. This includes: for each historical data point, determining the degree of distance from the center of the time window based on the difference between the original timestamp of the historical data and the timestamp of the midpoint of the time window to which the historical data belongs; and determining the time alignment weight within the time window based on the degree of distance from the center of the time window and the time sensitivity of the modality to which the historical data belongs.

[0093] The degree of distance from the center of the time window is negatively correlated with the difference between the original timestamp of the historical data and the timestamp of the middle moment of the time window to which the historical data belongs. That is, the greater the difference between the original timestamp of the historical data and the timestamp of the middle moment of the time window to which the historical data belongs, the greater the distance from the center of the time window; the smaller the difference between the original timestamp of the historical data and the timestamp of the middle moment of the time window to which the historical data belongs, the smaller the distance from the center of the time window.

[0094] The time alignment weight of historical data within a time window is negatively correlated with the degree of distance from the center of the time window. The time alignment weight of historical data within a time window is positively correlated with the time sensitivity of the modality to which the historical data belongs.

[0095] In one embodiment, the difference between the original timestamp of the historical data and the timestamp of the middle moment of the time window to which the historical data belongs can be calculated using methods such as difference or ratio.

[0096] In one embodiment, the difference between the original timestamp of the historical data and the timestamp of the middle moment of the time window to which the historical data belongs is calculated, and the degree of distance from the center of the time window is determined based on the ratio between this difference and the length of the base time window (i.e. the length of the time window to which the historical data belongs).

[0097] In one embodiment, the time alignment weight of the historical data within the time window can be determined by subtracting the distance from the center of the time window from 1, and then multiplying the difference by the time sensitivity of the modality to which the historical data belongs. The formula is as follows:

[0098]

[0099] in, Representing modes Historical data Time alignment weights within a time window. Representing modes Historical data The original timestamp. Representing historical data The timestamp of the middle moment of the time window to which it belongs. Indicates the length of the base time window. Representing modes Time sensitivity.

[0100] It's understandable that when aligning historical data from different modalities, the time alignment weights need to be adjusted based on the modality's sensitivity to time. For example, log data is event-triggered and strongly correlated with faults, making it more time-sensitive; while temperature and humidity data change slowly over time, resulting in lower time sensitivity. Therefore, calculating the time sensitivity of different modalities is crucial. This approach differentiates the importance of time for different modalities, avoiding the dilution of data for more time-sensitive modalities by treating all modalities equally. Furthermore, by unifying data from different sampling frequencies into a time window of a baseline length, the difference between the original timestamps of historical data and the timestamps of the midpoint of the corresponding time window is analyzed. Data that is far from the center of the time window is penalized, while key data that is close to the center within the time window is highlighted. For example, sensor data located at the edge of the time window will have a lower time alignment weight, while sensor data at the moment the log is generated will have a higher time alignment weight.

[0101] In the above embodiments, the time alignment weight of historical data within a time window can be accurately determined based on the degree of distance between the center of the time window of the historical data and the time sensitivity of the modality to which the historical data belongs.

[0102] In one embodiment, see Figure 3 For each time point, based on the differences between historical data and corresponding standard values ​​for each mode at that time point, the degree of multimodal conflict and the modal anomaly deviation of each mode at that time point are determined, including the following steps:

[0103] Step 301: For each time moment, calculate the difference between the historical data of each modality and the corresponding standard value at that time moment, and record it as the target difference.

[0104] It is understandable that, for any given moment, the greater the difference in the target of the modality, the higher the degree of anomaly reflected by that modality at that moment.

[0105] In one embodiment, the difference between historical data and the corresponding standard value can be calculated using methods such as difference or ratio.

[0106] Step 302: Determine the number of modes whose target differences are greater than the anomaly threshold, and use this number as the consensus mode count.

[0107] Among them, the number of consensus modes represents the number of modes that have a consensus on the state of the target device.

[0108] Step 303: Determine the multimodal conflict level of the target device at any given time based on the degree of dispersion of the target differences of each modality and the number of consensus modalities.

[0109] In one embodiment, the degree of multimodal conflict of the target device at any given time is positively correlated with the degree of dispersion of the target differences of each mode of the target device at that time, and negatively correlated with the number of consensus modes at that time.

[0110] In one embodiment, the dispersion of the target differences between the various modalities can be calculated using methods such as variance or standard deviation.

[0111] In one embodiment, the proportion of consensus modes to the total number of modes is calculated, then 1 is subtracted from the proportion, and the multimodal conflict level of the target device is determined by multiplying the result of the subtraction with the dispersion of the target differences of each mode.

[0112] In one embodiment, the degree of multimodal conflict of the target device can be determined according to the following formula:

[0113]

[0114] in, Indicates time The degree of multimodal conflict of the target device. Indicates time Lower mode The difference in objectives, i.e., time Lower mode Historical data and modalities The difference between the standard values. Indicates time The mean of the target differences for each modality. This represents the total number of modes. Indicates time The standard deviation of the target differences for each modality. This indicates the abnormal threshold. This represents the number of consensus modes, i.e., the number of modes whose objective differences exceed the anomaly threshold.

[0115] Understandable. This is a consensus penalty term. When most modalities are anomalous (high consensus), this term is small, reducing multimodal conflict. When only a few modalities are anomalous (low consensus), this term is large, increasing multimodal conflict. Multiplying the dispersion of the target difference by this consensus penalty term follows this logic: even with high dispersion (large differences between modalities), if most modalities are anomalous (small consensus penalty term value), the multimodal conflict will ultimately be suppressed, as this may indicate a serious failure affecting multiple components. Conversely, if only one modality is anomalous while others are normal, both the dispersion and consensus penalty term values ​​are high, resulting in a high level of multimodal conflict, suggesting the system's anomalousness may be unreliable.

[0116] Step 304: For each mode, determine the modal anomaly deviation degree of the mode at any given time based on the difference between the target difference of the mode and the mean of the target difference.

[0117] In one embodiment, the absolute value of the difference between the target difference of the modality and the mean of the target difference can be calculated. Then, the modality anomaly deviation degree of the modality can be determined based on the ratio of the absolute value of this difference to the mean of the target difference. The formula is as follows:

[0118]

[0119] in, Indicates time Lower mode Modal anomaly deviation. Indicates time Lower mode The difference in objectives, i.e., time Lower mode Historical data and modalities The difference between the standard values. Indicates time The mean of the target differences for each modality. Indicates time All The target difference mean of each modality. 0.1 represents a minimum constant to avoid cases where the denominator is 0.

[0120] It is understandable that in the above formula Characterization mode At any moment The modal anomaly deviation is calculated as the absolute deviation relative to the average anomaly performance. By adding the mean to the denominator, the obtained modal anomaly deviation is a relative ratio rather than an absolute value. This means that when overall anomalies are high, the same absolute difference will be considered a smaller and more reasonable modal anomaly deviation.

[0121] In the above embodiments, based on the dispersion of the target differences of each modality and the number of consensus modalities, the multimodal conflict degree of the target device at any given time can be accurately determined, thereby accurately measuring the magnitude of the divergence in the anomalies reflected by each modality. Since when there is an actual device anomaly, it is also necessary to determine which modality is the outlier evidence, i.e., at this time, the majority modality is more trusted than the individual modality performance, it is necessary to further determine the modal anomaly deviation degree of each modality at any given time based on the difference between the modality's target difference and the mean of the target difference.

[0122] In one embodiment, see Figure 4 Based on the degree of multimodal conflict and the modal anomaly deviation of each mode, the modal confidence of each mode at a given time is determined, including the following steps:

[0123] Step 401: Determine the modal dispute degree corresponding to each mode at each time point based on the degree of multimodal conflict and the modal anomaly deviation degree of each mode.

[0124] Modal controversy level measures the degree of controversy surrounding the historical data of any given modality. A higher modal controversy level indicates that the historical data for that modality is less reliable and more likely to be a source of noise or erroneous data that needs to be downweighted.

[0125] In one embodiment, the outlier degree of each mode is determined based on the modal anomaly deviation degree of each mode. For each mode individually, the modal controversy degree is determined based on the multimodal conflict degree and the outlier degree of that mode. The modal controversy degree of a mode is positively correlated with the multimodal conflict degree and also positively correlated with the outlier degree of that mode.

[0126] Among them, outlier degree is used to measure the degree to which the modal anomaly deviation of any mode deviates from the modal anomaly deviation of the other modes.

[0127] Step 402: Determine the modal credibility of each modality at each time point based on the modal controversy degree of each modality at each time point.

[0128] The modality credibility of a modality is negatively correlated with the modality controversy of that modality.

[0129] In one embodiment, modal confidence can be determined according to the following formula:

[0130]

[0131] in, Indicates time Lower mode Modal reliability. Indicates time Lower mode Modal controversy level.

[0132] It can be understood that the smaller the modal controversy in the above formula, the higher the modal credibility. When the modal controversy is close to 0, the modal credibility is close to 1 and reaches its maximum value. When the modal controversy is larger, the modal credibility is lower.

[0133] In the above embodiments, based on the degree of multimodal conflict and the degree of modal anomaly deviation of each modality, the modal controversy degree corresponding to each modality at each time can be accurately determined. Then, based on the modal controversy degree of each modality at each time, the modal credibility degree corresponding to each modality at each time can be accurately determined.

[0134] In one embodiment, determining the modal controversy level of each modality at a given time based on the degree of multimodal conflict and the modal anomaly deviation of each modality includes: determining the outlier degree of each modality based on the difference between the modal anomaly deviation of the modality and the mean of the modal anomaly deviation of the remaining modalities excluding the modality; and determining the modal controversy level of the modality at a given time based on the degree of multimodal conflict and the modal anomaly deviation.

[0135] In one embodiment, the difference between the modal anomaly deviation of a mode and the mean of the modal anomaly deviations of the remaining modes can be calculated by means of a ratio or difference.

[0136] In one embodiment, the modal controversy level of a mode at a given time is determined by the product of the multimodal conflict level and the outlier level of the mode at that time.

[0137] It's understandable that the modality controversy level is determined by multiplying the outlier level of a modality by the multimodal conflict level of the target device. The logic is as follows: In an environment with significant overall conflict, even a small relative deviation will be amplified and considered highly controversial; in an environment with good overall consensus, even a slightly larger relative deviation will have its controversy controlled at a low level. Therefore, the modality controversy level comprehensively reflects the magnitude of the controversy arising in a conflict environment. The higher the modality controversy level, the less reliable the data in that modality is, and the more likely it is to be a source of noise or erroneous data that needs to be downweighted.

[0138] In the above embodiments, the outlier degree of a mode can be accurately determined based on the difference between the modal outlier deviation degree of a mode and the mean of the modal outlier deviation degrees of the other modes. Then, based on the multimodal conflict degree and the outlier degree of a mode, the modal controversy degree of a mode at a given time can be accurately determined.

[0139] In one embodiment, determining the outlier degree of a mode based on the difference between the mode outlier deviation of the mode and the mean of the mode outlier deviations of the remaining modes excluding the mode includes: determining the outlier degree of the mode based on the ratio between the mode outlier deviation of the mode and the mean of the mode outlier deviations of the remaining modes excluding the mode.

[0140] In one embodiment, the outlierness of a mode can be determined according to the following formula:

[0141]

[0142] in, Indicates time Lower mode Modal controversy level. Indicates time Lower mode Modal anomaly deviation.

[0143] Indicates time Lower mode Other modes Modal anomaly deviation. This represents the total number of modes.

[0144] Indicates the mode Other modes Modal anomaly deviation The mean. Indicates time The degree of multimodal conflict of the target device.

[0145] In the above embodiments, the outlier degree of a mode can be accurately determined based on the ratio between the modal outlier deviation of a mode and the mean of the modal outlier deviations of the other modes.

[0146] In one embodiment, the method further includes: for each modality at each time step, determining the contribution of the modality to the anomalous performance at each time step based on the difference between the mapped feature vector of the modality at that time step and the reference feature vector of the modality, and the modality confidence of the modality at that time step; outputting the ranking of the contribution of each modality to the anomalous performance at each time step; and using the ranking of the contribution of each modality to the anomalous performance at each time step to analyze the cause of the anomalous performance.

[0147] The reference feature vector refers to the feature vector of the mode after mapping under normal conditions, which can be obtained by statistical analysis of normal data in the historical data of the mode.

[0148] In one embodiment, the product of the difference between the mapped feature vector of any modality and the reference feature vector of the modality at a given time step and the modality confidence at that time step is calculated and denoted as the target product. Then, the target product of any modality is normalized based on the sum of the target products of all modalities to obtain the contribution of any modality to the anomalous performance. The formula is as follows:

[0149]

[0150] in, Indicates time Lower mode Contribution to abnormal performance. Indicates time Lower mode Modal reliability. Indicates time Lower mode Modal reliability. This represents the total number of modes. Indicates time Lower mode The eigenvectors of the historical data. The eigenvectors of the historical data of the mode at time t. The reference eigenvector of the mode. Indicates time Lower mode The feature vector of historical data. Representing modes The reference feature vector. Representing modes The reference feature vector. Indicates modality The feature mapping function is used to map modes... The feature vectors of the data are mapped to a unified semantic space. Indicates modality The feature mapping function is used to map modes... The feature vectors of the data are mapped to a unified semantic space. Indicates time Lower mode The mapped feature vectors and modes The difference between the reference feature vectors. Indicates time Lower mode The mapped feature vectors and modes The difference between the reference eigenvectors is due to the fact that the formula is a normalization formula, so it loses its normalization meaning when all values ​​are 0. Greater than 0.

[0151] In one embodiment, the contribution of each modality to the abnormal performance at each time point can be ranked and displayed on the terminal. Operations personnel can analyze the cause of the anomaly based on the ranking displayed on the terminal.

[0152] In the above embodiments, for each modality at each time point, the contribution of the modality to the abnormal performance at each time point is determined based on the difference between the mapped feature vector of the modality at that time point and the reference feature vector of the modality, as well as the modality credibility at that time point. The ranking of the contribution of each modality to the abnormal performance at each time point is output. Operation and maintenance personnel can quickly locate the cause of the anomaly based on the ranking, thereby improving the operation and maintenance efficiency.

[0153] See Figure 5 This invention provides a device operation and maintenance system based on a multimodal large model. The system includes a memory and a processor. The memory stores executable program code. The processor calls and runs the executable program code from the memory to perform the following steps: Time-aligning the historical data of each modality of the target device; after time alignment, determining the degree of multimodal conflict and the degree of modal anomaly deviation of each modality at each time point based on the difference between the historical data of each modality at that time point and the corresponding standard value; determining the modal credibility of each modality at that time point based on the degree of multimodal conflict and the degree of modal anomaly deviation of each modality; mapping the feature vectors of the historical data of each modality at that time point to a unified semantic space, and performing a weighted summation of the mapped feature vectors of each modality based on the modal credibility of each modality to obtain the multimodal fusion feature vector at that time point; and judging the anomaly of the target device based on the multimodal fusion feature vector of the target device at each time point.

[0154] In one embodiment, time alignment of historical data for each modality of the target device includes: determining the time sensitivity of each modality based on the changes in historical data for each modality of the target device; dividing the historical data for each modality into several time windows according to the length of a basic time window; for each historical data point, determining the time alignment weight of the historical data within the time window based on the relative position of the historical data within its respective time window and the time sensitivity of the modality to which the historical data belongs; and time aligning the historical data for each modality of the target device according to the time alignment weight of each historical data point.

[0155] In one embodiment, determining the time sensitivity of each mode based on the changes in historical data of each mode of the target device includes: calculating the slope of the historical data of the target device in each mode at each acquisition time for each mode; and determining the time sensitivity of the mode based on the mean of the absolute values ​​of the slopes at each acquisition time.

[0156] In one embodiment, the relative position includes the degree of distance from the center of the time window. For each historical data point, the time alignment weight within the time window is determined based on the relative position of the historical data within its time window and the time sensitivity of the modality to which the historical data belongs. This includes: for each historical data point, determining the degree of distance from the center of the time window based on the difference between the original timestamp of the historical data and the timestamp of the midpoint of the time window to which the historical data belongs; and determining the time alignment weight within the time window based on the degree of distance from the center of the time window and the time sensitivity of the modality to which the historical data belongs.

[0157] In one embodiment, for each time moment, the degree of multimodal conflict of the target device and the modal anomaly deviation of each modality are determined based on the difference between the historical data of each modality at that time moment and the corresponding standard value. This includes: for each time moment, calculating the difference between the historical data of each modality at that time moment and the corresponding standard value, and recording it as the target difference; determining the number of modalities whose target differences are greater than an anomaly threshold, as the number of consensus modalities; determining the degree of multimodal conflict of the target device at that time moment based on the dispersion of the target differences of each modality and the number of consensus modalities; and for each modality, determining the modal anomaly deviation of the modality at that time moment based on the difference between the target difference of the modality and the mean of the target differences.

[0158] In one embodiment, the modal dispute degree corresponding to each modality at time t is determined based on the degree of multimodal conflict and the modal anomaly deviation degree of each modality; and the modal credibility degree corresponding to each modality at time t is determined based on the modal dispute degree of each modality at time t.

[0159] In one embodiment, for each mode, the outlier degree of the mode is determined based on the difference between the mode outlier deviation degree of the mode and the mean of the mode outlier deviation degrees of the other modes excluding the mode; and the mode controversy degree of the mode at a given time is determined based on the multimodal conflict degree and the outlier degree of the mode.

[0160] In one embodiment, the outlier degree of a mode is determined based on the ratio between the mode outlier deviation of the mode and the mean of the mode outlier deviations of the remaining modes excluding the mode.

[0161] In one embodiment, the processor further performs the following steps: for each modality at each time step, based on the difference between the mapped feature vector of the modality at that time step and the reference feature vector of the modality, and the modality confidence of the modality at that time step, determine the contribution of the modality to the abnormal performance at that time step; output the ranking of the contribution of each modality to the abnormal performance at each time step; the ranking of the contribution of each modality to the abnormal performance at each time step is used to analyze the cause of the anomaly.

[0162] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this invention.

[0165] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0166] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A device operation and maintenance method based on a multimodal large model, characterized in that, The method includes: Time-align the historical data of each mode of the target device; After time alignment, for each time point, based on the difference between the historical data of each mode and the corresponding standard value at that time point, the degree of multimodal conflict of the target device and the degree of modal anomaly deviation of each mode at that time point are determined; Based on the degree of multimodal conflict and the degree of modal anomaly deviation of each mode, the modal credibility of each mode at the given time is determined; The feature vectors of historical data of each modality at the specified time are mapped to a unified semantic space. Based on the modality credibility of each modality, the mapped feature vectors of each modality are weighted and summed to obtain the multimodal fusion feature vector at the specified time. Based on the multimodal fusion feature vectors of the target device at each time point, anomaly detection is performed on the target device; Methods for determining the degree of multimodal conflict and the modal anomaly deviation of each mode include: For each time point, calculate the difference between the historical data of each modality and the corresponding standard value at that time point, and record it as the target difference; The number of modes whose target differences exceed the anomaly threshold is determined as the number of consensus modes; The degree of multimodal conflict of the target device at the given time is determined based on the degree of dispersion of the target differences of each modality and the number of consensus modalities. For each mode, the modality anomaly deviation degree of the mode at the given time is determined based on the difference between the target difference of the mode and the mean target difference. Methods for determining the modal reliability of each mode include: Based on the degree of multimodal conflict and the degree of modal anomaly deviation of each modality, the modal dispute degree corresponding to each modality at the given time is determined. Based on the modality controversy degree of each modality at the given time, the modality credibility corresponding to each modality at the given time is determined.

2. The equipment operation and maintenance method based on a multimodal large model according to claim 1, characterized in that, The step of aligning the historical data of each mode of the target device by time includes: The time sensitivity of each mode is determined based on the changes in historical data of each mode of the target device; The historical data for each modality is divided according to the length of the basic time window to obtain several time windows; For each historical data point, the time alignment weight of the historical data within the time window is determined based on its relative position within the time window and the time sensitivity of the modality to which the historical data belongs. Based on the time alignment weight of each historical data, the historical data of each modality of the target device are time aligned; Methods for determining the time sensitivity of each mode include: For each mode, calculate the slope of the historical data of the target device at each acquisition time in the mode; The time sensitivity of the mode is determined based on the average absolute value of the slope at each acquisition time. The relative position includes the degree of distance from the center of the time window; the method for determining the time alignment weight includes: For each historical data point, the degree of distance from the center of the time window of the historical data is determined based on the difference between the original timestamp of the historical data and the timestamp of the middle moment of the time window to which the historical data belongs. The time alignment weight of the historical data within the time window is determined based on the degree of distance from the center of the time window and the time sensitivity of the modality to which the historical data belongs.

3. The equipment operation and maintenance method based on a multimodal large model according to claim 1, characterized in that, The step of determining the modal dispute degree corresponding to each mode at the given time based on the multimodal conflict degree and the modal anomaly deviation degree of each mode includes: For each mode, the outlier degree of the mode is determined based on the difference between the mode outlier deviation of the mode and the mean of the mode outlier deviation of the other modes. Based on the degree of multimodal conflict and the degree of outlier status of the mode, the modal controversy degree of the mode at the given time is determined.

4. The equipment operation and maintenance method based on a multimodal large model according to claim 3, characterized in that, Determining the outlier degree of a mode based on the difference between the mode outlier deviation of the mode and the mean of the mode outlier deviations of all other modes, includes: The outlier degree of a mode is determined by the ratio between the mode outlier deviation of the stated mode and the mean of the mode outlier deviations of all other modes.

5. The equipment operation and maintenance method based on a multimodal large model according to any one of claims 1 to 4, characterized in that, The method further includes: For each modality at each time point, the contribution of the modality to the anomalous behavior at that time point is determined based on the difference between the mapped feature vector of the modality at that time point and the reference feature vector of the modality, as well as the modality confidence at that time point. Output the ranking of the contribution of each modality to the abnormal performance at each time point; the ranking of the contribution of each modality to the abnormal performance at each time point is used to analyze the cause of the abnormality.

6. A device operation and maintenance system based on a multimodal large model, characterized in that, The system includes a memory and a processor; the memory is used to store executable program code. The processor is used to call and run the executable program code from the memory to implement the device operation and maintenance method based on a multimodal large model as described in any one of claims 1 to 5.

Images