Intelligent manufacturing data monitoring method and system based on multi-modal fusion

Abstract

The invention discloses an intelligent manufacturing data monitoring method and system based on multi-modal fusion, and relates to the field of intelligent manufacturing, and the system comprises a collection module which is used for collecting equipment operation data, environment perception data and production process image data in an intelligent manufacturing scene, and adding a sequence mark for each piece of data in each type of data; the storage module is used for receiving the marked data in the acquisition module, and establishing a dynamic hierarchical storage structure according to a preset storage logic in combination with marks so as to store the marked data in the acquisition module; according to the method, the abnormity is judged by calculating the collaborative deviation, the traceability information is recorded, graded early warning is performed according to the abnormity severity degree and the like, the traceability identifier is added, the product percent of pass and the equipment failure rate are predicted in combination with historical data and early warning records, and decision support is provided for the intelligent manufacturing process.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to an intelligent manufacturing data monitoring method and system based on multimodal fusion. Background Technology

[0002] As intelligent manufacturing is undergoing a deep digital upgrade, massive amounts of heterogeneous data, such as sensor time-series data, visual images, and process text, are generated from dimensions such as equipment operation, process parameters, and environmental conditions, which are used for production process and production equipment health monitoring.

[0003] The invention patent application with application number 202210708525.5 discloses a dynamic monitoring and analysis system for production data in intelligent manufacturing. The system aims to solve the problem that "the existing dynamic monitoring and analysis system for production data in intelligent manufacturing simply compares the acquired production data with preset values ​​and judges the production status corresponding to the current production data. It cannot predict the corresponding production data for future time based on the acquired data, and thus analyze the production status corresponding to future time in advance."

[0004] However, in smart manufacturing scenarios, people crave the ability to accurately and predictively grasp the operating status data and production data of smart manufacturing equipment in real time to ensure the stable progress of production tasks.

[0005] To address this, we propose a smart manufacturing data monitoring method and system based on multimodal fusion. Summary of the Invention

[0006] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a smart manufacturing data monitoring method and system based on multimodal fusion, which can effectively solve the problems of the existing technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions;

[0008] This invention discloses an intelligent manufacturing data monitoring system based on multimodal fusion, comprising:

[0009] The system comprises four modules: a data acquisition module (data acquisition module, storage module, and prediction module), a data storage module, and a prediction module. The acquisition module receives the tagged data from the acquisition module and, according to a pre-defined storage logic, establishes a dynamic hierarchical storage structure based on the tags. The visualization module retrieves data from each level of the storage module to construct a smart manufacturing data visualization model that maps parameters to images and time sequences to the environment. The monitoring module monitors the operational status of the smart manufacturing data visualization model, tracks the processing flow and results of multimodal data at each level in real time, and synchronously records the data source, processing step, and time corresponding to anomalies, using these as traceability information. The early warning module receives monitoring results and traceability information from the monitoring module, triggers early warning signals based on these results, and adds an anomaly traceability identifier to the early warning signals. The prediction module extracts historical data stored in the storage module and historical early warning information trigger records from the early warning module to predict trends in product qualification rate and equipment failure rate.

[0010] The data acquisition module is interconnected with a storage module via a wireless network. The storage module is interconnected with a visualization module via a wireless network. The visualization module is interconnected with a monitoring module via a wireless network. The monitoring module is interconnected with an early warning module via a wireless network. The early warning module is interconnected with a prediction module via a wireless network.

[0011] Furthermore, the data acquisition module dynamically adjusts the data acquisition frequency based on the equipment's operating load during operation:

[0012]

[0013] Where: f is the current data acquisition frequency; f0 is the preset basic acquisition frequency; k is the load influence coefficient, k∈(0,1]; P is the current operating load of the equipment; P n This is the rated load of the equipment;

[0014] During the operation of the acquisition module, outlier removal is performed on the raw data. The outlier determination formula is as follows: x i This indicates the currently collected data value. Let t represent the moving average of the first m collected data points, t represent the preset confidence coefficient, and s represent the moving average of the first m collected data points. m This represents the moving standard deviation of the first m collected data points. When a data point meets the outlier determination formula, it is marked as an outlier and removed. Simultaneously, based on... Complete the data for this moment.

[0015] Furthermore, during the operation phase of the storage module, the call frequency of each data is calculated to distinguish between high-frequency call data and low-frequency call data. The call frequency is the ratio of the actual number of times the data is called within a preset time window to the length of the time window.

[0016] When the call frequency is not less than the preset high-frequency threshold, the data is determined to be high-frequency call data; when the call frequency does not exceed the preset low-frequency threshold, it is determined to be low-frequency call data; when the call frequency is between the preset high-frequency threshold and the preset low-frequency threshold, the original storage level of the data is maintained.

[0017] Data storage hierarchy is dynamically adjusted based on the frequency of access:

[0018] Set the hierarchical adjustment trigger condition: ΔF = |F t2 -F t1 |>ΔF0,F t1 Indicates the call frequency of the previous time window, F t2 ΔF0 represents the call frequency for the current time window, and ΔF0 represents the preset call frequency change threshold. When the above triggering conditions are met, the data will be migrated to the corresponding high-frequency or low-frequency storage level.

[0019] Furthermore, the process by which the visualization module constructs an intelligent manufacturing data visualization model includes:

[0020] Establish a mapping relationship between equipment operating parameters and production image data, and express the strength of the mapping relationship as follows:

[0021]

[0022] Where: M p-i For parameter - image mapping intensity; T is the mapping calculation window length; P t The values ​​of the device operating parameters at time t; The mean of the parameters; I t Let be the image feature value at time t; The mean value of image features;

[0023] When a system user enters a specific threshold range of device operating parameters on the system's preset control panel, the visualization module automatically filters out the mapping intensity M. p-i For all moments falling within the threshold range, the system synchronously locates and displays the corresponding production image segments, annotating the image segments with the specific parameter values ​​for that moment. When a user selects a production image area on the control panel via clicks or bounding boxes, the visualization module automatically identifies the time range and image characteristics corresponding to that area. t Based on M p-iMatch the equipment operating parameters associated with the image features, and synchronously generate and display the parameter change curve within the time range. The horizontal axis of the curve represents time, the vertical axis represents the parameter value, and the parameter peak and valley values ​​at key time points are marked.

[0024] Establish a mapping relationship between time series data and environmental data, and express the mapping bias as follows:

[0025]

[0026] in,

[0027] In the formula: E t-e For time-environment mapping bias; T t ′ represents the standardized time series data at time t; E t ′ represents the standardized environmental data at time t; T t E t The original time series values ​​and original environmental values ​​are given; minT and maxT are the values ​​relative to the original time series value T. t Minimum and maximum values ​​of time-series data under the same statistical dimension; minE, maxE are the values ​​relative to the original environmental value E. t Minimum and maximum values ​​of environmental data under the same statistical dimension;

[0028] The visualization model is based on M p-i and E t-e Dynamically adjust mapping precision:

[0029] When M p-i When the intensity is less than the preset mapping intensity threshold, increase the sampling density of the corresponding device operating parameters and the dimension of image feature extraction.

[0030] When E t-e When the deviation exceeds a preset mapping threshold, the statistical window for time-series data and the granularity of environmental data collection are reduced; when M... p-i Greater than or equal to the preset mapping strength threshold and E t-e If the mapping deviation is less than or equal to the preset mapping deviation threshold, the current mapping configuration is maintained.

[0031] Among them, I t , It can be either the change in workpiece contour or the clarity of texture of equipment components.

[0032] Furthermore, the monitoring module determines anomalies by calculating the coordinated deviation of the data:

[0033]

[0034] In the formula: D is the coordination deviation value; n is the number of data types; a iv represents the bias weight of the i-th type of data; b and c represent the parameter-image mapping anomaly weight and the time-series-environment mapping anomaly weight, respectively; i,t This represents the current value of the i-th data type. M′ is the historical mean of the i-th data category; p-i For M p-i Abnormal fluctuation value; E′ t-e For E t-e Abnormal fluctuation values;

[0035] in, t represents the current time;

[0036] When D exceeds a preset anomaly threshold, the monitoring module determines that an anomaly exists in the current data processing flow and synchronously records the mapping type corresponding to the anomaly. The mapping type includes: parameter-image or time series-environment, as well as the source of the deviation and the timestamp. The monitoring module updates in real time. The update cycle should be consistent with the time window in the hierarchical adjustment trigger conditions;

[0037] in, These represent the historical mean of parameter-image mapping intensity and the historical mean of time-series-environment mapping deviation, respectively.

[0038] Furthermore, during the operation of the early warning module, early warning signals are triggered in stages based on the severity and scope of the anomaly:

[0039]

[0040] In the formula: L is the warning level; S, R, and M are the severity of the anomaly, the scope of the anomaly's impact, and the degree of disruption of the mapping relationship, respectively; α, β, and γ are the weights of severity, scope of impact, and disruption of the mapping relationship, respectively; δ is the level normalization coefficient; round(·) is the rounding function;

[0041] When L is 1, the warning module triggers a level 1 warning, corresponding to a minor local anomaly; when L is 2, it triggers a level 2 warning, corresponding to a moderate regional anomaly; when L is 3, it triggers a level 3 warning, corresponding to a global anomaly.

[0042] The abnormal source identification added to the early warning signal by the early warning module includes mapping type, deviation source and timestamp information, and the identification format corresponds to the sequence mark of the acquisition module.

[0043] in, N aff N represents the number of equipment and processes affected by the anomaly. total This represents the total number of equipment and processes in the system. δ is a preset value used to constrain the value of L to be within the range of 1-3.

[0044] Furthermore, during the operation of the prediction module, the product qualification rate prediction follows the following:

[0045]

[0046] In the formula: Y t+1 Y represents the product qualification rate for the next forecast period. t λ represents the actual product qualification rate for the current period; λ is the equipment failure rate impact coefficient, λ∈(0,1]; F t The equipment failure rate for the current period; μ is the impact coefficient of the early warning record, μ∈(0,1]; W t θ represents the percentage of warnings issued in the current period; θ is the mapping quality impact coefficient, θ∈(0,1]; This represents the average mapping deviation for the current period.

[0047] The prediction module predicts the trend of equipment failure rate according to:

[0048]

[0049] In the formula: F t+1 For the equipment failure rate in the next forecast period; F t λ represents the actual equipment failure rate for the current period. F λ is the equipment load influence coefficient. F ∈(0,1];P t P represents the average operating load of the equipment during the current cycle. n The rated load of the equipment; θ F For the parameter - image mapping intensity influence coefficient, θ F ∈(0,1]; The average parameter of the current period is the image mapping intensity; μ F μ is the impact coefficient for fault early warning records. F ∈(0,1];W F,t This represents the percentage of fault-related warnings during the current period.

[0050] The prediction period of the prediction module is positively correlated with the length T of the mapping calculation window; that is, the smaller T is, the shorter the prediction period.

[0051] Furthermore, during system operation, the efficiency of data interaction between modules is simultaneously optimized based on data transmission priority:

[0052]

[0053] In the formula: P trans E represents the data transmission priority; E represents the data urgency level, with a value of 1 for early warning data, 0.5 for historical data, and 0.8 for real-time data acquisition; D date D represents the current size of the transmitted data.max L represents the maximum amount of data that can be transmitted in a single system session. mod The current processing load of the data receiving module is the ratio of the number of tasks currently pending to the maximum number of tasks that the module can process; L max φ is the maximum processing load threshold of the data receiving module; M is the mapping correlation weight. rel To determine the correlation between data and bidirectional mapping, the parameter-image correlation data is set to 0.9, the time-series-environment correlation data is set to 0.8, and other data is set to 0.5; ω1, ω2, and ω3 are the urgency weight, data size weight, and module load weight, respectively.

[0054] Finally, the data transmission between each module is processed according to P. trans Sort by high to low, then transmit in order.

[0055] On the other hand, a smart manufacturing data monitoring method based on multimodal fusion includes:

[0056] The system collects equipment operation data, environmental perception data, and production process image data in intelligent manufacturing scenarios. Sequence tags are added to each data point, and the collection frequency is dynamically adjusted based on equipment load. Outliers are removed and supplemented in the collected data. Tagged data is received, and a dynamic hierarchical storage structure is established based on a preset logic: high-frequency data is stored at the upper level, and low-frequency data at the lower level. The data storage hierarchy is dynamically adjusted by calculating the data retrieval frequency within a preset time window. Data from each level is retrieved to establish mapping relationships between equipment operation parameters and production image data, and between time-series data and environmental data, to construct a visualization model. The model's mapping accuracy is dynamically adjusted based on mapping strength and deviation. The system monitors the operational status of the visualization model, determining whether the data processing flow is abnormal by calculating the data's collaborative deviation, and synchronously recording the source information of the abnormal data source, processing stage, time, and mapping type. Based on the severity of the abnormality, its impact range, and the degree of disruption of the mapping relationship, graded early warning signals are triggered, with an abnormal source identification corresponding to the sequence tag attached to the early warning signal. Historical stored data and historical early warning information trigger records are extracted, and combined with parameters such as the equipment failure rate impact coefficient and the early warning record impact coefficient, the system calculates and predicts the product qualification rate and equipment failure rate for the next cycle.

[0057] Compared with the known prior art, the technical solution provided by this invention has the following beneficial effects:

[0058] This invention provides a data monitoring method and system for intelligent manufacturing based on multimodal fusion. During execution, this method and system can accurately collect multiple types of data and add sequence tags. It dynamically adjusts the collection frequency based on the equipment operating load, and simultaneously removes outliers to complete the data, ensuring accurate and efficient data collection. It also constructs a dynamic hierarchical storage structure according to the call frequency, adjusts the data storage hierarchy in real time, and improves data access efficiency. At the same time, by constructing a visualization model and dynamically optimizing the mapping accuracy, the data presentation becomes more intuitive. It determines anomalies by calculating collaborative deviations, records traceability information, and provides graded early warnings based on the severity of anomalies, attaching traceability identifiers. Furthermore, it combines historical data and early warning records to predict product qualification rate and equipment failure rate, providing decision support for the intelligent manufacturing process. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0060] Figure 1 This is a schematic diagram of the structure of an intelligent manufacturing data monitoring system based on multimodal fusion.

[0061] Figure 2 This is a flowchart illustrating a smart manufacturing data monitoring method based on multimodal fusion. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0063] The present invention will be further described below with reference to embodiments.

[0064] Example 1:

[0065] This embodiment presents an intelligent manufacturing data monitoring system based on multimodal fusion, such as... Figure 1 As shown, it includes:

[0066] The data acquisition module is used to collect equipment operation data, environmental perception data, and production process image data in intelligent manufacturing scenarios, and to add sequence markers to each data item in each type of data.

[0067] During the operation of the data acquisition module, the data acquisition frequency is dynamically adjusted according to the equipment's operating load.

[0068]

[0069] Where: f is the current data acquisition frequency; f0 is the preset basic acquisition frequency; k is the load influence coefficient, k∈(0,1]; P is the current operating load of the equipment; P n This is the rated load of the equipment;

[0070] The above formula uses the preset basic acquisition frequency as a benchmark, combines the ratio between the current operating load of the equipment and the rated load, and introduces the load influence coefficient to determine the current data acquisition frequency. It can dynamically adapt the acquisition frequency according to the actual operating load of the equipment, avoid the problem of insufficient or redundant data acquisition caused by load fluctuations under a fixed frequency, and make the data acquisition both meet the monitoring needs and take into account resource efficiency.

[0071] During the data acquisition module's operation, outlier removal is performed synchronously on the acquired raw data. The outlier determination formula is as follows: x i This indicates the currently collected data value. Let t represent the moving average of the first m collected data points, t represent the preset confidence coefficient, and s represent the moving average of the first m collected data points. m This represents the moving standard deviation of the first m collected data points. When a data point meets the outlier determination formula, it is marked as an outlier and removed. Simultaneously, based on... Complete the data for this moment;

[0072] Among them, the more significant the impact of equipment operating load changes on production accuracy or safety, the larger the value of k; the more gradual the impact of equipment operating load changes on production status, the smaller the value of k.

[0073] The storage module is used to receive the marked data from the acquisition module, and establish a dynamic hierarchical storage structure based on the tags according to the preset storage logic, so as to store the marked data from the acquisition module.

[0074] During the operation of the storage module, the call frequency of each data is calculated to distinguish between high-frequency and low-frequency data. The call frequency is the ratio of the actual number of times the data is called within a preset time window to the length of the time window.

[0075] When the call frequency is not less than the preset high-frequency threshold, the data is determined to be high-frequency call data; when the call frequency does not exceed the preset low-frequency threshold, it is determined to be low-frequency call data; when the call frequency is between the preset high-frequency threshold and the preset low-frequency threshold, the original storage level of the data is maintained.

[0076] Data storage hierarchy is dynamically adjusted based on the frequency of access:

[0077] Set the hierarchical adjustment trigger condition: ΔF = |F t2 -F t1 |>ΔF0,F t1 Indicates the call frequency of the previous time window, F t2 The current time window is the call frequency, and ΔF0 is the preset call frequency change threshold. When the above triggering conditions are met, the data will be migrated to the corresponding high-frequency or low-frequency storage level.

[0078] The visualization module is used to retrieve data stored at various levels in the storage module and construct intelligent manufacturing data visualization models of parameter-mapped images and time-series-mapped environments.

[0079] The process of building a smart manufacturing data visualization model using the visualization module includes:

[0080] Establish a mapping relationship between equipment operating parameters and production image data, and express the strength of the mapping relationship as follows:

[0081]

[0082] Where: M p-i For parameter - image mapping intensity; T is the mapping calculation window length; P t The values ​​of the device operating parameters at time t; The mean of the parameters; I t Let be the image feature value at time t; The mean value of image features;

[0083] It should be noted that M p-i When applied to the trend prediction of the following equipment failure rates, its absolute value is used in the calculation of the trend prediction of equipment failure rates.

[0084] The above formula uses the length of the mapping calculation window as the statistical range. By quantifying the difference between the equipment operating parameter value and the mean parameter value at time t, and the difference between the image feature value and the mean image feature value at time t, it determines the strength of the mapping relationship between the equipment operating parameters and the production image data. It transforms the originally difficult-to-quantify parameter-image association into specific values, providing a precise basis for subsequent screening of production image segments at corresponding times or matching associated equipment operating parameters based on the mapping strength, and improving the operability of multimodal data association.

[0085] When a system user enters a specific threshold range of device operating parameters on the system's preset control panel, the visualization module automatically filters out the mapping intensity M. p-iFor all moments falling within the threshold range, the system synchronously locates and displays the corresponding production image segments, annotating the image segments with the specific parameter values ​​for that moment. When a user selects a production image area on the control panel via clicks or bounding boxes, the visualization module automatically identifies the time range and image characteristics corresponding to that area. t Based on M p-i Match the equipment operating parameters associated with the image features, and synchronously generate and display the parameter change curve within the time range. The horizontal axis of the curve represents time, the vertical axis represents the parameter value, and the parameter peak and valley values ​​at key time points are marked.

[0086] Establish a mapping relationship between time series data and environmental data, and express the mapping bias as follows:

[0087]

[0088] in,

[0089] In the formula: E t-e For time-environment mapping bias; T t ′ represents the standardized time series data at time t; E t ′ represents the standardized environmental data at time t; T t E t The original time series values ​​and original environmental values ​​are given; minT and maxT are the values ​​relative to the original time series value T. t Minimum and maximum values ​​of time-series data under the same statistical dimension; minE, maxE are the values ​​relative to the original environmental value E. t Minimum and maximum values ​​of environmental data under the same statistical dimension;

[0090] The above formula combines the original time series value and the original environmental value at time t with the maximum and minimum values ​​of the time series data and the environmental data under the same statistical dimension to obtain the standardized time series data and environmental data. Then, the mapping deviation is calculated based on the standardized data. The standardization process eliminates the comparison interference caused by the different dimensions of the time series data and environmental data, so that the mapping deviation can more accurately reflect the degree of matching between the two, and provides a reliable reference for adjusting the statistical window of the time series data and the granularity of environmental data collection according to the deviation.

[0091] The visualization model is based on M p-i and E t-e Dynamically adjust mapping precision:

[0092] When M p-i When the intensity is less than the preset mapping intensity threshold, increase the sampling density of the corresponding device operating parameters and the dimension of image feature extraction.

[0093] When E t-eWhen the deviation exceeds a preset mapping threshold, the statistical window for time-series data and the granularity of environmental data collection are reduced; when M... p-i Greater than or equal to the preset mapping strength threshold and E t-e If the mapping deviation is less than or equal to the preset mapping deviation threshold, the current mapping configuration is maintained.

[0094] Among them, I t , It can be either the change in workpiece contour or the clarity of texture of equipment components;

[0095] The monitoring module is used to monitor the operation status of the intelligent manufacturing data visualization model, track the processing flow and results of multimodal data at each level in real time, and synchronously record the data source, processing link and time corresponding to the anomaly, and record the data source, processing link and time corresponding to the anomaly as traceability information.

[0096] The monitoring module determines anomalies by calculating the coordination deviation of the data.

[0097]

[0098] In the formula: D is the coordination deviation value; n is the number of data types; a i v represents the bias weight of the i-th type of data; b and c represent the parameter-image mapping anomaly weight and the time-series-environment mapping anomaly weight, respectively; i,t This represents the current value of the i-th data type. M′ is the historical mean of the i-th data category; p-i For M p-i Abnormal fluctuation value; E′ t-e For E t-e Abnormal fluctuation values;

[0099] in, t represents the current time;

[0100] The above formula integrates the number of data types in the system, sets a deviation weight for each type of data, and sets anomaly weights for parameter-image mapping anomalies and time-series-environment mapping anomalies. By calculating the difference between the current value and the historical mean of each type of data, the abnormal fluctuation value of parameter-image mapping intensity, and the abnormal fluctuation value of time-series-environment mapping deviation, and integrating them according to weights, a collaborative deviation value is obtained. This multi-dimensional integration method can comprehensively consider the fluctuation of various types of data and the anomalies of key mapping relationships, avoid misjudgments caused by judging based on a single data or a single mapping anomaly, and improve the accuracy of anomaly judgment.

[0101] When D exceeds the preset anomaly detection threshold, the monitoring module determines that there is an anomaly in the current data processing flow and synchronously records the mapping type corresponding to the anomaly. The mapping type includes: parameter-image or time series-environment, as well as the source of the deviation and the timestamp; the monitoring module updates in real time. The update cycle should be consistent with the time window in the hierarchical adjustment trigger conditions;

[0102] in, These represent the historical mean of parameter-image mapping intensity and the historical mean of time-series-environment mapping bias, respectively.

[0103] The early warning module is used to receive monitoring results and source tracing information from the monitoring module, trigger early warning signals in a graded manner based on the monitoring results and source tracing information, and attach an abnormal source tracing identifier to the early warning signal;

[0104] During the operation of the early warning module, early warning signals are triggered in stages based on the severity and scope of the anomaly:

[0105]

[0106] In the formula: L is the warning level; S, R, and M are the severity of the anomaly, the scope of the anomaly's impact, and the degree of disruption of the mapping relationship, respectively; α, β, and γ are the weights of severity, scope of impact, and disruption of the mapping relationship, respectively; δ is the level normalization coefficient; round(·) is the rounding function;

[0107] The above formula comprehensively considers the severity of the anomaly, the scope of its impact, and the degree of disruption of the mapping relationship, assigning weights to each of the three factors. It then calculates the basic warning value by combining the level normalization coefficient, and finally determines the warning level through a rounding function. This multi-dimensional assessment ensures that the warning level accurately reflects the actual impact of the anomaly, and the tiered warning system facilitates staff to take targeted measures according to different levels.

[0108] When L is 1, the warning module triggers a level 1 warning, which corresponds to a minor local anomaly; when L is 2, it triggers a level 2 warning, which corresponds to a moderate regional anomaly; when L is 3, it triggers a level 3 warning, which corresponds to a global anomaly, and each warning level has a corresponding pre-set response measure.

[0109] The abnormal source identification added to the early warning signal by the early warning module includes mapping type, deviation source and timestamp information, and the identification format corresponds to the sequence mark of the acquisition module;

[0110] in, N aff N represents the number of equipment and processes affected by the anomaly. total This represents the total number of equipment and processes in the system. δ is a preset value used to constrain the value of L to be within the range of 1-3.

[0111] The prediction module is used to extract historical data stored in the storage module and historical warning information trigger records from the warning module to predict the trends of product qualification rate and equipment failure rate.

[0112] During the prediction module's operation phase, the product qualification rate prediction follows the following rules:

[0113]

[0114] In the formula: Y t+1 Y represents the product qualification rate for the next forecast period. t λ represents the actual product qualification rate for the current period; λ is the equipment failure rate impact coefficient, λ∈(0,1]; F t The equipment failure rate for the current period; μ is the impact coefficient of the early warning record, μ∈(0,1]; W t θ represents the percentage of warnings issued in the current period; θ is the mapping quality impact coefficient, θ∈(0,1]; This represents the average mapping deviation for the current period.

[0115] The above formula is based on the actual product qualification rate of the current period. It introduces the equipment failure rate influence coefficient, the current period equipment failure rate, the early warning record influence coefficient, the proportion of early warning times in the current period, the mapping quality influence coefficient, and the current period average mapping deviation. It integrates these factors closely related to product quality to calculate the predicted value, so that the prediction results can fit the key conditions such as the importance of equipment, the effectiveness of early warning, and the production accuracy requirements in actual production, thereby improving the reliability of the qualification rate prediction.

[0116] The prediction module's trend prediction of equipment failure rate follows:

[0117]

[0118] In the formula: F t+1 For the equipment failure rate in the next forecast period; F t λ represents the actual equipment failure rate for the current period. F λ is the equipment load influence coefficient. F ∈(0,1];P t P represents the average operating load of the equipment during the current cycle. n The rated load of the equipment; θ F For the parameter - image mapping intensity influence coefficient, θ F ∈(0,1]; The average parameter of the current period is the image mapping intensity; μ F μ is the impact coefficient for fault early warning records. F ∈(0,1];W F,t This represents the percentage of fault-related warnings during the current period.

[0119] The above formula takes the actual equipment failure rate of the current period as the benchmark and introduces the equipment load influence coefficient, the ratio of the average operating load of the equipment in the current period to the rated load, the parameter-image mapping intensity influence coefficient, the average parameter-image mapping intensity of the current period, the fault warning record influence coefficient, and the proportion of fault-related warning times in the current period. Combined with key factors affecting equipment failure such as equipment load sensitivity, fault-image correlation, and fault warning effectiveness, the prediction results are more in line with the actual operating characteristics of the equipment, thus improving the accuracy of failure rate prediction.

[0120] The prediction cycle of the prediction module is positively correlated with the length T of the mapping calculation window; that is, the smaller T is, the shorter the prediction cycle.

[0121] Among them, λ takes a larger value when the equipment is a core production equipment (significantly affecting product quality), and a smaller value when the equipment is a non-core auxiliary equipment (with negligible impact on product quality); λ F The larger the value of μ is when the equipment is sensitive to changes in operating load (increased load easily leads to failure), the smaller the value is when the equipment is highly adaptable to changes in operating load (load changes have little impact on failure). F The value is larger when the correlation between the warning and the product qualification rate / equipment failure rate is high (e.g., high warning accuracy, warning directly reflects target fluctuations), and smaller when the correlation is low (e.g., low warning accuracy, weak correlation between the warning and target fluctuations). The specific value is preset by the production process or the accuracy of the fault warning. Furthermore, μ is correlated with all types of warnings affecting product qualification. F Only related equipment failure warnings are associated, μ focuses on the degree of impact of the warnings on production results, μ F The focus is on the impact of fault warning on the probability of equipment failure. θ takes a larger value when the production precision requirement is higher and a smaller value when the production precision requirement is lower. F The higher the correlation between equipment failure and image features, the larger the value; the lower the correlation between equipment failure and image features, the smaller the value.

[0122] The initial sequence marker added to the data by the acquisition module is: acquisition time - device number - production process name. The preset storage logic is: high-frequency access data is stored in the upper layer and low-frequency access data is stored in the lower layer.

[0123] During system operation, the efficiency of data interaction between modules is optimized synchronously based on data transmission priority.

[0124]

[0125] In the formula: P trans E represents the data transmission priority; E represents the data urgency level, with a value of 1 for early warning data, 0.5 for historical data, and 0.8 for real-time data acquisition; D date D represents the current size of the transmitted data.max L represents the maximum amount of data that can be transmitted in a single system session. mod The current processing load of the data receiving module is the ratio of the number of tasks currently pending to the maximum number of tasks that the module can process; L max φ is the maximum processing load threshold of the data receiving module; M is the mapping correlation weight. rel To determine the correlation between data and bidirectional mapping, the parameter-image correlation data is set to 0.9, the time-series-environment correlation data is set to 0.8, and other data is set to 0.5; ω1, ω2, and ω3 are the urgency weight, data size weight, and module load weight, respectively.

[0126] The above formula integrates the urgency of the data, the ratio of the current data size to the maximum data size that can be transmitted in a single transmission, the current processing load and maximum processing load threshold of the data receiving module, and the correlation between the data and the bidirectional mapping. It also sets weights for urgency, data size, module load, and mapping correlation. After calculating the priority, the data is transmitted in descending order. This can balance the urgency of the data, the consumption of transmission resources, the carrying capacity of the receiving module, and the correlation with the core mapping, giving priority to the transmission of critical data, avoiding bandwidth waste or module overload, and optimizing the efficiency of data interaction between modules.

[0127] Finally, the data transmission between each module is processed according to P. trans Sort by high to low, then transmit in order;

[0128] Among them, the sum of ω1, ω2, ω3 and φ is 1 and all are positive numbers. ω1 takes a larger value when the system has higher requirements for the timeliness of the transmission of emergency data such as early warning data and real-time acquisition data, and a smaller value when the system has lower requirements for the timeliness of the transmission of emergency data such as early warning data and real-time acquisition data. ω2 takes a larger value when the system wants to prioritize the transmission of small-volume data to avoid large-volume data occupying bandwidth and affecting critical data interaction, and a smaller value when the system has lower sensitivity to data volume and pays more attention to other transmission priority factors. ω3 takes a larger value when the system needs to strictly avoid the data receiving module from processing delays or failures due to excessive load, and a smaller value when the system allows the receiving module to receive data under high load. φ takes a larger value when the system has higher requirements for the real-time performance of parameter-image and time-series-environment bidirectional mapping, and a smaller value when the system has lower requirements for the real-time performance of bidirectional mapping.

[0129] The data acquisition module is connected to the storage module via a wireless network. The storage module is connected to the visualization module via a wireless network. The visualization module is connected to the monitoring module via a wireless network. The monitoring module is connected to the early warning module via a wireless network. The early warning module is connected to the prediction module via a wireless network.

[0130] In this embodiment, the acquisition module collects equipment operation data, environmental perception data, and production process image data in a smart manufacturing scenario, and adds sequence tags to each data item in each data category. The storage module receives the tagged data from the acquisition module in real time, and establishes a dynamic hierarchical storage structure based on preset storage logic and tags to store the tagged data from the acquisition module. The visualization module runs in sequence to retrieve the stored data at each level in the storage module, and constructs a smart manufacturing data visualization model that maps parameters to images and time sequences to the environment. The monitoring module monitors the operation status of the smart manufacturing data visualization model, tracks the processing flow and results of multimodal data at each level in real time, and synchronously records the data source, processing link, and time corresponding to anomalies. The data source, processing link, and time corresponding to anomalies are recorded as traceability information. The early warning module receives the monitoring results and traceability information from the monitoring module, triggers early warning signals in stages based on the monitoring results and traceability information, and adds anomaly traceability identifiers to the early warning signals. Finally, the prediction module extracts the historical data stored in the storage module and the historical early warning information trigger records of the early warning module to predict the trend of product qualification rate and equipment failure rate.

[0131] In the above embodiments, when the system is implemented in a smart manufacturing scenario, it can dynamically adjust the data acquisition frequency according to the equipment load, eliminate and complete abnormal data, store data in layers according to the data call frequency to optimize resources, and associate equipment parameters with images, time series and environmental data for easy viewing. It can accurately determine data anomalies and trace their sources, issue warnings according to severity, predict product qualification rate and equipment failure rate, prioritize the transmission of key data, and effectively improve production monitoring efficiency and production process stability.

[0132] Example 2:

[0133] At the implementation level, based on Example 1, this example refers to... Figure 2 A further detailed description of the intelligent manufacturing data monitoring system based on multimodal fusion in Example 1 is provided below:

[0134] A smart manufacturing data monitoring method based on multimodal fusion includes:

[0135] Collect equipment operation data, environmental perception data, and production process image data in intelligent manufacturing scenarios, add sequence tags to each data point, dynamically adjust the collection frequency according to the equipment operating load, and remove and complete outliers in the collected data.

[0136] The system receives marked data and establishes a dynamic hierarchical storage structure based on a preset logic of storing high-frequency data in the upper layer and low-frequency data in the lower layer. The data storage hierarchy is dynamically adjusted by calculating the data call frequency within a preset time window.

[0137] Retrieve storage data from each level, establish mapping relationships between equipment operating parameters and production image data, and between time series data and environmental data to build a visualization model, and dynamically adjust the model mapping accuracy based on mapping strength and mapping deviation;

[0138] Monitor the running status of the visualization model, determine whether the data processing process is abnormal by calculating the collaborative deviation of the data, and synchronously record the source information of the abnormal data source, processing link, time and mapping type.

[0139] Based on the severity of the anomaly, the scope of its impact, and the degree of disruption of the mapping relationship, early warning signals are triggered in stages, and an anomaly tracing identifier corresponding to the sequence marker is added to the early warning signal;

[0140] Extract historical stored data and historical warning information trigger records, and combine them with parameters such as equipment failure rate impact coefficient and warning record impact coefficient to calculate and predict the product qualification rate and equipment failure rate for the next cycle.

[0141] In summary, the methods and systems described in the above embodiments can accurately collect multiple types of data and add sequence markers during execution. They can dynamically adjust the collection frequency based on the equipment's operating load, synchronously remove outliers to complete the data, ensuring accurate and efficient data collection. Furthermore, they can construct a dynamic hierarchical storage structure according to the call frequency, adjust the data storage hierarchy in real time, and improve data access efficiency. At the same time, by constructing a visualization model and dynamically optimizing the mapping accuracy, the data presentation becomes more intuitive. By calculating the collaborative deviation, anomalies are determined, traceability information is recorded, and graded warnings are issued based on the severity of the anomalies, with traceability identifiers attached. In addition, by combining historical data and warning records, product qualification rates and equipment failure rates are predicted, providing decision support for the intelligent manufacturing process.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal fusion-based intelligent manufacturing data monitoring system, characterized in that, The application comprises: a collection module for collecting equipment operation data, environment perception data and production process image data in an intelligent manufacturing scene, and adding sequence labels to each piece of data in each type of data; a storage module for receiving the labeled data in the collection module, establishing a dynamic hierarchical storage structure according to a preset storage logic and labels, and storing the labeled data in the collection module; a visualization module for calling the hierarchical storage data in the storage module, constructing an intelligent manufacturing data visualization model of parameter mapping images and time sequence mapping environments; a monitoring module for monitoring the running state of the intelligent manufacturing data visualization model, real-time tracking the processing flow and results of the multi-modal data at each level, and synchronously recording the data source, processing link and time corresponding to the abnormality as traceability information; an early warning module for receiving the monitoring results and traceability information in the monitoring module, triggering early warning signals based on the monitoring results and traceability information, and attaching abnormal traceability identifiers to the early warning signals; a prediction module for extracting historical data stored in the storage module and historical early warning information triggering records in the early warning module, and performing trend prediction on product qualification rate and equipment failure rate. The sequence labels added to the data by the collection module are initially: collection time-equipment number-production process name, and the preset storage logic is: high-frequency calling data is stored in the upper layer, and low-frequency calling data is stored in the lower layer. 2.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, The data collection frequency is dynamically adjusted according to the equipment operation load during the operation stage of the collection module: In the formula, f is the current data collection frequency; f0 is the preset basic collection frequency; k is a load influence coefficient, k ∈ (0, 1]; P is the current running load of the equipment; P n is the rated load of the equipment. The acquisition module runs a stage to synchronize the collected raw data to eliminate outliers, and the outlier determination formula is: x i represents the current acquisition data value, represents the sliding mean value of the previous m acquisition data, t represents the preset signal coefficient, s m represents the sliding standard deviation of the previous m acquisition data, when the data satisfies the outlier determination formula, the data is marked as an outlier and eliminated, and the data at this moment is supplemented based on the data. 3.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, The storage module calculates the calling frequency of each data to distinguish high-frequency calling data and low-frequency calling data during the operation stage of the storage module, and the calling frequency is the ratio of the actual calling number of data in a preset time window to the length of the time window; When the calling frequency is not less than a preset high-frequency threshold, the data is determined as high-frequency calling data; when the calling frequency is not more than a preset low-frequency threshold, the data is determined as low-frequency calling data; and when the calling frequency is between the preset high-frequency threshold and the preset low-frequency threshold, the original storage level of the data is maintained; The data storage level is dynamically adjusted according to the change of the calling frequency: Set the hierarchical adjustment trigger condition: ΔF = |F t2 -F t1 |>ΔF0,F t1 Indicates the call frequency of the previous time window, F t2 ΔF0 represents the call frequency for the current time window, and ΔF0 represents the preset call frequency change threshold. When the above triggering conditions are met, the data will be migrated to the corresponding high-frequency or low-frequency storage level. 4.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, The process of constructing the intelligent manufacturing data visualization model by the visualization module includes: establishing the mutual mapping relationship between the equipment operation parameters and the production image data, and the mapping relationship strength is represented as: In the formula, M p-i is the parameter-image mapping intensity; T is the mapping calculation window length; P t is the equipment operation parameter value at time t; is the parameter mean value; I t is the image characteristic value at time t; is the image characteristic mean value; When the system user inputs the specific threshold range of the equipment operation parameter on the control panel preset by the system, the visualization module automatically filters out the mapping intensity M p-i All the time points meeting the threshold range, the production image segment corresponding to the filtered time point is located and displayed synchronously, and the specific parameter value of the time point is labeled on the image segment; when the system user selects the production image area by clicking or box selecting operation on the control panel, the visualization module automatically identifies the time range and image feature I t corresponding to the image area, matches the equipment operation parameter associated with the image feature based on M p-i , synchronously generates and displays the parameter change curve in the time range, the horizontal axis of the curve is time, the vertical axis is parameter value, and the parameter peak and valley values of the key time points are labeled; establishing the mutual mapping relationship between the time sequence data and the environment data, and the mapping deviation is represented as: wherein, In the formula: E t-e is the time-series-environment mapping deviation; T t is the normalized time-series data at time t; E t is the normalized environment data at time t; T t , E t are the original time-series value and the original environment value; minT, maxT are the minimum and maximum values of the time-series data in the statistical dimension with the original time-series value T t ; minE, maxE are the minimum and maximum values of the environment data in the statistical dimension with the original environment value E t ; The visualization model is based on M p-i and E t-e Dynamic adjustment of mapping precision: When M p-i When the mapping intensity is less than the preset mapping intensity threshold, the sampling density of the corresponding device operating parameter and the image feature extraction dimension are increased. When E t-e When M p-i When E t-e When E wherein I t 、 is any of a workpiece profile variation, a device component texture definition. 5.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, The monitoring module determines the abnormality by calculating the cooperative deviation of the data: wherein: D is the synergistic bias value; n is the number of data types; a i is the bias weight of the ith data type; b, c are the parameter-image mapping abnormal weight, time-series-environment mapping abnormal weight, respectively; v i,t is the current value of the ith data type; is the historical mean value of the ith data type; M′ p-i is the abnormal fluctuation value of M p-i ; E′ t-e is the abnormal fluctuation value of E t-e ; wherein, t denotes the current time instant; When the D is greater than a preset abnormality determination threshold, the monitoring module determines that the current data processing flow has an abnormality, and synchronously records a mapping type corresponding to the abnormality, the mapping type including: parameter-image or time sequence-environment, and a bias source and a timestamp; the monitoring module updates in real time The update period and the time window in the level adjustment trigger condition are consistent. wherein, respectively represent the historical mean of the parameter-image mapping strength, the historical mean of the temporal-context mapping bias. 6.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, The early warning module triggers early warning signals based on the severity and influence range of the abnormality during the operation stage of the early warning module: wherein: L is the early warning level; S, R and M are the severity of the abnormality, the influence range of the abnormality and the mapping relationship damage degree, respectively; and a, β and γ are the severity weight, the influence range weight and the mapping damage weight, respectively; δ is the level normalization coefficient; and round(·) is the rounding function; When L is 1, the early warning module triggers a first-level early warning, which corresponds to a local slight abnormality; when L is 2, a second-level early warning is triggered, which corresponds to a regional moderate abnormality; When L is 3, a third-level early warning is triggered, which corresponds to a global abnormality. The abnormality traceability mark attached in the early warning signal by the early warning module contains mapping type, deviation source and timestamp information, and the mark format corresponds to the sequence mark of the collection module; wherein, N aff N represents the number of devices, processes affected by the anomaly total N represents the total number of devices, processes in the system, and δ is a preset value for constraining L to be in the interval [1, 3]. 7.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, The product qualification rate prediction of the prediction module in the running stage is subject to: In the formula, Y t+1 Y is the product yield of the next prediction period; t Y is the actual product yield of the current period; λ is the equipment failure rate influence coefficient, λ ∈ (0, 1]; F t F is the equipment failure rate of the current period; μ is the early warning record influence coefficient, μ ∈ (0, 1]; W t W is the proportion of early warning times in the current period; θ is the mapping quality influence coefficient, θ ∈ (0, 1]; Y is the average mapping deviation of the current period; The trend prediction of the prediction module on the equipment failure rate is subject to: wherein: F t+1 is the device failure rate of the next prediction period; F t is the actual device failure rate of the current period; λ F is the device load influence coefficient, λ F ∈(0,1]; P t is the average operating load of the device in the current period; P n is the rated load of the device; θ F is the parameter-image mapping intensity influence coefficient, θ F ∈(0,1]; is the average parameter-image mapping intensity in the current period; μ F is the failure early warning record influence coefficient, μ F ∈(0,1]; W F,t is the proportion of failure-related early warnings in the current period; The prediction cycle of the prediction module is positively correlated with the length T of the mapping calculation window, that is, the smaller T is, the shorter the prediction cycle is. 8.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, When the system is running, the data transmission priority is calculated to optimize the data interaction efficiency between modules: In the formula: P trans is the data transmission priority; E is the data emergency degree, the pre-warning data is set to 1, the historical data is set to 0.5, and the real-time acquisition data is set to 0.8; D date is the size of the current transmission data; D max is the maximum transmission data amount of the system at a time; L mod is the current processing load of the data receiving module, which is the ratio of the current number of tasks to be processed to the maximum processing task number of the module; L max is the maximum processing load threshold of the data receiving module; φ is the mapping correlation weight; M rel is the data and bidirectional mapping correlation degree, the parameter-image correlation data is set to 0.9, the time sequence-environment correlation data is set to 0.8, and other data is set to 0.5; ω1, ω2, ω3 are the emergency degree weight, the data size weight, and the module load weight, respectively; Finally, the data transmitted between the modules are sorted by P trans from high to low, and then transmitted in order. 9.The intelligent manufacturing data monitoring system based on multi-modal fusion of claim 1, wherein, The collection module is interactively connected with the storage module through a wireless network, the storage module is interactively connected with the visualization module through a wireless network, the visualization module is interactively connected with the monitoring module through a wireless network, the monitoring module is interactively connected with the early warning module through a wireless network, and the early warning module is interactively connected with the prediction module through a wireless network.

10. An intelligent manufacturing data monitoring method based on multi-modal fusion, the method being an implementation method of the intelligent manufacturing data monitoring system based on multi-modal fusion according to any one of claims 1-9, characterized in that, It includes: Collecting equipment running data, environment sensing data and production process image data in an intelligent manufacturing scene, adding sequence marks to each data, dynamically adjusting the collection frequency according to the equipment running load, and removing and completing abnormal values of the collected data; Receiving the data with completed marks, establishing a dynamic hierarchical storage structure according to the preset logic of storing high-frequency calling data in the upper layer and low-frequency calling data in the lower layer, dynamically adjusting the data storage level by calculating the data calling frequency in the preset time window; Retrieve the stored data at each level, establish the mapping relationship between the equipment running parameters and the production image data, the time series data and the environment data to construct a visualization model, and dynamically adjust the model mapping accuracy according to the mapping strength and mapping deviation; Monitor the running state of the visualization model, determine whether the data processing flow is abnormal by calculating the cooperative deviation, and record the traceability information of the corresponding data source, processing link, time and mapping type synchronously; Based on the severity of the abnormality, the influence range and the mapping relationship damage degree, trigger the early warning signal, and attach the abnormality traceability mark corresponding to the sequence mark in the early warning signal; Extract historical stored data and historical early warning information trigger records, combine equipment failure rate influence coefficient, early warning record influence coefficient and other parameters to calculate the product qualification rate and equipment failure rate in the next period.

Images