An industrial process supervision management system based on multi-modal data processing

Abstract

The application discloses an industrial process supervision management system based on multi-modal data processing, comprising a data acquisition module (100), a data fusion module (200), an anomaly detection module (300), a decision optimization module (400) and a visual interaction center (500), the data acquisition module (100) transmits multi-source sensing data to the data fusion module (200) in real time through an industrial Ethernet, the data fusion module (200) forms a multi-modal feature vector and sends the multi-modal feature vector to the anomaly detection module (300), the anomaly detection module (300) presets a threshold value and sends a joint alarm signal to the decision optimization module (400) and the visual interaction center (500). The application has the beneficial effect of realizing breakthroughs in detection speed, positioning accuracy and decision reliability through multi-modal data fusion and dynamic modeling, effectively solving technical bottlenecks such as timing misalignment, single-source misjudgment and insufficient cause-effect analysis existing in traditional industrial monitoring systems.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent control, and in particular to an industrial process supervision and management system based on multi-modal data processing. BACKGROUND

[0002] With the rapid development of Industry 4.0 and intelligent manufacturing technology, modern industrial production puts forward higher requirements for the real-time, accuracy and intelligent level of process monitoring systems. Traditional industrial monitoring systems mainly rely on single modal data (such as vibration signals or temperature parameters) for equipment state analysis, which has the following significant defects:

[0003] 1. Insufficient multi-source heterogeneous data fusion capability:

[0004] Existing systems usually use separate sensor networks to collect visual, acoustic and environmental parameter data, but lack a hardware-level time alignment mechanism, resulting in a timestamp deviation of multi-modal data exceeding milliseconds, which directly affects the timeliness of anomaly detection.

[0005] 2. Poor dynamic adaptability of anomaly detection:

[0006] Traditional methods rely on fixed threshold criteria, but when the equipment working condition changes dynamically with the production load, the static threshold will cause the false alarm rate to rise. In addition, existing dynamic modeling techniques cannot solve the problem of nonlinear coupling of multi-modal data, and cannot accurately define the dynamic threshold curve.

[0007] 3. Lack of explainability of intelligent decision-making:

[0008] Current systems mostly use black-box deep learning models, which can detect some abnormal events, but lack causal reasoning ability. Operators cannot trace the logical basis, leading to delayed or inappropriate control parameter adjustments, which seriously affects fault repair efficiency.

[0009] 4. Limited visual interaction effect:

[0010] Existing monitoring interfaces mostly use two-dimensional charts, which cannot intuitively present the full life cycle state of the equipment and the spatial distribution of anomalies. Although digital twin technology has been introduced, it has not yet achieved accurate spatiotemporal mapping of multi-modal data, especially lacking multi-dimensional traceability of abnormal events.

[0011] 5. Low efficiency of cross-domain collaborative control:

[0012] Independent monitoring-control architecture only supports one-way instruction transmission, and there is a lack of feedback correction mechanism between the decision unit and the execution mechanism. When the device response deviates from the expectation after parameter adjustment, the system cannot optimize the control strategy in real time, which may cause secondary abnormalities.

[0013] In view of the above problems and / or existing ones, an industrial process supervision and management system based on multi-modal data processing is proposed. SUMMARY

[0014] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification in order to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions are not to be construed as limiting the scope of the present application.

[0015] In view of the above problems and / or existing ones, an industrial process supervision and management system based on multi-modal data processing is proposed.

[0016] Therefore, the problem to be solved by the present application is how to provide a multi-modal data processing factory process supervision effect.

[0017] To solve the above technical problems, the present application provides the following technical solutions: an industrial process supervision and management system based on multi-modal data processing, comprising,

[0018] The data acquisition module, the data fusion module, the anomaly detection module, the decision optimization module and the visual interaction center:

[0019] The data acquisition module includes a visual sensor array, an acoustic monitoring unit and an environmental parameter sensor group, and transmits multi-source sensing data in real time to the data fusion module through an industrial Ethernet;

[0020] The data fusion module includes a timestamp alignment unit and a multi-modal feature generation unit, and the multi-modal feature generation unit forms a multi-modal feature vector and sends it to the anomaly detection module;

[0021] The anomaly detection module includes a dynamic modeling unit and a correlation decision unit, the dynamic modeling unit generates a real-time threshold curve of the equipment working condition, and receives the multi-modal feature vector of the data fusion module to calculate the feature deviation, and when the multi-source data anomaly confidence score exceeds the preset threshold, the correlation decision unit sends a joint alarm signal to the decision optimization module and the visual interaction center;

[0022] The decision optimization module includes a fault reasoning engine and a parameter control unit, the fault reasoning engine receives the multi-modal anomaly confidence score of the anomaly detection module, and drives the parameter control unit to send optimization instructions to the production line actuator.

[0023] The visualization interaction center couples raw data of the data collection module and analysis results of the anomaly detection module, and pushes a key alarm event to a remote monitoring terminal.

[0024] As a preferred scheme of the industrial process supervision and management system based on multi-modal data processing, the data collection module comprises:

[0025] The visual sensor array is arranged at a core working position of the production line to capture real-time data, the acoustic monitoring unit integrates a microphone array to collect equipment operation voiceprint signals, and the environmental parameter sensor group is configured to periodically detect vibration, temperature and humidity parameters, and transmit multi-source sensing data to the data fusion module in real time through an industrial Ethernet.

[0026] The visual sensor array comprises a multi-spectral industrial camera and an infrared thermal imager, the multi-spectral industrial camera covers the monitoring area of the production line in an interleaved arrangement, and the infrared thermal imager is installed on a rotating holder to dynamically track equipment heating points and generate temperature matrix data frames.

[0027] As a preferred scheme of the industrial process supervision and management system based on multi-modal data processing, the data fusion module comprises:

[0028] The timestamp alignment unit receives raw data from the data collection module, and uses a hardware-level pulse trigger mechanism to align video data, acoustic signals and environmental parameters, the multi-modal feature generation unit deploys a deep residual convolutional network to extract video semantic features, processes acoustic signals in parallel through a mel-frequency cepstral coefficient, and fuses multi-dimensional sensor time series data through a space-time feature splicing layer to form a unified multi-modal feature vector and send it to the anomaly detection module.

[0029] The hardware-level pulse trigger mechanism used by the timestamp alignment unit specifically comprises:

[0030] A GPS synchronous clock chip is embedded in the environmental parameter sensor group, which generates a signal every second to broadcast a global synchronization pulse to the visual sensor array and the acoustic monitoring unit, triggering the hardware timer of the three-modal data collection device to reset to zero.

[0031] As a preferred scheme of the industrial process supervision and management system based on multi-modal data processing, the real-time threshold curve of the dynamic modeling unit for equipment working conditions comprises:

[0032] The multi-modal feature vector is divided into windows with a window period of 30 seconds, the root mean square value of the vibration feature, the frequency energy total value of the video feature and the variance value of the acoustic cepstrum coefficient in each window are calculated, a dynamic probability distribution model of the features is generated based on a kernel density estimation algorithm, and a real-time threshold curve is determined according to the cumulative distribution function value in the interval [0.05, 0.95].

[0033] As a preferred scheme of the industrial process supervision and management system based on multi-modal data processing, the decision optimization module comprises:

[0034] The fault reasoning engine receives the multi-modal anomaly confidence score of the anomaly detection module, generates an interpretable causal chain analysis through a Bayesian network associated with a historical fault database, drives the parameter control unit to send optimization instruction messages to the production line actuator, and feeds back the control parameter adjustment log to the visual interaction center;

[0035] When the fault reasoning engine generates the causal chain analysis through the Bayesian network, a double reasoning mechanism is adopted:

[0036] The double reasoning mechanism comprises forward reasoning and reverse reasoning, the forward reasoning comprises traversing a conditional probability table in the historical fault database according to the current anomaly confidence score, and updating the edge probability distribution of the node event;

[0037] The reverse reasoning comprises reverse correcting the prior probability weight coefficient of the Bayesian network after the parameter control unit receives the state feedback of the actuator.

[0038] As a preferred scheme of the industrial process supervision and management system based on multi-modal data processing, the visual interaction center comprises:

[0039] The visual interaction center integrates a digital twin map and an event trace panel, connects through real-time data of an industrial bus, the digital twin map dynamically renders a three-dimensional virtual model of the production line and synchronously labels an abnormal area heat map, the event trace panel couples original data of the data acquisition module and analysis results of the anomaly detection module, supports synchronous playback of video monitoring, voiceprint spectrum and sensor waveform data according to a time axis, and pushes key alarm events to a remote monitoring terminal.

[0040] As a preferred scheme of the industrial process supervision and management system based on multi-modal data processing, the event trace panel comprises:

[0041] The synchronous playback function of the event trace panel is implemented in the following manner:

[0042] A three-modal data cache queue with a unified time reference is established, and when the user selects an abnormal event occurrence time point:

[0043] The video monitoring data is decoded and played from 5 seconds before the selected time point;

[0044] The voiceprint atlas reconstructs the energy time-frequency matrix of the frequency band below 20kHz during the abnormal period through short-time Fourier transform;

[0045] The sensor waveform data is dynamically visualized after the sampling rate is increased to 1kHz through cubic spline interpolation algorithm.

[0046] As a preferred scheme of the industrial process supervision and management system based on multi-modal data processing, the digital twin map dynamically renders a three-dimensional virtual model of the production line and synchronously labels an abnormal area heat map, which includes:

[0047] The abnormal event and its disposal process are superimposed and displayed on the abnormal area heat map.

[0048] In a second aspect, some embodiments of the present application provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0049] In a third aspect, some embodiments of the present application provide a computer readable medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method described in any of the implementation manners of the first aspect.

[0050] The present application has the beneficial effect of providing an industrial process supervision and management system based on multi-modal data processing, which realizes breakthroughs in detection speed, positioning accuracy and decision reliability through multi-modal data fusion and dynamic modeling. In particular, the hardware-level time synchronization, dual-modal correlation decision and Bayesian double reasoning mechanism effectively solve the technical bottlenecks of time sequence misalignment, single-source misjudgment and insufficient causal analysis in traditional industrial monitoring systems. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] In which:

[0054] Figure 1 The flow structure diagram of an industrial process supervision management system based on multi-modal data processing in embodiment 1. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0056] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other ways different from those described herein without departing from the scope of the present application, and those skilled in the art can make similar extensions without departing from the scope of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0057] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. In this specification, "in one embodiment" appearing in different places does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0058] Embodiment 1

[0059] Referring to Figure 1 For the first embodiment of the present application, the embodiment provides an industrial process supervision management system based on multi-modal data processing, which comprises,

[0060] The data acquisition module 100 includes a visual sensor array 101, an acoustic monitoring unit 102 and an environmental parameter sensor group 103. The visual sensor array 101 is deployed at the core workstations of the production line to capture real-time video stream data. The acoustic monitoring unit 102 integrates a microphone array acquisition device to collect equipment operation voiceprint signals. The environmental parameter sensor group 103 is configured to periodically detect vibration, temperature and humidity parameters, and transmit them to the data fusion module 200 through industrial Ethernet.

[0061] The visual sensor array 101 includes a multi-spectral industrial camera 101a covering the production line monitoring area in a staggered arrangement and an infrared thermal imager 101b installed on a rotating pan-tilt head dynamically tracking the device heat points and generating temperature matrix data frames.

[0062] The data fusion module 200 includes a timestamp alignment unit 201 achieving data synchronization of video data, acoustic signals and environmental parameters using a pulse trigger mechanism and a multi-modal feature generation unit 202 deploying a deep residual convolutional network to extract video semantic features, performing Mel cepstrum coefficient calculation on acoustic signals in parallel, and fusing multi-dimensional sensor time series data through a feature splicing layer.

[0063] The hardware-level pulse trigger mechanism used by the timestamp alignment unit 201 is as follows:

[0064] A GPS synchronization clock chip is embedded in the environmental parameter sensor group 103, which generates a signal every second to broadcast a global synchronization pulse to the visual sensor array 101 and the acoustic monitoring unit 102, triggering the hardware timer of the three-modal data acquisition device to reset to zero.

[0065] The anomaly detection module 300 includes a dynamic modeling unit 301 generating real-time threshold curves of device operating conditions based on sliding window statistical analysis and a correlation decision unit 302 using a bidirectional attention mechanism to compare the relevance of visual behavior features and acoustic features, outputting multi-modal anomaly confidence scores.

[0066] The dynamic modeling unit 301 generates real-time threshold curves of device operating conditions, including:

[0067] The multi-modal feature vector is divided into windows with a period of 30 seconds, the root mean square value of the vibration feature, the total frequency energy value of the video feature, and the variance value of the acoustic cepstrum coefficient in each window are calculated, the dynamic probability distribution model of the feature is generated based on the kernel density estimation algorithm, and the real-time threshold curve is determined according to the cumulative distribution function value in the [0.05, 0.95] interval.

[0068] The decision optimization module 400 includes a fault reasoning engine 401 generating an interpretable causal chain analysis by associating a historical fault database through a Bayesian network and a parameter control unit 402 converting optimization instructions into OPC UA protocol messages and sending them to the actuator;

[0069] When the fault reasoning engine 401 generates a causal chain analysis through a Bayesian network, a double reasoning mechanism is used:

[0070] The double reasoning mechanism includes forward reasoning and reverse reasoning, the forward reasoning includes traversing the conditional probability table in the historical fault database according to the current abnormal confidence score, updating the edge probability distribution of the node event;

[0071] The reverse reasoning includes correcting the prior probability weight coefficient of the Bayesian network in reverse when the parameter control unit 402 receives the state feedback of the actuator.

[0072] The visual interaction center 500 integrates the digital twin map 501 and the event tracing panel 502, the digital twin map 501 dynamically renders a three-dimensional virtual model of a production line and labels an abnormal area heat map, and the event tracing panel 502 supports synchronous playback of video monitoring, voiceprint spectrum and sensor waveform data according to a time axis. The digital twin map 501 dynamically renders a three-dimensional virtual model of a production line and synchronously labels an abnormal area heat map, the event tracing panel 502 couples original data of the data acquisition module 100 and analysis results of the abnormal detection module 300, supports synchronous playback of video monitoring, voiceprint spectrum and sensor waveform data according to a time axis, and pushes key alarm events to a remote monitoring terminal.

[0073] The event tracing panel 502 includes:

[0074] The synchronous playback function of the event tracing panel 502 is implemented in the following manner:

[0075] A three-modal data cache queue with a unified time reference is established, when a user selects an abnormal event occurrence time point:

[0076] Video monitoring data is decoded and played starting from 5 seconds before the selected time;

[0077] The voiceprint spectrum reconstructs an energy time-frequency matrix of a frequency band below 20 kHz during the abnormal period through short-time Fourier transform;

[0078] The sensor waveform data is dynamically visualized after the sampling rate is improved to 1 kHz through a cubic spline interpolation algorithm.

[0079] The abnormal event and its disposal process are superimposed and displayed on the abnormal area heat map.

[0080] Embodiment 2

[0081] The second embodiment of the application differs from the first embodiment in that it further includes a test preparation and implementation process:

[0082] Test preparation and implementation process:

[0083] This example selects a certain automobile manufacturing enterprise welding production line as the test scene, verifies the performance of the industrial process supervision and management system based on multi-modal data processing in complex industrial environment. The test object includes 12 welding robots, 8 conveyor belt drive motors and supporting cooling systems, and the total monitoring area covers 1200 square meters. The experimental period is 30 consecutive days, and the total amount of data collected is 2.3TB.

[0084] Hardware deployment:

[0085] The visual sensor array 101 is composed of 36 multispectral industrial cameras 101a and 6 infrared thermal imagers 101b, arranged in an interleaved manner 3 meters above the welding station track, covering all the welding gun working area. Each multispectral camera collects a 1280x1024 resolution video stream at 30fps, and the infrared thermal imager realizes 360° scanning through a rotating pan-tilt, with a temperature detection range of 0-500℃.

[0086] The acoustic monitoring unit 102 contains a 24-channel microphone array, with a sampling rate of 48kHz per channel, deployed around the key nodes of the equipment at a height of 1.5 meters, monitoring the welding arc soundprint and motor bearing abnormal noise.

[0087] The environmental parameter sensor group 103 contains 32 vibration sensors (range 0-50g) and temperature and humidity sensors (accuracy ±0.5℃), arranged at 1.2 meter intervals on the equipment base.

[0088] Data synchronization mechanism:

[0089] A hardware-level pulse trigger mechanism is used, with a GPS-synchronized clock chip built into the environmental parameter sensor group 103, generating a synchronization pulse signal every 1 second. Tests show that the timestamp alignment error of the three modal data is <1ms, meeting the requirements of the dynamic modeling unit 301 for time sequence consistency.

[0090] Abnormality detection process:

[0091] The dynamic modeling unit 301 analyzes the multi-modal feature vectors with a 30-second window, calculates the root mean square value (RMS) of vibration, video frequency energy (0-15Hz) and acoustic MFCC variance. When generating the dynamic threshold curve by kernel density estimation, the cumulative distribution function confidence interval is set to [0.05, 0.95]. When the feature deviation of any two modalities exceeds the threshold value, the correlation decision unit 302 triggers a joint alarm.

[0092] Decision optimization implementation:

[0093] The fault reasoning engine 401 accesses a database containing 1523 historical fault records, and sets the double inference parameters of the Bayesian network: the condition probability update step size a = 0.85 during forward inference, and the prior weight correction coefficient β = 0.72 during backward inference. The parameter control unit 402 controls the welding current, the conveyor belt speed, and other six types of actuators to realize closed-loop control.

[0094] Table 1: Comparison of timestamp alignment performance

[0095]

[0096]

[0097] Table 2: Abnormality detection response time table

[0098]

[0099] Table 3: Fault location accuracy

[0100]

[0101] Table 4: Parameter adjustment effectiveness

[0102]

[0103] Table 5: Heat map update delay table

[0104]

[0105] Table 6: Alarm event statistics

[0106]

[0107] Time synchronization performance (Table 1): The multi-modal data alignment error is reduced to sub-millisecond level (0.5-0.8ms) by the hardware-level pulse trigger mechanism, which reduces the error by 98.3% compared with the traditional software synchronization method. This feature ensures that the dynamic modeling unit 301 can accurately establish a spatio-temporal correlation model of multi-modal features, avoiding misjudgment caused by time sequence misalignment.

[0108] Abnormality detection efficiency (Table 2): The response time is shortened by 56.3%-76.5% by the dual-mode correlation detection, and the composite abnormality detection achieves a response speed of 1.2 seconds. Through the multi-source confidence fusion of the correlation decision unit 302, the false positive rate is controlled at 0.9%-3.2%, which is reduced by 42.7% compared with single-mode detection. This shows that the deep residual network of the multi-modal feature generation unit 202 and the MFCC feature extraction effectively improve the abnormality representation ability.

[0109] Decision optimization effect (Table 3-4): The double inference mechanism of the Bayesian network improves the fault positioning accuracy to 95.2%-98.5%, and the causal chain depth reaches 3-5 levels, which is 29.6 percentage points higher than the traditional expert system. The closed-loop response time of the parameter control unit 402 is less than 3 seconds, and the steady-state error is less than 1.2%, proving that the causal chain analysis of the fault reasoning engine 401 can effectively guide the actuator adjustment.

[0110] Visualization performance (Table 5): The heat map update delay of the digital twin map 501 is less than 92ms, which is 4.8 times higher than the traditional SCADA system, meeting the real-time monitoring requirements. The three-mode data synchronous playback function of the event trace panel 502 (Table 6) makes the effective alarm rate reach 98.3%, and the false alarm rate is less than 1.5%, which is significantly better than the existing single-dimensional alarm system.

[0111] The data of the embodiment prove that the system realizes breakthroughs in detection speed, positioning accuracy and decision reliability through multi-modal data fusion and dynamic modeling. In particular, the hardware-level time synchronization, double-modal correlation decision and Bayesian double inference mechanism effectively solve the technical bottlenecks of the traditional industrial monitoring system, such as timing misalignment, single-source misjudgment and insufficient causal analysis.

[0112] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application; the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance; in addition, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0113] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An industrial process supervisory management system based on multi-modal data processing, characterized in that, The application relates to a real-time monitoring system for production line equipment, which comprises a data acquisition module (100), a data fusion module (200), an anomaly detection module (300), a decision optimization module (400) and a visual interaction center (500). The data acquisition module (100) comprises a visual sensor array (101), an acoustic monitoring unit (102) and an environmental parameter sensor group (103), and multi-source sensing data is transmitted in real time to the data fusion module (200) through an industrial Ethernet; The data fusion module (200) comprises a timestamp alignment unit (201) and a multi-modal feature generation unit (202), the multi-modal feature generation unit (202) forms a multi-modal feature vector and sends the multi-modal feature vector to the anomaly detection module (300); The anomaly detection module (300) comprises a dynamic modeling unit (301) and a correlation decision unit (302), the dynamic modeling unit (301) generates a real-time threshold curve of equipment working conditions, receives a multi-modal feature vector of the data fusion module (200), calculates a feature deviation, and sends a joint alarm signal to the decision optimization module (400) and the visual interaction center (500) when multi-source data anomaly confidence score exceeds a preset threshold value; The decision optimization module (400) comprises a fault reasoning engine (401) and a parameter control unit (402), the fault reasoning engine (401) receives multi-modal anomaly confidence score of the anomaly detection module (300), drives the parameter control unit (402) to send an optimization instruction to a production line actuator, and the parameter control unit (402) sends the optimization instruction to the production line actuator; The visual interaction center (500) couples original data of the data acquisition module (100) and analysis results of the anomaly detection module (300), and pushes a key alarm event to a remote monitoring terminal; The data acquisition module (100) comprises: The visual sensor array (101), the acoustic monitoring unit (102) and the environmental parameter sensor group (103), the visual sensor array (101) is arranged at a core working position of a production line to capture real-time data, the acoustic monitoring unit (102) is integrated with a microphone array to collect equipment running voiceprint signals, the environmental parameter sensor group (103) is configured to periodically detect vibration, temperature and humidity parameters, and multi-source sensing data is transmitted in real time to the data fusion module (200) through an industrial Ethernet; The visual sensor array (101) comprises a multi-spectral industrial camera (101a) and an infrared thermal imager (101b), the multi-spectral industrial camera (101a) covers a production line monitoring area in an interlaced arrangement mode, and the infrared thermal imager (101b) is installed on a rotating holder to dynamically track equipment heating points and generate temperature matrix data frames; The dynamic modeling unit (301) generates a real-time threshold curve of equipment working conditions, and comprises: The multi-modal feature vector is divided with a window period of 30 seconds, the root mean square value of the vibration feature, the frequency energy total value of the video feature and the variance value of the acoustic cepstrum coefficient in each window are calculated, a dynamic probability distribution model of the features is generated based on a kernel density estimation algorithm, and a real-time threshold curve is determined according to the cumulative distribution function value in the interval [0.05, 0.95]; The decision optimization module (400) comprises: The fault reasoning engine (401) receives the multi-modal anomaly confidence score of the anomaly detection module (300), generates an interpretable causal chain analysis through a Bayesian network associated with a historical fault database, drives the parameter control unit (402) to send an optimization instruction message to a production line execution mechanism, and simultaneously feeds back a control parameter adjustment log to the visual interaction center (500); When the fault reasoning engine (401) generates a causal chain analysis through a Bayesian network, a double reasoning mechanism is adopted: The double reasoning mechanism comprises forward reasoning and reverse reasoning, the forward reasoning comprises traversing a conditional probability table in a historical fault database according to a current anomaly confidence score, and updating an edge probability distribution of a node event; The reverse reasoning comprises reverse correcting a prior probability weight coefficient of the Bayesian network after the parameter control unit (402) receives a state feedback of the execution mechanism.

2. The industrial process supervisory management system based on multi-modal data processing of claim 1, wherein, The data fusion module (200) comprises: The timestamp alignment unit (201) receives original data from the data acquisition module (100), adopts a hardware-level pulse trigger mechanism to realize alignment of video data, acoustic signals and environmental parameters, the multi-modal feature generation unit (202) deploys a deep residual convolutional network to extract video semantic features, simultaneously processes acoustic signals through a mel cepstrum coefficient calculation, and fuses multi-dimensional sensor time series data through a space-time feature splicing layer, forms a unified multi-modal feature vector and sends the feature vector to the anomaly detection module (300); The hardware-level pulse trigger mechanism adopted by the timestamp alignment unit (201) is as follows: A GPS synchronous clock chip is embedded in the environmental parameter sensor group (103), generates 1 signal per second, broadcasts a global synchronous pulse to the visual sensor array (101) and the acoustic monitoring unit (102), and triggers the hardware timer of the three-mode data acquisition equipment to reset to zero.

3. The industrial process supervisory management system based on multi-modal data processing of claim 1, wherein, The visual interaction center (500) comprises: The visual interaction center (500) integrates a digital twin map (501) and an event trace panel (502), is connected in real time through an industrial bus, the digital twin map (501) dynamically renders a three-dimensional virtual model of a production line and synchronously labels an abnormal area heat map, the event trace panel (502) couples original data of the data acquisition module (100) and analysis results of the anomaly detection module (300), supports synchronous playback of video monitoring, voiceprint spectrum and sensor waveform data according to a time axis, and pushes a key alarm event to a remote monitoring terminal.

4. The industrial process supervisory management system based on multi-modal data processing of claim 3, wherein, The event trace panel (502) comprises: The synchronous playback function of the event trace panel (502) is implemented as: A three-modal data cache queue with a unified time reference is established, and when a user selects an abnormal event occurrence time point: The video monitoring data is decoded and played from 5 seconds before the selected time point; The voiceprint spectrum is reconstructed by short-time Fourier transform to reconstruct the energy time-frequency matrix of the frequency band below 20 kHz during the abnormal period; The sensor waveform data is dynamically visualized after the sampling rate is improved to 1 kHz by the cubic spline interpolation algorithm.

5. The industrial process supervisory management system based on multi-modal data processing of claim 3, wherein, The digital twin map (501) dynamically renders a three-dimensional virtual model of the production line and synchronously labels an abnormal area heat map, comprising: The abnormal event and its disposal process are superimposed and displayed on the abnormal area heat map.

6. An electronic device, comprising: Comprise: One or more processors; A storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the system as claimed in any one of claims 1-5.

7. A computer-readable storage medium having stored thereon executable instructions that cause a processor-based system to perform steps comprising The instructions are executed by the processor to enable the processor to implement the system as claimed in any one of claims 1-5.

Images