Industrial process supervision and management system based on multi-modal data processing

Abstract

The invention discloses an industrial process supervision and management system based on multi-modal data processing, which comprises a data acquisition module (100), a data fusion module (200), an anomaly detection module (300), a decision optimization module (400) and a visual interaction center (500), and is characterized in that the data acquisition module (100) transmits multi-source sensing data to the data fusion module (200) in real time through an industrial Ethernet; the data fusion module (200) forms a multi-modal feature vector and sends the multi-modal feature vector to the anomaly detection module (300), and the anomaly detection module (300) presets a threshold value and sends a joint alarm signal to the decision optimization module (400) and the visual interaction center (500). The method has the advantages that breakthrough is achieved in the aspects of detection speed, positioning precision and decision reliability through multi-modal data fusion and dynamic modeling, and the technical bottlenecks of time sequence misalignment, single-source misjudgment, insufficient causal analysis and the like existing in a traditional industrial monitoring system are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control, and in particular to an industrial process supervision management system based on multimodal data processing. Background Art

[0002] With the rapid development of Industry 4.0 and intelligent manufacturing technologies, modern industrial production has placed higher demands on the real-time, accuracy, and intelligence level of process monitoring systems. Traditional industrial monitoring systems mainly rely on single-modal data (such as vibration signals or temperature parameters) for equipment status analysis, which has the following significant drawbacks:

[0003] 1. Insufficient ability to integrate multi-source heterogeneous data:

[0004] Existing systems typically use discrete sensor networks to collect visual, acoustic, and environmental parameter data separately, but lack a hardware-level timing alignment mechanism, resulting in multimodal data timestamp deviations exceeding the millisecond level, which directly affects the timeliness of anomaly detection.

[0005] 2. Poor dynamic adaptability of anomaly detection:

[0006] Traditional methods rely on fixed threshold criteria, but static thresholds can lead to increased false alarm rates when equipment operating conditions change dynamically with production loads. Furthermore, existing dynamic modeling techniques struggle to address the nonlinear coupling of multimodal data, making it impossible to accurately define dynamic threshold curves.

[0007] 3. Lack of explainability of intelligent decision-making:

[0008] Current systems often use black-box deep learning models, which, while capable of detecting some anomalies, lack causal reasoning capabilities. Operators are unable to trace the logical basis, leading to delayed or inappropriate adjustment of control parameters, severely impacting fault repair efficiency.

[0009] 4. Limited visual interaction effects:

[0010] Existing monitoring interfaces often rely on stacked two-dimensional charts, which fail to intuitively present the full lifecycle status of equipment and the spatial distribution of anomalies. Despite the introduction of digital twin technology, it has yet to achieve precise spatiotemporal mapping of multimodal data, particularly lacking the ability to trace anomalies across multiple dimensions.

[0011] 5. Inefficient cross-domain collaborative control:

[0012] The independent monitoring and control architecture only supports one-way command transmission and lacks a feedback correction mechanism between the decision-making unit and the actuator. When the device response deviates from expectations after parameter adjustment, the system cannot optimize the control strategy in real time, which may cause secondary anomalies.

[0013] To address these pain points, a new industrial process supervision and management system is urgently needed that deeply integrates multimodal sensing technology, supports dynamic threshold modeling, possesses explainable reasoning capabilities, and enables comprehensive monitoring to overcome the limitations of traditional solutions. Currently, there are no effective solutions on the market to address these issues. Summary of the Invention

[0014] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0015] In view of the above-mentioned and / or existing problems in an existing industrial process supervision management system based on multimodal data processing, the present invention is proposed.

[0016] Therefore, the problem to be solved by the present invention is how to provide a process supervision effect that can achieve a multimodal data processing factory.

[0017] In order to solve the above technical problems, the present invention provides the following technical solutions: an industrial process supervision and management system based on multimodal data processing, which includes:

[0018] Data acquisition module, data fusion module, anomaly detection module, decision optimization module and visualization interaction center:

[0019] The data acquisition module includes a visual sensor array, an acoustic monitoring unit and an environmental parameter sensor group, and transmits multi-source sensor data to the data fusion module in real time via industrial Ethernet;

[0020] The data fusion module includes a timestamp alignment unit and a multimodal feature generation unit, and the multimodal feature generation unit forms a multimodal feature vector and sends it to the anomaly detection module;

[0021] The anomaly detection module includes a dynamic modeling unit and an association judgment unit. The dynamic modeling unit generates a real-time threshold curve for the equipment operating condition and receives the multimodal feature vector from the data fusion module to calculate the feature deviation. When the confidence score of the anomaly of the multi-source data exceeds a preset threshold, the association judgment unit sends a joint alarm signal to the decision optimization module and the visualization interaction center.

[0022] The decision optimization module includes a fault inference engine and a parameter control unit. After the fault inference engine receives the multimodal anomaly confidence score from the anomaly detection module, the parameter control unit drives the optimization instruction to the production line execution mechanism.

[0023] The visualization interaction center couples the raw data of the data acquisition module with the analysis results of the anomaly detection module, and pushes key alarm events to the remote monitoring terminal.

[0024] As a preferred solution of the industrial process supervision and management system based on multimodal data processing described in the present invention, the data acquisition module includes:

[0025] A visual sensor array, an acoustic monitoring unit, and an environmental parameter sensor group. The visual sensor array is deployed at the core workstation of the production line to capture real-time data. The acoustic monitoring unit integrates a microphone array to collect the voiceprint signal of the device operation. The environmental parameter sensor group is configured to periodically detect vibration, temperature and humidity parameters, and transmit multi-source sensor data in real time to the data fusion module via industrial Ethernet.

[0026] The visual sensor array includes multispectral industrial cameras and infrared thermal imagers. The multispectral industrial cameras are arranged in an interlaced manner to cover the production line monitoring area. The infrared thermal imager is installed on a rotating pan-tilt platform to dynamically track the hot spots of the equipment and generate temperature matrix data frames.

[0027] As a preferred solution of the industrial process supervision and management system based on multimodal data processing described in the present invention, the data fusion module includes:

[0028] A timestamp alignment unit and a multimodal feature generation unit. After receiving the raw data from the data acquisition module, the timestamp alignment unit uses a hardware-level pulse triggering mechanism to align the video data, acoustic signals, and environmental parameters. The multimodal feature generation unit deploys a deep residual convolutional network to extract video semantic features, and in parallel processes the acoustic signals through Mel-frequency cepstral coefficient calculations. It also fuses the multi-dimensional sensor time series data through a spatiotemporal feature splicing layer to form a unified multimodal feature vector and sends it to the anomaly detection module.

[0029] The hardware-level pulse triggering mechanism adopted by the timestamp alignment unit is specifically:

[0030] A GPS synchronization clock chip is embedded in the environmental parameter sensor group. By generating one signal per second, a global synchronization pulse is broadcast to the visual sensor array and the acoustic monitoring unit, triggering the hardware timer of the trimodal data acquisition device to reset to zero.

[0031] As a preferred solution of the industrial process supervision and management system based on multimodal data processing described in the present invention, the dynamic modeling unit generates a real-time threshold curve of the equipment operating condition, including:

[0032] The multimodal feature vectors were divided into 30-second window periods. The root mean square value of the vibration feature, the total frequency domain energy of the video feature, and the variance of the acoustic cepstral coefficients in each window were calculated. A dynamic probability distribution model of the features was generated based on the kernel density estimation algorithm, and a real-time threshold curve was drawn in the interval of [0.05, 0.95] according to the cumulative distribution function value.

[0033] As a preferred solution of the industrial process supervision and management system based on multimodal data processing described in the present invention, the decision optimization module includes:

[0034] A fault inference engine and a parameter control unit. After receiving the multimodal anomaly confidence score from the anomaly detection module, the fault inference engine generates an interpretable causal chain analysis by associating the historical fault database with a Bayesian network. This drives the parameter control unit to send optimization instruction messages to the production line actuator and simultaneously feeds back the control parameter adjustment log to the visualization interaction center.

[0035] When the fault inference engine generates causal chain analysis through the Bayesian network, it adopts a dual inference mechanism:

[0036] The dual reasoning mechanism includes forward reasoning and backward reasoning. The forward reasoning includes traversing the conditional probability table in the historical fault database according to the current anomaly confidence score and updating the marginal probability distribution of the node event.

[0037] The reverse reasoning includes reversely correcting the prior probability weight coefficient of the Bayesian network after the parameter control unit receives the state feedback of the actuator.

[0038] As a preferred solution of the industrial process supervision and management system based on multimodal data processing described in the present invention, the visualization interaction center includes:

[0039] The visualization interaction center integrates the digital twin map and the event tracing panel, and connects through real-time data from the industrial bus. The digital twin map dynamically renders a three-dimensional virtual model of the production line and simultaneously marks the heat map of abnormal areas. The event tracing panel couples the raw data of the data acquisition module with the analysis results of the anomaly detection module, supports the synchronous playback of video surveillance, voiceprint maps and sensor waveform data according to the timeline, and pushes key alarm events to remote monitoring terminals.

[0040] As a preferred solution of the industrial process supervision management system based on multimodal data processing described in the present invention, the event tracing panel includes:

[0041] The synchronous playback function of the event tracing panel is implemented as follows:

[0042] A trimodal data cache queue with a unified time base is established. When the user selects the time point when an abnormal event occurs:

[0043] Video surveillance data starts decoding and playing 5 seconds before the selected time;

[0044] The voiceprint spectrum is reconstructed through short-time Fourier transform to obtain the energy time-frequency matrix of the frequency band below 20kHz during the abnormal period;

[0045] The sensor waveform data is dynamically visualized after the sampling rate is increased to 1kHz using the cubic spline interpolation algorithm.

[0046] As a preferred solution of the industrial process supervision and management system based on multimodal data processing described in the present invention, the digital twin map dynamically renders the three-dimensional virtual model of the production line and simultaneously annotates the abnormal area heat map, including:

[0047] Abnormal events and their handling processes are superimposed on the heat map of the abnormal area.

[0048] In a second aspect, some embodiments of the present invention provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation method of the above-mentioned first aspect.

[0049] In a third aspect, some embodiments of the present invention provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any one of the implementations of the first aspect is implemented.

[0050] The present invention provides an industrial process monitoring and management system based on multimodal data processing. Through multimodal data fusion and dynamic modeling, this system achieves breakthroughs in detection speed, positioning accuracy, and decision-making reliability. In particular, hardware-level time synchronization, bimodal association judgment, and Bayesian dual reasoning mechanisms effectively address technical bottlenecks such as timing inaccuracies, single-source misjudgments, and insufficient causal analysis in traditional industrial monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] in:

[0054] Figure 1 This is a process structure diagram of an industrial process supervision management system based on multimodal data processing in Example 1. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0057] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0058] Example 1

[0059] Reference Figure 1 This is the first embodiment of the present invention, which provides an industrial process supervision and management system based on multimodal data processing, which includes:

[0060] The data acquisition module 100 includes a visual sensor array 101, an acoustic monitoring unit 102, and an environmental parameter sensor group 103. The visual sensor array 101 is deployed at the core workstation of the production line to capture real-time video stream data. The acoustic monitoring unit 102 integrates a microphone array to collect the voiceprint signal of the device operation. The environmental parameter sensor group 103 is configured to periodically detect vibration, temperature and humidity parameters and transmit them to the data fusion module 200 via industrial Ethernet.

[0061] The visual sensor array 101 includes a multispectral industrial camera 101a and an infrared thermal imager 101b. The multispectral industrial camera 101a is arranged in an interlaced manner to cover the production line monitoring area. The infrared thermal imager 101b is installed on a rotating pan-tilt platform to dynamically track the hot spots of the equipment and generate temperature matrix data frames.

[0062] The data fusion module 200 includes a timestamp alignment unit 201 and a multimodal feature generation unit 202. The timestamp alignment unit 201 uses a pulse trigger mechanism to synchronize video data, acoustic signals, and environmental parameters. The multimodal feature generation unit 202 deploys a deep residual convolutional network to extract video semantic features, calculates Mel-frequency cepstral coefficients in parallel to process acoustic signals, and fuses multidimensional sensor time series data through a feature splicing layer.

[0063] The hardware-level pulse triggering mechanism adopted by the timestamp alignment unit 201 is specifically:

[0064] A GPS synchronization clock chip is embedded in the environmental parameter sensor group 103, which generates one signal per second and broadcasts a global synchronization pulse to the visual sensor array 101 and the acoustic monitoring unit 102, triggering the hardware timer of the trimodal data acquisition device to reset to zero.

[0065] The anomaly detection module 300 includes a dynamic modeling unit 301 and an association judgment unit 302. The dynamic modeling unit 301 generates a real-time threshold curve of the equipment operating condition based on sliding window statistical analysis. The association judgment unit 302 uses a bidirectional attention mechanism to compare the correlation between visual behavior features and acoustic features and outputs a multimodal anomaly confidence score.

[0066] The dynamic modeling unit 301 generates a real-time threshold curve of the equipment operating condition including:

[0067] The multimodal feature vectors were divided into 30-second window periods. The root mean square value of the vibration feature, the total frequency domain energy of the video feature, and the variance of the acoustic cepstral coefficients in each window were calculated. A dynamic probability distribution model of the features was generated based on the kernel density estimation algorithm, and a real-time threshold curve was drawn in the interval of [0.05, 0.95] according to the cumulative distribution function value.

[0068] The decision optimization module 400 includes a fault inference engine 401 and a parameter control unit 402. The fault inference engine 401 generates an interpretable causal chain analysis by associating a historical fault database with a Bayesian network. The parameter control unit 402 converts the optimization instructions into OPC UA protocol messages and sends them to the execution mechanism.

[0069] When the fault reasoning engine 401 generates a causal chain analysis through a Bayesian network, a dual reasoning mechanism is used:

[0070] The dual reasoning mechanism includes forward reasoning and backward reasoning. The forward reasoning includes traversing the conditional probability table in the historical fault database according to the current anomaly confidence score and updating the marginal probability distribution of the node event.

[0071] The reverse reasoning includes reversely correcting the prior probability weight coefficient of the Bayesian network after the parameter control unit 402 receives the state feedback of the actuator.

[0072] The visualization interaction center 500 integrates a digital twin map 501 and an event tracing panel 502. The digital twin map 501 dynamically renders a three-dimensional virtual model of the production line and annotates heat maps of abnormal areas. The event tracing panel 502 supports synchronized playback of video surveillance, voiceprint maps, and sensor waveform data along a timeline. The digital twin map 501 dynamically renders a three-dimensional virtual model of the production line and annotates heat maps of abnormal areas. The event tracing panel 502 combines the raw data from the data acquisition module 100 with the analysis results of the anomaly detection module 300, supports synchronized playback of video surveillance, voiceprint maps, and sensor waveform data along a timeline, and pushes key alarm events to remote monitoring terminals.

[0073] The event tracing panel 502 includes:

[0074] The synchronous playback function of the event tracing panel 502 is implemented as follows:

[0075] A trimodal data cache queue with a unified time base is established. When the user selects the time point when an abnormal event occurs:

[0076] Video surveillance data starts decoding and playing 5 seconds before the selected time;

[0077] The voiceprint spectrum is reconstructed through short-time Fourier transform to obtain the energy time-frequency matrix of the frequency band below 20kHz during the abnormal period;

[0078] The sensor waveform data is dynamically visualized after the sampling rate is increased to 1kHz using the cubic spline interpolation algorithm.

[0079] Abnormal events and their handling processes are superimposed on the heat map of the abnormal area.

[0080] Example 2

[0081] The second embodiment of the present invention is different from the first embodiment in that it also includes the following test preparation and implementation process:

[0082] Test preparation and implementation process:

[0083] This example uses a welding production line at an automobile manufacturer as a test scenario to verify the performance of an industrial process monitoring and control system based on multimodal data processing in a complex industrial environment. The test targets 12 welding robots, 8 conveyor drive motors, and their associated cooling systems, covering a total monitoring area of ​​1,200 square meters. The experiment lasted 30 consecutive days, collecting 2.3 TB of data.

[0084] Hardware deployment:

[0085] The visual sensor array 101 consists of 36 multispectral industrial cameras 101a and 6 infrared thermal imagers 101b, arranged in a staggered pattern on a 3-meter track above the welding stations, covering the entire welding torch operating area. Each multispectral camera captures a 1280×1024 resolution video stream at 30fps, while the infrared thermal imager uses a pan-tilt platform to achieve 360° scanning and a temperature detection range of 0-500°C.

[0086] The acoustic monitoring unit 102 includes a 24-channel microphone array with a sampling rate of 48kHz per channel. It is deployed at a height of 1.5 meters around key nodes of the equipment to monitor welding arc sound patterns and abnormal noises from motor bearings.

[0087] The environmental parameter sensor group 103 includes 32 vibration sensors (range 0-50g) and 32 temperature and humidity sensors (accuracy ±0.5°C), which are arranged on the device base at intervals of 1.2 meters.

[0088] Data synchronization mechanism:

[0089] Using a hardware-level pulse triggering mechanism, the environmental parameter sensor group 103 includes a built-in GPS synchronization clock chip, generating a synchronization pulse signal every second. Tests show that the timestamp alignment error of the three-modal data is less than 1ms, meeting the timing consistency requirements of the dynamic modeling unit 301.

[0090] Anomaly detection process:

[0091] The dynamic modeling unit 301 analyzes multimodal feature vectors using a 30-second window, calculating the vibration root mean square (RMS), video frequency energy (0-15Hz), and acoustic MFCC variance. When kernel density estimation generates the dynamic threshold curve, the cumulative distribution function confidence interval is set to [0.05, 0.95]. When the feature deviation between any two modalities exceeds the threshold, the association decision unit 302 triggers a joint alarm.

[0092] Decision optimization implementation:

[0093] Fault inference engine 401 accesses a database containing 1,523 historical fault records. The Bayesian network uses dual inference parameters: a conditional probability update step size α = 0.85 for forward inference and a priori weight correction coefficient β = 0.72 for backward inference. Parameter control unit 402 implements closed-loop control of six actuators, including welding current and conveyor speed.

[0094] Table 1: Timestamp alignment performance comparison

[0095]

[0096]

[0097] Table 2: Anomaly Detection Response Timeline

[0098]

[0099] Table 3: Fault location accuracy

[0100]

[0101] Table 4: Parameter adjustment effectiveness

[0102]

[0103] Table 5: Heatmap update delay table

[0104]

[0105] Table 6: Alarm event statistics

[0106]

[0107] Time Synchronization Performance (Table 1): The hardware-level pulse triggering mechanism reduces multimodal data alignment error to sub-millisecond levels (0.5-0.8ms), a 98.3% reduction compared to traditional software synchronization methods. This feature ensures that the dynamic modeling unit 301 can accurately establish a spatiotemporal correlation model of multimodal features, avoiding misjudgments caused by timing misalignment.

[0108] Anomaly Detection Efficiency (Table 2): Bimodal association detection reduces response time by 56.3%-76.5%, and composite anomaly detection achieves a response speed of 1.2 seconds. Through the multi-source confidence fusion of the association decision unit 302, the false alarm rate is controlled at 0.9%-3.2%, a 42.7% reduction compared to single-modal detection. This demonstrates that the deep residual network and MFCC feature extraction of the multimodal feature generation unit 202 effectively improve anomaly characterization capabilities.

[0109] Decision-making optimization results (Table 3-4): The Bayesian network's dual reasoning mechanism improves fault location accuracy to 95.2%-98.5%, and the causal chain depth reaches 3-5 levels, a 29.6 percentage point improvement over traditional expert systems. The closed-loop response time of parameter control unit 402 is less than 3 seconds, and the steady-state error is less than 1.2%, demonstrating that the causal chain analysis of fault reasoning engine 401 can effectively guide actuator adjustments.

[0110] Visualization Performance (Table 5): The digital twin map 501's heat map update latency is less than 92ms, a 4.8-fold improvement compared to traditional SCADA systems, meeting real-time monitoring requirements. The event tracing panel 502's trimodal data synchronous playback function (Table 6) achieves a 98.3% effective alarm rate and a false alarm rate of less than 1.5%, significantly outperforming existing single-dimensional alarm systems.

[0111] The data from this example demonstrates that this system, through multimodal data fusion and dynamic modeling, achieves breakthroughs in detection speed, positioning accuracy, and decision-making reliability. In particular, hardware-level time synchronization, bimodal association judgment, and Bayesian dual reasoning mechanisms effectively address technical bottlenecks such as timing inaccuracies, single-source misjudgments, and insufficient causal analysis, which plague traditional industrial monitoring systems.

[0112] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or components referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention; the terms "first", "second", and "third" are only used for descriptive purposes and should not be understood as indicating or implying relative importance. In addition, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, an indirect connection through an intermediate medium, or it can be internal communication between two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An industrial process supervision and management system based on multimodal data processing, characterized by: It includes a data acquisition module (100), a data fusion module (200), an anomaly detection module (300), a decision optimization module (400) and a visualization interaction center (500): The data acquisition module (100) comprises a visual sensor array (101), an acoustic monitoring unit (102) and an environmental parameter sensor group (103), and transmits multi-source sensor data to the data fusion module (200) in real time via industrial Ethernet; The data fusion module (200) includes a timestamp alignment unit (201) and a multimodal feature generation unit (202), wherein the multimodal feature generation unit (202) forms a multimodal feature vector and sends it to the anomaly detection module (300); The anomaly detection module (300) includes a dynamic modeling unit (301) and an associated judgment unit (302). The dynamic modeling unit (301) generates a real-time threshold curve of the equipment working condition and receives the multimodal feature vector from the data fusion module (200) to calculate the feature deviation. When the multi-source data anomaly confidence score exceeds a preset threshold, the associated judgment unit (302) sends a joint alarm signal to the decision optimization module (400) and the visualization interaction center (500). The decision optimization module (400) includes a fault inference engine (401) and a parameter control unit (402). After the fault inference engine (401) receives the multimodal anomaly confidence score from the anomaly detection module (300), the parameter control unit (402) is driven to send an optimization instruction to a production line execution mechanism. The visualization interaction center (500) couples the original data of the data acquisition module (100) and the analysis results of the anomaly detection module (300) to push key alarm events to the remote monitoring terminal.

2. The industrial process supervision and management system based on multimodal data processing according to claim 1 is characterized in that The data acquisition module (100) comprises: A visual sensor array (101), an acoustic monitoring unit (102) and an environmental parameter sensor group (103), wherein the visual sensor array (101) is deployed at a core workstation of a production line to capture real-time data, the acoustic monitoring unit (102) is integrated with a microphone array to collect device operation voiceprint signals, and the environmental parameter sensor group (103) is configured to periodically detect vibration, temperature and humidity parameters, and transmit multi-source sensor data to the data fusion module (200) in real time via industrial Ethernet; The visual sensor array (101) includes a multispectral industrial camera (101a) and an infrared thermal imager (101b). The multispectral industrial camera (101a) covers the production line monitoring area in a staggered arrangement. The infrared thermal imager (101b) is installed on a rotating pan-tilt platform to dynamically track the hot spots of the equipment and generate a temperature matrix data frame.

3. The industrial process supervision and management system based on multimodal data processing according to claim 1 is characterized in that: The data fusion module (200) comprises: A timestamp alignment unit (201) and a multimodal feature generation unit (202), wherein the timestamp alignment unit (201) receives the raw data from the data acquisition module (100) and uses a hardware-level pulse triggering mechanism to achieve alignment of the video data, the acoustic signal and the environmental parameters; the multimodal feature generation unit (202) deploys a deep residual convolutional network to extract video semantic features, and in parallel processes the acoustic signal by calculating the Mel-frequency cepstral coefficients, and fuses the multi-dimensional sensor time series data through a spatiotemporal feature splicing layer to form a unified multimodal feature vector and send it to the anomaly detection module (300); The hardware-level pulse triggering mechanism adopted by the timestamp alignment unit (201) is specifically: A GPS synchronization clock chip is embedded in the environmental parameter sensor group (103), which generates one signal per second and broadcasts a global synchronization pulse to the visual sensor array (101) and the acoustic monitoring unit (102), thereby triggering the hardware timer of the trimodal data acquisition device to reset to zero.

4. The industrial process supervision and management system based on multimodal data processing according to claim 3 is characterized in that: The dynamic modeling unit (301) generates a real-time threshold curve of the equipment operating condition including: The multimodal feature vectors were divided into 30-second window periods. The root mean square value of the vibration feature, the total frequency domain energy of the video feature, and the variance of the acoustic cepstral coefficients in each window were calculated. A dynamic probability distribution model of the features was generated based on the kernel density estimation algorithm, and a real-time threshold curve was drawn in the interval of [0.05, 0.95] according to the cumulative distribution function value.

5. The industrial process supervision and management system based on multimodal data processing according to claim 1 is characterized in that: The decision optimization module (400) includes: A fault inference engine (401) and a parameter control unit (402), wherein the fault inference engine (401) receives the multimodal anomaly confidence score from the anomaly detection module (300), generates an interpretable causal chain analysis by associating the historical fault database with a Bayesian network, drives the parameter control unit (402) to send an optimization instruction message to the production line execution mechanism, and simultaneously feeds back a control parameter adjustment log to the visualization interaction center (500); When the fault inference engine (401) generates a causal chain analysis through a Bayesian network, a dual inference mechanism is adopted: The dual reasoning mechanism includes forward reasoning and backward reasoning. The forward reasoning includes traversing the conditional probability table in the historical fault database according to the current anomaly confidence score and updating the marginal probability distribution of the node event. The reverse reasoning includes reversely correcting the prior probability weight coefficient of the Bayesian network after the parameter control unit (402) receives the state feedback of the actuator.

6. The industrial process supervision and management system based on multimodal data processing according to claim 1 is characterized in that: The visualization interaction center (500) includes: The visualization interaction center (500) integrates the digital twin map (501) and the event tracing panel (502), and is connected through the real-time data of the industrial bus. The digital twin map (501) dynamically renders the three-dimensional virtual model of the production line and simultaneously marks the abnormal area heat map. The event tracing panel (502) couples the original data of the data acquisition module (100) and the analysis results of the abnormality detection module (300), supports the synchronous playback of video monitoring, voiceprint maps and sensor waveform data according to the time axis, and pushes key alarm events to the remote monitoring terminal.

7. The industrial process supervision and management system based on multimodal data processing according to claim 6 is characterized in that: The event tracing panel (502) includes: The synchronous playback function of the event tracing panel (502) is implemented as follows: A trimodal data cache queue with a unified time base is established. When the user selects the time point when an abnormal event occurs: Video surveillance data starts decoding and playing 5 seconds before the selected time; The voiceprint spectrum is reconstructed through short-time Fourier transform to obtain the energy time-frequency matrix of the frequency band below 20kHz during the abnormal period; The sensor waveform data is dynamically visualized after the sampling rate is increased to 1kHz using the cubic spline interpolation algorithm.

8. The industrial process supervision management system based on multimodal data processing according to claim 6 is characterized in that: The digital twin map (501) dynamically renders the three-dimensional virtual model of the production line and simultaneously marks the abnormal area heat map, including: Abnormal events and their handling processes are superimposed on the heat map of the abnormal area.

9. An electronic device, characterized in that include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the system according to any one of claims 1 to 8.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that When the instructions are executed by a processor, the processor implements the system according to any one of claims 1 to 8.

Images