Multi-mode driven super-large scale neural network controller and danger early warning method thereof

Abstract

The invention provides a multi-mode-driven super-large-scale neural network controller and a danger early warning method thereof. The multi-mode-driven super-large-scale neural network controller comprises an audio and video acquisition module, a multi-sensor module, a data fusion module, an industrial data driving basic model, a controller decision module and an early warning and communication module. The intelligent controller and the method thereof have the beneficial effects that the intelligent controller and the method thereof integrate an audio and video system and multi-sensor information, perform cognitive calculation by using an audio and video neural network compatible with multiple data modalities, and complete environment intelligent evaluation and danger early warning; according to the invention, the audio and video neural network compatible with multiple data modals is applied to multi-modal data cognition and analysis of high-risk environments such as a coal mine for the first time; efficient fusion and cooperative processing of audio and video data and multi-sensor data are realized; an accurate and real-time danger early warning and accident response mechanism is provided, and the safety is remarkably improved; and an embedded platform is used to complete early warning of the intelligent controller.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control and safety early warning, and in particular relates to a multimodal driven ultra-large-scale neural network controller and its danger early warning method. Background Technology

[0002] In high-risk environments such as coal mining and industrial manufacturing, equipment operation and safety are always critical concerns. However, existing technologies generally suffer from the following problems: Data acquisition is limited: Traditional monitoring systems rely on only a single type of data source, such as video surveillance or temperature sensors, which makes it difficult to accurately reflect the overall state of the environment.

[0003] Insufficient analytical capabilities: Existing intelligent controllers are mostly based on rule models, which cannot effectively process complex multimodal data (such as audio, video, sensor signals, etc.).

[0004] Lack of real-time early warning: Most systems can only respond after an accident occurs, lacking an effective early warning mechanism.

[0005] Cognitive technologies based on multimodal compatible audio-visual neural networks can process complex multimodal data and possess stronger semantic understanding and reasoning capabilities; however, their application in high-risk industrial environments is currently lacking. Therefore, designing a multimodal-driven, ultra-large-scale neural network controller that integrates multimodal compatible audio-visual neural networks for real-time environmental assessment and early warning is of great significance. Summary of the Invention

[0006] In view of this, the present invention aims to propose a multimodal driven ultra-large-scale neural network controller and its hazard warning method to solve the technical problems of traditional systems being unable to fully integrate multi-source information, having insufficient analysis capabilities, and lacking real-time warning.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows: In a first aspect, the present invention provides a multimodal driven ultra-large-scale neural network controller, including an audio and video acquisition module, a multi-sensor module, a data fusion module, an industrial data driven basic model, a controller decision module, and an early warning and communication module. The audio and video acquisition module and the multi-sensor module communicate with the data fusion module. The data fusion module communicates with the controller decision module in sequence through the industrial data driven basic model. The controller decision module communicates with the early warning and communication module.

[0008] Furthermore, the audio and video acquisition module is used to acquire audio and video information in the work scene. The technical parameters of the audio and video acquisition module include video resolution, audio sampling rate, video compression encoding, and audio compression encoding.

[0009] Furthermore, the multi-sensor module is used to collect environmental parameters, including temperature, humidity, gas concentration, vibration, and pressure. The technical parameters of the multi-sensor module include the accuracy of the temperature sensor, the accuracy of the humidity sensor, the frequency range of the vibration sensor, and the gas concentration sensor.

[0010] Furthermore, the data fusion module is used to process audio and video data and multi-sensor data in a unified manner to eliminate timing errors and data inconsistencies; the technical parameters of the data fusion module include: data timing synchronization error ≤10ms, data processing throughput ≥100MB / s.

[0011] Furthermore, the industrial data-driven basic model utilizes a pre-trained large-scale multimodal model to perform semantic understanding and hazard assessment on the input data; the controller decision module generates control commands based on the analysis results of the industrial data-driven basic model and a preset rule base.

[0012] Furthermore, the early warning and communication module is used to achieve multi-terminal synchronous alarm and remote data transmission. It supports communication protocols such as Modbus TCP, MQTT, and RTSP.

[0013] Secondly, based on the same concept, the present invention also provides a danger warning method for a multimodal driven ultra-large-scale neural network controller, including three methods. Method 1 is a process of directly inputting structured data into a multimodal compatible audio-visual neural network. Method 2 is to collect data, set a neural network editing input strategy, and then input the data into a multimodal compatible audio-visual neural network. Method 3 is to collect data, preprocess the data structure, set a neural network editing input strategy, and then input the data into a multimodal compatible audio-visual neural network.

[0014] Furthermore, the process of directly inputting structured data into a multi-modal compatible audio-visual neural network includes: A1. Acquire raw data of the work scene in real time through audio and video acquisition module and multi-sensor module; A2. The data fusion module performs industrial data-driven basic model preprocessing, time synchronization, and data structuring on audio and video data and sensor data, and outputs structured multimodal data packets. Data preprocessing includes: Decode and synchronize audio and video data to ensure consistent timestamps; Normalize and convert the format of sensor data; Among them, timing synchronization: By using timestamp alignment technology, the timing synchronization error between audio / video data and sensor data is controlled to ≤10ms; Data structuring includes encapsulating audio / video data and sensor data into a unified multimodal data grid. A3. Input structured multimodal data packets into the industrial data-driven basic model for semantic understanding and hazard assessment; A4. The controller decision module generates control commands based on the analysis results of multi-data modality compatible audio and video neural networks and the preset rule base. A5. Multi-terminal synchronous alarm and remote data transmission are achieved through the early warning and communication module.

[0015] Furthermore, after acquiring data and setting the neural network editing input strategy, the data is input into a multi-modal compatible audio-visual neural network, specifically including: B1. Acquire raw data of the work scene in real time through audio and video acquisition modules and multi-sensor modules; B2. By using neural networks to extract features and optimize the input of the collected raw data, a multimodal data format suitable for multimodal audio and video neural network processing is generated; B3. Input the optimized multimodal feature data from the neural network into the industrial data-driven basic model for semantic understanding and hazard assessment; B4. The controller decision module generates control commands based on the analysis results of multi-data modality compatible audio and video neural networks and the preset rule base. B5. Multi-terminal synchronous alarm and remote data transmission are achieved through the early warning and communication module.

[0016] Furthermore, after data acquisition, data structure preprocessing, and setting the neural network editing input strategy, the data is input into a multi-modal compatible audio-visual neural network, specifically including: C1. Acquire raw data of the working scene in real time through audio and video acquisition module and multi-sensor module; C2. Clean, format, and perform preliminary feature extraction on the raw data to generate unified structured data, providing input for the neural network; C4. Input the optimized multimodal feature data into the industrial data-driven basic model for semantic understanding and hazard assessment; C5. The controller decision module generates control commands based on the analysis results of multi-data modality compatible audio and video neural networks and the preset rule base. C6. Multi-terminal synchronous alarm and remote data transmission are achieved through the early warning and communication module.

[0017] Compared with existing technologies, the multimodal driven ultra-large-scale neural network controller and its hazard warning method described in this invention have the following advantages: The multimodal driven ultra-large-scale neural network controller and its hazard warning method described in this invention are intelligent controllers and methods that integrate audio-visual systems and multi-sensor information, utilize multi-data modality compatible audio-visual neural networks for cognitive computing, and complete intelligent environmental assessment and hazard warning. This invention is the first to apply multi-data modality compatible audio-visual neural networks to multimodal data cognition and analysis in high-risk environments such as coal mines; it achieves efficient fusion and collaborative processing of audio-visual data and multi-sensor data; it provides accurate and real-time hazard warning and accident response mechanisms, significantly improving safety; and it uses an embedded platform to complete the intelligent controller's early warning. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the working principle of the multimodal driven ultra-large-scale neural network controller described in an embodiment of the present invention; Figure 2 This is a schematic diagram of the process of one method described in an embodiment of the present invention; Figure 3 This is a schematic diagram of the process of Method Two as described in an embodiment of the present invention; Figure 4 This is a schematic diagram of the process of Method 3 as described in the embodiment of the present invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0020] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] like Figures 1 to 4 As shown, the multimodal driven ultra-large-scale neural network controller mainly includes the following modules: audio and video acquisition module, multi-sensor module, data fusion module, industrial data driven basic model, controller decision module, and early warning and communication module. The audio and video acquisition module and multi-sensor module communicate with the data fusion module. The data fusion module communicates with the controller decision module through the industrial data driven basic model. The controller decision module communicates with the early warning and communication module.

[0024] This invention is the first to apply a multi-modal compatible audio-visual neural network to the cognition and analysis of multimodal data in high-risk environments such as coal mines; it achieves efficient fusion and collaborative processing of audio-visual data and multi-sensor data; it provides a precise and real-time hazard warning and accident response mechanism, significantly improving safety; and it uses an embedded platform to complete the intelligent controller warning.

[0025] The intelligent controller integrates an audio-visual system. The communication controller is used in scenarios such as conveyor belts and working surfaces. By integrating audio and video information and inputting multi-data modality compatible audio-video neural networks to recognize audio and video, and combining the information from various sensors in the controller system, it completes intelligent assessment of the working environment, intelligently provides the overall operating status and accident assessment of the system, and completes early warning of various dangerous accidents.

[0026] In a preferred embodiment of the present invention, the audio and video acquisition module is used to acquire audio and video information in the work scene.

[0027] Technical parameters: Video resolution: Supports 1080p to 4K, frame rate 15fps to 60fps; Audio sampling rate: 44.1kHz to 96kHz, signal-to-noise ratio ≥90dB; Video compression encoding: Supports H.264 / H.265; Audio compression encoding: AAC supported.

[0028] Application scenarios: monitoring belt operation noise, equipment noise at the work face, equipment operation screens, and the worker's working environment, etc.

[0029] In a preferred embodiment of the present invention, the multi-sensor module is used to collect environmental parameters and the start-up and shutdown of the equipment, including temperature, humidity, gas concentration, vibration, pressure, etc.

[0030] Technical parameters: Temperature sensor accuracy: ±0.5℃; Humidity sensor accuracy: ±3%; Vibration sensor frequency range: 1Hz~1kHz, accuracy ±5%; Gas concentration sensor: Supports monitoring of methane, carbon monoxide, and oxygen concentrations, with an error of ≤1%.

[0031] In a preferred embodiment of the present invention, the data fusion module is used to process audio and video data and multi-sensor data in a unified manner to eliminate timing errors and data inconsistencies.

[0032] Technical parameters: Data timing synchronization error ≤10ms; Data processing throughput ≥ 100MB / s.

[0033] In a preferred embodiment of the present invention, the industrial data-driven basic model utilizes a pre-trained large-scale multimodal model to perform semantic understanding and risk assessment on the input data.

[0034] In a preferred embodiment of the present invention, the controller decision module generates control commands based on the analysis results of a multi-data modality compatible audio-visual neural network and a preset rule base.

[0035] Output: Activate audible and visual alarm, adjust equipment operating parameters, and automatically trigger shutdown procedure.

[0036] In a preferred embodiment of the present invention, the early warning and communication module is used to realize multi-terminal synchronous alarm and remote data transmission. It supports communication protocols such as Modbus TCP, MQTT, and RTSP.

[0037] How a multimodal driven, ultra-large-scale neural network controller works: Data Acquisition and Fusion: The audio / video acquisition module and the multi-sensor module acquire environmental data in real time, and the data fusion module performs preprocessing and formatting. In this embodiment, the microphone, vibration sensor, temperature sensor, and camera on the traditional device are located around the device, in areas prone to failure. The data uses high-precision sensors, and formatting removes erroneous data. This algorithm is written into the sensors, and the data transmitted from the sensors is all valid, non-FF format data.

[0038] Multi-modal compatible audio and video neural network analysis: The fused multimodal data is input into the industrial data-driven basic model for hazard identification, environmental status assessment, and risk level determination. In this embodiment, since the operating scenario in coal mines is relatively simple, there are assessment mechanisms for environmental factors, equipment status, and interference factors, such as dangerous conditions involving people, vehicles, and goods. The judgment process is as follows: First, the video footage is frame-by-frame extracted for video feature identification. The multimodal compatible audio-video neural network is completed by a convolutional neural network. The characteristic values ​​of the output of the convolutional neural network are compared with the characteristic values ​​of the audio processed by the recurrent neural network during the same time period and the sensor data to ensure a unified timestamp.

[0039] Decision-making and control: Based on the analysis results, the controller decision module generates corresponding control commands.

[0040] In this embodiment, several states were analyzed, which can be roughly divided into three categories: Low risk: The environment and equipment are in normal condition, with no obvious risk factors.

[0041] Medium risk: Minor anomalies exist that may affect equipment performance but do not immediately threaten safety.

[0042] High risk: There is a serious anomaly that may lead to equipment failure or a safety accident.

[0043] The judgment strategy is: First, initial weighting coefficients are pre-defined based on the importance of each state in the coal working environment to safe production.

[0044] 1. Extract video features; 2. Extract audio features; 3. Extract sensor data features; 4. Shared attention layer analysis enables cross-modal fusion; 5. Weighted calculation of feature confidence; During the judgment process, a dynamic weight parameter is set, and the weight setting is confirmed to ensure that Risk_Score accurately reflects the risk level through downhole environment scenario testing.

[0045] Early warning output: The early warning and communication module sends early warning information to relevant personnel and triggers corresponding equipment responses. In this embodiment, the response response is equivalent to a notification subscription; clients that have subscribed to services on the device will receive notifications, such as those provided by the host computer service.

[0046] This invention also proposes a hazard warning method for a multimodal driven ultra-large-scale neural network controller, including three methods: Method 1 is a process of directly inputting structured data into a multimodal compatible audio-visual neural network; Method 2 is to collect data, set a neural network editing input strategy, and then input the data into a multimodal compatible audio-visual neural network; Method 3 is to collect data, preprocess the data structure, set a neural network editing input strategy, and then input the data into a multimodal compatible audio-visual neural network.

[0047] In a preferred embodiment of the present invention, the process of directly inputting structured data into a multi-modal compatible audio-visual neural network specifically includes: 1. Acquire raw data of the work scene in real time through audio and video acquisition modules and multi-sensor modules.

[0048] 2. The data fusion module preprocesses and structures audio and video data and sensor data to eliminate timing errors and data inconsistencies.

[0049] Data preprocessing: Decode and synchronize audio and video data to ensure consistent timestamps.

[0050] Normalize and convert the format of sensor data (e.g., convert analog signals to digital signals).

[0051] Timing synchronization: By using timestamp alignment technology, the timing synchronization error of audio and video data and sensor data is controlled to ≤10ms.

[0052] Data structuring: Encapsulate audio / video data and sensor data into a unified multimodal data format (such as JSON or Protobuf), including fields such as timestamp, data type (video / audio / sensor), data value, and metadata (device ID, location, etc.).

[0053] Data throughput: Ensure data processing throughput ≥100MB / s and support real-time stream processing.

[0054] Output: Generates structured multimodal data packets. Example format: json{ "timestamp": "2025-07-27T20:12:00.000Z", "video": { resolution: 1920x1080 "frame_rate": 30, "data": "<H.264 encoded frame> " }, "audio": { "sample_rate": 44100, "data": "<AAC encoded samples> " }, "sensors": { Temperature: 25.5 "humidity": 60.2, "vibration": 50.0, "methane": 0.01 }, "metadata": { "device_id": "DEV001", "location": "Workshop_A" } } 3. Input structured multimodal data into the industrial data-driven basic model for semantic understanding and hazard assessment.

[0055] Structured data packets are transmitted in real time via API or data pipeline to a multi-modal compatible audio-visual neural network. This network combines the multi-modal data to identify hazards (such as equipment overheating, belt misalignment, and gas leaks). It then outputs the risk level (e.g., low, medium, high) and a description of the specific event.

[0056] Output example: json{ "risk_level": "High", "events": [ {"type": "video", "description": "Belt misalignment detected", "confidence": 0.97}, {"type": "audio", "description": "Abnormal noise from motor", "confidence": 0.92}, {"type": "sensor", "description": "Methane concentration exceeds0.01%", "confidence": 0.99} ] } 4. The controller decision module generates control commands based on the analysis results of a multi-data modality compatible audio-visual neural network and a preset rule base; Control commands and warning information are sent to relevant equipment and personnel via Modbus TCP, MQTT, and RTSP.

[0057] 5. Achieve multi-terminal synchronous alarm and remote data transmission through the early warning and communication module. During testing, it was found that directly feeding the data into a multi-modal compatible audio-video neural network caused problems with the real-time understanding and processing of video data during model debugging. When processing large amounts of data, the understanding and output of instructions would deviate significantly.

[0058] In a preferred embodiment of the present invention, the process of acquiring data, setting a neural network editing input strategy, and then inputting the data into a multi-modal compatible audio-visual neural network specifically includes: 1. Acquire raw data of the work scene in real time through audio and video acquisition modules and multi-sensor modules.

[0059] 2. By using neural networks to extract features and optimize the input of the collected raw data, a multimodal data format suitable for multimodal audio and video neural network processing is generated.

[0060] To enhance data processing, feature extraction is added. Convolutional Neural Networks (CNNs) are used to extract frame-level features from video data, identifying key visual elements (such as equipment motion trajectories and worker actions). Recurrent Neural Networks (RNNs) or Transformers are used to extract temporal features from audio data, detecting abnormal sound patterns. Fully connected neural networks are used to reduce the dimensionality and enhance the features of sensor data, extracting key environmental parameters (such as temperature anomalies and vibration peaks).

[0061] Based on the input requirements of multi-modal compatible audio and video neural networks, data priorities are dynamically adjusted (e.g., prioritizing the processing of high-risk sensor data).

[0062] The attention mechanism is used to weight multimodal data to enhance key features (such as data related to anomalous events).

[0063] The unified data format is JSON or Protobuf, containing timestamps, feature vectors, metadata, etc. Example format: { "timestamp":2025-07-27T20:12:00.000Z", "video_features": { "motion_vector": [0.1, 0.2, 0.3], "object_detected": "belt_misalignment" }, "audio_features": { "frequency_peaks": [100, 500, 1200], "anomaly_score": 0.85 }, "sensor_features": { "temperature_anomaly": 1.2, "methane_level": 0.01 }, "metadata": { "device_id": "DEV001", "location": "Workshop_A" } } Timing synchronization: By using a neural network time alignment algorithm, the timing synchronization error of audio, video and sensor data is controlled to ≤200ms.

[0064] 3. Input the optimized multimodal feature data from the neural network into the industrial data-driven basic model for semantic understanding and hazard assessment.

[0065] 4. The controller decision module generates control commands based on the analysis results of the multi-data modality compatible audio and video neural network and the preset rule base.

[0066] 5. Multi-terminal synchronous alarm and remote data transmission are achieved through the early warning and communication module.

[0067] Testing Notes: For Method 2, due to hardware resource limitations, the timeliness of the output and extraction cannot be effectively judged. Secondly, there is a synchronization problem. Convolutional neural networks, recurrent neural networks, and fully connected neural networks are used for video, audio, and data. Time series and long short-term memory are used, and different layers are combined for mixed training with shared weights, but the training effect is not ideal.

[0068] In a preferred embodiment of the present invention, the process of acquiring data, preprocessing data structures, setting neural network editing input strategies, and then inputting the data into a multi-modal compatible audio-visual neural network specifically includes: 1. Acquire raw data of the work scene in real time through audio and video acquisition modules and multi-sensor modules.

[0069] 2. The raw data is cleaned, formatted, and preliminary feature extracted to generate unified structured data, providing high-quality input for the neural network.

[0070] Convert video, audio, and sensor data to a uniform time resolution (e.g., 60Hz, approximately 16.67ms / time): Video: Frame interpolation or downsampling to 60fps.

[0071] Audio: Segmented aggregation, generating audio segments every 16.67ms.

[0072] Sensor: Downsample or interpolate to 60Hz.

[0073] The unified data format is JSON, which includes timestamps, data values, and metadata. Example: { "timestamp": "2025-07-27T20:12:00.000Z", "video": {"frame_data": "<H.264 encoded frame> ", "resolution": "1920x1080"}, "audio": {"sample_data": "<AAC encoded samples> ", "sample_rate":44100}, "sensors": {"temperature": 25.5, "humidity": 60.2, "vibration": 50.0, "methane": 0.01}, "metadata": {"device_id": "DEV001", "location": "Workshop_A"} } 3. By using hybrid training of CNN, RNN and FCNN, combined with image recognition, semantic recognition and data processing strategies, optimized feature data is generated to solve the weight sharing problem.

[0074] A multimodal joint network is constructed, comprising CNN (video), RNN (audio), and FCNN (sensor) branches, sharing an intermediate feature fusion layer (attention mechanism). Lightweight methods are used to extract video features, and lightweight methods are used to extract audio temporal features (such as frequency peaks and anomaly ratings), simplifying the process to a 2-3 layer fully connected network. Sensor features (such as temperature anomalies and gas concentrations) are extracted, and a Transformer-based attention module fuses the multimodal features, outputting a unified feature vector. The shared layer uses a global attention module to fuse multimodal features, and the weights are optimized through joint training.

[0075] High-risk features are dynamically weighted using an attention mechanism.

[0076] The unified feature output format is JSON, which includes timestamps, feature vectors, and metadata. Example: { "timestamp": "2025-07-27T20:12:00.000Z", "video_features": {"motion_vector": [0.1, 0.2, 0.3], "object_detected": "belt_misalignment"}, "audio_features": {"frequency_peaks": [100, 500, 1200], "anomaly_score": 0.85}, "sensor_features": {"temperature_anomaly": 1.2, "methane_level":0.01}, "metadata": {"device_id": "DEV001", "location": "Workshop_A"} } Timing synchronization optimization: A time window (5ms) is introduced in the shared layer to align the multimodal feature output.

[0077] Use timestamp checksum buffers to ensure that the synchronization error of each network output is ≤50ms.

[0078] 4. Input the optimized multimodal feature data into the industrial data-driven basic model for semantic understanding and hazard assessment.

[0079] 5. The controller decision module generates control commands based on the analysis results of the multi-data modality compatible audio and video neural network and the preset rule base.

[0080] 6. Multi-terminal synchronous alarm and remote data transmission are achieved through the early warning and communication module. In actual use, small models of 0.6 and 1.7B can be used to verify the training effect of this method.

[0081] The network structure employed in this invention does not rely on large-scale natural language data for driving, but rather is based on a simplified neural network framework. Before input, the data first undergoes an embedded feature mapping process, where data from different application scenarios are preprocessed hierarchically according to a predetermined method, and then uniformly converted into a unified input to facilitate subsequent modeling and inference. The network structure used can be a lightweight convolutional neural network, whose architecture differs fundamentally from the MOE mechanism. The latter relies on task-level policy scheduling, while this invention uses a fixed data input channel to implement the processing flow, eliminating the need for additional dynamic scheduling.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal driven super large scale neural network controller and its hazard warning method, characterized in that: The application relates to an industrial data-driven safety monitoring system, which comprises an audio and video acquisition module, a multi-sensor module, a data fusion module, an industrial data-driven basic model, a controller decision module and an early warning and communication module, wherein the audio and video acquisition module and the multi-sensor module are in communication with the data fusion module, the data fusion module is in communication with the controller decision module through the industrial data-driven basic model, and the controller decision module is in communication with the early warning and communication module.

2. The multi-modal driven very large scale neural network controller of claim 1, wherein: The audio and video acquisition module is used for collecting audio and video information in a working scene, and the technical parameters of the audio and video acquisition module include video resolution, audio sampling rate, video compression encoding and audio compression encoding.

3. The multi-modal driven very large scale neural network controller of claim 1, wherein: The multi-sensor module is used for collecting environmental parameters, and the start and stop of the equipment, wherein the environmental parameters include temperature, humidity, gas concentration, vibration and pressure; the technical parameters of the multi-sensor module include temperature sensor accuracy, humidity sensor accuracy, vibration sensor frequency range and gas concentration sensor.

4. The multi-modal driven very large scale neural network controller of claim 1, wherein: The data fusion module is used for uniformly processing audio and video data and multi-sensor data, and eliminating time sequence error and data inconsistency. The technical parameters of the data fusion module include that the data time sequence synchronization error is less than or equal to 10 ms, and the data processing throughput is greater than or equal to 100 MB / s.

5. The multi-modal driven very large scale neural network controller of claim 1, wherein: The industrial data-driven basic model uses a pre-trained large-scale multi-modal model to perform semantic understanding and risk assessment on input data. The controller decision module generates control instructions based on the analysis results of the industrial data-driven basic model and a preset rule base.

6. The multi-modal driven very large scale neural network controller of claim 1, wherein: The early warning and communication module is used for realizing multi-terminal synchronous alarm and remote data transmission, and supports Modbus TCP, MQTT and RTSP communication protocols.

7. The method of hazard warning of the multi-modal driven super large scale neural network controller, applied to the multi-modal driven super large scale neural network controller of any one of claims 1-6, characterized in that: The application comprises three modes, wherein mode one is a process of directly structuring data and inputting the data into a multi-data modal compatible audio and video neural network; mode two is a process of collecting data, setting a neural network editing input strategy and then inputting the data into the multi-data modal compatible audio and video neural network; and mode three is a process of collecting data, data structure preprocessing, setting a neural network editing input strategy and then inputting the data into the multi-data modal compatible audio and video neural network.

8. The method of hazard alerting of a multi-modal driven very large scale neural network controller according to claim 6, characterized in that: The process of directly structuring data and inputting the data into the multi-data modal compatible audio and video neural network comprises the following steps: A1, acquiring original data of a working scene in real time through the audio and video acquisition module and the multi-sensor module; A2, performing industrial data-driven basic model preprocessing, time sequence synchronization and data structuring on audio and video data and sensor data through the data fusion module, and outputting a structured multi-modal data packet; The data preprocessing comprises the following steps: decoding and frame synchronization of the audio and video data, so as to ensure the consistency of time stamps; normalization and format conversion of the sensor data; The time sequence synchronization comprises the following steps: controlling the time sequence synchronization error of the audio and video data and the sensor data to be less than or equal to 10 ms by using a time stamp alignment technology; The data structuring comprises the following steps: encapsulating the audio and video data and the sensor data into a unified multi-modal data grid; A3, inputting the structured multi-modal data packet into the industrial data-driven basic model for semantic understanding and risk assessment; A4, generating control instructions through the controller decision module according to the analysis results of the multi-data modal compatible audio and video neural network and the preset rule base. A5. Realize multi-terminal synchronous alarm and remote data transmission through the early warning and communication module.

9. The method of hazard alerting of a multi-modal driven very large scale neural network controller according to claim 6, characterized in that: Collect data - set the input strategy of the neural network to input the multi-data modal compatible audio and video neural network, specifically including: B1. Real-time acquisition of original data of the working scene through the audio and video acquisition module and the multi-sensor module; B2. Feature extraction and input optimization of the collected original data through the neural network to generate multi-modal data format suitable for multi-data modal compatible audio and video neural network processing; B3. Input the multi-modal feature data optimized by the neural network into the industrial data driven basic model for semantic understanding and risk assessment; B4. Generate control instructions according to the multi-data modal compatible audio and video neural network analysis results and the preset rule base through the controller decision module; B5. Realize multi-terminal synchronous alarm and remote data transmission through the early warning and communication module.

10. The method of hazard alerting of a multi-modal driven very large scale neural network controller according to claim 6, characterized in that: Collect data - data structure preprocessing - set the input strategy of the neural network to input the multi-data modal compatible audio and video neural network, specifically including: C1. Real-time acquisition of original data of the working scene through the audio and video acquisition module and the multi-sensor module; C2. Clean, format and preliminarily extract features of the original data to generate unified structured data for neural network input; C4. Input the optimized multi-modal feature data into the industrial data driven basic model for semantic understanding and risk assessment; C5. Generate control instructions according to the multi-data modal compatible audio and video neural network analysis results and the preset rule base through the controller decision module; C6. Realize multi-terminal synchronous alarm and remote data transmission through the early warning and communication module.

Images