Industrial safety monitoring method, system and equipment based on multi-source perception and medium

Abstract

The invention discloses an industrial safety monitoring method, system and device based on multi-source perception, and a medium, mainly relates to the technical field of safety monitoring, and is used for solving the problems that an existing scheme has limitations in the aspects of perception stability, decision real-time performance and interaction efficiency, and a monitoring system for personnel-robot dynamic interaction is not formed yet. Comprising the steps of detecting whether a marked person and a robot have preset abnormal behaviors or not, and detecting whether an execution action sequence of the marked person triggers a robot scheduling instruction or not; when it is detected that the robot has the preset abnormal behavior, whether the preset abnormal behavior has a preset adjustment instruction or not is determined, and the corresponding preset adjustment instruction is triggered; when the preset adjustment instruction does not exist, sending a power cut-off instruction to the corresponding robot; and when it is detected that the marked person has the preset abnormal behavior, according to the real-time position of the marked person, determining a robot which triggers a language alarm, and issuing a preset person alarm voice to a voice alarm terminal of the corresponding robot.

Description

Technical Field

[0001] This application relates to the field of industrial safety monitoring technology, and in particular to an industrial safety monitoring method, system, equipment and medium based on multi-source sensing. Background Technology

[0002] In the field of industrial safety monitoring, traditional technologies mainly rely on manual inspections, physical isolation, and basic sensor alarms, which have drawbacks such as slow response, high false alarm rates, and limited functionality. For example, underground coal mines still use methods such as manual monitoring and infrared sensing, which are difficult to use for real-time and accurate identification of abnormal behavior.

[0003] While existing intelligent monitoring systems incorporate AI technologies, such as deep learning-based video analytics, they still face challenges in human-robot collaborative scenarios. High-speed movements of industrial robots can lead to serious personal injury, and existing solutions have limitations in perception stability (e.g., sensor anti-interference capabilities), real-time decision-making, and interaction efficiency (e.g., data silos between multiple systems). Furthermore, although multi-sensor fusion technology has been applied to scenarios such as track collision avoidance, a comprehensive monitoring system for dynamic human-robot interactions has yet to be established. Summary of the Invention

[0004] This application provides an industrial safety monitoring method, system, equipment, and medium based on multi-source sensing, to address the limitations of existing solutions in terms of sensing stability, real-time decision-making, and interaction efficiency, and the lack of a monitoring system for dynamic human-robot interaction.

[0005] Firstly, this application provides an industrial safety monitoring method based on multi-source sensing, the method comprising: Connect various monitoring devices configured on the workstation to the data acquisition terminal; Multi-source monitoring data is collected through the data acquisition terminal to obtain a human-machine fusion dynamic map of marked personnel and robots, wherein the dynamic map includes real-time location, execution action sequence and location sequence; By using the real-time positions, action sequences, and location sequences of the marked personnel and robots, and based on a safety rule base and a large language model, the system detects whether the marked personnel and robots exhibit any pre-set abnormal behaviors, and whether the action sequences of the marked personnel trigger robot scheduling instructions. When a robot is detected to exhibit pre-defined abnormal behavior, the system determines whether a pre-defined adjustment command exists for that behavior and triggers the corresponding command. If no pre-defined adjustment command exists, the system sends a power-off command to the corresponding robot. When a marked person is detected to exhibit pre-defined abnormal behavior, the system determines the robot that triggers the voice alarm based on the marked person's real-time location and sends a pre-defined warning voice message to the corresponding robot's voice alarm terminal.

[0006] In one implementation of this application, various monitoring devices configured on the workstation are connected to the data acquisition terminal, specifically including: All monitoring devices configured on the workstations are connected to the data acquisition terminal via a pre-set Ethernet interface.

[0007] In one implementation of this application, the data acquisition terminal includes at least a positioning device and a video acquisition terminal; Multi-source monitoring data is collected through a data acquisition terminal to obtain a human-robot fusion dynamic map, which includes real-time location, execution action sequence, and location sequence, specifically including: The data acquisition terminal collects multi-source monitoring data uploaded by various monitoring devices. Multi-source monitoring data corresponding to the same acquisition time are treated as a sensor array; The multi-source monitoring data generated by the positioning device is parsed into the real-time positions of personnel and robots, and then a position sequence corresponding to a preset time window is constructed. The multi-source data of video acquisition parameters are parsed into the actions performed by marked personnel and robots, and then the sequence of actions corresponding to the preset time window is constructed.

[0008] In one implementation of this application, before utilizing the real-time positions of the marked personnel and robot, the sequence of executed actions, and the position sequence, based on a safety rule base and a large language model, the method further includes: Obtain the rule base set for marking personnel and robots; establish the rule base set and identification code index based on the respective identification codes of personnel and robots; create a safety rule base using the rule base set, identification code index, and rule base set. Acquire sample data; the sample data includes: real-time location, execution action sequence, location sequence, and whether there are pre-defined abnormal behaviors. Using sample data, a well-trained large language model is obtained.

[0009] In one implementation of this application, the real-time positions, execution action sequences, and position sequences of the marking personnel and robot are used. Based on a safety rule base and a large language model, the system detects whether the marking personnel and robot exhibit any pre-defined abnormal behaviors, and detects whether the execution action sequence of the marking personnel triggers robot scheduling instructions. Specifically, this includes: Obtain the identification codes of the marked personnel and robots; extract the corresponding rule set from the security rule base based on the identification codes; By utilizing the location restriction rules, preset normal behavior sequence restriction rules, and preset normal position sequence restriction rules in the rule base set, it can be determined whether the marked personnel and robots have preset abnormal behaviors. When no abnormal behavior is detected, the real-time location, action sequence, and position sequence of the marked personnel or robot are input into the corresponding trained large language model to determine whether the preset abnormal behavior exists. The sequence of actions is input into a preset set of scheduling instruction sequences, and the robot scheduling instruction is triggered based on the similarity.

[0010] In one implementation of this application, when a preset abnormal behavior is detected in the robot, it is determined whether there is a preset adjustment instruction for the preset abnormal behavior, and the corresponding preset adjustment instruction is triggered, specifically including: Based on the preset mapping table between preset abnormal behaviors and preset adjustment instructions, determine whether there is a preset adjustment instruction for the current preset abnormal behavior; When present, the corresponding preset adjustment command is triggered.

[0011] In one implementation of this application, when a pre-defined abnormal behavior is detected in a marked person, the robot that triggers the voice alarm is determined based on the real-time location of the marked person, and a pre-defined warning voice message is sent to the corresponding robot's voice alarm terminal, specifically including: When a pre-defined abnormal behavior is detected in a marked person, the real-time location of the marked person is obtained; Based on the real-time location of the marked personnel, determine the nearest robot; Send a preset warning message to the voice alarm terminal of the nearest robot.

[0012] Secondly, this application provides an industrial safety monitoring system based on multi-source sensing, the system comprising: The access module is used to connect various monitoring devices configured on the workstation to the data acquisition terminal; The acquisition module is used to collect multi-source monitoring data through the data acquisition terminal to obtain a human-machine fusion dynamic map of marked personnel and robots, wherein the dynamic map includes real-time location, execution action sequence and location sequence; The response module utilizes the real-time positions, execution action sequences, and position sequences of the marked personnel and robots, based on a safety rule base and a large language model, to detect whether the marked personnel and robots exhibit preset abnormal behaviors, and whether the marked personnel's execution action sequences trigger robot scheduling instructions. When preset abnormal behaviors are detected in the robot, it determines whether there are preset adjustment instructions for the preset abnormal behaviors and triggers the corresponding preset adjustment instructions. If no preset adjustment instructions are found, a power-off instruction is sent to the corresponding robot. When preset abnormal behaviors are detected in the marked personnel, based on the marked personnel's real-time position, it determines the robot that triggers the voice alarm and sends a preset personnel warning voice message to the corresponding robot's voice alarm terminal.

[0013] Thirdly, this application provides an industrial safety monitoring device based on multi-source sensing, the device comprising: processor; And a memory containing executable code, which, when executed, causes the processor to perform an industrial safety monitoring method based on multi-source sensing, as described above.

[0014] Fourthly, this application provides a non-volatile computer storage medium storing computer instructions thereon, which, when executed, implement an industrial safety monitoring method based on multi-source sensing as described above.

[0015] As can be seen from the above technical solutions, this application has the following advantages: By integrating multiple sensors and acquiring real-time data, the system enhances the perception stability and decision-making efficiency of industrial safety monitoring. Specifically, by configuring various monitoring devices on workstations and connecting them to a unified data acquisition terminal, the system achieves simultaneous acquisition of multi-source data on the real-time positions, action sequences, and position sequences of personnel and robots. This integrated design effectively solves the data silo problem caused by scattered sensors or incompatible protocols in traditional solutions. Furthermore, through joint analysis using a safety rule base and a large language model, it can detect preset abnormal behaviors (such as robot malfunction or dangerous human actions) at millisecond speeds. The advantages directly brought by this technical solution include: 1) High-precision dynamic monitoring – cross-validation of multi-sensor data reduces the false alarm rate of a single device; 2) Hierarchical response mechanism – preset adjustment commands are prioritized for abnormal robot behavior, with power cut-off only triggered when ineffective, ensuring safety and reducing production interruptions; 3) Intelligent interaction capabilities – real-time two-way feedback in human-machine collaborative scenarios is achieved through directional warnings from voice alarm terminals.

[0016] Furthermore, a complete monitoring system covering dynamic human-robot interaction has been constructed, whose technical features directly address the shortcomings of existing systems in terms of interaction efficiency and adaptability. By associating the sequence of actions performed by marked personnel with robot scheduling instructions, the system can proactively identify operational intentions and predict potential conflicts (such as automatically triggering obstacle avoidance instructions when personnel accidentally enter high-risk areas). The introduction of a safety rule base makes anomaly judgments interpretable, while a large language model enhances the robustness of behavior recognition in complex scenarios. The direct effects of this combination of technologies include: 1) Full-process automation—no manual intervention is required from data collection to instruction issuance, and the response speed is improved by at least one order of magnitude compared to traditional solutions; 2) Flexible scalability—preset rules and models can be updated through configuration to adapt to different industrial scenarios; 3) Dual safety protection—simultaneously monitoring equipment status and personnel behavior, avoiding blind spots in single-dimensional monitoring. Attached Figure Description

[0017] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an industrial safety monitoring method based on multi-source sensing provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the internal structure of an industrial safety monitoring system based on multi-source sensing, provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the internal structure of an industrial safety monitoring device based on multi-source sensing, provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0023] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0024] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] The embodiment provides an industrial safety monitoring method based on multi-source sensing, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps: Step 110: Connect the various monitoring devices configured on the workstation to the data acquisition terminal.

[0026] In some embodiments, various monitoring devices configured on the workstation are connected to the data acquisition terminal, specifically including: All monitoring devices configured on the workstations are connected to the data acquisition terminal via a pre-set Ethernet interface.

[0027] In this embodiment, a "workstation" refers to a physical or logical work unit in an industrial environment used to perform specific production tasks. Its spatial scope needs to be defined according to the actual application scenario (such as assembly line workstations, warehousing operation areas, etc.). "Monitoring equipment" includes, but is not limited to, vision sensors (such as industrial cameras) and position trackers (such as UWB tags). These devices must have data output protocols compatible with the data acquisition terminal. The "data acquisition terminal" is a central processing unit responsible for summarizing and standardizing multi-source monitoring data. Its hardware configuration must meet real-time processing requirements (such as multi-core CPUs and dedicated FPGA acceleration modules). "Preset Ethernet interface" specifically refers to an industrial-grade network interface conforming to the IEEE 802.3 standard, using a TCP / IP protocol stack to achieve communication between devices. Interface types include RJ45, M12, etc., with a transmission rate of no less than 1Gbps to ensure data real-time performance. Accessing through a unified Ethernet interface can avoid data parsing delays caused by heterogeneous protocols (such as the mixing of Modbus and CAN buses) in traditional solutions, while also supporting plug-and-play device expansion.

[0028] Furthermore, the connection between monitoring equipment and data acquisition terminals must adhere to the following technical specifications: 1) The physical layer uses shielded twisted-pair (STP) or fiber optic transmission, and its electromagnetic interference resistance must meet the IEC 61000-4-3 standard; 2) The network layer is configured with static IP addresses or reserves a fixed address range via DHCP to ensure stable device identification; 3) The application layer data encapsulation format must include a timestamp (accuracy ≤ 1ms) and a device ID field to facilitate subsequent data correlation analysis. The technical benefits of this standardized access method include: 1) Reduced system integration complexity, requiring only physical connection and network configuration for device addition; 2) Improved data acquisition reliability, as the Ethernet protocol's retransmission mechanism effectively addresses transient noise in industrial environments; 3) Support for remote monitoring and diagnostics, enabling real-time device status queries via network management protocols (such as SNMP).

[0029] Step 120: Collect multi-source monitoring data through the data acquisition terminal to obtain a human-machine fusion dynamic map of the marked personnel and robot, wherein the dynamic map includes real-time location, execution action sequence and location sequence.

[0030] It should be noted that the data acquisition end includes at least a positioning device and a video acquisition end.

[0031] Specifically, multi-source monitoring data is collected through a data acquisition terminal to obtain a human-machine fusion dynamic map of marked personnel and robots. This dynamic map includes real-time location, execution action sequence, and location sequence, specifically including: The system collects multi-source monitoring data uploaded by various monitoring devices through the data acquisition terminal; it treats the multi-source monitoring data corresponding to the same acquisition time as a sensor array; it parses the multi-source monitoring data generated by the positioning device into the real-time positions of personnel and robots, and then constructs the position sequence corresponding to the preset time window; it parses the multi-source data of the video acquisition terminal parameters into the actions performed by personnel and robots, and then constructs the action sequence corresponding to the preset time window.

[0032] In this embodiment, the "data acquisition end" refers to the hardware system used to integrate and preprocess multi-source monitoring data. Its core components include: 1) a positioning device, which uses UWB (Ultra-Wideband) or LiDAR technology to achieve centimeter-level positioning accuracy through TOA (Time of Arrival) or TDOA (Time Difference of Arrival) algorithms, and outputs structured data containing device ID, three-dimensional coordinates (X, Y, Z), and timestamps; 2) a video acquisition end, which consists of an industrial camera (such as a global shutter CMOS sensor) and an embedded processor, supports H.264 real-time encoding, and extracts human key point coordinates and action category labels through models such as OpenPose or YOLO. The "multi-source monitoring data" includes raw sensor data (such as distance measurements from the positioning device and RGB frames from the camera) and intermediate processing results (such as position estimation after Kalman filtering and action recognition confidence). Each data stream needs to be synchronized at the microsecond level through the IEEE 1588 Precise Time Protocol (PTP). The "sensor array" is a multi-dimensional data matrix sorted by timestamp, and its data structure is defined as follows: t0[location data 1, video data 1]; t1[location data 2, video data 2]; etc.

[0033] Furthermore, the "location sequence" refers to a set of coordinate points continuously arranged at sampling intervals (e.g., 10ms) within a preset time window (e.g., 1 second). Its construction method includes: 1) applying sliding window mean filtering to the raw coordinates output by the positioning device to eliminate jump noise; 2) aligning the sampling time sequence of different devices using the Dynamic Time Warping (DTW) algorithm. The "action sequence" is generated by parsing from the video capture end. The specific process is as follows: 1) extracting the coordinates of 21 key points of the human body using the MediaPipe framework; 2) classifying the action features of consecutive frames based on an LSTM network (e.g., preset categories such as "reaching out" and "squatting"); 3) sorting the action labels within the window according to their occurrence time to form a sequence. This sequence data is ultimately stored in JSON format, with fields including "device_id", "timestamp", "sequence_type" (location / action), and "data_array" (numerical or category array), for subsequent analysis by the security rule base and large language model.

[0034] Step 130: Using the real-time positions, execution action sequences, and position sequences of the marked personnel and robots, based on the safety rule base and large language model, detect whether the marked personnel and robots have preset abnormal behaviors, and detect whether the execution action sequence of the marked personnel triggers robot scheduling instructions.

[0035] Specifically, by utilizing the real-time positions, action sequences, and location sequences of the marking personnel and robots, and based on a safety rule base and a large language model, the system detects whether the marking personnel and robots exhibit pre-defined abnormal behaviors, and whether the marking personnel's action sequences trigger robot scheduling instructions. This includes: Obtain the identification codes of the marked personnel and robots; extract the corresponding rule set from the safety rule base based on the identification codes; use the position restriction rules, preset normal behavior sequence restriction rules, and preset normal position sequence restriction rules in the rule set to determine whether the marked personnel and robots have preset abnormal behaviors; when no abnormal behaviors are found, input the real-time position, execution action sequence, and position sequence of the marked personnel or robots into the corresponding trained large language model to determine whether preset abnormal behaviors exist; input the execution action sequence into the preset scheduling instruction sequence set, and determine whether to trigger robot scheduling instructions based on similarity.

[0036] It should be noted that in this step, "marked personnel" refers to workplace personnel authorized and identified by the system. Their identity is bound by a unique identification code (such as an RFID wristband number or a work badge QR code), and a special tag is attached to the monitoring data to distinguish them from ordinary personnel. The "Safety Rule Base" is a structured database containing three core rules: 1) Position Restriction Rules—defining the dangerous areas within the working radius of each robot (e.g., a no-entry zone within 0.5 meters of the robotic arm's range of motion), judging violations by the spatial relationship between real-time coordinates and preset geometric boundaries (e.g., polygonal fences); 2) Preset Normal Behavior Sequence Restriction Rules—storing compliant action templates (e.g., standard operating procedures for assembly lines), using an edit distance algorithm to compare the difference between real-time action sequences and templates; 3) Preset Normal Position Sequence Restriction Rules—recording typical movement paths of equipment (e.g., AGV navigation routes), detecting abnormal displacements deviating from the trajectory using a Hidden Markov Model (HMM). The rule base is stored in key-value pairs. The index field is "device_id" + "rule_type", and the value field is the judgment condition in JSON format (such as {"range":[1.2, 3.5], "threshold": 0.8}).

[0037] "Large Language Model" specifically refers to a multimodal large language model based on the Transformer architecture. Its input layer integrates real-time location, location sequence (normalized by Min-Max), and action sequence (one-hot encoded), while the output layer is an anomaly probability distribution (softmax activation). The model enhances its robustness through adversarial training, fine-tuning it using 100,000 sets of labeled data collected from industrial scenarios (including 5% anomaly samples), with the loss function being Focal Loss (γ=2.0). "Preset abnormal behaviors" include, but are not limited to: 1) Personnel-related behaviors—entering high-risk areas, unauthorized operation of equipment, falls, etc.; 2) Robot-related behaviors—speeding (>1.5m / s), joint angle exceeding limits, repetitive failures, etc. "Robot scheduling instructions" are a standardized set of control instructions, including three types: "avoidance" (priority 3), "pause" (priority 2), and "emergency stop" (priority 1). The matching degree between the real-time action sequence and the instruction template is calculated using cosine similarity, and the corresponding instruction is triggered when the similarity is >0.85.

[0038] Supplementary explanation of the large language model training process: The training process for the large language model based on the Transformer architecture adopted in this application is as follows: 100,000 sets of labeled data (including 5% anomalous samples) collected from industrial scenes were used. The data sources included real-time location sequences fused from multiple sensors, motion capture data, and environmental context features. The raw data underwent rigorous cleaning: 1) Location data was filtered using sliding window mean filtering to eliminate noise; 2) Video data was processed using the MediaPipe framework to extract the coordinates of 21 keypoints, which were then classified into preset action categories (such as "reaching out" and "squatting") by an LSTM network. The model uses Focal Loss (γ=2.0) as the loss function and enhances robustness through adversarial training. The input layer integrates Min-Max normalized location sequences, one-hot encoded action sequences, and environmental features, while the output layer is an anomalous probability distribution (softmax activation). Training was conducted using an 8×NVIDIA A100 GPU cluster with a hybrid parallel strategy (data parallelism + model parallelism). The batch size was set to 32, and the learning rate was gradually decayed from 3e-5 using cosine annealing scheduling.

[0039] Step 140: When a robot is detected to have a preset abnormal behavior, determine whether there is a preset adjustment instruction for the preset abnormal behavior and trigger the corresponding preset adjustment instruction; if there is no preset adjustment instruction, send a power cut-off instruction to the corresponding robot; when a marked person is detected to have a preset abnormal behavior, determine the robot that triggers the voice alarm based on the real-time location of the marked person, and send a preset warning voice message to the voice alarm terminal of the corresponding robot.

[0040] In some embodiments, when a preset abnormal behavior is detected in the robot, it is determined whether there is a preset adjustment instruction for the preset abnormal behavior, and the corresponding preset adjustment instruction is triggered, specifically including: Based on the preset mapping table between preset abnormal behaviors and preset adjustment instructions, determine whether there is a preset adjustment instruction for the current preset abnormal behavior; When present, the corresponding preset adjustment command is triggered.

[0041] Specifically, when a pre-defined abnormal behavior is detected in a marked person, the robot that triggers the voice alarm is determined based on the marked person's real-time location. The robot then sends a pre-defined warning voice message to the corresponding robot's voice alarm terminal, including: When a pre-defined abnormal behavior is detected in a marked person, the real-time location of the marked person is obtained; based on the real-time location of the marked person, the nearest robot is determined; and a pre-defined warning voice message is sent to the voice alarm terminal of the nearest robot.

[0042] It should be noted that in this step, "preset abnormal behavior" refers to the predefined personnel or robot states that may cause danger in the industrial safety monitoring system, and their classification and judgment criteria must be clearly stated. Abnormal robot behavior includes, but is not limited to: 1) Overspeeding – triggered when the robot's movement speed exceeds a preset threshold (e.g., 1.5 m / s), with the speed monitored in real time by the encoder and compared with the safety rule base; 2) Joint angle exceeding limits – the robotic arm's rotation exceeds physical limits such as ±180°, calculated by combining IMU sensor data with the inverse kinematics model; 3) Repetitive failures – the same error occurs more than 3 times consecutively within a short period (e.g., 10 seconds) (e.g., positioning loss), with the failure frequency statistically analyzed through the state machine. For abnormal human behavior, the following are included: 1) Entering a high-risk area – UWB positioning coordinates entering within 0.5 meters of the robot's working radius, judged by a geometric fence algorithm; 2) Unauthorized operation – the action sequence is compared with a non-preset instruction template using an edit distance algorithm (similarity <0.7); 3) Fall detection – vertical acceleration (>2g) or attitude angle (pitch angle >60°) is calculated based on MediaPipe keypoint coordinates. The thresholds for all abnormal behaviors must be configurable and support dynamic updates to adapt to different scenario requirements.

[0043] The "Preset Adjustment Instructions" are a set of corrective control commands that the robot can execute. Their design must adhere to a hierarchical response principle: 1) Instruction Type – including "Decelerate to 50% of rated speed" (send speed setting value via CAN bus), "Retreat to safe coordinate point" (send target pose JSON data), "Restart fault module" (send hardware reset signal), etc.; 2) Execution Priority – Adjustment instructions have a lower priority than emergency stop instructions (response time < 5ms) but higher than regular scheduling instructions to ensure production continuity; 3) Parameter Specification – Each instruction must include execution parameters (e.g., {"target_speed": 0.5, "timeout": 5000}). If not completed within the timeout period, it automatically escalates to an emergency stop. Instructions are issued via industrial protocols (EtherCAT or Profinet) and must include CRC checksums and a retransmission mechanism (maximum 3 retries). The "Preset Mapping Table" is a database linking abnormal behaviors to instructions, stored using a hash table. The key is the abnormal behavior ID (e.g., "E1001"), and the value is the instruction type and parameter combination. Updates must be atomic through transaction locks.

[0044] The "Voice Alarm Terminal" is an audio output device integrated into the robot. Its technical implementation must meet the following requirements: 1) Hardware Specifications – Operating voltage 12VDC±10%, signal-to-noise ratio ≥60dB, supporting dynamic range compression (DRC) to adapt to industrial noise environments; 2) Communication Protocol – Receiving cloud voice packets via OPC UA or MQTT, supporting breakpoint resumption and version verification; 3) Content Management – ​​Preset warning voices are divided into three categories: danger warnings (e.g., "You have entered a dangerous area, please evacuate immediately"), operation instructions (e.g., "Please wear protective gloves before touching the equipment"), and emergency notifications (e.g., "Fire detected, please evacuate along the green channel"). Voice files use a 16kHz sampling rate and AAC encoding format, with a single message duration not exceeding 5 seconds. When abnormal human behavior is detected, the system calculates the nearest robot (Euclidean distance algorithm) based on UWB positioning coordinates (accuracy ±3cm), prioritizing terminals with CPU utilization <30% to ensure real-time performance, with voice latency controlled within 200ms.

[0045] The "Power Cut-off Command" is the highest priority control command, and its execution process must be strictly regulated: 1) Triggering Conditions – Triggered only when there is no corresponding adjustment command for abnormal robot behavior (mapping table lookup failure) or the adjustment command execution times out (>500ms); 2) Execution Method – The main power supply is cut off via a safety relay (compliant with IEC 60204-1 standard), while the emergency braking circuit is kept powered; 3) Status Feedback – An execution confirmation signal must be returned to the monitoring system within 50ms, and a log (including timestamp, anomaly ID, and operator ID) must be recorded. All control commands must support bidirectional communication verification. If no ACK confirmation is received, a redundant channel is activated to ensure reliability in industrial environments. The system design must meet functional safety (ISO13849-1 PLd level), and key modules adopt a dual-redundancy architecture.

[0046] In addition, this application Figure 2 This application provides an industrial safety monitoring system based on multi-source sensing. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes: The access module 210 is used to connect various monitoring devices configured on the workstation to the data acquisition terminal.

[0047] The acquisition module 220 is used to acquire multi-source monitoring data through the data acquisition terminal to obtain a human-machine fusion dynamic map of marked personnel and robots, wherein the dynamic map includes real-time location, execution action sequence and location sequence.

[0048] The data acquisition terminal should include at least a positioning device and a video acquisition terminal; The acquisition module 220 includes an acquisition unit, which is used to acquire multi-source monitoring data uploaded by various monitoring devices through a data acquisition terminal; Multi-source monitoring data corresponding to the same acquisition time are treated as a sensor array; The multi-source monitoring data generated by the positioning device is parsed into the real-time positions of personnel and robots, and then a position sequence corresponding to a preset time window is constructed. The multi-source data of video acquisition parameters are parsed into the actions performed by marked personnel and robots, and then the sequence of actions corresponding to the preset time window is constructed.

[0049] The response module 230 is used to detect whether the marked personnel and robot have preset abnormal behaviors based on the real-time positions, execution action sequences, and position sequences of the marked personnel and the robot, and to detect whether the execution action sequence of the marked personnel triggers the robot scheduling command based on the safety rule base and the large language model. When preset abnormal behaviors of the robot are detected, it determines whether there are preset adjustment commands for the preset abnormal behaviors and triggers the corresponding preset adjustment commands. When no preset adjustment commands are detected, it sends a power cut-off command to the corresponding robot. When preset abnormal behaviors of the marked personnel are detected, it determines the robot that triggers the language alarm based on the real-time position of the marked personnel and sends the preset personnel warning voice to the voice alarm terminal of the corresponding robot.

[0050] The above are method embodiments of this application. Based on the same inventive concept, this application also provides an industrial safety monitoring device based on multi-source sensing. Figure 3 As shown, the device includes: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform an industrial safety monitoring method based on multi-source sensing as described in the above embodiments.

[0051] Specifically, various monitoring devices configured on the server side at the workstation are connected to the data acquisition terminal. The data acquisition terminal collects multi-source monitoring data to obtain a dynamic map of the human-machine fusion of the marked personnel and robots. This dynamic map includes real-time location, execution action sequence, and location sequence. Using the real-time location, execution action sequence, and location sequence of the marked personnel and robots, based on a safety rule base and a large language model, it detects whether the marked personnel and robots exhibit preset abnormal behaviors, and whether the marked personnel's execution action sequence triggers robot scheduling instructions. When preset abnormal behaviors are detected in the robot, it determines whether there are preset adjustment instructions for the preset abnormal behaviors and triggers the corresponding preset adjustment instructions. When no preset adjustment instructions are found, a power-off instruction is sent to the corresponding robot. When preset abnormal behaviors are detected in the marked personnel, based on the marked personnel's real-time location, the robot that triggers the language alarm is determined, and a preset personnel warning voice is sent to the corresponding robot's voice alarm terminal.

[0052] In addition, embodiments of this application also provide a non-volatile computer storage medium storing executable instructions, which, when executed, implement the above-described industrial safety monitoring method based on multi-source sensing.

[0053] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An industrial safety monitoring method based on multi-source sensing, characterized in that, The method includes: Connect various monitoring devices configured on the workstation to the data acquisition terminal; Multi-source monitoring data is collected through the data acquisition terminal to obtain a human-machine fusion dynamic map of marked personnel and robots, wherein the dynamic map includes real-time location, execution action sequence and location sequence; By using the real-time positions, action sequences, and location sequences of the marked personnel and robots, and based on a safety rule base and a large language model, the system detects whether the marked personnel and robots exhibit any pre-set abnormal behaviors, and whether the action sequences of the marked personnel trigger robot scheduling instructions. When a robot is detected to exhibit pre-defined abnormal behavior, the system determines whether a pre-defined adjustment command exists for that behavior and triggers the corresponding command. If no pre-defined adjustment command exists, the system sends a power-off command to the corresponding robot. When a marked person is detected to exhibit pre-defined abnormal behavior, the system determines the robot that triggers the voice alarm based on the marked person's real-time location and sends a pre-defined warning voice message to the corresponding robot's voice alarm terminal.

2. The industrial safety monitoring method based on multi-source sensing according to claim 1, characterized in that, Connect various monitoring devices configured on the workstation to the data acquisition terminal, specifically including: All monitoring devices configured on the workstations are connected to the data acquisition terminal via a pre-set Ethernet interface.

3. The industrial safety monitoring method based on multi-source sensing according to claim 1, characterized in that, The data acquisition end should include at least a positioning device and a video acquisition end; Multi-source monitoring data is collected through a data acquisition terminal to obtain a human-machine fusion dynamic map of marked personnel and robots. This dynamic map includes real-time location, execution action sequence, and location sequence, specifically including: The data acquisition terminal collects multi-source monitoring data uploaded by various monitoring devices. Multi-source monitoring data corresponding to the same acquisition time are treated as a sensor array; The multi-source monitoring data generated by the positioning device is parsed into the real-time positions of personnel and robots, and then a position sequence corresponding to a preset time window is constructed. The multi-source data of video acquisition parameters are parsed into the actions performed by marked personnel and robots, and then the sequence of actions corresponding to the preset time window is constructed.

4. The industrial safety monitoring method based on multi-source sensing according to claim 1, characterized in that, Before utilizing the real-time positions, action sequences, and location sequences of marked personnel and robots, and based on a safety rule base and a large language model, the method further includes: Obtain the rule base set for marking personnel and robots; establish the rule base set and identification code index based on the respective identification codes of personnel and robots; create a safety rule base using the rule base set, identification code index, and rule base set. Acquire sample data; the sample data includes: real-time location, execution action sequence, location sequence, and whether there are pre-defined abnormal behaviors. Using sample data, a well-trained large language model is obtained.

5. The industrial safety monitoring method based on multi-source sensing according to claim 1, characterized in that, By utilizing the real-time positions, action sequences, and location sequences of the marked personnel and robots, and based on a safety rule base and a large language model, the system detects whether the marked personnel and robots exhibit pre-defined abnormal behaviors, and whether the action sequences of the marked personnel trigger robot scheduling instructions. Specifically, this includes: Obtain the identification codes of the marked personnel and robots; extract the corresponding rule set from the security rule base based on the identification codes; By utilizing the location restriction rules, preset normal behavior sequence restriction rules, and preset normal position sequence restriction rules in the rule base set, it can be determined whether the marked personnel and robots have preset abnormal behaviors. When no abnormal behavior is detected, the real-time location, action sequence, and position sequence of the marked personnel or robot are input into the corresponding trained large language model to determine whether the preset abnormal behavior exists. The sequence of actions is input into a preset set of scheduling instruction sequences, and the robot scheduling instruction is triggered based on the similarity.

6. The industrial safety monitoring method based on multi-source sensing according to claim 1, characterized in that, When a pre-defined abnormal behavior is detected in the robot, it is determined whether there is a pre-defined adjustment command for the pre-defined abnormal behavior, and the corresponding pre-defined adjustment command is triggered, specifically including: Based on the preset mapping table between preset abnormal behaviors and preset adjustment instructions, determine whether there is a preset adjustment instruction for the current preset abnormal behavior; When present, the corresponding preset adjustment command is triggered.

7. The industrial safety monitoring method based on multi-source sensing according to claim 1, characterized in that, When a pre-defined abnormal behavior is detected in a marked person, the robot that triggers the voice alarm is determined based on the marked person's real-time location. The robot then sends a pre-defined warning message to the corresponding robot's voice alarm terminal, specifically including: When a pre-defined abnormal behavior is detected in a marked person, the real-time location of the marked person is obtained; Based on the real-time location of the marked personnel, determine the nearest robot; Send a preset warning message to the voice alarm terminal of the nearest robot.

8. An industrial safety monitoring system based on multi-source sensing, characterized in that, The system includes: The access module is used to connect various monitoring devices configured on the workstation to the data acquisition terminal; The acquisition module is used to collect multi-source monitoring data through the data acquisition terminal to obtain a human-machine fusion dynamic map of marked personnel and robots, wherein the dynamic map includes real-time location, execution action sequence and location sequence; The response module utilizes the real-time positions, execution action sequences, and position sequences of the marked personnel and robots, based on a safety rule base and a large language model, to detect whether the marked personnel and robots exhibit preset abnormal behaviors, and whether the marked personnel's execution action sequences trigger robot scheduling instructions. When preset abnormal behaviors are detected in the robot, it determines whether there are preset adjustment instructions for the preset abnormal behaviors and triggers the corresponding preset adjustment instructions. If no preset adjustment instructions are found, a power-off instruction is sent to the corresponding robot. When preset abnormal behaviors are detected in the marked personnel, based on the marked personnel's real-time position, it determines the robot that triggers the voice alarm and sends a preset personnel warning voice message to the corresponding robot's voice alarm terminal.

9. An industrial safety monitoring device based on multi-source sensing, characterized in that, The device includes: processor; And a memory having executable code stored thereon, which, when executed, causes the processor to perform an industrial safety monitoring method based on multi-source sensing as described in any one of claims 1-6.

10. A non-volatile computer storage medium, characterized in that, It stores computer instructions, which, when executed, implement an industrial safety monitoring method based on multi-source sensing as described in any one of claims 1-6.

Images