Thermal power plant personnel dangerous behavior early warning method, system and equipment based on multi-modal identification and medium

Abstract

The invention discloses a thermal power plant personnel dangerous behavior early warning method, system and device based on multi-modal identification and a medium, and belongs to the technical field of device fault detection, and the method comprises the steps: obtaining and scheduling a front-end RTSP video stream to a multi-target tracking algorithm sequence, and completing personnel target detection, unique ID distribution and motion trail generation; behavior recognition and electronic fence area judgment are synchronously carried out on each target with the unique ID; and inputting an analysis result into a rule engine for multi-condition logic judgment, and triggering graded early warning feedback according to the risk grade. According to the method, on the basis of video stream management and a multi-target tracking technology, a unique identity label and a continuous movement track are established and maintained for each person, synchronous perception and association binding of individual behaviors and spatial positions are realized through parallel analysis tasks, and the limitation of single analysis dimension is broken through; and logic judgment and risk grading are performed on multi-source information based on a rule engine, so that the spanning from single alarm to comprehensive risk assessment is realized.

Description

Technical Field

[0001] This invention relates to the field of equipment fault detection technology, specifically to a method, system, equipment, and medium for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition. Background Technology

[0002] As a crucial link in energy supply, the safe, stable, and economical operation of thermal power plants' generating units is of paramount importance. With the development of the Industrial Internet of Things (IIoT) and digital technologies, modern thermal power plants have deployed numerous online monitoring systems, such as distributed control systems (DCS), condition monitoring systems (CMS), and laboratory information management systems. These systems continuously generate massive amounts of diverse operational and status data. Fully utilizing this multi-source data to construct accurate fault prediction models and provide early warnings of potential equipment failures has become a core technical means to ensure safe power plant production, reduce unplanned downtime losses, and optimize maintenance strategies. Therefore, developing a method and system for early warning of hazardous behaviors of personnel in thermal power plants that can effectively integrate multi-source heterogeneous data and improve prediction accuracy has significant practical significance and application value.

[0003] However, achieving efficient multi-source data fusion in existing technologies, especially when processing time-series data with significant differences in sampling frequencies, faces severe technical challenges. Process monitoring data from thermal power plants is typically high-frequency data at the second or even millisecond level, while laboratory analysis data reflecting slow system changes are low-frequency data at the hour or day level. Current mainstream processing methods usually employ upsampling or downsampling techniques to forcibly align these multi-rate data on the time axis. Downsampling methods match low-frequency data points by aggregating high-frequency data, but this can lead to the averaging or ignoring of critical transient anomalies, resulting in significant information loss. Conversely, upsampling methods expand low-frequency data to high-frequency time points by padding, but this creates a large amount of stair-step artifact data that does not conform to physical reality, misleading the model to learn false steady states, leading to prediction lag and distortion. This inherent contradiction in unifying data at different scales of change on a single time grid makes it difficult for existing methods to accurately interpret low-frequency trends while preserving high-frequency details, thus severely limiting the performance ceiling of fault prediction models.

[0004] Therefore, there is a need for an optimized intelligent technology control solution for thermal power plants. Summary of the Invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by this invention is that existing thermal power plant fault prediction methods, when processing multi-source heterogeneous time-series data (such as second-level process data and day-level laboratory data) with significant differences in sampling frequencies, suffer from the loss of key high-frequency transient signals or the introduction of low-frequency data artifacts due to their reliance on upsampling or downsampling techniques of a single time grid, which seriously restricts the accuracy and timeliness of the prediction model.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition, comprising,

[0008] RTSP video streams are captured by front-end devices and sent to a multi-target tracking algorithm sequence via video stream management. For each video stream, the channel header is identified, and a unique ID is assigned to the thermal power plant personnel based on the multi-target tracking algorithm, generating a motion trajectory. The thermal power plant personnel with unique IDs are tracked for synchronous analysis. The results of the synchronous analysis are input into a preset rule engine for real-time logical judgment. Based on the results of the real-time logical judgment, early warning triggers and feedback are provided.

[0009] As a preferred embodiment of the multimodal recognition-based method for early warning of hazardous behaviors of personnel in thermal power plants according to the present invention, the step of sending the RTSP video stream to the multi-target tracking algorithm sequence through video stream management includes,

[0010] The RTSP stream address is retrieved from the configuration library, and the decoding library is called to connect and decode the video data, converting it into image frames.

[0011] The decoded image frames are placed in the task queue and scheduled to ensure the timing of distribution.

[0012] Image frames are retrieved from the queue according to a preset strategy and distributed to idle multi-target tracking algorithm instances for processing.

[0013] As a preferred embodiment of the multimodal recognition-based method for early warning of hazardous behaviors of personnel in thermal power plants according to the present invention, the method of assigning unique IDs to personnel in thermal power plants and generating motion trajectories based on multi-target tracking algorithms includes:

[0014] For the input image frames, the target detection model is used to identify all personnel targets in the thermal power plant and obtain their location bounding boxes.

[0015] The bounding boxes of people detected in the current frame are matched with the targets tracked in the previous frame.

[0016] For each successfully matched ID, the motion trajectory sequence is updated based on the position coordinates of the current frame, forming a trajectory point sequence that changes over time.

[0017] As a preferred embodiment of the multimodal recognition-based early warning method for hazardous behaviors of personnel in thermal power plants according to the present invention, the step of tracking and synchronously analyzing personnel in thermal power plants with unique IDs includes:

[0018] For each person target with a unique ID, initiate the analysis task synchronously.

[0019] The analysis task extracts pose features from the image region where the personnel target is located, identifies specific actions, and compares the current position of the personnel target with the coordinates of a predefined danger zone.

[0020] The identified behavior types and regional status information are bound to the unique IDs of individuals, encapsulated into data packets, and then used by the rule engine.

[0021] The beneficial effect of this preferred technical solution is that it binds and encapsulates behavior type and regional status with a unique ID, enabling subsequent logical judgments to make accurate decisions based on fused information.

[0022] As a preferred embodiment of the method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition as described in this invention, the analysis task includes behavior recognition analysis based on skeletal key points and determination of electronic fence areas based on personnel location coordinates.

[0023] The process of inputting the results of synchronous analysis into a preset rule engine for real-time logical judgment includes...

[0024] The rules engine preloads a library of safety rules for thermal power plants defined in IF-THEN format.

[0025] After receiving the synchronous analysis results, the rule engine performs real-time matching of the behavior and location data in the data packet with the IF condition part of each rule.

[0026] Determine whether the following conditions are met simultaneously: whether the person's behavior is running, whether the person's location is in a high-voltage area, and whether the environmental sensor data is abnormal.

[0027] Based on the matching rule weights and condition combinations, the rule engine calculates the risk score and classifies the event into different risk levels according to preset thresholds.

[0028] The beneficial effects of this preferred technical solution are as follows: First, the use of skeletal key point analysis can effectively filter out interference such as clothing and lighting, accurately identify essential actions such as "running" and "falling down", and improve the robustness and accuracy of behavior recognition; Second, the rule engine realizes composite logical judgment of complex and dangerous scenarios by matching multi-dimensional conditions such as "behavior-location-environment" in real time (such as "running in a high-voltage area and the environment is abnormal"), reducing false alarms from single signals; Finally, by calculating a comprehensive risk score and classifying it, a refined early warning from "no risk" to "emergency" is realized.

[0029] As a preferred embodiment of the multimodal recognition-based early warning method for hazardous behaviors of personnel in thermal power plants according to the present invention, the step of triggering and responding to the early warning based on the results of real-time logical judgment includes:

[0030] Based on the logical judgment results, an early warning message containing personnel ID, location, screenshot and risk description is automatically assembled and routed to the preset feedback channel according to the risk level.

[0031] It executes multiple triggering actions, including pushing pop-up alarms to the monitoring center's large screen, sending push notifications to the mobile terminals of relevant safety personnel, and sending activation signals to the linked audible and visual alarm devices.

[0032] The beneficial effects of this preferred technical solution are as follows: First, it automatically assembles complete early warning messages containing personnel ID, location, visual evidence (screenshots), and risk descriptions, providing security personnel with intuitive, comprehensive, and traceable handling basis, greatly shortening the information verification and decision-making time; second, it intelligently routes warning information to different feedback channels according to the risk level, realizing differentiated distribution of early warning information, which avoids excessive interference from low-risk events and ensures strong reach of high-risk alarms; finally, by simultaneously pushing alarms to monitoring screens, mobile terminals, and on-site audio-visual equipment, it constructs a three-dimensional, instantaneous response network from central monitoring to on-site inspection and equipment linkage.

[0033] This invention provides a multimodal recognition-based early warning system for dangerous behaviors of personnel in thermal power plants.

[0034] To address the aforementioned technical problems, this invention provides the following technical solution: a system for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition, comprising: a data acquisition and flow management module, a personnel tracking and trajectory generation module, a multimodal behavior analysis module, an intelligent rule decision engine module, and an early warning distribution and feedback execution module.

[0035] The data acquisition and stream management module is responsible for capturing RTSP video streams through front-end devices and sending the RTSP video streams to the multi-target tracking algorithm sequence through video stream management.

[0036] The personnel tracking and trajectory generation module is responsible for identifying the header of each video stream, assigning a unique ID to the personnel of the thermal power plant based on a multi-target tracking algorithm, and generating a motion trajectory.

[0037] The multimodal behavior analysis module is responsible for tracking and analyzing thermal power plant personnel with unique IDs.

[0038] The intelligent rule decision engine module is responsible for inputting the results of synchronous analysis into a preset rule engine for real-time logical judgment.

[0039] The warning distribution and feedback execution module is responsible for triggering and responding to warnings based on the results of real-time logical judgments.

[0040] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the aforementioned method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition.

[0041] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the aforementioned method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition.

[0042] The beneficial effects of this invention are as follows: First, through the collaborative work of the video stream management module and the multi-target tracking algorithm, efficient and stable decoding and scheduling of multiple video streams are achieved, and a unique ID is assigned to each person target, generating a continuous trajectory. Technically, this ensures accurate and stable tracking and identity maintenance of multiple person targets in large-scale video surveillance scenarios, effectively avoiding target loss or ID jump problems, and providing a reliable data foundation for subsequent analysis.

[0043] Secondly, by simultaneously initiating behavior recognition and electronic fence judgment analysis tasks for each target with a unique ID, parallel and correlated perception of personnel behavior and spatial location is achieved. It can efficiently capture dangerous actions such as "running" and "falling" and dangerous location states such as "entering a high-voltage area" at the same time, and accurately bind the two at the individual level, overcoming the limitation of the disconnect between behavior and location analysis in existing technologies.

[0044] Ultimately, through multi-condition logical judgment and hierarchical early warning triggering mechanism based on the IF-THEN rule engine, real-time and accurate comprehensive risk assessment and decision-making can be performed on complex and dangerous scenarios. This reduces the false alarm rate of single behaviors and improves the accuracy of early warnings and the timeliness of emergency response. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 The above is a flowchart of an early warning method for dangerous behaviors of personnel in thermal power plants based on multimodal recognition, provided as an embodiment of the present invention.

[0047] Figure 2 This is an overall framework diagram of a hazardous behavior early warning system for personnel in thermal power plants based on multimodal recognition, provided as an embodiment of the present invention. Detailed Implementation

[0048] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0049] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition, including:

[0050] S1. Acquire RTSP video streams by capturing images through the front-end device, and send the RTSP video streams to the multi-target tracking algorithm sequence through video stream management.

[0051] S2. For each video stream, perform channel header identification, assign a unique ID to the personnel of the thermal power plant based on the multi-target tracking algorithm, and generate motion trajectories.

[0052] S3. Track personnel in thermal power plants with unique IDs for synchronous analysis.

[0053] S4. Input the results of the synchronous analysis into the preset rule engine for real-time logical judgment.

[0054] S5. Based on the results of real-time logical judgment, trigger early warnings and provide feedback.

[0055] Based on a multi-target tracking algorithm, a unique ID is assigned to each person and a motion trajectory is generated, ensuring stable and continuous tracking of multiple targets in complex scenarios and effectively avoiding target loss and identity confusion. Secondly, by simultaneously performing behavior recognition and electronic fence determination on each tracked target, real-time correlation and fusion of dangerous actions and dangerous location information are achieved, improving the ability to perceive complex risks such as "performing dangerous actions in dangerous areas." Finally, a rule engine is used to perform multi-condition logical judgments and hierarchical early warnings on the fused information, reducing the false alarm rate. A multi-channel linkage feedback mechanism achieves a second-level response closed loop from risk identification to on-site intervention, thereby effectively preventing safety accidents.

[0056] Example 2 is an embodiment of the present invention. Based on the previous embodiment, it provides a method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition, including:

[0057] In this embodiment, video stream management in S1, i.e., backflow forwarding, requires a high-quality video encoding technology due to the large amount of data. H.264 or H.265 encoding technology is employed, as it can significantly reduce the data volume while maintaining video clarity.

[0058] In a streaming media server (such as middleware configured with FFmpeg), the raw RTSP stream received from the front end is transcoded in real time. Specific command parameters are set to use hardware acceleration (such as -c:vh264_nvenc or h265_nvenc) to uniformly convert the video stream to H.264 or H.265 format, while controlling a constant quality factor through the -crf23 parameter. This dynamically reduces the bitrate to below 50% of the original stream while maintaining visual clarity for the human eye, significantly reducing network bandwidth and storage pressure.

[0059] When establishing a connection between the streaming media server and the client, the TLS 1.3 protocol is enforced. The specific process is as follows: the server deploys a digital certificate, and after the TCP handshake phase, a TLS handshake is performed to encrypt all transmitted video data packets. This ensures that even if data is intercepted during transmission, it cannot be decrypted, effectively preventing theft and man-in-the-middle attacks.

[0060] All video stream data is transmitted within an IPSec VPN tunnel. At the network layer, an IPSec gateway is deployed between the power plant monitoring network and the central analysis server, configured in tunnel mode, and uses the AES-256 algorithm to encrypt and encapsulate the entire IP data packet (including the data payload and the original IP header) before transmitting it over the public network.

[0061] In one alternative implementation, video stream management can be achieved by deploying streaming media processing tools such as FFmpeg as transcoding relays. Specifically, RTSP stream addresses are retrieved from the configuration library, FFmpeg is used for decoding and re-encoding, the video stream is converted into a standard protocol (such as HLS or WebRTC) in real time, and then pushed to a streaming media server (such as SRS or MediaSoup), from which it is distributed to subsequent analysis modules and the front-end display interface.

[0062] In another alternative implementation, video stream management can also be based on Python's OpenCV library combined with a lightweight WSGI server (such as Flask) to build a simple stream forwarding service. Specifically, the OpenCV VideoCapture class is used to read the RTSP stream, decode it into frame-by-frame images, then encode it into M-JPEG or MPEG format on the server side, and output it as a multimedia stream via an HTTP interface for the algorithm module to pull and analyze frame by frame.

[0063] Furthermore, in S1, the RTSP video stream is sent to the multi-target tracking algorithm sequence through video stream management, including the following steps A1-A3:

[0064] A1. Retrieve the RTSP stream address from the configuration library, call the decoding library to connect and decode, and convert the video data into image frames.

[0065] A2. Place the decoded image frames into the task queue and schedule them to ensure the timing of distribution;

[0066] A3. Retrieve image frames from the queue according to the preset strategy and distribute the image frames to idle multi-target tracking algorithm instances for processing.

[0067] In this application embodiment, the preset strategy in A3, namely the scheduling and allocation strategy of the task queue, specifically includes prioritizing the maintenance of one or more first-in-first-out (FIFO) or priority task queues to buffer image frames generated after decoding from the video stream. Each frame of data is encapsulated into a task unit containing metadata such as frame data, timestamp, and source video channel ID.

[0068] Maintain a resource pool of multi-target tracking algorithm instances and perform heartbeat monitoring on the status of each instance (such as idle, busy, offline).

[0069] The scheduler retrieves tasks from the task queue and allocates them to instances in the resource pool according to a preset strategy. The core strategies include:

[0070] The scheduler assigns tasks to each available instance in a fixed order, ensuring absolute fairness and is suitable for scenarios with homogeneous instance performance.

[0071] The scheduler queries the number of pending tasks or CPU load of each instance in real time and dynamically assigns new tasks to the instance with the lightest load. This is suitable for scenarios with heterogeneous instance performance or large load fluctuations, in order to maximize overall throughput.

[0072] Different priorities are assigned to the video streams in different critical areas. Frames from high-priority channels are scheduled and processed first, ensuring minimal analysis latency in key areas.

[0073] When an algorithm instance fails to process or times out, the scheduler will put the task back into the queue and assign it to other healthy instances to ensure that frames are not lost and improve robustness.

[0074] In one optional implementation, the preset strategy can be a round-robin strategy. Specifically, the load balancer distributes image frames in the task queue to each available multi-target tracking algorithm instance in a fixed order. This strategy is simple to implement, has low overhead, and can ensure that each algorithm instance is evenly distributed among computing tasks, making it suitable for scenarios where the processing capabilities of algorithm instances are homogeneous.

[0075] In another optional implementation, the preset strategy can be a dynamic load balancing strategy based on the minimum number of connections. Specifically, this involves real-time monitoring of the current task queue length or CPU utilization of each multi-target tracking algorithm instance. When a new frame needs to be allocated, it is always prioritized for allocation to the algorithm instance with the fewest pending tasks (i.e., the "most idle"). This strategy can more intelligently address differences in processing speed among different algorithm instances or instantaneous pressure fluctuations, maximizing overall throughput and making it more suitable for production environments with heterogeneous algorithm instance processing capabilities or large load variations.

[0076] In this embodiment of the application, the multi-target tracking algorithm YOLOv8 in S2 is used as the target detector, and it is combined with the optimized DeepSORT algorithm for data association, specifically including:

[0077] A pre-trained YOLOv8s model was used and fine-tuned through transfer learning on a thermal power plant personnel dataset to accurately adapt it to features such as on-site work clothes and safety helmets. During inference, the model performs forward computation on each frame of the image and outputs the bounding box coordinates and confidence scores of all personnel targets.

[0078] A motion model is established by using a Kalman filter to linearly predict the position of an existing trajectory in the next frame.

[0079] Each target region image detected by YOLOv8 is input into a parallel, lightweight OSNet network to extract a 128-dimensional feature vector, which is used to characterize the visual appearance of the target.

[0080] This approach comprehensively calculates motion correlation (Mahathano distance between predicted location and detection box) and appearance correlation (cosine similarity between feature vectors), and uses the Hungarian algorithm for optimal matching. A key improvement is the introduction of an adaptive weighting mechanism: when the target's trajectory is smooth, the motion model is prioritized; when the target reappears after occlusion, the weight of appearance features is automatically increased, effectively solving the ID switching problem.

[0081] In one alternative implementation, the multi-target tracking algorithm can be the StrongSORT algorithm. Specifically, it is implemented by introducing an appearance feature extractor (such as a Transformer-based ReID model) on top of DeepSORT to enhance appearance representation capabilities, and adding a camera motion compensation module (estimating the inter-frame affine transformation matrix through feature point matching) to eliminate interference from slight camera shake on motion prediction. This algorithm achieves more stable maintenance of target trajectories in complex occlusion scenarios through the deep fusion of appearance matching and motion compensation.

[0082] In another alternative implementation, the multi-target tracking algorithm can also be the BoT-SORT algorithm. Specifically, based on StrongSORT, it uses an IoU-Aware Kalman filter state vector and incorporates camera motion parameters and the confidence information of the detection box to optimize the calculation of the association cost matrix. This method further reduces the probability of target loss through more accurate motion modeling and detection quality awareness, making it particularly suitable for monitoring environments with fast-moving targets and frequent occlusion.

[0083] Furthermore, in S2, a unique ID is assigned to the personnel of the thermal power plant based on a multi-target tracking algorithm, and motion trajectories are generated, including the following steps B1-B3:

[0084] B1. For the input image frame, use the object detection model to identify all personnel targets in the thermal power plant and obtain their location bounding boxes.

[0085] B2. Associate and match the bounding boxes of people detected in the current frame with the targets tracked in the previous frame; (targets that are successfully matched retain their original IDs, while new targets that are not matched are assigned a new unique ID).

[0086] B3. For each successfully matched ID, update the motion trajectory sequence based on the position coordinates of the current frame to form a trajectory point sequence that changes over time.

[0087] In this embodiment, the object detection model in B1 adopts the DAMO-YOLO model. The specific technical solution is as follows: A large version of DAMO-YOLO is selected to fully utilize its more efficient Reparameterizable Generalized Efficient Layer Aggregation Network (GELAN) as the backbone and neck network. Before training, weights pre-trained on the ImageNet dataset are used for initialization, and knowledge distillation technology is introduced to allow the large model to learn feature representations from a larger set of teachers. When training on the thermal power plant personnel dataset, its unique Distributed Focus Loss (DFL) and label assignment strategy are enabled, making the model highly robust to scale variations and partial occlusion of personnel targets in complex industrial backgrounds, ultimately achieving extremely high detection accuracy and good inference speed.

[0088] In one alternative implementation, the object detection model can be the YOLOX model. Specifically, it employs the YOLOX-L architecture and uses its signature decoupled head to separate the classification and regression tasks, while using an advanced SimOTA label assignment strategy for dynamic positive sample matching. This approach eliminates the need for anchor-free mapping, simplifying the design process; however, its overall network structure is less efficient than the GELAN structure used in the core solution of this application in terms of feature fusion for densely packed small objects.

[0089] In another alternative implementation, the object detection model can also be the PP-YOLOE+ model. This scheme is specifically based on the CSPRESNet backbone network and uses a Path Aggregation Network (PAN) and Deformable Convolutional Network (DCN) to enhance feature extraction capabilities. Although this model performs well on public datasets, its adaptive optimization flexibility for specific scenarios (such as thermal power plants) is relatively low, and its Rectified Adam optimizer, for the customized dataset in this application, is less stable and efficient in terms of convergence speed and final accuracy than the training strategy adopted by DAMO-YOLO.

[0090] Furthermore, S3 involves tracking thermal power plant personnel with unique IDs for synchronized analysis, including the following steps C1-C3:

[0091] C1. Start the analysis task synchronously for each person target with a unique ID.

[0092] The analysis tasks specifically include behavior recognition analysis based on skeletal key points and electronic fence area determination based on their location coordinates.

[0093] C2. The analysis task extracts posture features from the image region where the personnel target is located, identifies specific actions, and compares the current position of the personnel target with the coordinates of a predefined danger zone.

[0094] Specifically, the system detects obvious features such as clothing, helmets, and faces of individuals, and then categorizes, manages, and summarizes these personnel by department and category.

[0095] Detection of not wearing a helmet and not wearing a reflective vest:

[0096] Algorithm logic: Detect the presence of a person (detect the person's head, shoulders, and size); detect the presence of any identifying objects (the shape, color, and position of a helmet, and the reflective strips on a reflective vest).

[0097] Unfastened seatbelt detection:

[0098] Algorithm logic: Detect the presence of a person (detect the person's head, shoulders, and size); detect the presence of a seat belt (a stripe that is different in color and shape from the person's jacket); detect whether the seat belt is attached on one side (whether there are continuous strips connecting to the surrounding structure, excluding the part on the person's body).

[0099] Personnel fall detection:

[0100] Algorithm logic: Detect the presence of a person (detect the person's head, shoulders, and size); detect whether the person is upright or nearly upright (detect the person's skeletal shape and head and shoulder position).

[0101] Smoking monitoring among personnel:

[0102] Algorithm logic: Identify whether an unloading platform exists, or manually draw the unloading platform area; identify whether there are objects resembling people within the range (detect head and shoulders).

[0103] Open flame detection and smoke detection:

[0104] Algorithm logic: Detect color-related open flame targets (obvious, bright, orange, or red); detect shape-related flame targets; detect smoke-like targets (water vapor, smoke, etc.); detect smoke concentration (light-blocking properties, pixel difference from the surrounding environment, smoke from ignition is relatively denser than water vapor).

[0105] Crowd gathering detection:

[0106] Algorithm logic: Define a people gathering detection zone; detect the presence of people (detect the head, shoulders, and size of the person); count the total number of people in the area of ​​interest, and determine whether the number of people detected can trigger an alarm.

[0107] Vehicle uncleaned inspection:

[0108] Algorithm logic: Detect the presence of a wheel; detect whether the wheel is in contact with the ground; detect whether there are large clumps of mud or thick dust on the tire.

[0109] Virtual electronic fence - electronic fence:

[0110] Algorithm logic: Utilizing intelligent video analytics technology, visual fences are set up for dangerous restricted areas or important zones. Real-time alerts are issued if anyone approaches the area or if unauthorized personnel enter.

[0111] C3. The identified behavior types and regional status information are bound to the unique IDs of the individuals, encapsulated into data packets, and then used by the rule engine.

[0112] Furthermore, the real-time logical judgment of the synchronous analysis results input into the preset rule engine in S4 includes the following steps D1-D4:

[0113] D1. The rule engine preloads the power plant safety rule library defined in IF-THEN format.

[0114] D2. After receiving the synchronous analysis results, the rule engine performs real-time matching of the behavior and location data in the data packet with the IF condition part of each rule.

[0115] D3. Determine whether the following conditions are met simultaneously: whether the person's behavior is running, whether the person's location is in a high-voltage area, and whether the environmental sensor data is abnormal.

[0116] D4. Based on the matching rule weights and condition combinations, the rule engine calculates the risk score and classifies the event into different risk levels according to preset thresholds.

[0117] Furthermore, a base weight W is preset for each security rule in the rule base. b This weight reflects the inherent severity of the hazardous scenario corresponding to the rule. For example, the weight of the rule "IF enter a high-risk area THEN alarm" is much higher than the weight of "IF not wearing a helmet THEN alarm".

[0118] Conditional confidence weighting: For each condition in the rule, a confidence coefficient C is assigned based on the reliability of its data source. i (The value range is usually from 0 to 1). For example, the confidence level C1 of "running" behavior clearly captured by a high-definition camera can be set to 0.95; while the confidence level C2 of "environmental anomaly" data from a single, weak sensor may only be 0.7.

[0119] Risk score calculation: When multiple conditions of a rule are met simultaneously, the rule engine uses a weighted summation or weighted product model to calculate the comprehensive risk score S triggered by that rule.

[0120]

[0121] Among them, V i Let be the Boolean value of the i-th condition (1 if satisfied, 0 if not satisfied), and n be the total number of conditions for this rule. This ensures that the rule is applied only when all conditions are satisfied (i.e., all V values ​​are satisfied). i The risk score S is non-zero only when = 1.

[0122] The same event may trigger multiple rules simultaneously. Assign risk scores S1, S2, S3...S to all these triggered rules. m The data is aggregated to obtain the final comprehensive risk score S. final .

[0123] The calculated comprehensive risk score S final Compared with the preset threshold range, low risk (S final <T1), medium risk (T1 < S) final <T2), high risk (T2<S) final <T3) and emergency (S) final >T3)

[0124] Furthermore, S5, based on the results of real-time logical judgments, triggers and provides feedback on early warnings, including the following steps E1-E2:

[0125] E1. Based on the logical judgment results, automatically assemble a warning message containing personnel ID, location, screenshot and risk description, and route it to the preset feedback channel according to the risk level.

[0126] E2. Execute multiple trigger actions, including pushing pop-up alarms to the monitoring center's large screen, sending push notifications to the mobile terminals of relevant safety personnel, and sending activation signals to the linked audible and visual alarm devices.

[0127] Example 3, referring to Figure 2 This embodiment of the present invention provides a multimodal recognition-based early warning system for hazardous behaviors of personnel in thermal power plants, comprising: a data acquisition and flow management module, a personnel tracking and trajectory generation module, a multimodal behavior analysis module, an intelligent rule decision engine module, and an early warning distribution and feedback execution module.

[0128] The data acquisition and stream management module is responsible for capturing RTSP video streams through front-end devices and sending the RTSP video streams to the multi-target tracking algorithm sequence through video stream management.

[0129] The personnel tracking and trajectory generation module is responsible for identifying the header of each video stream, assigning a unique ID to the personnel in the thermal power plant based on a multi-target tracking algorithm, and generating motion trajectories.

[0130] The multimodal behavior analysis module is responsible for tracking and analyzing thermal power plant personnel with unique IDs.

[0131] The intelligent rule decision engine module is responsible for inputting the results of synchronous analysis into the preset rule engine for real-time logical judgment.

[0132] The early warning distribution and feedback execution module is responsible for triggering and responding to early warnings based on the results of real-time logical judgments.

[0133] This embodiment also provides an electronic device applicable to a method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition as proposed in the above embodiment.

[0134] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition, as proposed in the above embodiment.

[0135] The storage medium proposed in this embodiment belongs to the same inventive concept as the method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0136] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0137] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition, characterized in that: include, RTSP video streams are captured by front-end devices and sent to multi-target tracking algorithm sequences through video stream management. For each video stream, the header of the channel is identified, a unique ID is assigned to the personnel of the thermal power plant based on a multi-target tracking algorithm, and a motion trajectory is generated; Track personnel at thermal power plants with unique IDs for synchronous analysis; The results of synchronous analysis are input into a preset rule engine for real-time logical judgment; Based on the results of real-time logical judgments, early warnings are triggered and feedback is provided.

2. The method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition as described in claim 1, characterized in that: The step of sending RTSP video streams to the multi-target tracking algorithm sequence through video stream management includes, The RTSP stream address is retrieved from the configuration library, and the decoding library is called to connect and decode the video data, converting it into image frames. The decoded image frames are placed in the task queue and scheduled to ensure the timing of distribution. Image frames are retrieved from the queue according to a preset strategy and distributed to idle multi-target tracking algorithm instances for processing.

3. The method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition as described in claim 2, characterized in that: The method of assigning unique IDs to personnel in thermal power plants and generating movement trajectories based on multi-target tracking algorithms includes, For the input image frames, the target detection model is used to identify all personnel targets in the thermal power plant and obtain their location bounding boxes; Associate and match the bounding boxes of people detected in the current frame with the targets tracked in the previous frame; For each successfully matched ID, the motion trajectory sequence is updated based on the position coordinates of the current frame, forming a trajectory point sequence that changes over time.

4. The method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition as described in claim 3, characterized in that: The tracking and synchronous analysis of thermal power plant personnel with unique IDs includes, For each individual target with a unique ID, initiate the analysis task synchronously; The analysis task extracts pose features from the image region where the personnel target is located, identifies specific actions, and compares the current position of the personnel target with the coordinates of a predefined danger zone; The identified behavior types and regional status information are bound to the unique IDs of individuals, encapsulated into data packets, and then used by the rule engine.

5. The method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition as described in claim 4, characterized in that: The analysis tasks include behavior recognition analysis based on skeletal key points and electronic fence area determination based on personnel location coordinates; The process of inputting the results of synchronous analysis into a preset rule engine for real-time logical judgment includes... The rules engine preloads a database of safety rules for thermal power plants defined in IF-THEN format; After receiving the synchronous analysis results, the rule engine performs real-time matching of the behavior and location data in the data packet with the IF condition part of each rule; Determine whether the following conditions are met simultaneously: whether the person's behavior is running, whether the person's location is in a high-voltage area, and whether the environmental sensor data is abnormal. Based on the matching rule weights and condition combinations, the rule engine calculates the risk score and classifies the event into different risk levels according to preset thresholds.

6. The method for early warning of hazardous behaviors of personnel in thermal power plants based on multimodal recognition as described in claim 4, characterized in that: The early warning triggering and feedback based on the results of real-time logical judgment includes, Based on the logical judgment results, the system automatically assembles warning messages containing personnel ID, location, screenshots, and risk descriptions, and routes them to preset feedback channels according to the risk level. It executes multiple trigger actions, including pushing pop-up alarms to the monitoring center's large screen, sending push notifications to the mobile terminals of relevant safety personnel, and sending activation signals to the linked audible and visual alarm devices.

7. A multimodal recognition-based early warning system for hazardous behaviors of personnel in thermal power plants, employing the multimodal recognition-based early warning method for hazardous behaviors of personnel in thermal power plants as described in any one of claims 1 to 6, characterized in that, include: The module includes data acquisition and flow management, personnel tracking and trajectory generation, multimodal behavior analysis, intelligent rule decision engine, and early warning distribution and feedback execution. The data acquisition and stream management module is responsible for capturing RTSP video streams through the front-end device and sending the RTSP video streams to the multi-target tracking algorithm sequence through video stream management. The personnel tracking and trajectory generation module is responsible for identifying the header of each video stream, assigning a unique ID to the personnel of the thermal power plant based on a multi-target tracking algorithm, and generating a motion trajectory. The multimodal behavior analysis module is responsible for tracking and synchronously analyzing personnel in thermal power plants who have unique IDs. The intelligent rule decision engine module is responsible for inputting the results of synchronous analysis into a preset rule engine for real-time logical judgment. The warning distribution and feedback execution module is responsible for triggering and responding to warnings based on the results of real-time logical judgments.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of any one of claims 1 to 6 for a method of early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for early warning of dangerous behaviors of personnel in thermal power plants based on multimodal recognition, as described in any one of claims 1 to 6.

Images