Substation entrance and exit behavior identification method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a transformer substation entrance and exit behavior recognition method and device, electronic equipment and a storage medium, and relates to the technical field of intelligent monitoring, and the method comprises the steps: obtaining video data of a transformer substation entrance and exit, and carrying out the preprocessing of the video data, and obtaining the preprocessed image data; performing target detection on the preprocessed image data by adopting a YOLO11 target detection model to obtain a target detection result; in combination with the station door state information and the time sequence data in the target detection result, a rule defined according to a bounding box and the station door state is set, and the target detection result is analyzed to judge the entering and exiting behavior of the target; analyzing an in-out mode in the in-out record by adopting a large language model, and predicting an abnormal behavior in combination with the time sequence data; according to the prediction result of the abnormal behavior, a preset external system is linked to execute corresponding operation; the method can improve the recognition precision of the entrance and exit behaviors of the transformer substation, realizes automatic monitoring and optimizes the utilization of computing resources.

Description

Technical Field

[0001] The present invention relates to the field of intelligent monitoring technology, and in particular to a method, device, electronic equipment and storage medium for identifying substation entrance and exit behaviors. Background Art

[0002] With the continued advancement of smart grid construction, substations, as crucial nodes in the power system, are undergoing a transition from manual monitoring to intelligent surveillance for safe operation. In this process, computer vision technology is gradually being introduced into substation monitoring systems, primarily for detecting people and vehicles at station entrances.

[0003] Currently, the industry generally uses target recognition systems based on traditional convolutional neural networks (CNNs) to achieve this function. These systems often utilize classic algorithm architectures such as YOLOv3 and Faster R-CNN. Through edge computing devices deployed on station surveillance cameras, they can achieve real-time detection of moving targets such as personnel and construction vehicles with approximately 90% accuracy. However, as power grid companies increase their demand for lean operations and maintenance management, these methodologies have certain application limitations, including the following: Insufficient dynamic behavior recognition: Based on single-frame image analysis, it is impossible to accurately determine the target's direction of movement, resulting in frequent misjudgments of station entry and exit records.

[0004] Weak scene understanding capabilities: The functional divisions within substations are complex, and the existing system cannot distinguish between normal inspections and abnormal behaviors, making it easy to make misjudgments.

[0005] Poor adaptability to complex environments: Substations are subject to complex environmental factors such as strong electromagnetic interference, extreme weather, and equipment reflections. The detection performance of traditional CNN models significantly decreases under these conditions, resulting in lower recognition accuracy.

[0006] Lack of multi-source data fusion capabilities: Limited to single visual dimension analysis, it is difficult to effectively integrate with multi-source data to achieve precise automated management.

[0007] Therefore, how to improve the accuracy of substation entrance and exit behavior identification, realize automated and refined management and control, and reduce deployment costs are technical problems that need to be urgently solved by technical personnel in this field. Summary of the Invention

[0008] The main purpose of the present invention is to provide a method, device, electronic device and storage medium for identifying substation entrance and exit behaviors, aiming to solve the problems of low accuracy in identifying substation entrance and exit behaviors, insufficient automation and refined management capabilities, and high computing resource requirements.

[0009] To achieve the above objectives, the present application provides, in a first aspect, a method for identifying substation entrance and exit behaviors, the method comprising: Acquire video data of a substation entrance and exit, and preprocess the video data to obtain preprocessed image data; Using the YOLO11 target detection model to perform target detection on the preprocessed image data to obtain a target detection result; Combining the station door state information and the time series data in the target detection results, a rule R(B, S(t)) defined based on the bounding box B and the station door state S(t) is set to analyze the target detection results to determine the target's entry and exit behavior; wherein, for a preset behavior, a long short-term memory network is used to predict the time series behavior of the time series data; the input features of the long short-term memory network include: a bounding box coordinate sequence of the target detected in the current frame, the target's category confidence, and at least one of the displacement vector of the target center point between consecutive frames; Generate entry and exit records based on the entry and exit behavior of the target; Using a large language model to analyze entry and exit patterns in the entry and exit records, and combining the time series data to predict abnormal behavior; Based on the prediction results of the abnormal behavior, the preset external system is linked to perform corresponding operations.

[0010] Optionally, the preprocessing includes: frame extraction of the video data, noise reduction using Gaussian filtering or median filtering, grayscale processing or RGB color retention, and image enhancement using histogram equalization or contrast stretching, one or more of the following.

[0011] Optionally, after performing target detection on the preprocessed image data using the YOLO11 target detection model, the method further includes: Using non-maximum suppression algorithm to remove redundant detection frames; and / or, Gaussian mixture model is used for background modeling.

[0012] Optionally, combining the station door status information and the time series data in the target detection result to analyze the target detection result to determine the target's entry and exit behavior includes: Combining the station door status information with the time series data in the target detection results, the target detection results are analyzed to distinguish the target's entry behavior, exit behavior, and in-station activities; wherein, for preset behaviors, a long short-term memory network is used to predict the time series behavior of the time series data; The input features of the long short-term memory network include at least one of a bounding box coordinate sequence of a target detected in a current frame, a category confidence of the target, and a displacement vector of a target center point between consecutive frames.

[0013] Optionally, the station door status information is acquired in real time by integrating with the access control system of the substation, or by performing image recognition on the door area in the video data.

[0014] Optionally, the preset external system includes an access control system, an alarm system or a remote management platform; The linkage preset external system performs corresponding operations, including triggering sound and light alarms, pushing notifications to management personnel, or recording relevant videos.

[0015] Optionally, the method further includes: The YOLO11 target detection model is transferred and learned using self-collected video data for substation scenarios. The model weights and hyperparameters are adjusted, and data augmentation technology is used to improve the model robustness.

[0016] A second aspect of the present application provides a substation entrance and exit behavior recognition device, comprising: An acquisition module is used to acquire video data of the substation entrance and exit, and preprocess the video data to obtain preprocessed image data; A target detection module is used to perform target detection on the preprocessed image data using a YOLO11 target detection model to obtain a target detection result; A behavior analysis module is used to analyze the target detection results by combining the station door status information and the time series data in the target detection results to determine the target's entry and exit behavior; A record generation module, configured to generate entry and exit records based on the entry and exit behavior of the target; a prediction module for analyzing entry and exit patterns in the entry and exit records using a large language model and predicting abnormal behavior in combination with the time series data; The management module is used to link a preset external system to perform corresponding operations based on the prediction results of the abnormal behavior.

[0017] A third aspect of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the first aspect and any possible implementation thereof.

[0018] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in the first aspect.

[0019] The present application provides a method, device, electronic device and storage medium for identifying substation entrance and exit behaviors. The method comprises the following steps: acquiring video data of the substation entrance and exit, and preprocessing the video data to obtain preprocessed image data; performing target detection on the preprocessed image data using a YOLO11 target detection model to obtain a target detection result; analyzing the target detection result in combination with station door status information and time series data in the target detection result to determine the target's entry and exit behavior; generating entry and exit records based on the target's entry and exit behavior; analyzing the entry and exit patterns in the entry and exit records using a large language model, and predicting abnormal behavior based on the time series data; and linking a preset external system to perform corresponding operations based on the predicted result of the abnormal behavior. The method can improve the accuracy of identifying substation entrance and exit behaviors, realize automated monitoring, and optimize computing resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] in: Figure 1 A flow chart of a method for identifying substation entrance and exit behaviors provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a substation entrance and exit behavior recognition device provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0023] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0024] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0025] YOLO (You Only Look Once) involved in the embodiments of the present application is an efficient target detection algorithm that is widely used in real-time image recognition and has low computing resource requirements.

[0026] The small-scale, large-language model mentioned in the embodiments of this application refers to a language model that is relatively small in scale but can effectively provide decision support in specific tasks. In the embodiments of this application, it is typically at the 1.5B level and can be run on edge devices.

[0027] The substation mentioned in the embodiments of the present application is a facility used for voltage conversion, distribution and transmission in the power system. The substation involved in the present application mainly includes external entrance and exit areas and their monitoring facilities.

[0028] The image recognition mentioned in the embodiment of the present application is a process of analyzing and identifying image content through computer vision technology. In the embodiment of the present application, image recognition technology can be mainly used to monitor the entry and exit of people and vehicles at the substation gate.

[0029] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0030] See also Figure 1 , is a flow chart of a method for identifying substation entrance and exit behaviors provided by an embodiment of the present application, such as Figure 1 As shown, the method includes: 101. Obtain video data of a substation entrance and exit, and preprocess the video data to obtain preprocessed image data.

[0031] The execution subject of the method in the embodiment of the present application can be a substation entrance and exit behavior recognition device, which can be implemented using a terminal device in actual application, such as a server or a computer.

[0032] Specifically, the existing cameras in the substation can be used to collect video data of the entrances and exits in real time, ensuring that the cameras cover all entrance and exit locations, and ensuring the clarity of the video data and a sufficient frame rate to ensure the continuity of the moving target.

[0033] In an optional embodiment, the above-mentioned preprocessing includes: frame extraction of the above-mentioned video data, noise reduction processing using Gaussian filtering or median filtering, grayscale processing or RGB color retention processing, and one or more of image enhancement using histogram equalization or contrast stretching.

[0034] Furthermore, video data preprocessing can be performed on edge computing devices, including but not limited to: Frame extraction: extract key frames at fixed intervals or in a motion-triggered manner; Noise reduction processing: Use Gaussian filtering, median filtering and other algorithms to reduce video noise and improve image quality; Color processing: grayscale conversion is performed to reduce computational complexity while retaining RGB data for color feature learning; Image enhancement: Use methods such as histogram equalization and contrast stretching to improve detection accuracy in low-light or overexposed scenes.

[0035] 102. Use the YOLO11 target detection model to perform target detection on the preprocessed image data to obtain the target detection result.

[0036] After data preparation is completed, the target detection and entry and exit judgment process can be executed.

[0037] This application utilizes the YOLO11 object detection model. The YOLO11 model, described in the embodiments of this application, is a lightweight object detection model independently improved upon the YOLOv5 framework. Key improvements focus on computing power adaptability, detection accuracy, and multi-scale fusion capabilities. Building on the YOLOv5s architecture, the original CSPDarknet is replaced with the lightweight MobileNetV3-Large to reduce model parameters and improve inference efficiency on edge devices. The Bidirectional Feature Pyramid Network (BiFPN) replaces the traditional PANet architecture to achieve efficient multi-scale feature fusion and improve detection accuracy for small objects (such as pedestrians at a distance). A Decoupled Head (DHE) separates classification and regression tasks, reducing inter-task interference and improving classification accuracy and bounding box regression accuracy. A Squeeze-and-Excitation (SE) attention module is added at the junction of the backbone and neck to dynamically adjust channel weights, thereby enhancing the responsiveness of key area features. This is particularly applicable to substation scenarios where the similarity between people and background is high. Based on the deployment environment, the combination of pruning and INT8 quantization technology can significantly reduce the model size and computing load while ensuring that the detection accuracy is basically not reduced, meeting the real-time inference needs of edge computing devices.

[0038] The training process of the model in this application may include: Transfer learning: Using the YOLO11 model weights pre-trained on large general datasets (such as COCO and ImageNet) as the initial weights, and then fine-tuning them on the substation-specific dataset constructed in S13. This approach can accelerate model convergence and improve performance on specific tasks, especially when the amount of labeled data is relatively limited.

[0039] Model training: Set appropriate training hyperparameters. For example, choose Adam or SGD as the optimizer, set the initial learning rate to 1e-3, and adopt a learning rate decay strategy (such as cosine annealing). The batch size should be based on the GPU memory capacity of the edge computing device, for example, 8 or 16. The number of training epochs should be determined based on the performance on the validation set, for example, 50-200 epochs of fine-tuning.

[0040] Video data can be manually annotated to construct a high-quality training dataset, specifically encompassing entry, exit, and on-site activity. Data cleaning can also be performed to remove blurry or heavily occluded samples. The dataset is then divided proportionally (e.g., 80% training, 10% validation, and 10% test), and data augmentation techniques (random cropping, rotation, and flipping) can be employed to improve model generalization.

[0041] During the object detection process, the preprocessed image frame to be detected can be input into the trained YOLO11 model. The input image I is a tensor of H×W×C (H is the height, W is the width, and C is the number of channels). The feature extraction network f(I;θ) inside the model will calculate the feature map F, which is a tensor of H'×W'×D (H'<H, W'<W, and D is the feature dimension), where θ is the parameter weight of the model.

[0042] The YOLO11 model usually includes a backbone network, a neck network, and a head network. The backbone network is responsible for extracting image features at different levels; the neck network (such as FPN, PANet, BiFPN) is responsible for fusing features from different levels of the backbone network to enhance the detection ability for objects of different scales; the head network then makes predictions based on the fused feature map, outputting the bounding box B={(x, y, w, h, c)} of the object and the corresponding class confidence P(c). Here, (x,y) are the coordinates of the center point of the bounding box, w and h are the width and height of the bounding box, and c is the object class.

[0043] In an optional implementation, after performing object detection on the preprocessed image data using the YOLO11 object detection model, the method further includes: Using the non-maximum suppression algorithm to remove redundant detection boxes; and / or, Using the Gaussian mixture model for background modeling.

[0044] In the embodiments of the present application, post-processing can be performed on the original detection boxes output by the model to improve the accuracy of the detection results. Specifically, in the embodiments of the present application, the following can be used: Non-maximum suppression (NMS): Remove redundant detection boxes of the same object. Set an IoU (Intersection over Union) threshold (such as 0.45 or 0.5). When the IoU of multiple detection boxes is greater than this threshold and they belong to the same class, retain the detection box with the highest confidence and suppress other detection boxes.

[0045] Gaussian mixture model (GMM) background modeling (optional): For static scenes, the GMM can be used to model the background, and foreground moving objects can be segmented by background subtraction, thereby improving the segmentation accuracy of the objects and reducing false detections caused by background interference. The number K of Gaussian distributions in the GMM model can be set to 3 to 5 according to the scene complexity.

[0046] 103. In combination with the station door state information and the time series data in the above-mentioned target detection results, set a rule R(B, S(t)) defined according to the bounding box B and the station door state S(t), and analyze the above-mentioned target detection results to determine the target's entry and exit behavior; wherein, for the preset behavior, a long short-term memory network is used to predict the time series behavior of the above-mentioned time series data; the input features of the above-mentioned long short-term memory network include: the bounding box coordinate sequence of the target detected in the current frame, the category confidence of the target, and at least one of the displacement vector of the target center point between consecutive frames.

[0047] Specifically, behavior analysis and entry and exit judgment can be further performed. This step is one of the core of the present invention, and is mainly intended to accurately distinguish the target's "entry", "exit" behavior and "in-station activities".

[0048] Optionally, the above-mentioned station door status information is obtained in real time by integrating with the access control system of the above-mentioned substation, or obtained by performing image recognition on the door body area in the above-mentioned video data.

[0049] Specifically, for station door status information, a deep learning target detection model can be used to detect the status of the door itself, such as whether the sliding door is in the closed position; the image change features of the door area (position, occlusion, light and shadow changes, etc.) can be extracted through the surveillance camera to assist in judging the current door status; in cases where multiple people pass or there is severe occlusion, it can be fused with the access control status signal to enhance robustness.

[0050] In an optional embodiment, the above-mentioned target detection results are analyzed in combination with the station door status information and the time series data in the target detection results to determine the target's entry and exit behavior, including: Combined with the above-mentioned station door status information and the time series data in the above-mentioned target detection results, the above-mentioned target detection results are analyzed to distinguish the target's entry and exit behaviors and activities within the station; among them, for the preset behaviors, the above-mentioned time series data are used to predict the temporal behavior; The input features of the long short-term memory network include at least one of a bounding box coordinate sequence of a target detected in a current frame, a category confidence of the target, and a displacement vector of a target center point between consecutive frames.

[0051] Specifically, combining the station door status (open / closed) information S(t) and time series data, a rule R(B, S(t)) can be set to distinguish "entry", "exit" and "station activities". Among them, S(t) represents the status of the station door, which may be "open" or "closed" at time t. R(B, S(t)) is a rule defined based on the bounding box B and the station door status S(t) to determine the target's entry and exit behavior. The judgment logic may include: If target B is outside the station at time t and enters the station at time t+1, and the station door status is "open", it is judged as "entering the station"; If target B is inside the station at time t and outside the station at time t+1, and the station door status is "open", it is judged as "out of the station"; If target B stays in the station area for a long time and does not cross the access control area, it is judged as "station activity".

[0052] Furthermore, for predefined behaviors (primarily complex entry and exit behaviors), such as multiple people passing simultaneously or targets being temporarily occluded, the system can use an LSTM network h(t) = LSTM(h(t-1), F(t)) for temporal behavior prediction. Here, h(t) represents the hidden state of the LSTM network at time t, h(t-1) is the hidden state of the LSTM at the previous time step, and F(t) represents the feature input at the current time step t. This formula indicates that the LSTM updates h(t) using historical information h(t-1) and current features F(t) for temporal behavior prediction. This enhances the recognition of briefly occluded or temporarily lost targets, generating the final behavioral decision A(t). Simultaneously, a trajectory tracking algorithm maintains identity consistency for target B, avoiding recognition errors caused by temporary occlusions.

[0053] The input features F(t) are designed to integrate visual detection, door status, spatial location information, and temporal behavior patterns, thereby enhancing the ability to discriminate complex behaviors (such as entry and exit, staying in place, occlusion reappearance, etc.). Specifically, they may include: 1. Image object detection features The feature vector extracted from the intermediate layer output of the YOLO11 network mainly includes: Confidence of category c (such as people, cars, etc.); Bounding box location information: center point coordinates (x, y); 2. Access control status characteristics S(t): current door status (open / closed, numerical code 0 / 1); 3. Spatial location information The relative distance between the target and the center point of the entrance and exit (normalized coordinate difference); The displacement direction vector between the target center point and the center of gravity of the bounding box; 4. Time characteristics Δt: the time interval between the current frame and the previous frame; Dwell time estimation: The cumulative presence time of the target in the camera's field of view (aggregated by tracking ID).

[0054] In an optional embodiment, the above method further includes: The YOLO11 target detection model is transferred to the self-collected video data for the substation scene, and the model weights and hyperparameters are adjusted. The data augmentation technology is also used to improve the robustness of the model.

[0055] To further improve and maintain model performance, model fine-tuning and data augmentation can be performed. These operations can be continued after initial deployment, forming a closed loop of online learning and optimization.

[0056] S31. Transfer learning is performed on the YOLO11 model using video data collected from substations. The model weights and hyperparameters are adjusted to make it more suitable for substation scenarios. For example, a cosine annealing learning rate adjustment strategy is used to optimize the training process, and a class-balanced loss function is used to address class imbalance.

[0057] S32. Improve model robustness through various data augmentation techniques. In addition to the basic data augmentation mentioned in the previous steps, you can also use more advanced data augmentation strategies and apply them dynamically during model training or fine-tuning: Geometric transformations: random rotation (e.g., 0°-15°), scale transformation (e.g., scaling factor between 0.8-1.2), random cropping, and affine transformation; Color and appearance transformation: brightness adjustment (e.g. ±20%), contrast adjustment, saturation adjustment, hue jitter, Gaussian blur, motion blur; Advanced enhancement methods: MixUp (mixing two images and their labels proportionally), CutMix (cutting a region from one image and pasting it onto another, mixing the labels accordingly), and Mosaic (combining four images into one for training). These methods can effectively improve the model's generalization ability and robustness to occlusion and complex backgrounds.

[0058] S33. Use online learning methods to continuously optimize the model after actual deployment to adapt to new environmental changes. This may include: Through active learning strategies, training data is automatically screened from high-confidence error detection samples of the model, reducing the burden of manual labeling.

[0059] Using knowledge distillation technology, lightweight sub-models learn features from the main model to improve online reasoning efficiency.

[0060] Combined with a federated learning framework (suitable for multi-site deployments), this system, deployed across multiple substations, enables collaborative training of a global model without sharing raw sensitive data. Edge devices at each site train the model on local data, then upload the model parameters (or parameter updates) to a central server for aggregation, updating the global model. This updated global model is then distributed to each site. This helps leverage diverse data across sites, improving the model's overall performance and generalization capabilities.

[0061] S34. To address the uncertainty of test results, Bayesian deep learning methods are used to quantify the uncertainty of model outputs, improving the ability to handle low-confidence results. Prediction results with high uncertainty can be flagged, prompting the system or operator to further verify or serving as a basis for active learning sample screening, thereby improving the ability to handle low-confidence results and reducing false positives and negatives.

[0062] 104. Generate entry and exit records based on the entry and exit behaviors of the above targets.

[0063] To maintain the identity of the same target in consecutive frames, target tracking algorithms such as DeepSORT, FairMOT, or simple trackers based on Kalman filtering and the Hungarian algorithm can be used. This helps handle situations such as short-term occlusion and intersection of targets, ensuring the consistency of behavioral analysis.

[0064] Finally, the aforementioned rule judgment results, LSTM prediction results (if enabled), and trajectory information can be combined to generate the final behavior decision (entry, exit, and in-station activities).

[0065] 105. Use a large language model to analyze the entry and exit patterns in the above entry and exit records, and combine the above time series data to predict abnormal behavior.

[0066] After completing behavior recognition, the system will assist in decision-making and interact with external systems.

[0067] Generate entry and exit records and store them in the database. Based on the determined behavioral decision A(t), the system automatically generates structured entry and exit records. Each record should contain at least: the target's unique ID (maintained through trajectory tracking), target category (person / vehicle), event timestamp, behavior type (entry / exit / in-station activity), duration of stay (for in-station activity or entry and exit), station door status at the time, associated snapshot image path or short video clip path, detection confidence, etc. These records are stored in a relational database (such as MySQL, PostgreSQL) or NoSQL database for subsequent query, analysis, and report generation.

[0068] In this application, a small-scale large language model (LLM) is introduced to analyze inbound and outbound patterns and predict abnormal behavior in combination with time series data. This step is to enhance the intelligence level of the system and to extract deeper information from the original detection results and records.

[0069] Specifically, a large language model with a smaller parameter size (e.g., 1.5B) can be used, such as a lightweight version of ChatGLM-6B or an open-source model fine-tuned for a specific task. This LLM is fine-tuned for substation safety management scenarios, performing instruction or few-shot prompting. Its input can be a structured event description (e.g., "time T, target type P, behavior A, location L, confidence C, associated history H"), and its output can be a natural language risk assessment, event summary, potential cause analysis, or action recommendations.

[0070] Entry and exit pattern analysis: LLM can analyze historical entry and exit records to identify common entry and exit patterns (e.g., regular entry and exit times for a specific work group, frequency of visits by a specific vehicle).

[0071] Abnormal behavior prediction / detection: Combine time series analysis models (such as RNN, Transformer, or simpler statistical models such as ARIMA) to model historical entry and exit data, predict future entry and exit trends, and mark behaviors that significantly deviate from the trend as potential anomalies.

[0072] Optionally, use an unsupervised anomaly detection algorithm (such as Isolation Forest, set n_estimators=100; or DBSCAN, set eps and min_samples according to the data distribution characteristics, for example, eps=0.5, min_samples=5) to identify atypical entry and exit events, such as frequent activities during non-working hours, long stays in unauthorized areas, and unusual entry and exit combinations.

[0073] LLM can provide semantic interpretation and risk rating for abnormal events detected by these algorithms.

[0074] In order to enhance the intelligent decision-making capability of the system, in the embodiment of the present application, a small-volume, large-language model (with a parameter volume of approximately 1.5B) is introduced on the basis of target detection and behavior recognition to perform semantic analysis, trend recognition, and auxiliary judgment of abnormal behavior on structured information such as identified entry and exit behaviors, time patterns, and access control records.

[0075] The core design and deployment features of this language model are as follows: 1. Model type and deployment method: This model, based on open-source Transformer language model architectures (such as GPT-Neo, OPT, and BLOOM), builds a customized language model for power substation scenarios through fine-tuning of instructions. The model has a parameter scale of approximately 150 million to 1.5 billion, and is lightweight, low-latency, and edge-deployable, compatible with computing platforms such as Jetson Nano, Orin, and RK3588.

[0076] 2. Training and adaptation methods: A high-quality question-and-answer dataset covering substation access control scenarios was constructed, encompassing typical tasks such as abnormal behavior analysis, entry and exit record logic judgment, and access control permission verification. The LoRA (Low-Rank Adaptation) method was applied to quickly and cost-effectively fine-tune the pre-trained model. Industry vocabulary expansion and intent recognition modules were introduced to improve the model's accuracy in understanding characteristic behavior labels such as "trespassing" and "illegal stay."

[0077] 3. Implementation of decision-making support function: The system encodes the detected behavior results (personnel ID, time, door status, behavior type, etc.) into a natural language prompt input model; the model output results include: behavior explanation, whether an alarm is triggered, whether it is abnormal, cause analysis, recommended measures, etc.; example output: "At 02:14 on May 23, 2025, an engineering vehicle with license plate number Dian A×××× was detected entering the 110kV ×× substation. This vehicle is not on the nighttime pass whitelist. It is recommended to trigger an alarm and notify the administrator for verification." 4. Model usage scenarios: Real-time semantic interpretation and alarm judgment of entry and exit behavior; combining historical data to generate trend analysis and policy recommendations; automatic summary, archiving and log generation of entry and exit behavior data; linkage control system to achieve automatic control of access control response according to authorization status.

[0078] This language model effectively makes up for the shortcomings of traditional visual recognition systems in semantic understanding and management suggestion generation, enabling the monitoring system of this application to have human-like judgment capabilities and improving the depth and explainability of intelligent management and control.

[0079] 106. Based on the prediction results of the above abnormal behavior, the preset external system is linked to perform corresponding operations.

[0080] In the embodiment of the present application, based on the obtained prediction results of abnormal behavior, a preset external system can be linked to achieve an automated response.

[0081] In an optional embodiment, the aforementioned preset external system includes an access control system, an alarm system, or a remote management platform; The above-mentioned linkage preset external system performs corresponding operations, including triggering sound and light alarms, pushing notifications to management personnel, or recording relevant videos.

[0082] Specific examples include but are not limited to: Access control system linkage: For entry behavior identified as authorized personnel, a command can be automatically sent to the access control system to open the door; for unauthorized personnel or suspicious behavior, the door will remain closed or triggered to lock.

[0083] Alarm system linkage: When preset abnormal behaviors are detected (such as illegal intrusion, prolonged loitering, triggering of electronic fences, etc.), the system automatically triggers the sound and light alarm and sends an alarm notification to the management personnel via SMS, App push, email, etc. The notification content can include the event type, time, location, captured pictures, etc.

[0084] Remote management platform linkage: Identification results and alarm information are pushed in real time to the remote monitoring center or the manager's mobile terminal. Managers can remotely view real-time video, play back recordings, confirm alarms, and take appropriate measures (such as remote calls, manual door opening, etc.). The system also automatically records relevant video clips as evidence.

[0085] Further optionally, the embodiments of the present application also include automatic monitoring and management functions to improve the overall management efficiency and intelligence level of the substation.

[0086] S51. The system supports remote real-time monitoring, allowing managers to view real-time entry and exit data and alarm information through an interface. The system provides an interactive interface via a web or mobile application, displaying the entry and exit of people and vehicles and providing alarm notifications. For example, if unauthorized entry or unusual oversight is detected, the system automatically sends real-time alarm information to managers and provides event playback for subsequent analysis.

[0087] S52. Provides historical data analysis and generates statistical reports to optimize substation management strategies. The system stores all inbound and outbound data and uses data analysis tools (such as pivot tables and time series analysis) to extract valuable management information. For example, managers can optimize substation work schedules by analyzing inbound and outbound trends over a specific time period, or develop more effective security strategies through long-term analysis of abnormal events. Furthermore, the system can use machine learning methods to mine historical data to predict possible future abnormalities and improve substation safety.

[0088] S53 supports integration with existing substation management systems to achieve integrated intelligent management. The system connects with access control, attendance, and security systems through standardized interfaces (such as REST APIs or MQTT protocols), enabling data sharing and collaborative control. For example, upon detecting a person entering the station, data can be automatically synchronized with the attendance system. When abnormal behavior is detected, the security system can be linked to trigger an alarm or cameras to automatically track and record. The system also supports cloud deployment for multi-site management, enabling remote centralized monitoring of substations across regions, improving management efficiency and security.

[0089] Current image recognition technology still has limitations in substation personnel management, particularly in the accuracy of identifying human activity. Currently, most image recognition systems cannot accurately distinguish between routine substation work activities, such as normal inspections and equipment maintenance, and the movement of people entering or leaving the substation. This limitation makes the system prone to false positives and false negatives. For example, routine inspections by staff within the substation may be mistaken for abnormal behavior, while unauthorized intrusions may not be promptly identified. This not only affects the efficiency of substation security management, but may also cause operations and maintenance personnel to frequently handle invalid alarms due to system misjudgments, reducing work efficiency and compromising the substation's security capabilities.

[0090] Current image recognition systems also have significant shortcomings in automation and refined management. Although some substations have installed video surveillance systems, these systems are primarily used for video archiving and manual retrospective analysis, rather than real-time intelligent processing. The existing management model relies heavily on manual intervention, resulting in slow response to abnormal behavior and inability to achieve precise automated management. Furthermore, existing systems often fail to fully utilize camera resources for efficient management, failing to form a unified and coordinated intelligent monitoring network. For example, cameras in different areas may operate independently, failing to share analytical data, resulting in blind spots and limiting security assurance capabilities in critical areas.

[0091] Traditional image recognition algorithms typically require high computing resources, limiting their large-scale application in substation environments. These algorithms often require the installation of additional high-performance computing equipment, such as GPU servers or dedicated AI computing units. These devices are expensive and require additional power supply and maintenance, increasing substation management costs. For some substations located in remote areas with limited resources, installing high-performance computing equipment is even more difficult, further restricting the practical application of image recognition technology. In addition, high computing resource requirements may also cause delays in system data processing, affecting the effectiveness of real-time monitoring. Summary: (1) Insufficient accuracy: Most existing image recognition technologies cannot distinguish between normal activities of personnel in the substation and those entering and leaving the substation, resulting in serious false alarms or missed alarms; (2) Lack of automation and refined management: Existing systems fail to achieve refined automated management and control based on video surveillance; (3) High computing resource requirements: Traditional image recognition algorithms usually have high requirements for computing equipment and require the installation of additional computing equipment, which increases the management cost of the substation.

[0092] Therefore, how to reduce computing resource requirements while ensuring recognition accuracy is an important direction for optimizing the current substation intelligent monitoring system.

[0093] This application aims to address the current deficiencies in the identification and monitoring management of entry and exit behaviors at substation gates, focusing on improving recognition accuracy, achieving automated monitoring, and optimizing computing resource utilization.

[0094] The embodiment of the present application proposes a method for identifying behaviors at the entrances and exits of a substation, which is used to distinguish between the entry and exit behaviors at the substation entrance and the normal activities within the station. At present, many image recognition technologies cannot accurately distinguish between the entry and exit of people or vehicles and normal behaviors such as daily inspections and equipment maintenance within the station, resulting in serious false alarms or missed reports. This application optimizes the recognition algorithm and combines behavioral pattern analysis with target tracking technology to ensure that people and vehicles entering and exiting can be accurately identified, effectively reduce misjudgments, and improve the accuracy of entry and exit management at the substation entrance. This improvement not only enhances the security management and control capabilities of the substation, but also reduces the workload of operation and maintenance personnel who need to conduct additional verification due to false alarms.

[0095] This application realizes the automated monitoring of the substation entrance and automatically generates detailed record information based on the recognition results. Through intelligent analysis of entry and exit behaviors, the system can record the entry and exit of personnel and vehicles in real time and form traceable log data. In addition, this application also introduces a small-scale large language model for decision support, enabling the system to provide intelligent security management suggestions based on monitoring data, such as abnormal situation warnings, permission verification, entry and exit record analysis, etc. Compared with the traditional method that relies on manual monitoring and retrospective analysis, this management model that combines automation and intelligence has greatly improved the safety and management efficiency of substations.

[0096] The method in this application makes full use of the existing cameras and computing resources in the substation, avoiding the addition of high-performance computing equipment, thereby effectively reducing management costs and improving the universality and feasibility of the system. Traditional image recognition systems usually require the additional deployment of high-computing servers or AI computing equipment, but this application optimizes the algorithm so that existing computing resources can support efficient operation. This not only reduces hardware investment and maintenance costs, but also makes the system easier to promote and apply to various substations, including small and medium-sized substations with limited equipment resources.

[0097] Based on the description of the aforementioned method embodiment, an embodiment of the present application further provides a substation entrance and exit behavior identification device.

[0098] Figure 2 This is a schematic diagram of the structure of a substation entrance and exit behavior recognition device provided in an embodiment of the present application. Figure 2 As shown, the substation entrance and exit behavior recognition device 200 includes: An acquisition module 210 is configured to acquire video data of a substation entrance and exit, and preprocess the video data to obtain preprocessed image data; The target detection module 220 is used to perform target detection on the preprocessed image data using the YOLO11 target detection model to obtain a target detection result; The behavior analysis module 230 is configured to combine the station door state information and the time series data in the target detection results, set a rule R(B, S(t)) defined based on the bounding box B and the station door state S(t), and analyze the target detection results to determine the target's entry and exit behavior. A long short-term memory network is used to predict the time series behavior of the time series data for a preset behavior. The input features of the long short-term memory network include at least one of the bounding box coordinate sequence of the target detected in the current frame, the target's category confidence, and the displacement vector of the target's center point between consecutive frames. A record generation module 240 is used to generate entry and exit records based on the entry and exit behavior of the target; Prediction module 250, configured to analyze entry and exit patterns in the entry and exit records using a large language model, and predict abnormal behavior in combination with the time series data; The management module 260 is used to link a preset external system to perform corresponding operations based on the prediction results of the above abnormal behavior.

[0099] Understandably, Figure 2 The relevant contents of each module in the above method embodiment have been described in detail, and the details can be referred to the contents of the method embodiment; Figure 2 The provided substation entrance and exit behavior recognition device 200 can perform the following Figure 1Any steps in the illustrated embodiment will not be described in detail here.

[0100] In one embodiment of the present application, an electronic device is also provided. Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302. The memory 302 stores a computer program. When the computer program is executed by the processor 301, the following operations are performed: Figure 1 The electronic device 300 may further include an input / output device, etc. In a specific embodiment, the electronic device may be a terminal device, etc.

[0101] In one embodiment, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program. When the computer program is executed by the processor 301, the processor 301 executes any step in the above method embodiment.

[0102] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0103] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for identifying substation entrance and exit behaviors, characterized in that: The method comprises: Acquire video data of a substation entrance and exit, and preprocess the video data to obtain preprocessed image data; Using the YOLO11 target detection model to perform target detection on the preprocessed image data to obtain a target detection result; Combining the station door state information and the time series data in the target detection results, a rule R(B, S(t)) defined based on the bounding box B and the station door state S(t) is set to analyze the target detection results to determine the target's entry and exit behavior; wherein, for a preset behavior, a long short-term memory network is used to predict the time series behavior of the time series data; the input features of the long short-term memory network include: a bounding box coordinate sequence of the target detected in the current frame, the target's category confidence, and at least one of the displacement vector of the target center point between consecutive frames; Generate entry and exit records based on the entry and exit behavior of the target; Using a large language model to analyze entry and exit patterns in the entry and exit records, and combining the time series data to predict abnormal behavior; Based on the prediction results of the abnormal behavior, the preset external system is linked to perform corresponding operations.

2. The substation entrance and exit behavior identification method according to claim 1 is characterized in that: The preprocessing includes: performing frame extraction on the video data, performing noise reduction processing using Gaussian filtering or median filtering, performing grayscale processing or retaining RGB color processing, and performing one or more of the following image enhancements using histogram equalization or contrast stretching.

3. The substation entrance and exit behavior identification method according to claim 2 is characterized in that: After performing target detection on the preprocessed image data using the YOLO11 target detection model, the method further includes: Using non-maximum suppression algorithm to remove redundant detection frames; and / or, Gaussian mixture model is used for background modeling.

4. The substation entrance and exit behavior identification method according to claim 1 is characterized in that: The station door status information is obtained in real time by integrating with the access control system of the substation, or by performing image recognition on the door area in the video data.

5. The substation entrance and exit behavior identification method according to claim 1 is characterized in that: The preset external system includes an access control system, an alarm system or a remote management platform; The linkage preset external system performs corresponding operations, including triggering sound and light alarms, pushing notifications to management personnel, or recording relevant videos.

6. The substation entrance and exit behavior identification method according to claim 1, characterized in that: The method further comprises: The YOLO11 target detection model is transferred and learned using self-collected video data for substation scenarios. The model weights and hyperparameters are adjusted, and data augmentation technology is used to improve the model robustness.

7. A substation entrance and exit behavior recognition device, characterized in that: include: An acquisition module is used to acquire video data of the substation entrance and exit, and preprocess the video data to obtain preprocessed image data; A target detection module is used to perform target detection on the preprocessed image data using a YOLO11 target detection model to obtain a target detection result; A behavior analysis module is configured to combine the station door state information and the time series data in the target detection results, set a rule R(B, S(t)) defined based on the bounding box B and the station door state S(t), and analyze the target detection results to determine the target's entry and exit behavior. A long short-term memory network is used to predict the time series behavior of the time series data for a preset behavior. The input features of the long short-term memory network include at least one of the bounding box coordinate sequence of the target detected in the current frame, the target's category confidence, and the displacement vector of the target's center point between consecutive frames. A record generation module, configured to generate entry and exit records based on the entry and exit behavior of the target; a prediction module for analyzing entry and exit patterns in the entry and exit records using a large language model and predicting abnormal behavior in combination with the time series data; The management module is used to link a preset external system to perform corresponding operations based on the prediction results of the abnormal behavior.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

Images