Fence crossing behavior identification method and device, storage medium and electronic equipment

Abstract

The invention provides a fence crossing behavior identification method and apparatus, a storage medium and an electronic device. The method comprises the steps of obtaining video data including image data of a plurality of workers; detecting a target person in the video data by adopting a Center Net network, and obtaining a detection frame representing the position of the target person in the next frame; predicting the position of the target person at the next moment according to the current frame image of the video data by adopting a self-adaptive dynamic Kalman filtering algorithm to obtain a prediction frame; and in combination with the detection frame and the prediction frame, obtaining a target position frame of the target person at the next moment, and determining that the target person has a fence crossing behavior under the condition that the target position frame and the fence marking area at least partially coincide. According to the method, the recognition cost is reduced based on deep learning, the working efficiency is improved, the behavior of crossing the fence can be strictly prevented, meanwhile, a more advanced detection strategy is adopted, the calculated amount is greatly reduced, a motion prediction module is added, and the early warning function is achieved.

Description

Technical Field

[0001] The present application relates to the field of behavior recognition, and in particular, to a method for identifying a behavior of crossing a substation fence, a device for identifying a behavior of crossing a substation fence, a computer-readable storage medium, and an electronic device. Background Art

[0002] There are many high-voltage live equipment in the substation, so fences must be installed during the construction, maintenance, and civil engineering of the substation to prevent the construction team or maintenance personnel from mistakenly entering the live compartment. During large-scale civil engineering and equipment overhaul, due to the lack of safety awareness of different construction teams, or for the convenience of walking, there is a phenomenon of crossing the fence.

[0003] Common anti-crossing fence measures in substations are mainly for safety supervision personnel to use cameras to monitor or replay the station screen from time to time to find out whether there is any crossing of the fence. The behavior of crossing the fence is found by manually checking the monitoring screen.

[0004] The manual spot check of surveillance videos by the safety supervision department consumes a lot of manpower, and it is difficult to detect the phenomenon of crossing the fence in time due to the fatigue of the monitoring personnel. Secondly, the spot check of each surveillance video is prone to omissions of the behavior of crossing the fence, making it difficult to truly detect the behavior of crossing the fence in a timely and complete manner.

[0005] Some substations will install infrared curtains to detect fence crossing, but this requires the deployment of a large number of infrared sensor devices around each device in the substation. In addition, due to the different lengths of equipment maintenance and civil construction cycles in the substation, if the infrared device is turned on for a long time, it is easy for the device to overheat and burn. Four dynamic lifting poles need to be installed around a device in the substation, and an additional power supply circuit needs to be designed to ensure that the power supply of each device in the station is not affected. Although real-time monitoring can be achieved, a lot of financial resources need to be invested to deploy infrared curtains around the fence.

[0006] Existing deep learning-based methods mainly use bounding box regression and classification methods to detect and identify fence crossing behavior. YOLOv3 uses a single image as the input of the entire neural network, uses a logical classifier to complete label classification and divide the image into multiple regions, and predicts bounding boxes and probabilities in each region. This type of method will generate a large number of candidate boxes during bounding box regression, which greatly increases the amount of calculation; at the same time, this method is only suitable for the detection and identification of people crossing fences, and cannot issue warning signals in time when people cross fences. Summary of the invention

[0007] The main purpose of the present application is to provide a method for identifying the behavior of crossing the substation fence, a device for identifying the behavior of crossing the substation fence, a computer-readable storage medium and an electronic device, so as to at least solve the problem that the method for identifying the behavior of crossing the fence in the prior art has a large amount of calculation and a low prediction accuracy for people who have the behavior of crossing the fence.

[0008] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for identifying the behavior of crossing a substation fence is provided, comprising: acquiring video data, the video data including image data of multiple staff members; using a CenterNet network to detect a target person in the video data, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members; using an adaptive dynamic Kalman filtering algorithm to predict the position of the target person at the next moment according to the current frame image of the video data, and obtain a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment; combining the detection frame and the prediction frame to obtain a target position frame of the target person at the next moment, wherein when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

[0009] Optionally, a CenterNet network is used to detect the target person in the video data, and a detection frame representing the position of the target person in the next frame is obtained, including: using DLA34 as the backbone network of the CenterNet network to detect the video data and obtain an initial feature map; decoupling the initial feature map to obtain parameter information of the detection frames of all the staff members, and the parameter information of the detection frame includes the center position, width and height of the detection frame; using a spatiotemporal attention algorithm to identify the features of each of the staff members, determine the target person and the detection frame of the target person, and determine the ID information of the detection frame of the target person.

[0010] Optionally, a spatiotemporal attention algorithm is used to identify the features of each of the staff members to determine the target person and the detection frame of the target person, including: determining a first moving speed of the target person according to the center point of the detection frame of the target person in the previous frame of the video data and the center point of the detection frame of the target person in the current frame, the first moving speed being the moving speed of the target person from the center point of the detection frame of the target person in the previous frame to the center point of the detection frame of the target person in the current frame; determining a second moving speed of the target person according to the center point of the detection frame of the target person in the current frame of the video data and the center point of the detection frame of the target person in the next frame, the second moving speed being the moving speed of the target person from the center point of the detection frame of the target person in the current frame to the center point of the detection frame of the target person in the next frame; confirming the target person according to the intersection-union ratio formula Determine the target intersection-over-union ratio of the detection frame of the target person in the next frame to determine the target person and the detection frame of the target person, where TIoU is the target intersection-over-union ratio of the detection frame of the target person in the next frame, K 32 is the second moving speed, K 21 is the first moving speed, and CIoU is the center intersection over union ratio of the detection box.

[0011] Optionally, an adaptive dynamic Kalman filter algorithm is used to predict the position of the target person at the next moment according to the current frame image of the video data to obtain a prediction frame, including: according to the prediction formula Determine the position of the target person at the next moment and obtain the prediction frame, where: is the state estimation of the target person at time t-1, representing the prediction frame of the target person, F t is the state transition model at the current moment, It is the tracking result of the target person at time t-1, representing the tracking frame at time t-1.

[0012] Optionally, combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment includes: according to the cumulative error formula δ t k-predict =δ t-1 track +δ t-1 k-predict , determine the cumulative error of the adaptive dynamic Kalman filter algorithm at the next moment, where δ t k-predict is the cumulative error of the adaptive dynamic Kalman filter algorithm at the next moment, δ t-1 track is the error of the target position frame of the target person at the current moment, δt-1 k-predict is the cumulative error of the adaptive dynamic Kalman filter algorithm at the current moment; according to the modified tracking formula δ t track =δ t detection +μδ t k-predict , determine the error of the target position frame of the target person at the next moment, where δ t track is the error of the target position frame of the target person at the next moment, δ t detection is the error of the detection frame at the current moment, and μ is a coefficient; based on the error of the target position frame of the target person, the target position frame of the target person at the next moment is determined in combination with the detection frame and the prediction frame.

[0013] Optionally, based on the error of the target position frame of the target person, combining the detection frame and the prediction frame, determining the target position frame of the target person at the next moment includes: updating the Kalman filter formula Determine the initial position frame of the target person at the next moment, where K t is the gain of the adaptive dynamic Kalman filter at time t, P t|t-1 is the estimated covariance matrix at time t-1 to time t, I is the input image, H t is the observation model at time t, R t is the observation noise at time t, and z t The fusion of the results represents the initial position frame of the target person at time t. is the state estimation of the target person at time t-1, representing the prediction box of the target person, z t is the output result of the target detector at time t, the target detector is used to detect the detection frame of the target person, P t|t The estimated covariance matrix updated at time t for the next moment; the target position frame of the target person at the next moment is determined according to the error of the target position frame of the target person and the initial position frame of the target person at the next moment.

[0014] Optionally, according to the modified tracking formula δ t track =δ t detection +μδ t k-predictBefore determining the error of the target position frame of the target person at the next moment, the method further includes: determining the confidence of the detection frame, and when the confidence of the detection frame is greater than a preset confidence, determining the coefficient to be 0; when the confidence of the detection frame is less than or equal to the preset confidence, determining the coefficient to be 1.

[0015] According to another aspect of the present application, a device for identifying fence crossing behavior in a substation is provided, comprising: an acquisition unit, for acquiring video data, wherein the video data includes image data of multiple staff members; a detection unit, for detecting a target person in the video data using a CenterNet network, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members; a prediction unit, for predicting the position of the target person at the next moment according to a current frame image of the video data using an adaptive dynamic Kalman filtering algorithm, and obtaining a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment; a determination unit, for combining the detection frame and the prediction frame to obtain a target position frame of the target person at the next moment, wherein when the target position frame at least partially overlaps with a fence marking area, it is determined that the target person has crossed the fence.

[0016] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for identifying the behavior of crossing the substation fence.

[0017] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the identification methods of crossing the substation fence.

[0018] Applying the technical solution of the present application, the above-mentioned identification method of crossing the fence of the substation first obtains video data, which includes image data of multiple staff members; then uses the CenterNet network to detect the target person in the video data, and obtains a detection frame representing the position of the target person in the next frame, and the target person is one of the multiple staff members; then uses an adaptive dynamic Kalman filter algorithm to predict the position of the target person at the next moment according to the current frame image of the video data, and obtains a prediction frame, which is used to represent the position of the target person at the next moment; finally, the detection frame and the prediction frame are combined to obtain the target position frame of the target person at the next moment, wherein, when the target position frame and the fence marking area at least partially overlap, it is determined that the target person has crossed the fence. The method directly uses the real-time picture of the monitoring camera in the station, saving a lot of investment costs; supervises the entire movement process in the fence from the perspective of multi-target tracking, and obtains the border and category of the target based on the detection strategy of the anchor point, thereby improving the calculation efficiency; adds a motion trajectory prediction strategy, and pre-warns the personnel who are about to cross the fence to ensure the safety of the personnel, and solves the problem that the method for identifying the behavior of crossing the fence in the prior art has a large amount of calculation and low prediction accuracy for the personnel who have crossed the fence. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 A hardware structure block diagram of a mobile terminal for performing a method for identifying a behavior of crossing a substation fence provided in an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of a flow chart of a method for identifying a behavior of crossing a substation fence provided according to an embodiment of the present application is shown;

[0022] Figure 3 A structural block diagram of a DLA-34 backbone network provided according to an embodiment of the present application is shown;

[0023] Figure 4 A schematic diagram of a DLASeg structure in a DLA-34 backbone network provided according to an embodiment of the present application is shown;

[0024] Figure 5 A structural block diagram of a network of a CenterNet-based anchor-free multi-target tracking substation fence anti-crossing monitoring method provided according to an embodiment of the present application is shown;

[0025] Figure 6A schematic diagram showing a TIoU effect on a target person and other surrounding objects provided in an embodiment of the present application is shown;

[0026] Figure 7 A schematic diagram of a flow chart of a Kalman filter algorithm provided according to an embodiment of the present application is shown;

[0027] Figure 8 A structural block diagram of a device for identifying fence crossing behavior of a substation provided according to an embodiment of the present application is shown.

[0028] The above drawings include the following reference numerals:

[0029] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] As introduced in the background technology, the common anti-crossing fence measures in the substation are mainly to use the camera to monitor or replay the station screen from time to time to find out whether there is a situation of crossing the fence. In order to improve the efficiency of manual viewing of monitoring screens, the prior art proposes to install infrared devices around each device in the substation, dynamically raise and lower the infrared device pole to form an infrared curtain, and use infrared real-time sensing to cross the fence, and send out an alarm signal in time. This method is affected by factors such as high temperature weather, rainy days or obstacles. The sensitivity of the infrared device. In addition, the existing deep learning-based methods for detecting and identifying the behavior of crossing the fence are mostly implemented by target detection methods such as YOLOv3, which uses drones to shoot real-time video and transmits it to the background for analysis and calculation. YOLOv3 generates 9 anchor frames for a pedestrian in advance, and then selects the candidate frame with the highest probability as the detection frame through calculation. This type of method will generate a large number of candidate frames during bounding box regression, which greatly increases the amount of calculation. In order to solve the problem that the existing method for identifying the behavior of crossing the fence has a large amount of calculation and a low prediction accuracy for people who will cross the fence, the embodiments of the present application provide a method for identifying the behavior of crossing the substation fence, a device for identifying the behavior of crossing the substation fence, a computer-readable storage medium and an electronic device.

[0034] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0035] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal for a method for identifying a behavior of crossing a substation fence according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0036] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the Z method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] In this embodiment, a method for identifying the behavior of crossing the substation fence running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0038] Figure 2 FIG. 1 is a flow chart of a method for identifying a behavior of crossing a substation fence according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0039] Step S201, obtaining video data, wherein the video data includes image data of multiple staff members;

[0040] Specifically, real-time video is shot by drones to obtain video data including multiple staff members, providing a basic database for the subsequent use of CenterNet network to detect target personnel in video data.

[0041] Step S202, using CenterNet network to detect the target person in the video data, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members;

[0042] Specifically, YOLOv3 uses CenterNet as the feature extraction network, skips the step of generating candidate boxes, and optimizes the problem of Darknet-53 network generating a large number of candidate boxes, resulting in reduced computing speed. At the same time, the CenterNet network is suitable for target persons of different sizes, has a wide range of applications, and improves detection speed and detection accuracy.

[0043] The CenterNet network is used to detect the target person in the video data, and a detection frame representing the position of the target person in the next frame is obtained, including the following steps:

[0044] Step S301, using DLA34 as the backbone network of the CenterNet network, detecting the video data to obtain an initial feature map;

[0045] Step S302, decoupling the initial feature map to obtain parameter information of the detection frames of all the staff members, where the parameter information of the detection frames includes the center position, width and height of the detection frames;

[0046] Step S303, using a spatiotemporal attention algorithm to identify the features of each of the above-mentioned staff members, determine the above-mentioned target person and the detection frame of the above-mentioned target person, and determine the ID information of the detection frame of the above-mentioned target person.

[0047] Specifically, using DLA34 as the backbone network of the CenterNet network can better balance speed and detection accuracy, while being able to fuse multiple layers of underlying feature information. Figure 3 As shown in the figure, the DLA-34 backbone network adopts more skip-layer connections to better integrate multi-level semantic features.

[0048] Furthermore, the specific process of detecting and processing the input video data is as follows: Figure 4 As shown in the figure, the DLA34-base module is used to complete the hierarchical fusion, the DLAUP module is responsible for iterative fusion, and the IDAUP is the upsampling operation module. The detection heads are divided into three branches: heat map (hm_head), center offset (wh_head) and bounding box regression (reg_head). Among them, the heat map branch is responsible for predicting the probability of the existence of the target on the feature map. The center value of the object is expected to be 1, and the greater the distance between the neighboring value and the center value, the smaller the value. The heat map is processed by non-maximum suppression, and the center position of the target person can be predicted as The feature map is 4 times downsampled relative to the input image, and the center of the target person in the original image is The center offset branch is used to predict the offset of the target center. The bounding box regression branch predicts the width and height of each object. Adding the predicted object center to the target center offset in the heat map can predict the position of the target center at the next moment.

[0049] Among them, the spatiotemporal attention algorithm is used to identify the characteristics of each of the above-mentioned staff members, and the above-mentioned target personnel and the detection frame of the above-mentioned target personnel are determined, which also includes the following steps:

[0050] Step S3031, determining a first moving speed of the target person according to the center point of the detection frame of the target person in the previous frame in the video data and the center point of the detection frame of the target person in the current frame, wherein the first moving speed is the moving speed of the target person from the center point of the detection frame of the target person in the previous frame to the center point of the detection frame of the target person in the current frame;

[0051] Step S3032, determining a second moving speed of the target person according to the center point of the detection frame of the target person in the current frame and the center point of the detection frame of the target person in the next frame in the video data, wherein the second moving speed is a moving speed of the target person from the center point of the detection frame of the target person in the current frame to the center point of the detection frame of the target person in the next frame;

[0052] Step S3033: confirm the formula based on the intersection-and-union ratio Determine the target intersection-over-union ratio of the detection frame of the target person in the next frame to determine the target person and the detection frame of the target person, where TIoU is the target intersection-over-union ratio of the detection frame of the target person in the next frame, K 32 is the second moving speed, K 21 is the first moving speed, and CIoU is the center intersection over Union ratio of the detection box.

[0053] Specifically, Figure 5 As shown in the figure, the target detection network is CenterNet, which obtains the width, height and center of the target by decoding the feature map, thereby obtaining an accurate detection frame and avoiding the mechanism of generating a large number of candidate frames. In order to track moving workers, spatiotemporal attention is added to the re-identification module to obtain unique identity ID information. The spatiotemporal attention algorithm can accurately lock the target person and improve the tracking accuracy of the target person.

[0054] Furthermore, TIoU takes into account the displacement size and direction of the target on the basis of CIoU (Complete IoU), that is, it embeds the speed of the target per unit time, thus indirectly incorporating time information. Figure 6 As shown, Figure 6As shown, the input of the network is two consecutive frames of images. The displacement of the target center point of the current frame (T = 2) and the target center point of the previous frame (T = 1) is introduced into CIoU to obtain the speed K of the target in the previous frame. 21 , calculate the displacement between the current frame and the next frame (T=3), that is, the speed K of the target in the current frame 32 When the video sampling frequency is high, the target displacement in each frame is very small, and it can be roughly regarded as the target moving in a straight line at a uniform speed. Therefore, K 21 and K 32 The values ​​of will be roughly equal or approximately equal within a very small range. This provides a theoretical basis for distinguishing the IoU of the target object and its surrounding objects, which is close to the IoU of the target object itself in the previous frame. When the objects around the target also overlap with the target, such as the dotted box at time T=2 in the figure overlaps with the target box at time T=3, its K 32 The IoU value of the path and V 32 The IoU values ​​of the paths are equal, which may easily cause the identity ID of the tracked target to switch to its surrounding objects, resulting in tracking failure. However, considering its speed K 32 With V 32 The values ​​of are very different (in different directions), so the objects can be effectively distinguished, allowing the tracked target to transmit the original identity ID.

[0055] Among them, TIoU is defined by cosine distance. The loss function of TIoU can be jointly expressed by formula (1), formula (2), and formula (3):

[0056]

[0057] In formula (1), b and b gt are the bounding boxes B and B gt The center point of the image, ρ() represents the Euclidean distance and c is the left diagonal of the minimum enclosing box of the two bounding boxes, Wgt represents the width of the detected person's true label box, hgt represents the height of the detected person's true label box; w is the width of the person box obtained by the detection algorithm, and h is the height of the person box obtained by the detection algorithm. In the method, β=2 is conducive to helping the model improve the positioning and detection accuracy of the IoU target person. More importantly, it can speed up the learning of the target person in the previous frame, thereby improving the final target person tracking accuracy.

[0058] Step S203, using an adaptive dynamic Kalman filter algorithm, predicting the position of the target person at the next moment according to the current frame image of the video data, and obtaining a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment;

[0059] Specifically, the adaptive dynamic Kalman filter algorithm can achieve frame-by-frame prediction, that is, predict the target position of the next frame based on the result of the previous frame, and can fuse multi-source information to make the prediction more accurate. In addition, the calculation process of the Kalman filter is relatively simple, which can improve the detection efficiency.

[0060] Among them, the Kalman filter algorithm is a data fusion algorithm that can fuse measurement data from different channels to obtain an optimal estimate. Kalman filtering estimates the motion trajectory of the target at the next moment based on the assumption that the target is moving in a uniform linear straight line. It is a motion model centered on the estimate and corrected by the observation value. In multi-target tracking, the Kalman filter only needs the state estimate of the target in the previous frame and the observation value of the target in the current frame to complete the motion trajectory prediction.

[0061] There are two situations in which position prediction using motion information based on adaptive dynamic Kalman filtering is used: when the target is occluded and the appearance information is unreliable, the motion feature information of the target trajectory is used to predict the target's position information when it is occluded. In particular, when the target is completely occluded, only motion trajectory prediction can be used to ensure that the target can maintain its identity ID when it reappears in the picture after leaving the occluder; when the target enters or leaves the picture: most of the target itself is outside the picture, and the Re-ID branch cannot obtain reliable appearance feature information, so motion trajectory prediction is used to supplement the missing associated dependencies at this time.

[0062] Among them, an adaptive dynamic Kalman filter algorithm is used to predict the position of the target person at the next moment according to the current frame image of the video data to obtain a prediction frame, including the following steps:

[0063] Step S2031, according to the prediction formula Determine the position of the target person at the next moment and obtain the prediction frame, where: is the state estimation of the target person at time t-1, representing the prediction box of the target person, F t is the state transition model at the current moment, It is the tracking result of the target person at time t-1, representing the tracking frame at time t-1.

[0064] Specifically, the Kalman filter process can be described as two stages: prediction and update. The prediction stage can be represented by the above prediction formula and the covariance matrix formula (4):

[0065]

[0066] Among them, P t|t-1 is the estimated covariance matrix of the target personnel at time t-1, P t-1|t-1is the estimated covariance matrix updated at time t for the next moment, Q t is the covariance matrix of the prediction process noise at time t. Prediction is to estimate the current system state using the state at the previous moment.

[0067] The adaptive dynamic Kalman filter reduces the cumulative error problem caused by the long time of Kalman filtering to a certain extent and corrects the drift of the tracking frame.

[0068] Step S204, combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

[0069] Specifically, the Kalman filter of the motion trajectory prediction model is improved, and an adaptive Kalman filter design adaptive coefficient μ is proposed to alleviate the error accumulation problem in the prediction and estimation process and optimize the detection accuracy.

[0070] Combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment includes the following steps:

[0071] Step S401, according to the cumulative error formula δ t k-predict =δ t-1 track +δ t-1 k-predict , determine the cumulative error of the adaptive dynamic Kalman filter algorithm at the next moment, where δ t k-predict is the cumulative error of the above adaptive dynamic Kalman filter algorithm at the next moment, δ t-1 track is the error of the target position frame of the target person at the current moment, δ t-1 k-predict is the cumulative error of the above adaptive dynamic Kalman filter algorithm at the current moment;

[0072] Step S402, according to the modified tracking formula δ t track =δ t detection +μδ t k-predict , determine the error of the target position frame of the target person at the next moment, where δ t track is the error of the target position frame of the target person at the next moment, δ t detection is the error of the above detection frame at the current moment, μ is the coefficient;

[0073] Step S403, based on the error of the target position frame of the target person, combined with the detection frame and the prediction frame, determine the target position frame of the target person at the next moment.

[0074] Specifically, the error δ of the final tracking result track Mainly caused by the uncertainty error δ caused by the target detector detection detection and the cumulative error δ generated in the Kalman filter prediction phase k-predict From the cumulative error formula and the corrected tracking formula, it can be concluded that the error accumulation of the Kalman filter is to infer the prediction frame at the current moment based on the tracking frame at the previous moment. The prediction frame at the current moment accumulates the error of the tracking frame at the previous moment and transfers it to the current tracking frame.

[0075] Wherein, based on the error of the target position frame of the target person, combined with the detection frame and the prediction frame, determining the target position frame of the target person at the next moment includes the following steps:

[0076] Step S4031, update the formula according to Kalman filter Determine the initial position frame of the target person at the next moment, where K t is the gain of the adaptive dynamic Kalman filter at time t, P t|t-1 is the estimated covariance matrix at time t-1 to time t, I is the input image, H t is the observation model at time t, R t is the observation noise at time t, and z t The fusion of the results represents the initial position frame of the target person at time t. is the state estimation of the target person at time t at time t-1, representing the prediction box of the target person, z t is the output result of the target detector at time t, which is used to detect the detection frame of the target person, P t|t The estimated covariance matrix updated at time t for the next moment;

[0077] Step S4032, determining the target position frame of the target person at the next moment according to the error of the target position frame of the target person and the initial position frame of the target person at the next moment.

[0078] Specifically, the update uses the a priori knowledge about the system at the current moment, that is, the current frame detection result obtained by the target detector, and fuses it with the estimated state of the system to obtain the optimal estimate of the tracking system at the current moment.

[0079] Furthermore, if Figure 7As shown in the figure, the adaptive dynamic Kalman filter is observation-centric and abandons the original estimation-centric Kalman filter. After the advanced target detector detects the target, when the confidence of the target is greater than 0.8, the detection frame can be considered reliable. Therefore, after discarding the prediction frame of the Kalman filter, the detection frame is directly used as the tracking frame and then data association is performed to obtain the target's identity ID. On the contrary, when the detection frame obtained by target detection is smaller than the set threshold, the detection frame and the prediction frame of the Kalman filter are fused to obtain the tracking frame. In this way, when the detection frame has a high confidence, the prediction error accumulation of the Kalman filter is interrupted. The high confidence detection frame reduces the error accumulation of the Kalman filter prediction to a certain extent and can correct the excessive drift of the tracking frame in a timely manner.

[0080] Among them, according to the modified tracking formula δ t track =δ t detection +μδ t k-predict Before determining the error of the target position frame of the target person at the next moment, the method further comprises the following steps:

[0081] Step S501, determining the confidence of the above detection frame,

[0082] Step S502, when the confidence of the detection frame is greater than a preset confidence, determining the coefficient to be 0;

[0083] Step S503: When the confidence of the detection frame is less than or equal to a preset confidence, the coefficient is determined to be 1.

[0084] Specifically, a coefficient μ is added to the modified tracking formula. When μ=0, it means that the confidence of the detection frame is high and the prediction frame of the Kalman filter is discarded; when μ=1, the original Kalman filter mechanism is still used. Adaptive performance is that when there are many targets in the picture or the occlusion is serious, there are often a lot of trajectory drift problems. At this time, the function adopted by our μ is adaptive to make it appear 0 with a higher probability and 1 with a low probability.

[0085] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for identifying fence crossing behavior of the present application will be described in detail below in combination with specific embodiments.

[0086] This embodiment relates to a specific substation fence anti-crossing monitoring algorithm based on CenterNet anchor-free multi-target tracking. The specific implementation steps are given in Table 1:

[0087] Table 1. Research steps of substation fence anti-crossing monitoring method based on CenterNet anchor-free multi-target tracking

[0088]

[0089]

[0090] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0091] The embodiment of the present application also provides a device for identifying the behavior of crossing the fence of a substation. It should be noted that the device for identifying the behavior of crossing the fence of a substation in the embodiment of the present application can be used to execute the method for identifying the behavior of crossing the fence of a substation provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0092] The following introduces the device for identifying the substation fence crossing behavior provided in an embodiment of the present application.

[0093] Figure 8 Schematic diagram of a device for identifying fence crossing behavior of a substation according to an embodiment of the present application. Figure 8 As shown, the device includes: an acquisition unit 10, a detection unit 20, a prediction unit 30 and a determination unit 40, the acquisition unit 10 is used to acquire video data, the above video data includes image data of multiple staff members; the detection unit 20 is used to use the CenterNet network to detect the target person in the above video data, and obtain a detection frame representing the position of the above target person in the next frame, and the above target person is one of the multiple staff members; the prediction unit 30 is used to use an adaptive dynamic Kalman filter algorithm to predict the position of the above target person at the next moment according to the current frame image of the above video data, and obtain a prediction frame, and the above prediction frame is used to represent the position of the above target person at the next moment; the determination unit 40 is used to combine the above detection frame and the above prediction frame to obtain a target position frame of the above target person at the next moment, wherein, when the above target position frame at least partially overlaps with the fence marking area, it is determined that the above target person has crossed the fence.

[0094] The above-mentioned substation fence crossing behavior identification device of the present application includes: an acquisition unit, a detection unit, a prediction unit and a determination unit, the acquisition unit is used to acquire video data, the video data includes image data of multiple staff members; the detection unit is used to use the CenterNet network to detect the target person in the video data, and obtain a detection frame representing the position of the target person in the next frame, and the target person is one of the multiple staff members; the prediction unit is used to use an adaptive dynamic Kalman filtering algorithm to predict the position of the target person at the next moment according to the current frame image of the video data, and obtain a prediction frame, and the prediction frame is used to represent the position of the target person at the next moment; the determination unit is used to combine the detection frame and the prediction frame to obtain a target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence. The device directly uses the real-time images from the surveillance cameras in the station, saving a lot of investment costs; it monitors the entire movement process within the fence from the perspective of multi-target tracking, and obtains the target's border and category based on the anchor point detection strategy to improve computing efficiency; it adds a motion trajectory prediction strategy to pre-warn people who are about to cross the fence to ensure their safety, and solves the problem that the existing method of identifying the behavior of crossing the fence has a large amount of calculation and low prediction accuracy for people who are going to cross the fence.

[0095] In some examples, the detection unit includes a first detection module, a second detection module and a third detection module. The first detection module is used to use DLA34 as the backbone network of the CenterNet network to detect the video data and obtain an initial feature map. The second detection module is used to decouple the initial feature map to obtain parameter information of the detection frames of all the staff members. The parameter information of the detection frame includes the center position, width and height of the detection frame. The third detection module is used to use a spatiotemporal attention algorithm to identify the characteristics of each of the staff members, determine the target person and the detection frame of the target person, and determine the ID information of the detection frame of the target person. Real-time video is captured by a drone to obtain video data including multiple staff members, providing a basic database for the subsequent use of the CenterNet network to detect target personnel in video data.

[0096] In some instances, the third detection module also includes a first detection submodule, a second detection submodule and a third detection submodule, the first detection submodule is used to determine the first moving speed of the target person according to the center point of the detection frame of the target person in the previous frame of the video data and the center point of the detection frame of the target person in the current frame, the first moving speed being the moving speed of the target person from the center point of the detection frame of the target person in the previous frame to the center point of the detection frame of the target person in the current frame; the second detection submodule is used to determine the second moving speed of the target person according to the center point of the detection frame of the target person in the current frame of the video data and the center point of the detection frame of the target person in the next frame, the second moving speed being the moving speed of the target person from the center point of the detection frame of the target person in the current frame to the center point of the detection frame of the target person in the next frame; the third detection submodule is used to confirm the formula according to the intersection-union ratio Determine the target intersection-over-union ratio of the detection frame of the target person in the next frame to determine the target person and the detection frame of the target person, where TIoU is the target intersection-over-union ratio of the detection frame of the target person in the next frame, K 32 is the second moving speed, K 21 is the first moving speed, and CIoU is the center intersection-over-union ratio of the detection frame. YOLOv3 uses CenterNet as the feature extraction network, skipping the step of generating candidate frames. At the same time, the CenterNet network is applicable to target persons of different sizes, has a wide range of applications, and improves detection speed and detection accuracy.

[0097] In some embodiments, the prediction unit is used to predict the Determine the position of the target person at the next moment and obtain the prediction frame, where: is the state estimation of the target person at time t-1, representing the prediction box of the target person, F t is the state transition model at the current moment, The tracking result of the target person at time t-1 represents the tracking frame at time t-1. The adaptive dynamic Kalman filter algorithm can realize frame-by-frame prediction, that is, predict the target position of the next frame based on the result of the previous frame, and can integrate multi-source information to make the prediction more accurate.

[0098] In some examples, the determination unit includes a first determination module, a second determination module and a third determination module, wherein the first determination module is configured to determine the error according to the cumulative error formula δ t k-predict =δ t-1 track +δ t-1 k-predict, determine the cumulative error of the adaptive dynamic Kalman filter algorithm at the next moment, where δ t k-predict is the cumulative error of the above adaptive dynamic Kalman filter algorithm at the next moment, δ t-1 track is the error of the target position frame of the target person at the current moment, δ t-1 k-predict is the cumulative error of the adaptive dynamic Kalman filter algorithm at the current moment; the second determination module is used to correct the tracking formula δ t track =δ t detection +μδ t k-predict , determine the error of the target position frame of the target person at the next moment, where δ t track is the error of the target position frame of the target person at the next moment, δ t detection is the error of the detection frame at the current moment, and μ is the coefficient; the third determination module is used to determine the target position frame of the target person at the next moment based on the error of the target position frame of the target person, combined with the detection frame and the prediction frame. The adaptive dynamic Kalman filter reduces the cumulative error problem caused by the long time of the Kalman filter to a certain extent and corrects the drift of the tracking frame.

[0099] In some examples, the third determination module further includes a first determination submodule and a second determination submodule, wherein the first determination submodule is configured to update the formula according to the Kalman filter. Determine the initial position frame of the target person at the next moment, where K t is the gain of the adaptive dynamic Kalman filter at time t, P t|t-1 is the estimated covariance matrix at time t-1 to time t, I is the input image, H t is the observation model at time t, R t is the observation noise at time t, and z t The fusion of the results represents the initial position frame of the target person at time t. is the state estimation of the target person at time t at time t-1, representing the prediction box of the target person, z t is the output result of the target detector at time t, which is used to detect the detection frame of the target person, P t|tThe estimated covariance matrix for the next moment updated at time t; the second determination submodule is used to determine the target position frame of the target person at the next moment according to the error of the target position frame of the target person and the initial position frame of the target person at the next moment. The current frame detection result obtained by the target detector is fused with the estimated state of the system to obtain the optimal estimate of the tracking system at the current moment.

[0100] In some examples, the determination unit further includes a fourth determination module, a fifth determination module and a sixth determination module, wherein the fourth determination module is used to determine the confidence of the detection frame, the fifth determination module is used to determine the coefficient to be 0 when the confidence of the detection frame is greater than the preset confidence, and the sixth determination module is used to determine the coefficient to be 1 when the confidence of the detection frame is less than or equal to the preset confidence. The Kalman filter of the motion trajectory prediction model is improved, and an adaptive Kalman filter is proposed to design an adaptive coefficient, which alleviates the error accumulation problem in the prediction and estimation process and optimizes the detection accuracy.

[0101] The above-mentioned identification device for substation crossing fence behavior includes a processor and a memory, and the above-mentioned acquisition unit and the like are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0102] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to solve the problem that the existing method for identifying fence crossing behavior has a large amount of calculation and a low prediction accuracy for people who have fence crossing behavior.

[0103] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0104] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for identifying the behavior of crossing the substation fence.

[0105] Specifically, the identification method of crossing the substation fence behavior includes:

[0106] Step S201, obtaining video data, wherein the video data includes image data of multiple staff members;

[0107] Step S202, using CenterNet network to detect the target person in the video data, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members;

[0108] Step S203, using an adaptive dynamic Kalman filter algorithm, predicting the position of the target person at the next moment according to the current frame image of the video data, and obtaining a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment;

[0109] Step S204, combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

[0110] Optionally, a CenterNet network is used to detect the target person in the video data, and a detection frame representing the position of the target person in the next frame is obtained, including: using DLA34 as the backbone network of the CenterNet network to detect the video data and obtain an initial feature map; decoupling the initial feature map to obtain parameter information of the detection frames of all the staff members, and the parameter information of the detection frame includes the center position, width and height of the detection frame; using a spatiotemporal attention algorithm to identify the features of each of the staff members, determine the target person and the detection frame of the target person, and determine the ID information of the detection frame of the target person.

[0111] Optionally, a spatiotemporal attention algorithm is used to identify the features of each of the above-mentioned staff members to determine the above-mentioned target person and the detection frame of the above-mentioned target person, including: determining a first moving speed of the above-mentioned target person according to the center point of the detection frame of the above-mentioned target person in the previous frame of the above-mentioned video data and the center point of the detection frame of the above-mentioned target person in the current frame, the first moving speed being the moving speed of the above-mentioned target person from the center point of the detection frame of the above-mentioned target person in the previous frame to the center point of the detection frame of the above-mentioned target person in the current frame; determining a second moving speed of the above-mentioned target person according to the center point of the detection frame of the above-mentioned target person in the current frame of the above-mentioned video data and the center point of the detection frame of the above-mentioned target person in the next frame, the second moving speed being the moving speed of the above-mentioned target person from the center point of the detection frame of the above-mentioned target person in the current frame to the center point of the detection frame of the above-mentioned target person in the next frame; confirming the formula according to the intersection-union ratio Determine the target intersection-over-union ratio of the detection frame of the target person in the next frame to determine the target person and the detection frame of the target person, where TIoU is the target intersection-over-union ratio of the detection frame of the target person in the next frame, K 32is the second moving speed, K 21 is the first moving speed, and CIoU is the center intersection over Union ratio of the detection box.

[0112] Optionally, an adaptive dynamic Kalman filter algorithm is used to predict the position of the target person at the next moment according to the current frame image of the video data to obtain a prediction frame, including: according to the prediction formula Determine the position of the target person at the next moment and obtain the prediction frame, where: is the state estimation of the target person at time t-1, representing the prediction box of the target person, F t is the state transition model at the current moment, It is the tracking result of the target person at time t-1, representing the tracking frame at time t-1.

[0113] Optionally, combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment includes: according to the cumulative error formula δ t k-predict =δ t-1 track +δ t-1 k-predict , determine the cumulative error of the adaptive dynamic Kalman filter algorithm at the next moment, where δ t k-predict is the cumulative error of the above adaptive dynamic Kalman filter algorithm at the next moment, δ t-1 track is the error of the target position frame of the target person at the current moment, δ t-1 k-predict is the cumulative error of the adaptive dynamic Kalman filter algorithm at the current moment; according to the modified tracking formula δ t track =δ t detection +μδ t k-predict , determine the error of the target position frame of the target person at the next moment, where δ t track is the error of the target position frame of the target person at the next moment, δ t detection is the error of the detection frame at the current moment, μ is the coefficient; based on the error of the target position frame of the target person, combined with the detection frame and the prediction frame, the target position frame of the target person at the next moment is determined.

[0114] Optionally, based on the error of the target position frame of the target person, combined with the detection frame and the prediction frame, determining the target position frame of the target person at the next moment includes: updating the Kalman filter formula Determine the initial position frame of the target person at the next moment, where K t is the gain of the adaptive dynamic Kalman filter at time t, P t|t-1 is the estimated covariance matrix at time t-1 to time t, I is the input image, H t is the observation model at time t, R t is the observation noise at time t, and z t The fusion of the results represents the initial position frame of the target person at time t. is the state estimation of the target person at time t at time t-1, representing the prediction box of the target person, z t is the output result of the target detector at time t, which is used to detect the detection frame of the target person, P t|t is the estimated covariance matrix updated at time t for the next moment; according to the error of the target position frame of the target person and the initial position frame of the target person at the next moment, the target position frame of the target person at the next moment is determined.

[0115] Optionally, according to the modified tracking formula δ t track =δ t detection +μδ t k-predict Before determining the error of the target position frame of the target person at the next moment, the method further includes: determining the confidence of the detection frame, and when the confidence of the detection frame is greater than the preset confidence, determining the coefficient to be 0; when the confidence of the detection frame is less than or equal to the preset confidence, determining the coefficient to be 1.

[0116] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the program executes the method for identifying the behavior of crossing the substation fence when running.

[0117] Specifically, the identification method of crossing the substation fence behavior includes:

[0118] Step S201, obtaining video data, wherein the video data includes image data of multiple staff members;

[0119] Step S202, using CenterNet network to detect the target person in the video data, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members;

[0120] Step S203, using an adaptive dynamic Kalman filter algorithm, predicting the position of the target person at the next moment according to the current frame image of the video data, and obtaining a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment;

[0121] Step S204, combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

[0122] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0123] Step S201, obtaining video data, wherein the video data includes image data of multiple staff members;

[0124] Step S202, using CenterNet network to detect the target person in the video data, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members;

[0125] Step S203, using an adaptive dynamic Kalman filter algorithm, predicting the position of the target person at the next moment according to the current frame image of the video data, and obtaining a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment;

[0126] Step S204, combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

[0127] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0128] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0129] Step S201, obtaining video data, wherein the video data includes image data of multiple staff members;

[0130] Step S202, using CenterNet network to detect the target person in the video data, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members;

[0131] Step S203, using an adaptive dynamic Kalman filter algorithm, predicting the position of the target person at the next moment according to the current frame image of the video data, and obtaining a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment;

[0132] Step S204, combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

[0133] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0140] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0142] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0143] 1) The above-mentioned identification method of crossing the fence of the substation of the present application first obtains video data, which includes image data of multiple staff members; then the CenterNet network is used to detect the target person in the video data, and a detection frame representing the position of the target person in the next frame is obtained, and the target person is one of the multiple staff members; then the adaptive dynamic Kalman filter algorithm is used to predict the position of the target person at the next moment according to the current frame image of the video data, and a prediction frame is obtained, and the prediction frame is used to represent the position of the target person at the next moment; finally, the target position frame of the target person at the next moment is obtained by combining the detection frame and the prediction frame, wherein, when the target position frame and the fence marking area at least partially overlap, it is determined that the target person has crossed the fence. The method directly uses the real-time picture of the monitoring camera in the station, saving a lot of investment costs; the entire movement process in the fence is supervised from the perspective of multi-target tracking, and the detection strategy based on the anchor point is used to obtain the border and category of the target, thereby improving the calculation efficiency; the motion trajectory prediction strategy is added to pre-warn the personnel who are about to cross the fence to ensure the safety of the personnel, and solve the problem that the behavior recognition method of crossing the fence in the prior art has a large amount of calculation and the prediction accuracy of the personnel who have crossed the fence is low.

[0144] 2) The above-mentioned substation fence crossing behavior identification device of the present application includes: an acquisition unit, a detection unit, a prediction unit and a determination unit. The acquisition unit is used to acquire video data, and the video data includes image data of multiple staff members; the detection unit is used to use the CenterNet network to detect the target person in the video data, and obtain a detection frame representing the position of the target person in the next frame, and the target person is one of the multiple staff members; the prediction unit is used to use an adaptive dynamic Kalman filtering algorithm to predict the position of the target person at the next moment according to the current frame image of the video data, and obtain a prediction frame, and the prediction frame is used to represent the position of the target person at the next moment; the determination unit is used to combine the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence. The device directly uses the real-time images from the surveillance cameras in the station, saving a lot of investment costs; it monitors the entire movement process within the fence from the perspective of multi-target tracking, and obtains the target's border and category based on the anchor point detection strategy to improve computing efficiency; it adds a motion trajectory prediction strategy to pre-warn people who are about to cross the fence to ensure their safety, and solves the problem that the existing method of identifying the behavior of crossing the fence has a large amount of calculation and low prediction accuracy for people who are going to cross the fence.

[0145] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying the behavior of crossing the fence of a substation, characterized in that: include: Acquire video data in real time, wherein the video data includes image data of multiple staff members; Using a CenterNet network to detect a target person in the video data, and obtaining a detection frame representing the position of the target person in the next frame, wherein the target person is one of the multiple staff members; Adopting an adaptive dynamic Kalman filter algorithm, according to the current frame image of the video data, predicting the position of the target person at the next moment, and obtaining a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment; The detection frame and the prediction frame are combined to obtain a target position frame of the target person at the next moment, wherein, when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

2. The method according to claim 1, characterized in that: The CenterNet network is used to detect the target person in the video data, and a detection frame representing the position of the target person in the next frame is obtained, including: Using DLA34 as the backbone network of the CenterNet network, detecting the video data to obtain an initial feature map; Decoupling the initial feature map to obtain parameter information of the detection frames of all the staff members, wherein the parameter information of the detection frames includes the center position, width and height of the detection frames; A spatiotemporal attention algorithm is used to identify the features of each of the staff members, determine the target person and the detection frame of the target person, and determine the ID information of the detection frame of the target person.

3. The method according to claim 2, characterized in that Using a spatiotemporal attention algorithm to identify the features of each staff member, determining the target person and the detection frame of the target person, including: Determine a first moving speed of the target person according to the center point of the detection frame of the target person in the previous frame in the video data and the center point of the detection frame of the target person in the current frame, wherein the first moving speed is the moving speed of the target person from the center point of the detection frame of the target person in the previous frame to the center point of the detection frame of the target person in the current frame; Determine a second moving speed of the target person according to the center point of the detection frame of the target person in the current frame and the center point of the detection frame of the target person in the next frame in the video data, wherein the second moving speed is a moving speed of the target person from the center point of the detection frame of the target person in the current frame to the center point of the detection frame of the target person in the next frame; According to the intersection and union ratio confirmation formula Determine the target intersection-over-union ratio of the detection frame of the target person in the next frame to determine the target person and the detection frame of the target person, where TIoU is the target intersection-over-union ratio of the detection frame of the target person in the next frame, K 32 is the second moving speed, K 21 is the first moving speed, and CIoU is the center intersection over union ratio of the detection box.

4. The method according to claim 1, characterized in that: Adopting an adaptive dynamic Kalman filter algorithm, according to the current frame image of the video data, predicting the position of the target person at the next moment, and obtaining a prediction frame, including: According to the prediction formula Determine the position of the target person at the next moment and obtain the prediction frame, where: is the state estimation of the target person at time t-1, representing the prediction frame of the target person, F t is the state transition model at the current moment, It is the tracking result of the target person at time t-1, representing the tracking frame at time t-1.

5. The method according to claim 1, characterized in that Combining the detection frame and the prediction frame to obtain the target position frame of the target person at the next moment includes: According to the cumulative error formula δ t k-predict =δ t-1 track +δ t-1 k-predict , determine the cumulative error of the adaptive dynamic Kalman filter algorithm at the next moment, where δ t k-predict is the cumulative error of the adaptive dynamic Kalman filter algorithm at the next moment, δ t-1 track is the error of the target position frame of the target person at the current moment, δ t-1 k-predict is the cumulative error of the adaptive dynamic Kalman filter algorithm at the current moment; According to the modified tracking formula δ t track =δ t detection +μδ t k-predict , determine the error of the target position frame of the target person at the next moment, where δ t track is the error of the target position frame of the target person at the next moment, δ t detection is the error of the detection frame at the current moment, and μ is the coefficient; Based on the error of the target position frame of the target person, the target position frame of the target person at the next moment is determined in combination with the detection frame and the prediction frame.

6. The method according to claim 5, characterized in that Determining the target position frame of the target person at a next moment based on the error of the target position frame of the target person in combination with the detection frame and the prediction frame includes: According to the Kalman filter update formula Determine the initial position frame of the target person at the next moment, where K t is the gain of the adaptive dynamic Kalman filter at time t, P t|t-1 is the estimated covariance matrix at time t-1 to time t, I is the input image, H t is the observation model at time t, R t is the observation noise at time t, for and z t The fusion of the results represents the initial position frame of the target person at time t. is the state estimation of the target person at time t-1, representing the prediction box of the target person, z t is the output result of the target detector at time t, the target detector is used to detect the detection frame of the target person, P t|t The estimated covariance matrix updated at time t for the next moment; The target position frame of the target person at the next moment is determined according to the error of the target position frame of the target person and the initial position frame of the target person at the next moment.

7. The method according to claim 5, characterized in that According to the modified tracking formula δ t track =δ t detection +μδ t k-predict Before determining the error of the target position frame of the target person at the next moment, the method further includes: Determine the confidence of the detection box, When the confidence of the detection frame is greater than a preset confidence, determining the coefficient to be 0; When the confidence of the detection frame is less than or equal to a preset confidence, the coefficient is determined to be 1.

8. A device for identifying fence crossing behavior in a substation, characterized in that: include: An acquisition unit, used for acquiring video data, wherein the video data includes image data of a plurality of staff members; A detection unit, configured to detect a target person in the video data using a CenterNet network, and obtain a detection frame representing a position of the target person in a next frame, wherein the target person is one of the plurality of staff members; A prediction unit, configured to use an adaptive dynamic Kalman filter algorithm to predict the position of the target person at the next moment according to the current frame image of the video data, and obtain a prediction frame, wherein the prediction frame is used to represent the position of the target person at the next moment; A determination unit is used to combine the detection frame and the prediction frame to obtain a target position frame of the target person at the next moment, wherein when the target position frame at least partially overlaps with the fence marking area, it is determined that the target person has crossed the fence.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for identifying the behavior of crossing the substation fence as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the identification method of crossing the substation fence behavior described in any one of claims 1 to 7.

Images