A behavior tracking method for moving targets

Abstract

The present invention discloses a method for tracking the behavior of a moving target in the field of image phase information tracking and processing technology, aiming to solve the technical problems of slow response and high environmental requirements for identifying moving objects. It includes: capturing changes in the on-site environment according to event streams and timestamps, tracking moving targets to overcome environmental obstacles and identify images of moving targets; fusion of three-channel spatiotemporal events, supplementing the characteristics of time information and asynchronous updates in event streams; extracting multiple feature planes in multiple stages, so that the fused feature map has information of different depths; by storing events of different polarities in different event channels, avoiding the situation of positive and negative polarity cancellation and thus preventing information loss, so as to better use the fused feature map to track moving targets; using a target detection model to obtain the position parameters and size parameters of the moving target from the fused feature map, and formulating control instructions, it has the advantages of high precision, real-time, robustness, adaptability and scalability.

Description

Technical Field

[0001] The invention relates to a method for tracking the behavior of a moving target, and belongs to the technical field of image information tracking and processing. Background Art

[0002] With the rapid development of power systems and the continuous expansion of substations, ensuring safe and efficient operation in substations has become an important issue. The safety of personnel in substations is particularly critical because the high-voltage electrical environment is extremely dangerous. Traditional personnel monitoring methods mainly rely on fixed surveillance cameras and manual inspections, which have shortcomings such as limited coverage and inability to respond and identify dangerous behaviors in real time.

[0003] In recent years, with the development of image processing technology and sensor technology, some new monitoring methods have been proposed to improve monitoring efficiency and safety. However, these methods usually rely on only a single type of image data, such as traditional RGB images, and lack in-depth understanding and accurate detection of human behavior in complex environments. For example, in the case of insufficient lighting or severe occlusion, a monitoring system that relies only on RGB images is difficult to accurately identify human behavior, and thus cannot effectively warn of potential dangers.

[0004] In addition, existing human detection and tracking technologies face great challenges in dealing with highly dynamic scenarios, such as fast-moving people or sudden dangerous behaviors in the work environment, which require the monitoring system to have the ability to respond quickly and with high accuracy. Traditional technologies often find it difficult to achieve the best balance between real-time and accuracy.

[0005] In summary, the existing behavior recognition methods used in power system inspection and security have the problems of slow response and high environmental requirements for tracking moving targets. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method for tracking the behavior of a moving target that can reliably track and identify the moving target in a low-quality environment such as insufficient illumination or severe occlusion.

[0007] To achieve the above objectives, the present disclosure is implemented by adopting the following technical solutions:

[0008] In a first aspect, the present disclosure provides a method for tracking the behavior of a moving target.

[0009] Acquire an event stream of a moving target, wherein the event stream includes a timestamp and polarity corresponding to the event, store events of different polarities in the event stream in a positive event channel and a negative event channel respectively, and store the timestamp in the time channel;

[0010] The positive event channel, the negative event channel and the time channel are fused as a three-channel spatiotemporal event;

[0011] Use the trained convolutional neural network to extract multiple feature planes from the three-channel spatiotemporal events in multiple stages;

[0012] The feature planes extracted at different stages are fused into the same feature map, and the target detection model is used to obtain the position parameters and size parameters of the moving target from the fused feature map. The target state of the moving target is predicted based on the position parameters and size parameters, and control instructions are formulated based on the target state to track the moving target.

[0013] In some embodiments of the first aspect,

[0014] The storing of events of different polarities in the event stream in the positive event channel and the negative event channel respectively includes:

[0015] Obtain definition rules for positive and negative events;

[0016] Classify the events of the event stream according to the defined rules, and store the events of different polarities in the positive event channel or the negative event channel according to the classification results;

[0017] The fusion of the positive event channel, the negative event channel and the time channel as a three-channel spatiotemporal event includes:

[0018] The event stream feature plane output formula is used to output the events in the positive event channel and the negative event channel as different feature planes. The event stream feature plane output formula is as follows:

[0019] ;

[0020] ;

[0021] In the formula, The event flow characteristic plane formed by positive events, is the total number of positive events in the positive event channel, The event flow characteristic plane formed by negative events, is the total number of negative events in the negative event channel;

[0022] Obtain a time surface feature plane according to the timestamp in the time channel;

[0023] The event stream feature plane composed of positive events, the event stream feature plane composed of negative events and the time plane feature plane are fused to represent the coded event frame data, and the coded event frame data is output as a three-channel spatiotemporal event.

[0024] In some embodiments of the first aspect,

[0025] The fusion formula for obtaining the event frame data representing the encoding is as follows:

[0026] F event = [ f EC + ⊕ f TS ⊕ f EC − ] ;

[0027] In the formula, To represent the encoded event frame data, is the time feature plane, For splicing operation;

[0028] The formula for obtaining the time plane feature plane is as follows:

[0029] ;

[0030] In the formula, is the timestamp of the latest event in the event stream, is the timestamp of the input event, is the constant decay rate parameter.

[0031] In some embodiments of the first aspect, the multi-stage extraction of multiple feature planes from the three-channel spatiotemporal events using a trained convolutional neural network includes:

[0032] The convolutional neural network is a ResNet-50 network, which includes five stages. The ResNet-50 network is used to extract the second stage feature plane, the third stage feature plane and the fourth stage feature plane from the event frame data representing the encoding. The feature plane output formulas of the three stages are as follows:

[0033] ;

[0034] ;

[0035] ;

[0036] In the formula, is the characteristic plane of the second stage, It is the feature-level fusion operation in the ResNet-50 network. is the image frame data of the second stage, is the event frame data of the second stage, is the characteristic plane of the third stage, is the image frame data of the third stage, is the event frame data of the third stage, is the characteristic plane of the fourth stage, is the image frame data of the fourth stage, It is the event frame data of the fourth stage.

[0037] In some embodiments of the first aspect, a method for outputting a characteristic plane of any one of the second stage characteristic plane, the third stage characteristic plane and the fourth stage characteristic plane is as follows:

[0038] Input the event frame data of this stage into the first spatial attention module;

[0039] The first spatial attention module uses an average pooling layer to aggregate the spatial information of event frame data and generate feature maps;

[0040] Inputting the feature map into a multilayer perceptron having one hidden layer to obtain an attention coefficient vector of the corresponding channel;

[0041] The adaptive enhancement or suppression channel feature is obtained according to the attention coefficient vector of the corresponding channel; the adaptive enhancement or suppression channel feature acquisition formula is as follows:

[0042] ;

[0043] ;

[0044] In the formula, is the attention coefficient vector corresponding to the channel, is the sigmoid activation function (also known as the logistic activation function or S-type function), A multilayer perceptron with hidden layers. is the average pooling operation, is the event frame data of this stage, is the image frame data of this stage, is element-wise multiplication, To adaptively enhance or suppress channel characteristics.

[0045] The feature plane of the corresponding stage is obtained according to the adaptively enhanced or suppressed channel features.

[0046] In some embodiments of the first aspect, obtaining a feature plane of a corresponding stage according to adaptively enhancing or suppressing channel features includes:

[0047] Inputting the adaptively enhanced or suppressed channel features into a second spatial attention module, the second spatial attention module comprising an average pooling layer and a maximum pooling layer, and the second spatial attention module outputs two two-dimensional feature maps respectively passing through the average pooling layer and the maximum pooling layer;

[0048] Concatenate two two-dimensional feature maps and convolve the concatenated two-dimensional feature maps to generate an effective two-dimensional attention coefficient matrix;

[0049] According to the two-dimensional attention coefficient matrix and the adaptive enhancement or suppression channel features, the feature plane obtained by the spatial attention mechanism is obtained.

[0050] The feature plane obtained by the spatial attention mechanism is spliced ​​with the image frame data of the corresponding stage to obtain the feature plane of the corresponding stage. The formula for obtaining the feature plane of the corresponding stage is as follows:

[0051] M s = σ ( f 3 × 3 ( AvgPool ( U c ); MaxPool ( U c ))) = σ ( f 3 × 3 ([ F avg S ⊕ F max S ]) ;

[0052] ;

[0053] ;

[0054] In the formula, is a two-dimensional attention coefficient matrix, is the convolution kernel size The convolution operation, Maximize pooling operations, is the 2D feature map generated by the average pooling layer in the second spatial attention module, is the 2D feature map generated by the max pooling layer in the second spatial attention module, is the feature plane obtained through the spatial attention mechanism, and F is the feature plane of the corresponding stage.

[0055] In some embodiments of the first aspect,

[0056] The target detection model is used to obtain the position parameters and speed parameters of the moving target from the fused feature map, including:

[0057] A retina network RetinaNet target detection model is used to obtain position parameters and size parameters of the moving target from the fused feature map, wherein a focal loss function is used to replace the original loss function of the RetinaNet detection model;

[0058] The method of predicting the target state of the moving target according to the position parameter and the size parameter includes:

[0059] The position parameters of the fused feature map and the speed parameters corresponding to the position parameters are converted into a state vector. The conversion formula of the state vector is as follows:

[0060] z t = [ x t y t v x t v y t ] ,

[0061] In the formula, t is the frame number of the fused feature map, For the t The state vector of the frame fusion feature map, and Respectively t The position parameters of the frame, and Respectively and Corresponding speed parameters;

[0062] The uniform motion model is used to represent the state update of the moving target. The state update formula of the moving target is as follows:

[0063] z t + 1 = A z t + w t A = [ 1 0 Δ t 0 0 1 0 Δ t 0 0 1 0 0 0 0 1 ] ,

[0064] Where A is the state transfer matrix, is the time step, is the process noise, To update to t +1 frame's state vector;

[0065] The control instructions are formulated according to the target state to track the moving target, including:

[0066] Acquire the center of the field of view, acquire the control angle according to the center of the field of view and the states of the moving target before and after the update, and output a control instruction for tracking the moving target according to the control angle;

[0067] The control angle is obtained by the following formula:

[0068] ,

[0069] e t = [ e x t e y t ] = [ x c − x t y c − y t ] ,

[0070] e t + 1 = [ e x t + 1 e y t + 1 ] = [ x c − x t + 1 y c − y t + 1 ] ,

[0071] In the formula, and are the control angle, , , They are proportional gain, integral gain and differential gain respectively, and the coordinates of the center of the field of view are , is the control error of the tth frame, and for The amount in For the t +1 frame control error, and for The amount in and for The weight,i It is the frame count sequence of the fused feature map.

[0072] In some embodiments of the first aspect, acquiring an image of a moving target for behavior recognition includes:

[0073] The method of using the fused feature map to track the moving target and obtain the image of the moving target for behavior recognition includes:

[0074] Output the gimbal camera tracking instructions based on the fused feature map;

[0075] Acquire the PTZ image and new event stream of the moving target, and fuse the PTZ image and the new event stream into a comprehensive image;

[0076] A first a priori frame is set, wherein the aspect ratio of the first a priori frame is fixed to 1:1; a second a priori frame and a third a priori frame are set, wherein the aspect ratios of the second a priori frame and the third a priori frame are reciprocal of each other;

[0077] Using the first a priori frame, the second a priori frame and the third a priori frame to locate the behavior of the moving target in the integrated image;

[0078] Perform behavior recognition on moving targets within the prior frame.

[0079] In a second aspect, the present disclosure further provides an electronic terminal, comprising a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method for tracking the behavior of a moving target provided in any embodiment of the first aspect are performed.

[0080] In a third aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method for tracking the behavior of a moving target provided in any one of the embodiments of the first aspect are implemented.

[0081] In a fourth aspect, the present disclosure further provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method for tracking the behavior of a moving target provided in any embodiment of the first aspect are implemented.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] The behavior tracking method of a moving target provided by the present invention captures changes in the on-site environment according to event streams and timestamps, and tracks the moving target so as to overcome environmental obstacles and identify the image of the moving target; the fusion of three-channel spatiotemporal events supplements the characteristics of time information and asynchronous update in the event stream; multiple feature planes are extracted in multiple stages so that the fused feature map has information of different depths; by storing events of different polarities in different event channels respectively, the situation of positive and negative polarity cancellation is avoided, thereby preventing information loss, so as to better use the fused feature map to track the moving target; a target detection model is used to obtain the position parameters and size parameters of the moving target from the fused feature map, and the target state is predicted based on this information to formulate control instructions. The method for tracking the moving target has the advantages of high precision, real-time performance, robustness, adaptability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0085] Figure 1 It is a flow chart of the method for tracking the behavior of a moving target provided in this embodiment. DETAILED DESCRIPTION

[0086] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0087] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0088] Embodiment 1

[0089] Figure 1 This is a flow chart of a method for tracking the behavior of a moving target in the first embodiment of the present invention. This flow chart only shows the logical sequence of the method described in this embodiment. In other possible embodiments of the present invention, different methods may be used without conflict. Figure 1 The steps shown or described are accomplished in the order shown.

[0090] The method for tracking the behavior of a moving target provided in this embodiment can be applied to a terminal and can be executed by a power system inspection device, which can be implemented by software and / or hardware, and can be integrated in a terminal, such as any smart phone, tablet computer or computer device with communication function. Figure 1 The method of this implementation specifically includes the following steps:

[0091] Acquire an event stream of a moving target, wherein the event stream includes a timestamp and polarity corresponding to the event, store events of different polarities in the event stream in a positive event channel and a negative event channel respectively, and store the timestamp in the time channel;

[0092] The positive event channel, the negative event channel and the time channel are fused as a three-channel spatiotemporal event;

[0093] Use the trained convolutional neural network to extract multiple feature planes from the three-channel spatiotemporal events in multiple stages;

[0094] The feature planes extracted at different stages are fused into the same feature map, and the target detection model is used to obtain the position parameters and size parameters of the moving target from the fused feature map. The target state of the moving target is predicted based on the position parameters and size parameters, and control instructions are formulated based on the target state to track the moving target.

[0095] Optionally, the event stream is acquired through an event camera, which is a new type of visual sensor that only detects brightness changes of a single pixel and outputs event frame data in an asynchronous manner, achieving target detection in high dynamic range (140dB) scenarios, high spatiotemporal resolution (μs level) and low power consumption.

[0096] The event camera will output an event stream only when it detects a moving target. The event stream contains multiple asynchronously output events. These events contain the timestamp, pixel coordinates and polarity of each event. The polarity indicates whether the brightness of the pixel increases or decreases. Different events are divided into positive and negative events by polarity. However, the same event may sometimes trigger both positive and negative events.

[0097] The edges of moving targets are often where brightness changes are most obvious. More events are triggered near the edges of the target. The event stream is used to display the spatial location information of the events on the image coordinate plane and the outline of the target object.

[0098] In order to avoid information loss caused by cancellation due to triggering events of different polarities, i.e., positive and negative, in the same pixel, events of different polarities are stored in different event stream channels.

[0099] The event stream retains the spatial information of the edges of moving targets, but loses the rich temporal information and asynchronous update characteristics of events. Therefore, an event representation based on the time surface is added as a time channel in addition to the event stream channel.

[0100] Multi-stage feature extraction can gradually and deeply explore the inherent laws of the three-channel spatiotemporal event frame data and extract richer and deeper features. These features are crucial for subsequent behavior tracking and target object recognition, and can improve the accuracy and robustness of recognition.

[0101] The above mentioned acquisition of images of moving targets for behavior recognition includes tracking the moving target with a fused feature map and then using an RGB camera to image and recognize the moving target; or an event camera combined with an RGB camera to detect multi-source image information, and represent the action of the moving target through event streams and spatiotemporal events on the time plane.

[0102] The moving target behavior tracking method provided in this embodiment can be used to detect dangerous actions of substation workers.

[0103] It is not difficult for those skilled in the art to see that in the prior art, moving targets are often detected only through RGB cameras. To achieve high-efficiency and high-accuracy identification, good background light is required, and there should be no large-area occlusion, otherwise the moving target will be out of the detection range. However, in actual working conditions represented by substations, the environment is often dim, and the existing behavior tracking methods for moving targets have poor environmental adaptability. The behavior tracking method for moving targets provided in this embodiment continuously tracks the moving target so that the moving target is continuously within the detection range, which facilitates the behavior identification of the moving target, and in the moving target, the acquired image can also be combined with event stream and timestamp data to perform multi-source detection on the moving target. The acquisition of event streams and timestamps has low requirements for ambient light, and can be adapted to situations where the field of view is blocked.

[0104] The behavior tracking method of a moving target provided in this embodiment captures changes in the on-site environment according to event streams and timestamps, and tracks the moving target so as to overcome environmental obstacles and identify the image of the moving target; the fusion of three-channel spatiotemporal events supplements the characteristics of time information and asynchronous updates in the event stream; multiple feature planes are extracted in multiple stages so that the fused feature map has information of different depths; and by storing events of different polarities in different event channels, the cancellation of positive and negative polarities is avoided, thereby preventing information loss, so as to better use the fused feature map to track the moving target.

[0105] Embodiment 2

[0106] This embodiment provides a method for tracking the behavior of a moving target, which is improved on the basis of the first embodiment. For details not provided, please refer to the first embodiment.

[0107] Embodiment 1 mentions that events of different polarities in the event stream are stored in the positive event channel and the negative event channel respectively. Optionally, in a further but non-limiting embodiment of the present application, specifically,

[0108] Obtain definition rules for positive and negative events;

[0109] Classify the events of the event stream according to the defined rules, and store the events of different polarities in the positive event channel or the negative event channel according to the classification results;

[0110] Embodiment 1 mentions fusing the positive event channel, the negative event channel and the time channel as a three-channel spatiotemporal event, as one embodiment thereof, including:

[0111] The event stream feature plane output formula is used to output the events in the positive event channel and the negative event channel as different feature planes. The event stream feature plane output formula is as follows:

[0112] ;

[0113] ;

[0114] In the formula, The event flow characteristic plane formed by positive events, for The total number of positive events in the positive event channel at the position, The event flow characteristic plane formed by negative events, for The total number of negative events in the negative event channel at the position, and All are pixel coordinates.

[0115] Obtain a time surface feature plane according to the timestamp in the time channel;

[0116] The event stream feature plane composed of positive events, the event stream feature plane composed of negative events and the time plane feature plane are fused to represent the coded event frame data, and the coded event frame data is output as a three-channel spatiotemporal event.

[0117] As one of the embodiments, the fusion formula for obtaining the event frame data representing the encoding is as follows:

[0118] F event = [ f EC + ⊕ f TS ⊕ f EC − ] ;

[0119] In the formula, To represent the encoded event frame data, is a real number set, H and W are the height and width of the event frame data, 3 is the number of channels, is the time plane feature plane. In this embodiment, the time plane feature plane is composed of all time stamps. For splicing operation.

[0120] By fusing the event stream feature plane and the time plane feature plane, the contour information and movement direction of the moving target in motion can be better obtained.

[0121] In addition, this embodiment also provides a method for obtaining a time feature plane:

[0122] The time plane is calculated by using the exponential kernel function to obtain the characteristics of past events. The formula for obtaining the time plane feature plane is as follows:

[0123] ;

[0124] In the formula, is the timestamp of the latest event in the event stream, is the timestamp of the input event, is a constant decay rate parameter. The timestamp is obtained from the event stream, that is, the total time of the event stream segment. The constant decay rate parameter converts the pixel value of the obtained time surface from [ 0 , 1 ] Zoom to [ 0 , 255 ] As a time channel, is the output of the time plane, For time, and are pixel coordinates, is the spatial coordinate position The timestamp of the input event.

[0125] It is not difficult for those skilled in the art to see that the essence of calculating the time plane by using the exponential kernel function to obtain the characteristics of past events is to assign different weights to each time according to the timestamp of the event, assigning larger weights to newer events and smaller weights to older events. Therefore, the pixel value of the time plane is the motion history function at the corresponding coordinate position. Larger pixel values ​​correspond to newer target movements, and the difference in pixel values ​​can better reflect the traces of events triggered by the target when it moves, that is, the direction of the target movement.

[0126] As mentioned in the first embodiment, a trained convolutional neural network is used to extract multiple feature planes from a three-channel spatiotemporal event in multiple stages. Optionally, in a further but non-limiting embodiment of the present application,

[0127] The convolutional neural network is a ResNet-50 network. The ResNet-50 network consists of five stages, including an input stage and four feature extraction stages with different structures. The ResNet-50 network is used to extract the second stage feature plane, the third stage feature plane, and the fourth stage feature plane from the event frame data representing the code. Due to the differences in the convolutional structures, the extracted feature planes of the above three feature planes gradually deepen. The output formulas of the feature planes of the three stages are as follows:

[0128] ;

[0129] ;

[0130] ;

[0131] In the formula, is the characteristic plane of the second stage, It is the feature-level fusion operation in the ResNet-50 network. is the image frame data of the second stage, is the event frame data of the second stage, is the characteristic plane of the third stage, is the image frame data of the third stage, is the event frame data of the third stage, is the characteristic plane of the fourth stage, is the image frame data of the fourth stage, It is the event frame data of the fourth stage; Represented as a low-dimensional feature, and Represented as high-dimensional features, low-dimensional features have more detailed information and can improve classification accuracy, and high-dimensional features have stronger semantic information and can provide accurate target locations. Optionally, the number of input channels and the size of feature planes corresponding to FFCSA modules of different scales are different, namely 512 60 × 80, 1024 30 × 40, and 2048 15 × 20.

[0132] As one of the embodiments, the second stage is also called the initial stage, including the first convolutional layer conv1 and the first residual block group conv2_x, the third stage is the intermediate stage, including the third residual block group conv3_x and the fourth residual block group conv4_x, and the fourth stage is the depth stage, including the fifth residual block group conv5_x.

[0133] The module integrates channel and spatial attention mechanisms to integrate the features of event frame data with the features of 2D image data. This process strengthens the feature representation ability of the data, enabling the model to more accurately identify and understand the actions and behaviors of moving objects.

[0134] The event number channel and the time plane channel contain the spatial and temporal information of the event respectively. The channel attention can adaptively adjust the weight of the spatiotemporal channel to better integrate the features of the spatiotemporal channel. The spatial attention assigns different weights to different spatial positions to enhance the target area of ​​interest while weakening the irrelevant background area.

[0135] For any of the above-mentioned second-stage feature planes, third-stage feature planes, and fourth-stage feature planes, this embodiment also provides an output method:

[0136] The event frame data of this stage Input to the first spatial attention module, the event frame data is represented by feature maps, is a three-dimensional feature tensor, Indicates the number of channels, represents the height of the feature map, Indicates the width of the feature map;

[0137] The first spatial attention module uses an average pooling layer to aggregate the spatial information of event frame data and generate feature maps;

[0138] Input the feature map into a multilayer perceptron with one hidden layer to obtain the attention coefficient vector of the corresponding channel;

[0139] The adaptive enhancement or suppression channel features are obtained according to the attention coefficient vector of the corresponding channel; the formula for obtaining the adaptive enhancement or suppression channel features is as follows:

[0140] ;

[0141] ;

[0142] In the formula, is the attention coefficient vector corresponding to the channel, is the sigmoid activation function (also known as the logistic activation function or S-type function), A multilayer perceptron with hidden layers. is the average pooling operation, is the event frame data of this stage, is the image frame data of this stage, is element-wise multiplication, To adaptively enhance or suppress channel characteristics.

[0143] The feature plane of the corresponding stage is obtained according to the adaptively enhanced or suppressed channel features.

[0144] For the above-mentioned method of obtaining the feature plane of the corresponding stage according to the adaptive enhancement or suppression channel features, as one embodiment:

[0145] The adaptive enhanced or suppressed channel features are input into the second spatial attention module, which includes an average pooling layer and a maximum pooling layer. The second spatial attention module outputs two two-dimensional feature maps that pass through the average pooling layer and the maximum pooling layer respectively. and ;

[0146] Concatenate two two-dimensional feature maps and convolve the concatenated two-dimensional feature maps to generate an effective two-dimensional attention coefficient matrix;

[0147] According to the two-dimensional attention coefficient matrix and the adaptive enhancement or suppression channel features, the feature plane obtained by the spatial attention mechanism is obtained.

[0148] The feature plane obtained by the spatial attention mechanism is spliced ​​with the image frame data of the corresponding stage to obtain the feature plane of the corresponding stage. The formula for obtaining the feature plane of the corresponding stage is as follows:

[0149] M s = σ ( f 3 × 3 ( AvgPool ( U c ); MaxPool ( U c ))) = σ ( f 3 × 3 ([ F avg S ⊕ F max S ]) ;

[0150] ;

[0151] ;

[0152] In the formula, is a two-dimensional attention coefficient matrix, is the convolution kernel size The convolution operation, Maximize pooling operations, is the 2D feature map generated by the average pooling layer in the second spatial attention module, is the 2D feature map generated by the max pooling layer in the second spatial attention module, is the feature plane obtained through the spatial attention mechanism, and F is the feature plane of the corresponding stage.

[0153] The feature planes of multiple stages are fused into a fused feature map to track the moving target. Optionally, in a further but non-limiting embodiment of the present application, for the use of the target detection model mentioned in the first embodiment to obtain the position parameters and speed parameters of the moving target from the fused feature map, specifically,

[0154] The RetinaNet target detection model is used to obtain the position parameters and size parameters of the moving target from the fused feature map. The focal loss function is used to replace the original loss function of the RetinaNet detection model, which effectively solves the category imbalance problem and improves the model's learning ability for difficult samples.

[0155] The target state of the moving target is predicted based on the position parameter and the size parameter mentioned in the first embodiment. Specifically,

[0156] The position parameters of the fused feature map and the speed parameters corresponding to the position parameters are converted into a state vector. The conversion formula of the state vector is as follows:

[0157] z t = [ x t y t v x t v y t ] ,

[0158] In the formula, t is the frame number of the fused feature map, For the t The state vector of the frame fusion feature map, and Respectively t The position parameters of the frame, and Respectively and Corresponding speed parameters;

[0159] The uniform motion model is used to represent the state update of the moving target. The state update formula of the moving target is as follows:

[0160] z t + 1 = A z t + w t A = [ 1 0 Δ t 0 0 1 0 Δ t 0 0 1 0 0 0 0 1 ] ,

[0161] Where A is the state transfer matrix, is the time step, is the process noise, To update to t +1 frame's state vector;

[0162] The control instructions mentioned in the first embodiment are formulated according to the target state to track the moving target. Specifically,

[0163] Obtaining the center of the field of view, obtaining the control angle according to the center of the field of view and the state of the moving target before and after the update, and outputting a control instruction for tracking the moving target according to the control angle;

[0164] The formula for obtaining the control angle is as follows:

[0165] ,

[0166] e t = [ e x t e y t ] = [ x c − x t y c − y t ] ,

[0167] e t + 1 = [ e x t + 1 e y t + 1 ] = [ x c − x t + 1 y c − y t + 1 ] ,

[0168] In the formula, and are the control angle, , , They are proportional gain, integral gain and differential gain respectively, and the coordinates of the center of the field of view are , is the control error of the tth frame, and for The amount in For the t +1 frame control error, and for The amount in and for The weight, i It is the frame count sequence of the fused feature map.

[0169] When in use, the control angle can be given to the gimbal camera so that the moving target is always in the center of the gimbal camera's field of view.

[0170] Embodiment 1 mentions obtaining an image of a moving target for behavior recognition. In the current target detection based on a priori frame, the a priori frame is generally set manually, and the default aspect ratio parameter is {1:2, 1:1, 2:1}, and the default scale parameter is ,The quality of the prior box design will affect the training and convergence speed of the model.

[0171] Output the PTZ camera tracking instruction according to the fused feature map, so that the PTZ camera tracks the moving target and creates recognition conditions. The PTZ camera is different from the event camera that obtains the event stream. The PTZ camera obtains conventional images. The PTZ camera can be an RGB camera or an infrared camera. The PTZ image is a grayscale image or a multi-color channel image.

[0172] Obtain the PTZ image and new event stream of the moving target, and fuse the PTZ image and the new event stream into a comprehensive image; the comprehensive image combines the PTZ image and the event stream, and the two data types complement each other;

[0173] Optionally, in a further but non-limiting embodiment of the present application, when identifying a moving target based on an image output by a gimbal camera, a differential evolution search algorithm is used to optimize the scale and aspect ratio of the prior frame to maximize the overlapping area between the real frame and the prior frame, further improving the tracking accuracy during the identification process, specifically including:

[0174] Set the first a priori frame, and the aspect ratio of the first a priori frame is fixed to 1:1; set the second a priori frame and the third a priori frame, and the aspect ratios of the second a priori frame and the third a priori frame are reciprocal to each other (that is, if one scale is 1:3, the other scale is 3:1);

[0175] Using the first prior frame, the second prior frame and the third prior frame to locate and classify the behavior of the moving target in the comprehensive image;

[0176] Perform behavior recognition on moving targets within the prior frame.

[0177] The three aspect ratios of the prior boxes correspond to three scale parameters, which respectively represent the ratio of the prior box to the original image.

[0178] The behavior tracking method of the moving target mentioned in the first embodiment can be applied to the detection of dangerous actions of the substation workers, including not wearing a helmet correctly, crossing the line / breaking into the line, not wearing long-sleeved work clothes, etc. Specifically, the detection confidence threshold is set , when the confidence level of the dangerous situation ( The system reports the detection results to the upper computer and keeps tracking the dangerous behavior.

[0179] Optionally, as one of the application scenarios of the behavior tracking method of moving targets, the event camera and the RGB camera can be used in synergy. The video pan-tilt is selected as a carrier, and the event camera and the RGB camera are installed on the video pan-tilt. The event camera detects the on-site environment. When a dangerous behavior is detected, the pan-tilt moves and continuously tracks the target of the dangerous action. After determining that there is a dangerous behavior, an alarm message is sent to the host computer, and the position and trajectory of the dangerous action are monitored in real time. In addition, the captured RGB image is preprocessed, including image correction, denoising, alignment and normalization. The feature-level fusion module based on the channel and spatial attention mechanism is combined to perform feature fusion on the image frame and event frame data. The verification and decision module confirms the result and uploads it to the host computer. At the same time, the event camera and the RGB camera run synchronously to collect event frame data and two-dimensional image data in the substation in real time. The event camera is responsible for capturing environmental changes in the substation, such as light changes, moving objects, etc., and its advantage lies in the high temporal resolution of dynamic scenes. The RGB camera provides high-quality color images and can capture detailed visual information of the scene. This step ensures that the two data types complement each other and provides rich raw information for subsequent image data fusion.

[0180] Embodiment 3

[0181] This embodiment provides an electronic terminal, including a processor and a memory connected to the processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the steps of the method for tracking the behavior of a moving target provided in Embodiment 1 or 2 are performed.

[0182] Embodiment 4

[0183] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method for tracking the behavior of a moving target provided in the first or second embodiment are implemented.

[0184] Embodiment 5

[0185] This embodiment also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for tracking the behavior of a moving target provided in Embodiment 1 or 2.

[0186] The electronic terminal, computer-readable storage medium, and computer program product provided in the above embodiments have the same technical effects as those in Embodiment 1 or 2, and will not be described in detail here.

[0187] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0188] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0192] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0193] For those skilled in the art, the specific meanings of the above terms in the present invention can be understood through specific circumstances.

[0194] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for tracking the behavior of a moving target, characterized in that: Acquire an event stream of a moving target, wherein the event stream includes a timestamp and polarity corresponding to the event, store events of different polarities in the event stream in a positive event channel and a negative event channel respectively, and store the timestamp in the time channel; The positive event channel, the negative event channel and the time channel are fused as a three-channel spatiotemporal event; Use the trained convolutional neural network to extract multiple feature planes from the three-channel spatiotemporal events in multiple stages; The feature planes extracted at different stages are fused into the same feature map. The target detection model is used to obtain the position parameters and size parameters of the moving target from the fused feature map. The target state of the moving target is predicted based on the position parameters and size parameters. The control instructions are formulated based on the target state to track the moving target. The events of different polarities in the event stream are stored in the positive event channel and the negative event channel respectively. include, Obtain definition rules for positive and negative events; Classify the events of the event stream according to the defined rules, and store the events of different polarities in the positive event channel or the negative event channel according to the classification results; The fusion of the positive event channel, the negative event channel and the time channel as a three-channel spatiotemporal event includes: The event stream feature plane output formula is used to output the events in the positive event channel and the negative event channel as different feature planes. The event stream feature plane output formula is as follows: ； ； In the formula, The event flow characteristic plane formed by positive events, is the total number of positive events in the positive event channel, The event flow characteristic plane formed by negative events, is the total number of negative events in the negative event channel; Obtain a time surface feature plane according to the timestamp in the time channel; The event stream feature plane composed of the positive events, the event stream feature plane composed of the negative events and the time plane feature plane are integrated into event frame data representing the coding, and the event frame data representing the coding is output as a three-channel spatiotemporal event; The fusion formula for obtaining the event frame data representing the encoding is as follows: ； In the formula, To represent the encoded event frame data, is the time feature plane, For splicing operation; The formula for obtaining the time plane feature plane is as follows: ； In the formula, is the timestamp of the latest event in the event stream, is the timestamp of the input event, is the constant decay rate parameter.

2. The method for tracking the behavior of a moving target according to claim 1, characterized in that: The method uses a trained convolutional neural network to extract multiple feature planes from three-channel spatiotemporal events in multiple stages, including: The convolutional neural network is a ResNet-50 network, which includes five stages. The ResNet-50 network is used to extract the second stage feature plane, the third stage feature plane and the fourth stage feature plane from the event frame data representing the encoding. The feature plane output formulas of the three stages are as follows: ； ； ； In the formula, is the characteristic plane of the second stage, It is the feature-level fusion operation in the ResNet-50 network. is the image frame data of the second stage, is the event frame data of the second stage, is the characteristic plane of the third stage, is the image frame data of the third stage, is the event frame data of the third stage, is the characteristic plane of the fourth stage, is the image frame data of the fourth stage, It is the event frame data of the fourth stage.

3. The method for tracking the behavior of a moving target according to claim 2, characterized in that: Output method of any one of the second stage feature plane, the third stage feature plane and the fourth stage feature plane: Input the event frame data of this stage into the first spatial attention module; The first spatial attention module uses an average pooling layer to aggregate the spatial information of event frame data and generate feature maps; Inputting the feature map into a multilayer perceptron having one hidden layer to obtain an attention coefficient vector of the corresponding channel; The adaptive enhancement or suppression channel feature is obtained according to the attention coefficient vector of the corresponding channel; the adaptive enhancement or suppression channel feature acquisition formula is as follows: ； ； In the formula, is the attention coefficient vector corresponding to the channel, is the sigmoid activation function, A multilayer perceptron with hidden layers. is the average pooling operation, is the event frame data of this stage, is the image frame data of this stage, is element-wise multiplication, To adaptively enhance or suppress channel characteristics; The feature plane of the corresponding stage is obtained according to the adaptively enhanced or suppressed channel features.

4. The method for tracking the behavior of a moving target according to claim 3, characterized in that: The step of obtaining the feature plane of the corresponding stage according to the adaptive enhancement or suppression channel features includes: Inputting the adaptively enhanced or suppressed channel features into a second spatial attention module, the second spatial attention module comprising an average pooling layer and a maximum pooling layer, and the second spatial attention module outputs two two-dimensional feature maps respectively passing through the average pooling layer and the maximum pooling layer; Concatenate two two-dimensional feature maps and convolve the concatenated two-dimensional feature maps to generate an effective two-dimensional attention coefficient matrix; According to the two-dimensional attention coefficient matrix and the adaptive enhancement or suppression channel features, the feature plane obtained by the spatial attention mechanism is obtained. The feature plane obtained by the spatial attention mechanism is spliced ​​with the image frame data of the corresponding stage to obtain the feature plane of the corresponding stage. The formula for obtaining the feature plane of the corresponding stage is as follows: ； ； ； In the formula, is a two-dimensional attention coefficient matrix, is the convolution kernel size The convolution operation, Maximize pooling operations, is the 2D feature map generated by the average pooling layer in the second spatial attention module, is the 2D feature map generated by the max pooling layer in the second spatial attention module, is the feature plane obtained through the spatial attention mechanism, and F is the feature plane of the corresponding stage.

5. The method for tracking the behavior of a moving target according to any one of claims 1 to 4, characterized in that: The target detection model is used to obtain the position parameters and speed parameters of the moving target from the fusion feature map, including: A RetinaNet target detection model is used to obtain position parameters and size parameters of a moving target from the fused feature map, wherein a focal loss function is used to replace an original loss function of the RetinaNet detection model; The method of predicting the target state of the moving target according to the position parameter and the size parameter includes: The position parameters of the fused feature map and the speed parameters corresponding to the position parameters are converted into a state vector. The conversion formula of the state vector is as follows: ， Where t is the frame number of the fused feature map, is the state vector of the fused feature map of the tth frame, and are the position parameters of the tth frame, and Respectively and Corresponding speed parameters; The uniform motion model is used to represent the state update of the moving target. The state update formula of the moving target is as follows: ， Where A is the state transfer matrix, is the time step, is the process noise, is the state vector updated to the t+1 frame; The control instructions are formulated according to the target state to track the moving target, including: Acquire the center of the field of view, acquire the control angle according to the center of the field of view and the states of the moving target before and after the update, and output a control instruction for tracking the moving target according to the control angle; The control angle is obtained by the following formula: ， ， ， In the formula, and are the control angle, , , They are proportional gain, integral gain and differential gain respectively, and the coordinates of the center of the field of view are , is the control error of the tth frame, and for The amount in is the control error of the t+1th frame, and for The amount in and for , i is the frame count number of the fused feature map.

6. An electronic terminal, characterized in that: It comprises a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method for tracking the behavior of a moving target as claimed in any one of claims 1 to 5 are executed.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for tracking the behavior of a moving target described in any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method for tracking the behavior of a moving target described in any one of claims 1 to 5 are implemented.

Images