Adaptive scene surveillance video target detection method and system based on feature fusion

Abstract

The present invention discloses an adaptive scene monitoring video target detection method and system based on feature fusion, which solves the problem that the temporal context information is not fully integrated and the detection speed is slow in the video target detection in the airport scene monitoring scene. The technical scheme is as follows: Step 1: Calculate the optical flow motion field and temporal feature consistency discriminant matrix Qk2i of the current frame; Step 2: Determine whether the current frame is a key frame according to the temporal adaptive key frame dynamic scheduling strategy combined with the discriminant matrix Qk2i; Step 3: If the current frame is a key frame, input the current frame into the feature extraction network to obtain a feature map, and then input the feature maps of the current frame and the previous key frame into the adaptive weight network for weighted fusion to obtain the final feature map of the current frame; Step 4: If the current frame is not a key frame, a spatial adaptive local feature update method is used, and features are extracted for some areas through a convolutional network, and other areas use the optical flow motion field and the feature map of the previous key frame to calculate the current area features, and then the two parts of the area features are fused to obtain the feature map of the current frame; Step 5: Input the feature map of the current frame into the classification and positioning network to obtain the detection result. The present invention can improve the accuracy and detection speed of airport scene surveillance video target detection methods for airport aircraft and vehicle position data in complex scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of video target detection in computer vision, and more specifically, relates to an adaptive scene surveillance video target detection method and system based on feature fusion in an airport environment, which can identify and detect airport aircraft and vehicle targets. Background Art

[0002] Object detection is a very important research topic in the field of computer vision. Object detection models can not only be applied to various security monitoring systems, autonomous driving systems, drones and other fields, but also have a wide range of commercial uses, such as face recognition, license plate recognition, medical image analysis, etc. With the continuous development of deep learning technology, more and more excellent object detection algorithms have been proposed, which makes object detection more accurate, faster and more efficient in processing large amounts of data.

[0003] Deep learning is an artificial intelligence technology that achieves machine learning by simulating the structure and function of the human brain neural network. In terms of image target detection, deep learning has many advantages, such as high accuracy, fast speed, and the ability to process large amounts of data. Therefore, deep learning has been widely used in image target detection. However, in terms of video target detection, it faces more complex difficulties and challenges. First, the amount of video data is huge and it is very time-consuming to process, which requires the algorithm to have higher speed and efficiency. Secondly, the targets in the video may move, be blocked, deform, and so on, which will bring difficulties to target detection. In addition, video target detection requires real-time performance, which places higher requirements on the speed and efficiency of the algorithm.

[0004] Before the emergence of deep learning, traditional target detection methods can generally be divided into three parts: region selection (sliding window), feature extraction (SIFT, HOG, etc.) and classifier (SVM, Adaboost, etc.). For example, the Viola-Jones detector uses a sliding window to check whether the target exists in the window. Its main problems are two-fold: on the one hand, the sliding window selection strategy is not targeted, has high time complexity, and has window redundancy; on the other hand, the manually designed features have poor robustness. In the era of deep learning, target detection can be divided into two categories: "two-stage detection" and "one-stage detection". The former frames detection as a "coarse to fine" process, while the latter is "one-step". For example, RCNN is a typical two-stage target detection algorithm based on deep learning. It first selects possible object boxes from a set of object candidate boxes through a selective search algorithm, then resizes the images in these selected object boxes to images of a fixed size and feeds them into the CNN model to extract features. Finally, the extracted features are sent to the classifier to predict whether there is a target to be detected in the image of the object box, and further predict which category the detected target belongs to.

[0005] However, these video object detection methods often adopt the same processing method for all frames, or select fixed frames as key frames for object detection, resulting in extremely slow detection and recognition speed. At the same time, these methods often ignore the timing information between different frames in the video, which is crucial for improving detection accuracy and speed. Therefore, how to accurately capture the timing information between different contexts in the video and adopt different processing strategies is worthy of in-depth study, which can make an important contribution to improving the accuracy and speed of video object detection. Summary of the invention

[0006] The purpose of the present invention is to address the problems existing in the above-mentioned prior art and to provide an adaptive scene surveillance video target detection method and system based on feature fusion in an airport environment, which can achieve a balance between detection speed and detection accuracy while fully integrating the features of the temporal context.

[0007] The technical solution to achieve the purpose of the present invention is: an adaptive scene surveillance video target detection method and system based on feature fusion, the method comprising the following steps:

[0008] Step 1: Determine a video stream including a target to be detected, wherein the video stream includes a multi-frame image sequence, and the image includes the target to be detected;

[0009] Step 2: Use the ResNet network as the feature extraction network Nfeat, use the RFCN network as the classification and positioning network Ntask, design a convolutional neural network as the weight network Nw, and design an optical flow network FlowNet based on the convolutional neural network;

[0010] Step 3: If the current frame is the first frame of the video stream, the current frame is selected as a key frame, and the feature extraction network Nfeat is used to extract the features of the current frame image, and then its feature map is directly input into the classification and positioning network Ntask for classification and positioning to obtain the target detection result;

[0011] Step 4: If the current frame is not the first frame of the video stream, the optical flow motion field and feature temporal consistency discrimination matrix Qk2i of the current frame and the previous key frame are calculated according to the time-series adaptive key frame dynamic scheduling strategy to determine whether the current frame is a key frame or a non-key frame;

[0012] Step 5: If the current frame is a key frame, the feature extraction network Nfeat is used to extract the features of the current frame image, and the fused feature map of the feature map of the current frame and the previous key frame is calculated after being aggregated by the weight network Nw, and then classified and located to obtain the target detection result;

[0013] Step 6: If the current frame is a non-key frame, a spatially adaptive local feature update method is used to calculate the feature map of the current frame based on the consistency discriminant matrix Qk2i obtained in step 4 and the feature map of the previous key frame, and then classification and positioning are performed to obtain the target detection result.

[0014] Further, step 1: determining a video stream including a target to be detected, wherein the video stream includes a sequence of multiple frames of images, and the images include the target to be detected;

[0015] Furthermore, in step 2, the ResNet network is used as the feature extraction network Nfeat, RFCN is used as the classification and positioning network Ntask, a convolutional neural network is designed as the weight network Nw, and an optical flow network FlowNet based on the convolutional neural network is designed, which specifically includes:

[0016] Step 2-1: Construct a ResNet-based feature extraction network Nfeat to calculate the feature map of the image. Here, the modified ResNet-101 model is used.

[0017] ResNet101 discards the last classification layer, changes the stride of the first block of conv5 to 1, and applies the holing algorithm to all 3x3 convolution kernels in conv5, which ensures that the receptive field of the model remains unchanged. Therefore, the overall stride of Nfeat is 16, that is, the output of Nfeat is 1 / 16 of the original image.

[0018] At the end of conv5, a 3x3 convolution layer needs to be added to reduce the feature channel dimension to 1024. The modified ResNet-101 model needs to be pre-trained according to the dataset. The image of the current key frame is input into the Nfeat network, and the feature map of the key frame is output.

[0019] Step 2-2: Construct a classification and positioning network Ntask based on RFCN to calculate the classification and positioning of the current frame. First, it is necessary to extract the feature map of the current frame image and then input it into the Ntask network for processing.

[0020] In the Ntask network, a series of convolution and pooling operations are first performed on the input feature map to extract the features. Then, with the help of some structures similar to the fully connected layer, the extracted features are reduced in dimension and adjusted, and finally the target classification and position information corresponding to the current frame are output. This position information includes the coordinates of the upper left corner of the target box and parameters such as the width and height of the box.

[0021] Step 2-3: Adopt the feature fusion method of adaptive weights. Construct a weight network based on convolutional neural network, which includes 3 convolution layers and 3 activation layers. Figure 1 The input network starts from the last pooling layer, and after the last pooling layer, it is processed by the cosine similarity function to obtain the importance weight. According to the weight, the weighted sum of the two feature maps is obtained to obtain the feature map of the new key frame;

[0022] Step 2-4: Construct an optical flow network FlowNet based on FlowNetCorr to calculate the optical flow motion field, reduce the number of convolution kernels in the FlowNetCorr network by half, and increase the running speed. Input the current frame and the previous key frame into FlowNet together, and output the optical flow motion field of the current frame. The optical flow motion field is an important concept in computer vision, which is used to describe the movement of objects in an image sequence. It represents the motion vector of each pixel in an image, that is, the moving direction and speed of each pixel in the image sequence. The calculation of the optical flow motion field is based on an assumption: between adjacent image frames, the pixels on the same object have the same motion. Therefore, by comparing the pixel values ​​in adjacent image frames, the motion vector of each pixel can be calculated, thereby obtaining the optical flow motion field.

[0023] Further, step 3: if the current frame is the first frame of the video stream, the current frame is selected as a key frame, the feature extraction network Nfeat is used to extract the features of the current frame image, and then its feature map is directly input into the classification and positioning network Ntask for classification and positioning to obtain the target detection result;

[0024] Furthermore, in step 4, if the current frame is not the first frame of the video stream, the optical flow motion field and the feature temporal consistency discrimination matrix Qk2 of the current frame and the previous key frame are calculated according to the timing adaptive key frame dynamic scheduling strategy to determine whether the current frame is a key frame or a non-key frame, specifically including:

[0025] Step 4-1: Input the current frame and the previous key frame into the optical flow network FlowNet described in step 2-4, and output the optical flow motion field between the current frame and the previous key frame;

[0026] Step 4-2: Based on the optical flow motion field obtained in step 4-1, calculate the feature temporal consistency discriminant matrix Qk2i. The specific calculation method is: set a threshold. If the position offset of each pixel point on the optical flow motion field exceeds this threshold, the corresponding element of the matrix Qk2i is set to 1, otherwise it is set to 0; if the offset exceeds the threshold, we believe that the motion amplitude of the object at this pixel point is large.

[0027] Step 4-3: Determine whether the current frame is a key frame or a non-key frame based on the feature temporal consistency discriminant matrix Qk2i; specifically: set a threshold, if the proportion of element values ​​in Qk2i that are 1 exceeds the threshold, the current frame is selected as a key frame, otherwise it is selected as a non-key frame. This is because when the proportion of 1 exceeds the threshold, the offset of more pixels in the image also exceeds the threshold; we believe that in the current image, the object has a large motion amplitude, so it needs to be selected as a key frame.

[0028] Furthermore, in step 5, if the current frame is a key frame, the feature extraction network Nfeat is used to extract the features of the current frame image, and the fused feature map of the feature map of the current frame and the previous key frame is calculated after being aggregated by the weight network Nw, and then classified and located to obtain the target detection result; specifically including:

[0029] Step 5-1: Input the current frame into the feature extraction network Nfeat described in step 2-1 to obtain a feature map;

[0030] Step 5-2: Combine the features of the current frame and the previous key frame Figure 1 The weight network Nw described in step 2-3 is input to obtain a fused feature map after the feature maps of the two frames are aggregated. The fused feature map represents the feature map of the current frame. Different adjacent key frames are iterated in this way to achieve the purpose of transmitting important information in the entire video.

[0031] Step 5-3: Input the fused feature map into the classification and positioning network Ntask described in step 2-2 for classification and positioning to obtain the target detection result.

[0032] Furthermore, in step 6, if the current frame is a non-key frame, a spatially adaptive local feature updating method is used to calculate the feature map of the current frame according to the consistency discriminant matrix Qk2i obtained in step 4 and the feature map of the previous key frame, and then classify and locate the target to obtain the target detection result; specifically including:

[0033] Step 6-1: According to the feature temporal consistency discriminant matrix Qk2i obtained in step 4-2, the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 0 are divided into area A, and the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 1 are divided into area B. Area A and area B together constitute the current frame;

[0034] Step 6-2: For area A of the current frame, the feature map of area A of the current frame is calculated by combining the optical flow motion field obtained in step 4-1 and the feature map of the previous key frame; because we believe that the pixel offset of area A is small, in order to achieve the purpose of detection speed, the feature map of area A of the current frame is predicted by using the feature map of the previous key frame and the optical flow motion field between them.

[0035] Step 6-3: For the B region of the current frame, use the feature extraction network to perform convolution on the local region B to extract features. Because we believe that the pixel offset in the B region is large, if we continue to use the method of step 6-2, it will have an adverse effect on the detection accuracy. Considering this, we use the convolution feature extraction method for the B region.

[0036] Step 6-4: Recombine the features of area A and area B of the current frame to obtain the feature map of the current frame, and then input it into the classification and positioning network Ntask for classification and positioning to obtain the detection result. The fused feature map of the current frame reduces the calculation speed while ensuring the accuracy of feature extraction. Compared with the simple global convolution feature extraction or the prediction calculation based on the optical flow motion field, this method has an overall performance improvement.

[0037] The present invention also discloses an adaptive scene surveillance video target detection system based on feature fusion, characterized in that the system comprises:

[0038] The feature map acquisition module uses the ResNet-based feature extraction network Nfeat as the backbone network to extract feature maps of key frames in the video;

[0039] The classification and positioning acquisition module uses the RFCN-based classification and positioning network Ntask to calculate the classification and positioning of the image content according to the feature map of each frame in the video;

[0040] The feature fusion module constructs a weighted network based on a convolutional neural network. The feature map of the current frame is combined with the feature map of the previous key frame. Figure 1 The input network starts from the last pooling layer, and after the processing, it is processed by the cosine similarity function, and finally the importance weight is obtained. According to the weight, the two feature maps are weighted fused to obtain the feature map of the new key frame;

[0041] The optical flow motion field calculation module constructs an optical flow network FlowNet based on a convolutional neural network to calculate the optical flow motion field. The current frame and the previous key frame are input into FlowNet together, and the optical flow motion field of the current frame is output;

[0042] The key frame selection module adopts an adaptive decision-making method to calculate the temporal consistency discriminant matrix Qk2i of the temporal features of the current frame, and then adaptively determines whether to select the current frame as a key frame based on the matrix.

[0043] The feature map calculation module is used to calculate the feature map of the current frame. For key frames, the feature map is calculated using the method of step 5; for non-key frames, the feature map is calculated using the method of step 6.

[0044] According to an embodiment of the adaptive scene surveillance video target detection system based on feature fusion of the present invention, the feature map acquisition module adopts the ResNet-based feature extraction network Nfeat as the backbone network to extract the feature map of the key frame in the video, wherein ResNet101 discards the last classification layer, modifies the stride of the first block of conv5 to 1, and applies the holing algorithm to all 3x3 convolution kernels in conv5, so as to ensure that the receptive field of the model remains unchanged. Therefore, the overall step size of Nfeat is 16, that is, the output of Nfeat is 1 / 16 of the original image. At the end of conv5, a 3x3 convolution layer needs to be added to reduce the feature channel dimension to 1024. The modified ResNet-101 model needs to be pre-trained according to the data set first. The image of the current key frame is input into the Nfeat network, and the feature map of the key frame is output.

[0045] According to an embodiment of the adaptive scene surveillance video target detection system based on feature fusion of the present invention, the feature fusion module adopts an adaptive weight feature fusion method. A weight network based on a convolutional neural network is constructed, and the network includes 3 convolution layers and 3 activation layers. The feature map of the current frame is combined with the feature map of the previous key frame. Figure 1 The input network starts from the last pooling layer, and after the last pooling layer, it is processed by the cosine similarity function to obtain the importance weight. According to the weight, the weighted sum of the two feature maps is obtained to obtain the feature map of the new key frame;

[0046] According to an embodiment of the adaptive scene surveillance video target detection system based on feature fusion of the present invention, the key frame selection module is further configured to perform the following processing:

[0047] Input the current frame and the previous key frame into the optical flow network FlowNet described in step 2-4, and output the optical flow motion field between the current frame and the previous key frame;

[0048] According to the optical flow motion field obtained in step 4-1, the feature temporal consistency discriminant matrix Qk2i is calculated. The calculation method is as follows: a threshold is set. If the position offset of each pixel point on the optical flow motion field exceeds this threshold, the corresponding element of the matrix Qk2i is set to 1, otherwise it is set to 0;

[0049] According to the feature temporal consistency discriminant matrix Qk2i, it is determined whether the current frame is a key frame or a non-key frame.

[0050] According to an embodiment of the adaptive scene surveillance video target detection system based on feature fusion of the present invention, the feature map calculation module is further configured to perform the following processing:

[0051] If the current frame is a key frame, the current frame and the previous key frame are respectively input into the feature extraction network Nfeat described in step 2-1 to obtain feature maps corresponding to the two frames;

[0052] The features of the two frames Figure 1 Input the weight network Nw to obtain a fused feature map after the feature maps of the two frames are aggregated;

[0053] The fused feature map is input into the classification and positioning network Ntask for classification and positioning to obtain the target detection result.

[0054] If the current frame is a non-key frame, according to the discriminant matrix Qk2i, the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 0 are divided into area A, and the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 1 are divided into area B. Area A and area B together constitute the current frame;

[0055] For area A of the current frame, the feature map of area A of the current frame is calculated by combining the optical flow motion field obtained in step 4-1 and the feature map of the previous key frame;

[0056] For area B of the current frame, the feature extraction network is used to perform convolution on the local area B to extract features.

[0057] The features of area A and area B of the current frame are recombined to obtain the feature map of the current frame, which is then input into the classification and positioning network Ntask for classification and positioning to obtain the detection result.

[0058] Compared with the prior art, the present invention has the following beneficial effects: First, in the process of feature fusion, the adaptive scene surveillance video target detection method based on feature fusion of the present invention calculates the adaptive weights of the features of the previous key frame and the features of the current frame, and then fuses them, so that when the target is detected in the current frame, the semantic information of all previous key frames is combined. Second, in the process of key frame selection, the adaptive scene surveillance video target detection method based on feature fusion of the present invention adopts a temporal adaptive key frame dynamic scheduling strategy. In the polling process of each frame, it dynamically calculates and decides whether the current frame is selected as a key frame, and has the adaptability to time series changes. Third, in the process of calculating the feature map of non-key frames, the adaptive scene surveillance video target detection method based on feature fusion of the present invention takes into account that non-key frames will have large local changes relative to key frames, and the feature calculation using optical flow propagation is prone to errors. Therefore, a spatial adaptive local feature update method is adopted to avoid this problem.

[0059] The above three points make this method different from other methods in video target detection, achieving an excellent balance between detection accuracy and speed, and obtaining stable detection results in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or characteristics may have the same or nearby reference numerals.

[0061] Figure 1 The overall framework schematic diagram of an embodiment of the adaptive scene surveillance video target detection method based on feature fusion of the present invention is shown.

[0062] Figure 2 Shows Figure 1 Schematic diagram of the weight network Nw and feature fusion method in the overall framework shown.

[0063] Figure 3 Shows Figure 1 The network structure of the optical flow network FlowNet in the overall framework is shown. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with the drawings and specific embodiments. Note that the aspects described below in conjunction with the drawings and specific embodiments are only exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0065] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0066] The purpose of the present invention is to address the problems existing in the above-mentioned prior art and to provide an adaptive scene surveillance video target detection method and system based on feature fusion in an airport environment, which can achieve a balance between detection speed and detection accuracy while fully integrating the features of the temporal context.

[0067] Figure 1 The overall framework of an embodiment of the adaptive scene surveillance video target detection method based on feature fusion of the present invention is shown. Figure 1 , the implementation steps of the method of this embodiment are described in detail as follows.

[0068] Step 1: Determine a video stream including a target to be detected, wherein the video stream includes a multi-frame image sequence, and the image includes the target to be detected;

[0069] In step 2, the ResNet network is used as the feature extraction network Nfeat, RFCN is used as the classification and positioning network Ntask, a convolutional neural network is designed as the weight network Nw, and an optical flow network FlowNet based on the convolutional neural network is designed, which specifically includes:

[0070] Step 2-1: Construct a ResNet-based feature extraction network Nfeat to calculate the feature map of the image. Here, the modified ResNet-101 model is used.

[0071] ResNet101 discards the last classification layer, changes the stride of the first block of conv5 to 1, and applies the holing algorithm to all 3x3 convolution kernels in conv5, which ensures that the receptive field of the model remains unchanged. Therefore, the overall stride of Nfeat is 16, that is, the output of Nfeat is 1 / 16 of the original image.

[0072] At the end of conv5, a 3x3 convolution layer needs to be added to reduce the feature channel dimension to 1024. The modified ResNet-101 model needs to be pre-trained according to the dataset. The image of the current key frame is input into the Nfeat network, and the feature map of the key frame is output.

[0073] Step 2-2: Construct a classification and positioning network Ntask based on RFCN to calculate the classification and positioning of the current frame. First, it is necessary to extract the feature map of the current frame image and then input it into the Ntask network for processing.

[0074] In the Ntask network, a series of convolution and pooling operations are first performed on the input feature map to extract the features. Then, with the help of some structures similar to the fully connected layer, the extracted features are reduced in dimension and adjusted, and finally the target classification and position information corresponding to the current frame are output. This position information includes the coordinates of the upper left corner of the target box and parameters such as the width and height of the box.

[0075] Step 2-3: Adopt the feature fusion method of adaptive weights. Construct a weight network based on convolutional neural network, which includes 3 convolution layers and 3 activation layers. Figure 1 The input network starts from the last pooling layer, and after the last pooling layer, it is processed by the cosine similarity function to obtain the importance weight. According to the weight, the weighted sum of the two feature maps is obtained to obtain the feature map of the new key frame;

[0076] Step 2-4: Construct an optical flow network FlowNet based on FlowNetCorr to calculate the optical flow motion field, reduce the number of convolution kernels in the FlowNetCorr network by half, and increase the running speed. Input the current frame and the previous key frame into FlowNet together, and output the optical flow motion field of the current frame. The optical flow motion field is an important concept in computer vision, which is used to describe the movement of objects in an image sequence. It represents the motion vector of each pixel in an image, that is, the moving direction and speed of each pixel in the image sequence. The calculation of the optical flow motion field is based on an assumption: between adjacent image frames, the pixels on the same object have the same motion. Therefore, by comparing the pixel values ​​in adjacent image frames, the motion vector of each pixel can be calculated, thereby obtaining the optical flow motion field.

[0077] Step 3: If the current frame is the first frame of the video stream, the current frame is selected as a key frame, and the feature extraction network Nfeat is used to extract the features of the current frame image, and then its feature map is directly input into the classification and positioning network Ntask for classification and positioning to obtain the target detection result;

[0078] In step 4, if the current frame is not the first frame of the video stream, the optical flow motion field and feature temporal consistency discrimination matrix Qk2 of the current frame and the previous key frame are calculated according to the time-series adaptive key frame dynamic scheduling strategy to determine whether the current frame is a key frame or a non-key frame, specifically including:

[0079] Step 4-1: Input the current frame and the previous key frame into the optical flow network FlowNet described in step 2-4, and output the optical flow motion field between the current frame and the previous key frame;

[0080] Step 4-2: Based on the optical flow motion field obtained in step 4-1, calculate the feature temporal consistency discriminant matrix Qk2i. The specific calculation method is: set a threshold. If the position offset of each pixel point on the optical flow motion field exceeds this threshold, the corresponding element of the matrix Qk2i is set to 1, otherwise it is set to 0; if the offset exceeds the threshold, we believe that the motion amplitude of the object at this pixel point is large.

[0081] Step 4-3: Determine whether the current frame is a key frame or a non-key frame based on the feature temporal consistency discriminant matrix Qk2i; specifically: set a threshold, if the proportion of element values ​​in Qk2i that are 1 exceeds the threshold, the current frame is selected as a key frame, otherwise it is selected as a non-key frame. This is because when the proportion of 1 exceeds the threshold, the offset of more pixels in the image also exceeds the threshold; we believe that in the current image, the object has a large motion amplitude, so it needs to be selected as a key frame.

[0082] In step 5, if the current frame is a key frame, the feature extraction network Nfeat is used to extract the features of the current frame image, and the fusion feature map of the feature map of the current frame and the previous key frame is calculated after being aggregated by the weight network Nw, and then classified and located to obtain the target detection result; specifically including:

[0083] Step 5-1: Input the current frame into the feature extraction network Nfeat described in step 2-1 to obtain a feature map;

[0084] Step 5-2: Combine the features of the current frame and the previous key frame Figure 1 The weight network Nw described in step 2-3 is input to obtain a fused feature map after the feature maps of the two frames are aggregated. The fused feature map represents the feature map of the current frame. Different adjacent key frames are iterated in this way to achieve the purpose of transmitting important information in the entire video.

[0085] Step 5-3: Input the fused feature map into the classification and positioning network Ntask described in step 2-2 for classification and positioning to obtain the target detection result.

[0086] In step 6, if the current frame is a non-key frame, a spatial adaptive local feature update method is used to calculate the feature map of the current frame according to the consistency discriminant matrix Qk2i obtained in step 4 and the feature map of the previous key frame, and then classification and positioning are performed to obtain the target detection result; specifically including:

[0087] Step 6-1: According to the feature temporal consistency discriminant matrix Qk2i obtained in step 4-2, the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 0 are divided into area A, and the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 1 are divided into area B. Area A and area B together constitute the current frame;

[0088] Step 6-2: For area A of the current frame, the feature map of area A of the current frame is calculated by combining the optical flow motion field obtained in step 4-1 and the feature map of the previous key frame; because we believe that the pixel offset of area A is small, in order to achieve the purpose of detection speed, the feature map of area A of the current frame is predicted by using the feature map of the previous key frame and the optical flow motion field between them.

[0089] Step 6-3: For the B region of the current frame, use the feature extraction network to perform convolution on the local region B to extract features. Because we believe that the pixel offset in the B region is large, if we continue to use the method of step 6-2, it will have an adverse effect on the detection accuracy. Considering this, we use the convolution feature extraction method for the B region.

[0090] Step 6-4: Recombine the features of area A and area B of the current frame to obtain the feature map of the current frame, and then input it into the classification and positioning network Ntask for classification and positioning to obtain the detection result. The fused feature map of the current frame reduces the calculation speed while ensuring the accuracy of feature extraction. Compared with the simple global convolution feature extraction or the prediction calculation based on the optical flow motion field, this method has an overall performance improvement.

[0091] For the specific definition of the adaptive scene surveillance video target detection system based on feature fusion of the present invention, please refer to the definition of the adaptive scene surveillance video target detection system based on feature fusion of the present invention above, which will not be repeated here. Each module in the adaptive scene surveillance video target detection system based on feature fusion of the present invention can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0092] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An adaptive scene surveillance video target detection method based on feature fusion, characterized in that: The method comprises the following steps: Step 1: Determine a video stream including a target to be detected, wherein the video stream includes a multi-frame image sequence, and the image includes the target to be detected; Step 2: Use the ResNet network as the feature extraction network Nfeat, use the RFCN network as the classification and positioning network Ntask, design a convolutional neural network as the weight network Nw, and design an optical flow network FlowNet based on the convolutional neural network; Step 3: If the current frame is the first frame of the video stream, the current frame is selected as a key frame, and the feature extraction network Nfeat is used to extract the features of the current frame image, and then its feature map is directly input into the classification and positioning network Ntask for classification and positioning to obtain the target detection result; Step 4: If the current frame is not the first frame of the video stream, the optical flow motion field and feature temporal consistency discrimination matrix Qk2i of the current frame and the previous key frame are calculated according to the time-series adaptive key frame dynamic scheduling strategy to determine whether the current frame is a key frame or a non-key frame; Step 5: If the current frame is a key frame, the feature extraction network Nfeat is used to extract the features of the current frame image, and the fused feature map of the feature map of the current frame and the previous key frame is calculated after being aggregated by the weight network Nw, and then classified and located to obtain the target detection result; Step 6: If the current frame is a non-key frame, a spatially adaptive local feature update method is used to calculate the feature map of the current frame based on the consistency discriminant matrix Qk2i obtained in step 4 and the feature map of the previous key frame, and then classification and positioning are performed to obtain the target detection result.

2. The method for adaptive scene surveillance video target detection based on feature fusion according to claim 1 is characterized in that: In step 2, the ResNet network is used as the feature extraction network Nfeat, RFCN is used as the classification and positioning network Ntask, a convolutional neural network is designed as the weight network Nw, and an optical flow network FlowNet based on the convolutional neural network is designed, which specifically includes: Step 2-1: Construct a ResNet-based feature extraction network Nfeat to calculate the feature map of the image; here we use the modified ResNet-101 model, in which ResNet-101 discards the last classification layer, changes the stride of the first block of conv5 to 1, applies the holing algorithm to all 3x3 convolution kernels in conv5, and adds a 3x3 convolution layer after conv5 to reduce the feature channel dimension to 1024. Input the image of the current key frame into the Nfeat network, and output the feature map of the key frame; Step 2-2: Construct a classification and positioning network Ntask based on RFCN to calculate the classification and positioning of the current frame, input the feature map calculated by the current frame into the Ntask network, and output the classification and positioning of the current frame; Step 2-3: Adopt the feature fusion method of adaptive weights, construct a weight network based on convolutional neural network, input the feature map of the current frame and the feature map of the previous key frame into the network together, and after the last pooling layer is processed, the importance weight is obtained by processing through the cosine similarity function. According to the weight, the two feature maps are weightedly fused to obtain the feature map of the new key frame; Step 2-4: Construct an optical flow network FlowNet based on a convolutional neural network to calculate the optical flow motion field. Input the current frame and the previous key frame into FlowNet together, and output the optical flow motion field of the current frame.

3. The method for adaptive scene surveillance video target detection based on feature fusion according to claim 1 is characterized in that: In step 4, if the current frame is not the first frame of the video stream, the optical flow motion field and feature temporal consistency discrimination matrix Qk2i of the current frame and the previous key frame are calculated according to the time-series adaptive key frame dynamic scheduling strategy to determine whether the current frame is a key frame or a non-key frame, specifically including: Step 4-1: Input the current frame and the previous key frame into the optical flow network FlowNet described in step 2-4, and output the optical flow motion field between the current frame and the previous key frame; Step 4-2: According to the optical flow motion field obtained in step 4-1, calculate the feature temporal consistency discriminant matrix Qk2i. The calculation method is: set a threshold. If the position offset of each pixel point on the optical flow motion field exceeds this threshold, the corresponding element of the matrix Qk2i is set to 1, otherwise it is set to 0; Step 4-3: Determine whether the current frame is a key frame or a non-key frame based on the feature temporal consistency discrimination matrix Qk2i.

4. The method for adaptive scene surveillance video target detection based on feature fusion according to claim 1 is characterized in that: In step 5, if the current frame is a key frame, the feature extraction network Nfeat is used to extract the features of the current frame image, and the fusion feature map of the feature map of the current frame and the previous key frame is calculated after being aggregated by the weight network Nw, and then classified and located to obtain the target detection result; specifically including: Step 5-1: Input the current frame into the feature extraction network Nfeat described in step 2-1 to obtain a feature map; Step 5-2: Input the feature maps of the current frame and the previous key frame into the weight network Nw described in step 2-3 to obtain a fused feature map after the feature maps of the two frames are aggregated; Step 5-3: Input the fused feature map into the classification and positioning network Ntask described in step 2-2 for classification and positioning to obtain the target detection result.

5. The method for adaptive scene surveillance video target detection based on feature fusion according to claim 3 is characterized in that: In step 6, if the current frame is a non-key frame, a spatial adaptive local feature update method is used to calculate the feature map of the current frame according to the consistency discriminant matrix Qk2i obtained in step 4 and the feature map of the previous key frame, and then classification and positioning are performed to obtain the target detection result; specifically including: Step 6-1: According to the feature temporal consistency discriminant matrix Qk2i obtained in step 4-2, the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 0 are divided into area A, and the current frame pixel points corresponding to the area where the element value of the matrix Qk2i is 1 are divided into area B. Area A and area B together constitute the current frame; Step 6-2: For area A of the current frame, the feature map of area A of the current frame is calculated by combining the optical flow motion field obtained in step 4-1 and the feature map of the previous key frame; Step 6-3: For area B of the current frame, use the feature extraction network to perform convolution on the local area B to extract features; Step 6-4: Recombine the features of area A and area B of the current frame to obtain the feature map of the current frame, and then input it into the classification and positioning network Ntask for classification and positioning to obtain the detection result.

6. Adaptive scene surveillance video target detection system based on feature fusion, characterized in that: The adaptive scene surveillance video target detection system based on feature fusion is applied to the adaptive scene surveillance video target detection method based on feature fusion according to any one of claims 1 to 5, and the system comprises: The feature map acquisition module uses the ResNet-based feature extraction network Nfeat as the backbone network to extract feature maps of key frames in the video; The classification and positioning acquisition module uses the RFCN-based classification and positioning network Ntask to calculate the classification and positioning of the image content according to the feature map of each frame in the video; The feature fusion module constructs a weighted network based on a convolutional neural network, inputs the feature map of the current frame and the feature map of the previous key frame into the network, and after the last pooling layer is processed, it is processed by the cosine similarity function to obtain the importance weight. According to the weight, the two feature maps are weightedly fused to obtain the feature map of the new key frame. The optical flow motion field calculation module constructs an optical flow network FlowNet based on a convolutional neural network to calculate the optical flow motion field. The current frame and the previous key frame are input into FlowNet together, and the optical flow motion field of the current frame is output; The key frame selection module uses an adaptive decision-making method to calculate the temporal consistency discriminant matrix Qk2i of the temporal features of the current frame, and then adaptively determines whether to select the current frame as a key frame based on the matrix; The feature map calculation module is used to calculate the feature map of the current frame. For key frames, the feature map is calculated using the method of step 5; for non-key frames, the feature map is calculated using the method of step 6.

Images