A method for recognizing a moving target based on bidirectional cross attention

Abstract

The application provides a motion target recognition method based on bidirectional cross attention, comprising the following steps: collecting a video frame sequence containing a background and a motion target, generating a background frame aligned with a current frame according to a historical background sample; extracting features of the current frame and the background frame, taking the features of the current frame as a query vector, taking the features of the background frame as a key vector and a value vector, calculating a first cross attention between the current frame and the background frame, and generating a motion feature of the current frame according to the first cross attention; taking the features of the background frame as a query vector, taking the features of the current frame as a key vector and a value vector, calculating a second cross attention between the background frame and the current frame, and suppressing environmental interference in the motion feature according to the second cross attention to obtain a motion enhanced feature; and inputting the motion enhanced feature into a neural network to obtain a recognition result of the motion target.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of moving object recognition, and particularly relates to a moving object recognition method based on bidirectional cross attention. BACKGROUND

[0002] Moving object recognition refers to detecting and classifying moving objects in streaming videos. As moving objects usually carry key semantic information, effectively distinguishing them from static objects or backgrounds has become a crucial and rapidly developing field in computer vision research. This technology is closely related to multiple application scenarios of intelligent video analysis, including intrusion detection, video monitoring anomaly detection, and event-based video understanding behavior recognition.

[0003] At present, the field of computer vision has carried out in-depth research on moving object recognition (MOR) related tasks. Moving object segmentation is a task that can be regarded as a pixel-level binary classification problem: foreground represents moving objects, and background corresponds to static scenes. Typical moving object segmentation (MOS) methods include temporal frame difference method, optical flow method, and background subtraction method. In addition, object detection is also a related task, the core of which is to locate and classify specified category objects. MOR, as a combination of MOS and object detection, aims to realize joint detection and classification of moving objects in video sequences. The MOR methods proposed in recent years can be divided into two categories: segmentation-based detection methods and spatiotemporal object detection methods. Segmentation-based detection methods usually obtain foreground masks through MOS first, and then input the masks into the object detection network to obtain category and bounding box information; spatiotemporal object detection methods usually input multiple historical images into a deep learning network, estimate background and motion information, and perform object regression and classification on the motion saliency feature map.

[0004] Although the existing moving object segmentation method can effectively capture the motion information, the result usually lacks category semantics, limiting the practical application value. The target detection method cannot distinguish between moving and stationary targets due to the lack of temporal information. In some scenarios, detecting all targets frame by frame will produce redundant results, reducing the efficiency of manual decision-making. For the MOR method, the segmentation-based detection method has the problem that the moving object is often partially segmented or split into multiple regions, resulting in a decrease in the recall rate of the detection network, and the mask range of non-target objects in complex scenes may be much larger than the real target, directly inputting the foreground mask will significantly reduce the detection performance. Although some researchers have proposed a connected region detection algorithm to alleviate the target splitting problem, the problem of missegmentation caused by the lack of semantics has not been solved. The spatiotemporal target detection method has the problem that the sliding window frame input method limits the network's ability to model continuous time series. If a background area is continuously obscured by a slowly moving target within the sliding window, the network will not be able to effectively reconstruct the background, and increasing the input frame number to improve the background estimation accuracy will increase the computational burden. SUMMARY

[0005] The present application provides a moving target recognition method based on bidirectional cross-attention, which solves the problem of inaccurate positioning of moving targets in existing moving target recognition methods, resulting in poor detection performance.

[0006] To solve the above technical problems, the present application provides a moving target recognition method based on bidirectional cross-attention, comprising the following steps:

[0007] Step S1: Collecting a video frame sequence containing background and moving targets, generating a background frame aligned with the current frame according to historical background samples;

[0008] Step S2: Extracting the features of the current frame and the background frame, taking the features of the current frame as the query vector, the features of the background frame as the key vector and the value vector, calculating the first cross-attention between the current frame and the background frame, and generating the motion features of the current frame according to the first cross-attention; taking the features of the background frame as the query vector, the features of the current frame as the key vector and the value vector, calculating the second cross-attention between the background frame and the current frame, and suppressing the environmental interference in the motion features according to the second cross-attention to obtain the motion enhanced features;

[0009] Step S3: Inputting the motion enhanced features into a neural network to obtain the recognition result of the moving target.

[0010] Preferably, step S1 comprises the following steps:

[0011] Step S11: obtaining a current frame and N historical background samples of each pixel point in the current frame, and constructing a background model B(x)={B1(x), B2(x),..., B i N i (x) is the i-th background sample in B(x), B i (x)={c i (x), w i (x), a i (x)}; c i (x), w i (x) and a i (x) are color intensity value, weight and activation flag of B i (x), respectively;

[0012] Step S12: calculating the weight sum W valid (x) of the activated background sample for each pixel point x in the current frame:

[0013]

[0014]

[0015] R dynamic,i (x) = R scale ·d min (x i ) ± R inc / dec ;

[0016] In the above formula, dist(c(x), c i (x)) is the Euclidean distance between c(x) and c i (x); c(x) is the pixel intensity value of the current frame; R dynamic，i (x) is the distance threshold value of the i-th background sample; d min (x i ) is the background complexity measure; R scale is the proportion factor; R inc / dec is the increment / decrement of the distance threshold value;

[0017] If W valid (x) is greater than the set weight threshold value, x is classified as background, otherwise x is marked as foreground;

[0018] Step S13: calculating the mean value of all background samples to generate a background image Background(x):

[0019]

[0020] Preferably, the expression of the first cross attention in step S2 is:​​

[0021]

[0022] wherein Attn t2b is the first cross-attention; Softmax denotes an activation function; Q t2b is the query vector of the first cross-attention; K b , V b are the key vector and the value vector of the first cross-attention.

[0023] Preferably, the expression of the motion feature in step S2 is:

[0024] M motion = Sigmoid(Conv 1×1 ([Attn t2b , F self ]) ) ⊙ F self ;

[0025]

[0026]

[0027] wherein M motion is the motion feature; Sigmoid, Softmax denote an activation function; Conv 1×1 denotes a 1x1 convolution operation; Attn t2b is the first cross-attention; F self is the residual output; denotes element-wise multiplication; LayerNorm denotes a layer normalization operation; F t is the feature of the current frame; is the self-attention transformation matrix; Attn self is the self-attention of the current frame; Q t , K t , V t are the query vector, the key vector and the value vector of the current frame, respectively.

[0028] Preferably, the expression of the second cross-attention in step S2 is:

[0029]

[0030] wherein Attn b2t is the second cross-attention; Softmax denotes an activation function; Q b2t is the query vector of the second cross-attention; K t , V t are the key vector and the value vector of the current frame, respectively.

[0031] Preferably, the expression of the motion enhancement feature is:

[0032] F out = Conv 3×3 ([Attn b2t , F trans ])+F t ;

[0033] F trans = W2·GELU(W1·M motion );

[0034] In the formula, F out is the motion enhancement feature; Conv 3×3 represents a 3x3 convolution operation; Attn b2t is the second cross-attention; F trans is the motion feature after nonlinear feature transformation; F t is the feature of the current frame; W1 and W2 are feature transformation matrices respectively; GELU represents an activation function; and M motion is the motion feature.

[0035] Preferably, step S3 comprises the following steps:

[0036] Step S31: constructing a neural network comprising a heat map branch and a bounding box regression branch;

[0037] Step S32: performing horizontal and vertical average pooling operations on the motion enhancement feature to generate spatial feature encoding, calculating attention weights of the heat map branch and the bounding box regression branch according to the spatial feature encoding, and generating attention maps for the heat map branch and the bounding box regression branch according to the attention weights respectively;

[0038] Step S33: the heat map branch outputs the category of the motion target and the position of the center of the motion target, and the bounding box regression branch outputs the offset of the position of the center of the motion target;

[0039] Step S34: aligning the position of the center of the motion target according to the offset to obtain the recognition result of the motion target.

[0040] Preferably, after constructing the neural network in step S31, a heat map loss and a bounding box regression loss are designed respectively, the neural network is trained according to the heat map loss and the bounding box regression loss, and the parameters of the neural network are updated through back propagation.

[0041] Preferably, the expression of the heat map loss is:

[0042]

[0043] In the formula, Lheat is the focal loss; N represents the number of labeled boxes; a, b are the hyperparameters of the focal loss; represents the predicted confidence at position (x, y); represents the predicted confidence of class c at position (x, y); M xy represents the true response value at position (x, y).

[0044] Preferably, the expression of the bounding box regression loss is:

[0045]

[0046]

[0047] In the above formula, L reg is the bounding box regression loss; N represents the number of valid samples whose Gaussian kernel values exceed 0.3; GIoU() is the GIoU loss function; represents the decoded prediction box at pixel position (i, j); B n represents the true labeled box of the nth moving target; W ij is the regression area R n of the nth moving target; a n is the area of the bounding box of the nth moving target; G n (i, j) is the Gaussian kernel value at position (i, j); G n (x, y) is the Gaussian kernel value at position (x, y).

[0048] The beneficial effects of the present application at least include:

[0049] 1. By storing multiple historical background samples for each pixel point in the current frame to construct a background model, compared with the traditional single background model, the multi-sample storage method can better capture the dynamic changes and complexity of the background, making the background model more representative, thereby more accurately generating a background frame aligned with the current frame, providing a more reliable background reference for subsequent motion target detection.

[0050] 2. The cross-attention mechanism is used to extract the motion features of the current frame, the first cross-attention is calculated to generate motion features, which accurately focuses on the motion part different from the background in the current frame, and enhances the expression of the motion features; the second cross-attention is calculated to suppress the environmental interference in the motion features, further improving the robustness of the features. This two-way cross-attention calculation method not only ensures the effective extraction of the motion features, but also effectively reduces the influence of environmental interference factors, so that the motion enhanced features more accurately reflect the true information of the moving target. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 a method flowchart of an embodiment of the present application;

[0052] Figure 2 a framework diagram of motion target recognition based on background auxiliary heat map of an embodiment of the present application;

[0053] Figure 3 a framework structure diagram of background estimation based on bidirectional cross attention and motion target recognition based on coordinate attention in an embodiment of the present application;

[0054] Figure 4 an output diagram of a heat map branch of an embodiment of the present application;

[0055] Figure 5 a diagram of superimposing a heat map on an original image in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0057] The most widely used algorithm in the field of motion target segmentation at present is the background subtraction algorithm. The core principle is to construct the static part in the scene into a background model, and identify the target in motion as foreground, thereby realizing effective separation of the foreground target and the background. Such as calculating the average value or median value of a plurality of consecutive frames to construct a background model. This kind of simplest background modeling algorithm can play a good effect in a controlled indoor environment. However, once in a complex outdoor environment, its performance is greatly discounted. In order to effectively overcome this difficult problem, such as Gaussian mixture model (GMM) and other probability methods emerge as the times require. This method represents the pixel distribution as a mixture of weighted Gaussian distribution, thereby being able to more flexibly capture the characteristics of complex background. Although the probability method can meet the needs of many practical application scenarios, it cannot be ignored that natural images do not necessarily strictly follow the law of Gaussian distribution.

[0058] The SACON background modeling method is another way, which is no longer obsessed with finding a probability density function that highly matches the actual pixel distribution, but instead maintains a fixed number of first-in-first-out cache, which stores the most recently observed background values to build the background model. Specifically, if the new observation value is highly matched with the majority of samples in the background model of the corresponding pixel position, it is determined as background. However, the ViBe algorithm points out that the first-in-first-out updating strategy has the disadvantage of limiting the time coverage of the background model, so its performance is not satisfactory when dealing with slow-moving targets. In order to improve and optimize the ViBe algorithm, the PBAS algorithm incorporates the idea of control system theory, which enables the background model to dynamically adjust the update threshold according to the complexity of the background, thereby enhancing the environmental adaptability. WeSamBE improves the background sample updating strategy of the ViBe algorithm by introducing a weight mechanism, that is, when updating the background model, the least efficient sample is replaced, rather than simply replacing the oldest or random sample. This weight mechanism greatly optimizes the efficiency of the sample-based background model, thereby improving the overall performance. Overall, these improvement measures are to break through the limitations of traditional background modeling techniques and further enhance the accurate representation and self-adaptability of complex background environments.

[0059] Target detection algorithms can be mainly divided into two categories: one-stage and two-stage methods. One-stage target detection methods can directly predict the bounding box and class probability of each target instance. YOLO series, RetinaNet, and keypoint-based detection networks belong to this category. Two-stage target detection methods achieve target detection through two steps: the first step is to generate a set of candidate regions (Region Proposals), which are candidate positions in the image that may contain targets; the second step is to classify each candidate region to determine whether it is a target or background, and to fine-tune the coordinates of the bounding box. In the two-stage detection method, typical representatives include Fast RCNN, Faster RCNN, Mask R-CNN, and the two-stage version of CenterNet. In recent years, models based on Transformer, especially visual Transformer, such as DETR and deformable DETR, have made remarkable progress in target detection. These models have the advantage of utilizing long-range dependencies in images, thereby improving the overall performance of the detection system. The success of DETR has inspired researchers' enthusiasm for exploring end-to-end target detection networks, aiming to eliminate the non-differentiable post-processing step of Non-Maximum Suppression (NMS) in traditional detectors. Experimental results show that these methods have shown full competitiveness in the most advanced detectors using NMS, and have effectively demonstrated the potential advantages of end-to-end detection processes.

[0060] By combining the advantages of the above methods, the embodiment of the present application provides a motion target recognition method based on bidirectional cross attention as shown in Figure 1 The motion target recognition method based on bidirectional cross attention includes the following steps:

[0061] Step S1: Collect a sequence of video frames containing background and motion targets, and generate a background frame aligned with the current frame according to historical background samples.

[0062] Specifically, N historical background samples are stored for each pixel point x in the current frame, a background model of the current frame is constructed, the foreground and background in the background model are distinguished, and a background frame aligned with the current frame is generated.

[0063] To solve the problems of inaccurate motion target region segmentation and excessive dependence on sliding window length in extracting motion information in existing motion target recognition methods, the embodiment of the present application proposes a motion target recognition method based on bidirectional cross attention as shown in Figure 2The new framework is shown. The framework first adopts an improved background updating strategy, breaks through the time limit of the traditional first-in first-out (FIFO) sliding window through a historical background cache; secondly, the background samples and the current frame are jointly input into a key point-based detection network, so that the network autonomously determines the target position and category.

[0064] As Figure 3 shown, the target recognition model of the embodiment of the application includes two main modules: a background estimation module based on bidirectional cross attention and a moving target recognition module based on coordinate attention. The model aims to efficiently detect and analyze moving targets in a given video sequence.

[0065] The dynamic background estimation based on bidirectional cross attention is realized through a background modeling technique based on the current frame and the background frame. This process can accurately estimate the dynamic background of the current frame. After generating the dynamic background, the dynamic background and the current frame are jointly input into a neural network, and the goal of the network is to locate and classify the moving target.

[0066] The motion feature extraction based on bidirectional cross attention is based on the information extraction of the current frame and the background frame. First, the background frame is obtained by the following method: a background model is constructed by maintaining the N most recently observed background values of each pixel point, and the background model in the embodiment of the application is defined as:

[0067] B(x)={B1(x),B2(x),...,B i (x),...,B N (x)};

[0068] In the formula, B(x) is the background model of the pixel point x in the current frame; B i (x) is the i-th background sample in B(x).

[0069] Each background sample contains three parts:

[0070] B i (x)={c i (x),w i (x),a i (x)},i=1,2,...,N;

[0071] In the formula, c i (x), w i (x), a i (x) are the color intensity value, weight and activation flag of B i (x), respectively.

[0072] The activation flag is determined by the following formula:

[0073]

[0074] where dist(c(x), c i (x)) is the Euclidean distance between c(x) and c i (x); c(x) is the pixel intensity value of the current frame; R dynamic，i (x) is the distance threshold value of the i-th background sample.

[0075] In the embodiment of the present application, the final distance is calculated by the sum of absolute differences of three channels, and the distance threshold value is dynamically adjusted according to the scene complexity:

[0076] R dynamic,i (x) = R scale · d min (x i ) ± R inc / dec ;

[0077] where d min (x i ) is the background complexity measure; R scale is the proportion factor; and R inc / dec is the increment / decrement of the distance threshold value.

[0078] The foreground and background are distinguished by F(x):

[0079]

[0080]

[0081] In the above formula, W valid (x) is the sum of weights of the activated background samples; and #min is a preset threshold value.

[0082] When F(x) = 0, the pixel point x is determined as background. After the classification of the foreground and background is completed, each pixel point classified as background is updated with a probability of 1 / T, where T is a time sub-sampling factor used to control the background model update frequency. The update method can cover a wider time range while processing a limited sample amount.

[0083] For the sample B(x) in the background model B i (x) that needs to be updated, a reward-punishment strategy and a minimum weight update mechanism are adopted. The weight w i (x) of the activated sample in the background model will be increased by a certain value, and the weight of the non-activated sample will be decreased by another value, so as to maintain the weight balance of the background model as a whole. When the background model is updated, the new background sample will replace the sample with the lowest weight in the background model.

[0084] The background samples in the same spatial position can constitute a pseudo frame, and the value of each pixel point in the pseudo frame comes from different time points. Compared with the multiple frames obtained by the sliding window, the pseudo frames have a wider time coverage range and can replace the continuous frames obtained by the sliding window for background estimation. In the embodiment of the application, the three-channel background Background(x) is finally obtained by averaging the sample dimension:

[0085]

[0086] In summary, the weight-based background update in the embodiment of the application includes the following steps:

[0087] Step S11: obtaining a current frame I t , and constructing a background model B t-1 (x).

[0088] Step S12: calculating W t (x) for each pixel point x in I valid , if W valid (x) >= #min, then x is classified as background, and B t-1 (x) is updated by the minimum weight replacement strategy, otherwise x is marked as foreground. t

[0089] Step S13: calculating the sample mean to generate a background image Background(x).

[0090] Step S2: extracting the features of the current frame and the background frame, taking the features of the current frame as the query vector, the features of the background frame as the key vector and the value vector, calculating the first cross-attention between the current frame and the background frame, and generating the motion feature of the current frame according to the first cross-attention; taking the features of the background frame as the query vector, the features of the current frame as the key vector and the value vector, calculating the second cross-attention between the background frame and the current frame, and suppressing the environmental interference in the motion feature according to the second cross-attention to obtain the motion-enhanced feature.

[0091] After obtaining the background image corresponding to the current frame, the improved deep aggregation network DLA-34 is used as the backbone network of the motion recognition model to extract the features of the motion target. The improved DLA-34 adds a cross-layer skip connection between the shallow layer and the deep layer, so that it can better capture the target detail information and context features in multi-scale feature learning. However, this model is easily disturbed by dynamic background, and has problems such as feature space aliasing and high missing rate. In the embodiment of the application, the enhanced current frame and its corresponding background frame are used as input: the two frames of images are spliced along the channel dimension to form a 6-channel tensor, and the backbone features are extracted from the tensor by using the backbone network.

[0092] ​After the backbone feature extraction is completed, unlike the traditional feature pyramid FPN structure, the embodiment of the application does not perform classification and regression of multi-scale targets on each level feature map, but directly uses a single layer feature map with a down-sampling rate of 4 to perform full-scale target detection, which avoids the common scale assignment problem in the FPN-based method. The prediction mechanism based on the single layer feature map has higher prediction efficiency while ensuring the same prediction effect of the prediction accuracy.

[0093] Meanwhile, the embodiment of the application proposes a bidirectional cross-attention mechanism to realize robust extraction of motion features through four-stage operations.

[0094] Step S21: calculating the self-attention Attn of the current frame self :

[0095]

[0096] In the formula, Softmax represents an activation function; Q t , K t , V t are respectively query, key and value vectors of the current frame; d is the dimension of the key vector; R represents a real number; C is the number of target categories; H and W are the size of the input image.

[0097] According to the self-attention Attn of the current frame self , the residual output F self is calculated:

[0098]

[0099] In the formula, LayerNorm represents a layer normalization operation; F t is the backbone feature; is a self-attention transformation matrix.

[0100] This step strengthens the internal structure representation of the moving target and provides a high-discriminative query vector Query for the subsequent steps.

[0101] Step S22: calculating the cross-attention of the current frame to the background frame.

[0102] The residual output F self of the current frame is taken as the query vector Q t2b , and the feature of the background frame is taken as the key vector and the value vector K b , V b , and the cross-attention Attn of the current frame to the background frame is calculated: t2b :

[0103]

[0104] The motion region is located by highlighting the moving target inconsistent with the background in the current frame, and a motion feature M is generated motion :

[0105] M motion = Sigmoid(Conv 1×1 ([Attn t2b , F sef ])) * F self ;

[0106] wherein Sigmoid represents an activation function; Conv 1×1 represents a 1*1 convolution operation; and * represents element-wise multiplication.

[0107] Step S23: performing nonlinear feature transformation on the motion feature, compressing redundant information, and enhancing the expression ability of the motion feature.

[0108] F trans = W2*GELU(W1*M motion ) e R C×H×W ;

[0109] wherein F trans is the motion feature after nonlinear feature transformation; W1 and W2 are feature transformation matrices; and GELU represents an activation function.

[0110] Step S24: calculating cross-attention from the background frame to the current frame.

[0111] The background frame is taken as a query vector Q b2t , and the feature of the current frame is taken as a key vector and a value vector K t , V t , and cross-attention Attn b2t from the background frame to the current frame is calculated:

[0112]

[0113] The dynamic interference is suppressed by using background information, the discrimination between the moving target and the background is improved, and a motion enhanced feature F out is calculated:

[0114] F out = Cov 3×3 ([Attn b2t , F trans ]) + F t ;

[0115] wherein Conv 3×3 represents a 3*3 convolution operation.

[0116] The embodiment of the application realizes the cooperative optimization of motion region positioning and environment constraint injection through the motion feature extraction based on the bidirectional cross attention.

[0117] Step S3: constructing a motion target recognition module, inputting the motion enhanced feature into the motion target recognition module to obtain a recognition result of the motion target.

[0118] Specifically, the motion target recognition module based on the coordinate attention in the embodiment of the application includes two branches: a heat map branch and a bounding box regression branch. The heat map branch is responsible for target classification and positioning, and its dimension is CxH / 4xW / 4. In the embodiment of the application, the motion target is divided into two specific categories: pedestrians and vehicles. In the heat map, the center position of the motion target is assigned a response value of 1, which decays outward along a two-dimensional Gaussian function, and the static target and the background region are assigned a response value of 0, indicating that they are classified as non-motion targets. The expression of the two-dimensional Gaussian function is:

[0119]

[0120] In the formula, x and y represent the coordinates of the center of the bounding box in the original image; and represents the coordinates of the center of the bounding box after being mapped to the scale of the feature map; δ c represents the standard deviation related to the target scale.

[0121] Figure 4 The visualization result of the heat map is shown in (a) and (b) of FIG. 6, Figure 4 (a) of FIG. 6 corresponds to the heat map of the car category, Figure 4 (b) of FIG. 6 corresponds to the heat map of the pedestrian category. Figure 5 The superimposition effect of the heat map and the original image is shown in (a) and (b) of FIG. 7, Figure 5 (a) of FIG. 7 is the superimposition effect of the car category, Figure 5 (b) of FIG. 7 is the superimposition effect of the pedestrian category. Only moving objects will produce corresponding Gaussian kernel responses. Only the motion target has a corresponding Gaussian kernel response in the heat map, and the response value of the heat map also represents the confidence of the target category. In the inference stage, a 3x3 maximum pooling layer is used to implement non-maximum suppression on the heat map, and candidate regions with a response value lower than 0.3 are removed.

[0122] The improved focal loss function L heat The heat map branch is trained as follows:

[0123]

[0124] In the formula, N1 represents the number of bounding boxes; and α and β are hyperparameters of the focal loss, and in the embodiment of the application, α=2 and β=4 are set; is the predicted heat map, represents the predicted confidence at the position (x, y). represents the predicted confidence of class c at location (x, y); M is the real heat map; M xy represents the real response value at location (x, y).

[0125] The bounding box regression branch is responsible for target size regression. Embodiments of the present application perform size regression on the location of the Gaussian kernel in the heat map, and only regress the location where the response value is higher than 0.3. For each pixel location (i, j) participating in regression in the heat map, first multiply it by a down-sampling factor n to map it back to the network input size, and then multiply the 4-dimensional vector output by this location by a scaling factor to obtain the distance of this location to the four edges of the original image. The predicted bounding box of the final location (i, j) can be represented as:

[0126]

[0127] wherein, are the x coordinates of the upper left corner and the lower right corner of the bounding box, respectively; are the y coordinates of the upper left corner and the lower right corner of the bounding box, respectively; s is a scale scaling factor.

[0128] Embodiments of the present application introduce GIoU loss when calculating the bounding box regression loss. First, the Gaussian kernel value is used as a weight to balance the GIoU loss, and a weight based on the area of the target box is applied to normalize the bounding box regression loss. The expression of the weight based on the area of the target box is:

[0129]

[0130] wherein, W ij is the contribution weight of each sample in the regression area R n of the nth moving target to the loss; a n is the area of the bounding box of the nth moving target; G n (i, j) is the Gaussian kernel value at the (i, j) location; G n (x, y) is the Gaussian kernel value at the (x, y) location.

[0131] The expression of the bounding box regression loss L reg is:

[0132]

[0133] wherein, N2 represents the number of effective samples whose Gaussian kernel value is higher than 0.3; represents the decoded predicted bounding box at pixel location (i, j); B n represents the real labeled bounding box of the nth moving target; GIoU() is the GIoU loss function.

[0134] To enhance the interaction between the heat map branch and the bounding box branch, and improve the ability to capture the relevant patterns and relationships of the input features, the embodiment of the present application introduces a coordinate attention mechanism on the feature map extracted by the backbone network. At the same time, in order to bridge the gap between the classification and positioning tasks, a task alignment module is designed. Compared with the decoupling head commonly used by mainstream target detection networks to handle classification and positioning tasks, the task alignment module of the embodiment of the present application realizes joint optimization of tasks through spatial feature encoding and task-specific attention.

[0135] Where the spatial feature encoding captures the spatial characteristics of the feature map through horizontal / vertical average pooling operations. Specifically, the feature map output by the backbone network is subjected to horizontal and vertical average pooling to obtain horizontal feature encoding and vertical feature encoding . After transposition and splicing, the global coordinate feature map is obtained Where C, H, and W are the channel number, height, and width of the feature map, respectively.

[0136] Task-specific attention refers to generating exclusive attention maps for the heat map and bounding box branches. The heat map and bounding box regression branches apply coordinate attention based on their respective task requirements on X coordinate , respectively. The coordinate attention calculation process is as follows:

[0137] First, the spatial information in the vertical and horizontal directions is encoded through a nonlinear function, i.e.

[0138] f = δ (F1 ([X h , X w ])) ;

[0139] In the formula, f is a nonlinear function; F1 represents a 1x1 convolution function; [, ] represents a splicing operation; δ represents a nonlinear activation function.

[0140] Separate f to obtain f h and f w , and then convert them to the channel number of the feature Figure 1 through a 1x1 convolution function, i.e.

[0141] g h = σ (F h (f h )) ;

[0142] g w = σ (F w (f w )).

[0143] In the formula, g h and g w are the attention weight maps in the height direction and the width direction, respectively; Fh , F w are activation functions in the height and width directions respectively; f h , f w are 1x1 convolution functions in the height and width directions respectively.

[0144] By multiplying the generated attention weight with the original feature map X backbone , the task-aware attention feature map X attention is obtained. In particular, a residual connection is introduced in the heat map branch, and the prediction result is based on the element-wise summation of X attention and X backbone . This strategy effectively promotes task-aligned learning and avoids performance loss caused by independent operation of the double branches.

[0145] The embodiment of the present application proposes a new MOR framework, a cross-attention background auxiliary heat map learning framework, which can effectively estimate the motion information of the current frame. First, a dynamic background matching the current frame and the background frame is generated through background modeling technology, and a scene-adaptive dynamic background model is constructed to establish a feature interaction channel between the background estimation result and the real-time observation data, thereby effectively enhancing the discrimination between the moving target and the background interference area in a complex scene.

[0146] A key point-based detector network is designed to extract robust motion features from complex scenes by taking the current frame and the pre-generated background as input.

[0147] A double-task cooperative target positioning network is designed to synchronously learn the target existence probability distribution and the spatial geometric parameters through a unified network. The heat map is used to encode the spatial likelihood distribution of the target center, and the offset compensation mechanism is combined to realize pixel-level positioning. The lightweight regression branch is used to directly output the target boundary parameters. The developed network directly predicts the center heat map and the boundary box of each moving target without post-processing such as non-maximum suppression (NMS).

[0148] The method of the embodiment of the present application achieves a breakthrough result on the CDNET-MOR dataset: the average precision mean (mAP50) reaches 87.1%, which refreshes the current optimal performance record.

[0149] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, and only the preferred embodiments of the present application are expressed, which are more specific and detailed. However, it should not be understood as a limitation on the scope of the present application. As long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present application.

[0150] It should be noted that, for the person of ordinary skill in the art, several modifications and improvements can be made without departing from the inventive concept, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for motion target recognition based on bidirectional cross attention, characterized in that, The method comprises the following steps: Step S1: collecting a video frame sequence containing a background and a moving target, and generating a background frame aligned with a current frame according to a historical background sample; Step S2: extracting features of the current frame and the background frame, taking the features of the current frame as a query vector, the features of the background frame as a key vector and a value vector, calculating a first cross-attention between the current frame and the background frame, generating a moving feature of the current frame according to the first cross-attention, taking the features of the background frame as a query vector, the features of the current frame as a key vector and a value vector, calculating a second cross-attention between the background frame and the current frame, suppressing environmental interference in the moving feature according to the second cross-attention, and obtaining a moving enhanced feature; Step S3: inputting the moving enhanced feature into a neural network to obtain a recognition result of the moving target.

2. The method of claim 1, wherein the method is a bidirectional cross-attention based moving object recognition method. Step S1 comprises the following steps: Step S11: obtaining a current frame and N historical background samples of each pixel point in the current frame, and constructing a background model B(x) = {B1(x), B2(x),..., B i (x),..., B N (x)}. i (x) is the i-th background sample in B(x), B i (x) = {c i (x), w i (x), a i (x)}. i (x), w i (x), a i (x) are color intensity value, weight and activation flag of B i (x), respectively. Step S12: Calculate the sum of weights w of activated background samples for each pixel point x in the current frame valid (x): R dynamic,i (x) = R scale · d min (x i ) ± R inc / dec ; In the above equation, dist(c(x), c i (x)) is the Euclidean distance between c(x) and c i (x); c(x) is the pixel intensity value of the current frame; R dynamic，i (x) is the distance threshold value of the i-th background sample; d min (x i ) is the background complexity measure; R scale is the scale factor; R inc / dec is the increment / decrement of the distance threshold value; If W valid (x) is greater than a set weight threshold, then x is classified as background, otherwise x is marked as foreground. Step S13: calculating a mean value of all background samples to generate a background image Background(x):

3. The method of claim 1, wherein the method is a bidirectional cross-attention based moving object recognition method. The expression of the first cross-attention in step S2 is: where Attn t2b is the first cross attention; Softmax denotes an activation function; Q t2b is the query vector of the first cross attention; K b , V b are the key vector and the value vector of the first cross attention.

4. The method of claim 1, wherein the method is a bidirectional cross-attention based moving object recognition method. The expression of the moving feature in step S2 is: M motion = Sigmoid(Conv 1×1 ([Attn t2b , F self ])) * F self ; In the formula, M motion is a motion feature; Sigmoid, Softmax represent activation functions; Conv 1×1 represents 1x1 convolution operation; Attn t2b is the first cross attention; F self is the residual output; represents element-wise multiplication; LayerNorm represents layer normalization operation; F t is the feature of the current frame; is the self-attention transformation matrix; Attn self is the self-attention of the current frame; Q t , K t , V t are the query vector, key vector and value vector of the current frame, respectively.

5. The method of claim 1, wherein the method is a bidirectional cross-attention based moving object recognition method. The expression of the second cross-attention in step S2 is: where Attn b2t is the second cross attention; Softmax denotes an activation function; Q b2t is the query vector of the second cross attention; K t , V t are the key vector and the value vector of the current frame, respectively.

6. The method of claim 1, wherein the method is a bidirectional cross-attention based moving object recognition method. The expression of the moving enhanced feature is: F out = Conv 3×3 ([Attn b2t , F trans ])+F t ; F trans = W2 GELU(W1 M motion ); where F out is the motion enhanced feature; Conv 3×3 denotes a 3x3 convolution operation; Attn b2t is the second cross attention; F trans is the motion feature after nonlinear feature transformation; F t is the feature of the current frame; W1, W2 are feature transformation matrices, respectively; GELU denotes an activation function; M motion is a motion feature.

7. The method of claim 1, wherein the method is a bidirectional cross-attention based moving object recognition method. Step S3 comprises the following steps: Step S31: constructing a neural network comprising a heat map branch and a bounding box regression branch; Step S32: performing horizontal and vertical average pooling operations on the moving enhanced feature to generate a spatial feature code, calculating attention weights of the heat map branch and the bounding box regression branch according to the spatial feature code, and generating an attention map for the heat map branch and the bounding box regression branch according to the attention weights; Step S33: the heat map branch outputs a class of the moving target and a position of a center of the moving target, and the bounding box regression branch outputs an offset of the position of the center of the moving target; Step S34: aligning the position of the center of the moving target according to the offset to obtain a recognition result of the moving target.

8. The method of claim 7, wherein the method is a bidirectional cross-attention based moving object recognition method. After constructing the neural network in step S31, a heat map loss and a bounding box regression loss are designed respectively, the neural network is trained according to the heat map loss and the bounding box regression loss, and parameters of the neural network are updated through back propagation.

9. The method of claim 8, wherein the method is a bidirectional cross-attention based moving object recognition method. The expression of the heat map loss is: In the formula, L heat is the heatmap loss; N represents the number of labeled boxes; and a and b are hyperparameters of the focal loss; represents the predicted confidence at the position (x, y); represents the predicted confidence of the class c at the position (x, y); and M xy represents the true response value at the position (x, y).

10. The method of claim 8, wherein the method is a bidirectional cross-attention based moving object recognition method. The expression of the bounding box regression loss is: In the above formula, L reg is the bounding box regression loss; N represents the number of effective samples whose Gaussian kernel values exceed 0.3; GIoU() is a GIoU loss function; represents the decoded prediction frame at pixel position (i,j); B n represents the true label frame of the nth moving target; W ij Regression area R for the nth moving target n Contribution weight of each sample to loss; a n Area of the bounding box for the nth moving target; G n Gaussian kernel value at (i, j) position; G n Gaussian kernel value at (x, y) position.

Images