A scene adaptive video tracking system and method for fusing images

Abstract

The application discloses a scene self-adaptive video tracking system and method of fused images, relates to the technical field of image processing, and respectively acquires the weight of image features of a subject object and a subordinate object in a space-time feature, performs feature enhancement on the collected original subject image features and subordinate image features through weighted calculation, forms a feature pair corresponding to a video frame by using the subject image features and the subordinate image features after feature enhancement, combines the frame feature variation law and the time sequence feature variation law of a plurality of memory feature groups, obtains the space-time variation features of the memory feature groups, acquires the image features of the subject object in a target frame and the features of the subject object in a previous frame of the target frame, calculates a tracking contribution factor of the image features of the subject object, calculates the contribution degree of the image features of the subordinate object to the feature variation of the target frame in the target frame, and strengthens the image features of the subject object in the target frame.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a scene adaptive video tracking system and method for fused images. BACKGROUND

[0002] In the field of video intelligent analysis, target feature extraction and dynamic tracking are core technical links of target object tracking, and the tracking performance of target objects directly affects the reliability of upper-layer applications such as scene perception, behavior analysis and decision prediction. Traditional video target tracking algorithms, such as correlation filtering, twin networks or optical flow method, usually run based on a single mode, that is, a fixed strategy is used to process all objects in a video stream, and the dynamic perception ability of the primary and secondary relationships of targets in a scene is lacking. This limitation is particularly prominent in complex dynamic scenes where the main object is disturbed by external interference. SUMMARY

[0003] The present application aims to provide a scene adaptive video tracking system and method for fused images to solve the problems in the prior art.

[0004] To achieve the above object, the present application provides the following technical scheme: a scene adaptive video tracking method for fused images, the method comprising:

[0005] Step S100: acquiring a video frame sequence, extracting static features in each video frame and continuous change features of a local range in the video frame, performing feature splicing on the static features and the continuous change features to obtain spatio-temporal features of the video frame;

[0006] Step S200: marking a main object and a subordinate object in a historical acquisition record of the video frame, respectively acquiring weights of image features of the main object and the subordinate object in the spatio-temporal features from the image features of the main object and the image features of the subordinate object in the video frame, performing feature enhancement on the original main image features and the subordinate image features acquired through weighted calculation, and forming a feature pair corresponding to the video frame by using the feature-enhanced main image features and the subordinate image features in each video frame.

[0007] Step S300: extracting image features and position information of the main object, image features and position information of the subordinate object in each video frame, and combining a memory feature group composed of the position change of the last frame with the memory feature group to obtain spatio-temporal change features of the memory feature group.

[0008] Step S400: Take a frame in the video sequence as the target frame, obtain the image features of the main object in the target frame and the features of the main object in the previous frame of the target frame, obtain the historical correlation between the image features and subordinate features of the main object, and calculate the tracking contribution factor of the image features of the main object.

[0009] Step S500: Calculate the contribution of the image features of subordinate objects to the feature changes in the target frame, collect the image features of the main objects in all video frames before the target frame, and enhance the image features of the main objects in the target frame.

[0010] Furthermore, step S100 includes:

[0011] Step S101: Obtain the historical acquisition sequence V of video frames, V = I1, I2, I3, ..., I T Among them, I1, I2, I3, ... and I T These represent the 1st, 2nd, 3rd, ..., and Tth video frames, respectively, and the main object in each video frame is identified.

[0012] Step S102: Take the t-th video frame I in the current video sequence t As the target frame, features are extracted from the target frame using a residual neural network to obtain ResNet(I). t ), where ResNet represents a residual neural network;

[0013] Step S103: Form the target frame association sequence L from the a1 video frames before the target frame to the a2 video frames after the target frame. t Temporal information of the target frame associated sequence is obtained through convolutional kernels (Conv3D(L)). t ), where Conv3D represents a three-dimensional convolutional layer;

[0014] Step S104: Convert ResNet(I) t ) and Conv3D(L t The spatiotemporal features F of the target frame are obtained by channel splicing. t .

[0015] Conv3D can utilize temporal information to extract spatiotemporal dynamic features, such as motion speed and acceleration information, while ResNet extracts static features in the single-frame space of video frames, such as texture and shape information. By fusing static and dynamic features, the model's understanding of the temporal context is improved, especially when dealing with occlusion or motion blur, which can help the model better infer the current position of the target.

[0016] Furthermore, step S200 includes:

[0017] Step S201: Obtain an image feature separation function GM of the subject object, the image feature separation function of the subject object comprising a convolution layer, an activation function and a normalization layer, wherein the image feature separation function of the subject object comprises three convolution layers of the same size, three different sizes of feature information of the image feature of the subject object are obtained respectively, a weight matrix of each convolution layer is set to respond to the feature information, the normalization layer comprises a batch normalization layer, and the activation layer comprises a ReLU activation layer.

[0018] Step S202: Obtain an image feature separation function GS of the subordinate object, the image feature separation function of the subordinate object comprising a convolution layer, an activation function and a normalization layer, wherein the image feature separation function of the subordinate object comprises three convolution layers, three different sizes of feature information of the image feature of the subject object are obtained respectively, according to the size of the feature information from large to small, a weight matrix of the corresponding size is used to respond to the feature information, a convolution result of a previous convolution layer is output through the normalization layer and the activation layer between two convolution layers, the normalization layer comprises an instance normalization layer, and the activation layer comprises a LeakyReLU activation layer.

[0019] The image feature separation function GM of the subject object and the image feature separation function GS of the subordinate object are different in construction, batch normalization layers, i.e., Batch Norm normalization layers, are used for recognition of the image feature of the subject object, the batch normalization layers are more likely to extract the commonality of the image feature of the subject object, and thus the feature of the subject object is obtained.

[0020] Instance normalization layers, i.e., Instance Norm normalization layers, are used for recognition of the subordinate object, the instance normalization layers are more likely to extract the stylized feature of the subordinate object, such as a light and shadow feature in a scene, and thus the feature of the subordinate object is obtained.

[0021] Step S203: Obtain a separation feature M of the image feature of the subject object raw t ,

[0022] M raw t = GM (F t ), obtain a separation feature S of the image feature of the subordinate object raw t S raw t = GS (F t );

[0023] Step S204: Obtain a weight A of the subject object in a target frame M (F t ),

[0024] where AvgPool(Ft) denotes average pooling of Ft, and MaxPool(Ft) denotes max pooling of Ft.

[0025] C M 1 and C M 2 are weight matrices, where C M 1 denotes a weight matrix of image features of the subject object in the feature frame, and C M 2 denotes a weight matrix of corresponding regions of the subject object on all regions of the feature frame;

[0026] Step S205: obtaining the weight A S (F t ) of the dependent object in the target frame,

[0027] C S 1 and C S 2 are weight matrices, where C S 1 denotes a weight matrix of image features of the subject object in the feature frame, and C S 2 denotes a weight matrix of corresponding regions of the subject object on all regions of the feature frame;

[0028] C M 1 is responsive to the subject object's own image, such as edge detection and shape detection, C M 2 is responsive to the region where the subject object is located, and the weight on the region in the target frame, such as setting the region where the subject object is located in the target frame as the central region, and concentrating the positive weights of the weight matrix C M 2 in the central region;

[0029] C S 1 learns the features of the dependent object's image, such as motion blur features and shadow features, C S 2 is responsive to the region where the dependent object is located, and disperses the positive weights of the weight matrix C S 2 in the edge region of the target frame except the central region

[0030] Step S206: calculating the spatially weighted image features of the subject object M t = M raw t ⊙ A M (F t ), and the spatially weighted image features of the dependent object S t = S raw t ⊙ A S (F t ), where denotes Hadamard product;

[0031] Step S207: The image features of the subject object and the image features of the subordinate object in the target frame are collected to form a feature pair (M t , S t ).

[0032] Further, step S300 includes:

[0033] Step S301: The target frame and all the previous k video frames of the target frame are merged into a target frame sequence FR, and the feature pairs of each video frame in the target frame sequence are obtained to form a frame feature sequence Y, Y = (M t-k , S t-k ), (M t-k+1 , S t-k+1 ), (M t-k+2 , S t-k+2 ), …, (M t , S t ), wherein (M t-k , S t-k ) represents the feature pair corresponding to the kth video frame before the target frame, and (M t , S t ) represents the feature pair corresponding to the target frame.

[0034] Step S302: The feature pair (M i , S i ) corresponding to the ith video frame in the target sequence is obtained, the position vector p i of the subject object in the ith video frame is obtained, and the position vector p i-1 of the subject object in the video frame corresponding to the i-1th feature pair is obtained, the displacement vector Δp i of the ith feature pair is calculated, Δp i = p i -p i-1 .

[0035] Step S303: The memory feature group m i corresponding to the feature pair (M i , S i ) corresponding to the ith video frame is calculated, m i = (ROIAlign(M i , p i ), Hist(S i ), Δp i ), wherein Hist(S i ) represents the distribution probability of all colors in S i , and ROIAlign represents a region of interest alignment method.

[0036] RoI Align, Region of Interest Align, is a feature alignment technology for object detection and instance segmentation tasks;

[0037] Step S304: Assemble the memory feature groups of the target frame and all the previous k video frames of the target frame to form a memory feature sequence ME, ME: (m t-k , m t-k+1 , m t-k+2 , …, m t ), wherein m t-k represents the memory feature group corresponding to the kth video frame before the target frame, m t represents the memory feature group corresponding to the target frame, and the global dependency feature h TCN t , h TCN t = Dilated-Conv (ME), Dilated-Conv represents a dilated convolution network;

[0038] Step S305: Obtain the time sequence feature of the memory feature group through the long short-term memory network, h LSTM t = LSTM (h t-1 , m t );

[0039] Step S306: Perform feature fusion on the global dependency feature and the time sequence feature through the normalization operation to obtain the spatio-temporal change feature H t , H t = LayerNorm (Y1·h TCN t + Y2·h LSTM t ), wherein LayerNorm represents the normalization operation, and Y1 and Y2 represent weight matrices, respectively.

[0040] Further, step S400 includes:

[0041] Step S401: Obtain the image feature M t of the spatially weighted main object and the image feature S t of the spatially weighted subordinate object in the target frame, and obtain the image feature M t-1 of the spatially weighted main object in the previous video frame of the target frame.

[0042] Step S402: Calculate the similarity of M t and M t-1 , denoted as sim t , and the similarity sim tNormalized to the (0,1) interval, the occlusion confidence O of the main object in the target frame is obtained. t When sim t When O approaches 0 t tending towards 1, when sim t When O approaches 1 t It tends towards 0;

[0043] Step S403: Establish the correlation vector gt between the spatially weighted image features of the target frame and the spatially weighted image features of the subject object in the previous frame, and use it as the input vector of the gated recurrent unit network. t =[AvgPool(M t )⊕MaxPool(S t )⊕h t-1 ⊕O t ], where ⊕ represents the concatenation operation, AvgPool(M t ) indicates that for M t Perform average pooling, MaxPool(S) t ) indicates max pooling on St;

[0044] Step S404: Obtain the gated recurrent unit network model and establish the update gate z. t , z t =σ(W z ·g t +b z ), where W z This indicates updating the gate weight matrix, b z The update gate bias term is labeled, σ represents the Sigmoid activation function, and a reset gate r is constructed. t r t =σ(W r ·g t +b r ), where W r Denotes the reset gate weight matrix, b r This represents the reset gate bias term, σ represents the Sigmoid activation function, and the candidate hidden state. Among them W h U represents the candidate state weight matrix. h Let denot be the candidate state input weight matrix, tanh be the tanh activation function, and h be the hidden state output gate. t , Where ⊙ represents the Hadamard product operation;

[0045] Step S405: Calculate the tracking contribution factor β t ,β t =σ(W β ·ht +b β ), wherein W β represents a contribution factor weight matrix, b β represents a contribution factor output bias term.

[0046] In order to improve the similarity calculation efficiency, the first order norm is used to calculate the similarity, and the similarity is quantified through the ratio relationship, wherein ||·||1 represents the first order norm operation;

[0047] When M t is similar to M t-1 , the value of M t -M t-1 is close to 0, and O t is close to 0. In order to prevent the tracking loss of the subject object caused by the fact that part of the features of the subject object cannot be directly observed, such as occlusion or part of the subject object leaving the field of view, the first order norm of the local feature change is selected to evaluate the feature change of the subject object. At the same time, when the first order norm is used to calculate the similarity, the complexity of the calculation is reduced.

[0048] W z represents an update gate weight matrix, which is used to control the proportion of historical memory retention, b z represents an update gate bias term, which is used to adjust the threshold of the gate activation, W r represents a reset gate weight matrix, which determines the fusion mode of the historical state to the current input, W h represents a candidate state weight matrix, which processes the reset historical state, U h represents a candidate state input weight matrix, which processes the current input g t , and Wβ represents an output gate weight matrix, which maps the hidden state to the contribution factor β.

[0049] Further, the step S500 comprises:

[0050] Step S501: calculating a query vector Q of the image features of the subordinate object in the target frame, Q=W q ·S t , wherein W q represents a query weight matrix, calculating a key vector K t-j corresponding to the jth video frame before the target frame, K t-j =W k ·H t-j , wherein W k represents a key weight matrix, and H t-j represents the spatio-temporal change feature corresponding to the jth video frame before the target frame, calculating a value vector R t-j , R t-j =W v ·Ht-j W v Represents the value weight matrix;

[0051] Step S502: Calculate the attention weight α of the j-th video frame preceding the target frame. t-j , Where T represents matrix transpose, and d represents the dimension of the spatially weighted image features of the subordinate objects;

[0052] Step S503: Calculate the attention Attn(St, Ht) between the image features and spatiotemporal variation features of the subordinate object.

[0053] Step S504: Collect the image features of the main object from the j-th video frame preceding the target frame, compensate for the features of the main object in the target frame, and obtain the optimized features of the main object in the target frame.

[0054] When the image features of the main object are drastically lost in the target frame compared to the previous frame, O t It tends to 1, at which point β t tending towards 1, through β t Inheriting the variation features of subordinate objects from frames preceding the target frame, the characteristics of the main object are inferred in the target frame. These characteristics are then enhanced, and the enhanced features of the main object in the target frame are output.

[0055] When tracking features of the main object in the target frame, O t β approaches 0 at this point t The value tends towards 0, and the tracking of the main object is maintained based on the characteristics of the main object in the target frame.

[0056] To better implement the above methods, a scene-adaptive video tracking system based on image fusion is proposed. The system includes: a spatiotemporal feature management module, a feature pair management module, a spatiotemporal change feature management module, a tracking contribution factor management module, and a feature enhancement module.

[0057] The system includes a spatiotemporal feature management module for collecting and recording video frames and managing the spatiotemporal features of each video frame; a feature pair management module for managing the enhanced main image features and subordinate image feature groups in each video frame; a spatiotemporal change feature management module for managing the spatiotemporal change features of memory feature groups; a tracking contribution factor management module for calculating the tracking contribution factor of the image features of the tracking main object in the video frame sequence; and a feature enhancement module for enhancing the features of the main object in the target frame.

[0058] Further, the space-time feature management module comprises a residual neural network feature extraction unit, a three-dimensional convolution layer extraction unit and a space-time feature splicing unit.

[0059] The residual neural network feature extraction unit is configured to extract the subject object feature of the target frame through a residual neural network, the three-dimensional convolution layer extraction unit is configured to extract the time sequence information of the target frame associated sequence through a three-dimensional convolution layer, and the space-time feature splicing unit is configured to splice the features of the subject object to obtain the space-time feature of the target frame.

[0060] Further, the feature pair management module comprises a subject object feature separation unit, a subordinate object feature separation unit, a weight calculation unit, a feature weighting unit and a feature pair management unit.

[0061] The subject object feature separation unit is configured to separate the image features of the subject object in the video frame through a subject object image feature separation function, the subordinate object feature separation unit is configured to separate the image features of the subordinate object in the video frame through a subordinate object image feature separation function, the weight calculation unit is configured to calculate the weight values of the image features of the subject object and the image features of the subordinate object in the video frame through a weight function respectively, the feature weighting unit is configured to weight the image features of the subject object and the image features of the subordinate object, and the feature pair management unit is configured to manage the feature pairs corresponding to each video frame.

[0062] Further, the space-time change feature management module comprises a frame feature sequence management unit, a memory feature group management unit, a memory feature sequence management unit and a space-time change feature management unit.

[0063] The frame feature sequence management unit is configured to manage the frame feature sequence, the memory feature group management unit is configured to manage the memory feature group corresponding to each video frame, the memory feature sequence management unit is configured to collect the memory feature groups to obtain the memory feature sequence, and the space-time change feature management unit is configured to obtain the space-time change feature of the memory feature sequence by extracting the global dependency feature and the time sequence feature of the memory feature sequence.

[0064] Further, the tracking contribution factor management module comprises a similarity quantization unit, an input vector management unit, a gated recurrent unit network management unit and a tracking contribution factor management unit.

[0065] The similarity quantization unit is configured to compare the similarity of the image features of the subject object in the target frame and the previous frame of the target frame, and quantize the similarity, the input vector management unit is configured to manage the input vector of the gated recurrent unit network, the gated recurrent unit network management unit is configured to manage the parameters of the gated recurrent unit network model, and the tracking contribution factor management unit is configured to calculate the tracking contribution factor.

[0066] Further, the feature enhancement module comprises an attention weight management unit, an attention management unit and an optimization output unit.

[0067] The attention weight management unit is configured to manage the attention weight of the video frame before the target frame, the attention management unit is configured to manage the attention of the image feature and the spatio-temporal change feature of the subordinate object, and the optimization output unit is configured to output the optimization feature of the subject object when the image feature of the subject object in the target frame is lost.

[0068] Compared with the prior art, the present application has the following advantages:

[0069] 1. The features of the subject object and the subordinate object in the video frame are decoupled, the features of the subject object and the subordinate object are collected respectively, the feature changes of the subject object and the subordinate object are inferred respectively, and the feature aliasing is prevented in the feature extraction process, so that the features of each part cannot be accurately extracted.

[0070] 2. The historical relationship between the feature changes of the subject object and the subordinate object is captured through the spatio-temporal memory, the state transition probability of the state change of the subject object is established, and the system is prevented from excessively intervening in the compensation of the subject object, so as to cause damage to the real features.

[0071] 3. An elastic compensation mechanism is established to fuse the causal reasoning compensation and the environmental compensation, so as to suppress the interference and improve the tracking ability of the features of the subject object. BRIEF DESCRIPTION OF DRAWINGS

[0072] Fig. 1 FIG. 1 is a structural schematic diagram of a scene adaptive video tracking system according to the present application;

[0073] Fig. 2 FIG. 2 is a flowchart of a scene adaptive video tracking method according to the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0075] Embodiment: As shown in the accompanying drawings, the present application provides a technical solution, a scene adaptive video tracking system and method fusing images; Figs. 1-2

[0076] The method comprises:

[0077] ​Step S100: obtaining a video frame sequence, extracting static features in each video frame and continuous change features of a local range in the video frame, splicing the static features and the continuous change features to obtain spatio-temporal features of the video frame;

[0078] In step S100, the following steps are included:

[0079] Step S101: obtaining a historical acquisition sequence V of video frames, V=I1, I2, I3, …, IT T , I2, I3, …, and IT T respectively represent the 1st, 2nd, 3rd, …, and Tth video frames, and a subject object in each video frame is identified;

[0080] Step S102: taking the tth video frame It in the current video sequence as a target frame, performing feature extraction on the target frame through a residual neural network to obtain ResNet(It) t , wherein ResNet represents the residual neural network; t

[0081] Step S103: forming a target frame associated sequence L t from a1 video frames before the target frame to a2 video frames after the target frame, obtaining time sequence information Conv3D(L t of the target frame associated sequence through a convolution kernel, wherein Conv3D represents a three-dimensional convolution layer;

[0082] Step S104: splicing ResNet(It) t and Conv3D(L t ) to obtain spatio-temporal features F t of the target frame.

[0083] F t = RestNet(It) t ⊕Conv3D(L t ), wherein represents a splicing operation;

[0084] In an embodiment, a1 and a2 are both 2, so that Lt is a target frame associated sequence composed of 5 frames of I t-2 , I t-1 , I t , I t+1 , and IT t+2 . If future frames of the target frame cannot be obtained, a2 is 0, and the range of video frames in the target frame associated sequence is controlled by adjusting a1.

[0085] ​Step S200: marking the subject object and the subordinate object in the historical acquisition record of the video frame, obtaining the weight of the image feature of the subject object and the image feature of the subordinate object in the spatio-temporal feature from the image feature of the subject object and the image feature of the subordinate object in the video frame, performing feature enhancement on the original subject image feature and the subordinate image feature acquired by weighted calculation, and forming the feature-enhanced subject image feature and the subordinate image feature in each video frame into a feature pair corresponding to the video frame;

[0086] In the embodiment, when tracking a person, for example, the image features of the limbs, joints and costumes of the person can be taken as the image features of the subject object, and the image of the environment where the person is located can be taken as the image features of the subordinate object; when tracking a vehicle, the image features of the parts of the vehicle such as the windows, doors and wheels, or the texture features of the vehicle image can be taken as the image features of the subject object, and the image features of the road, sky and distant buildings can be taken as the image features of the subordinate object.

[0087] The step S200 includes:

[0088] Step S201: obtaining the image feature separation function GM of the subject object, the image feature separation function of the subject object including a convolution layer, an activation function and a normalization layer, wherein the image feature separation function of the subject object includes three convolution layers of the same size, respectively obtaining three different size feature information of the image feature of the subject object, setting the weight matrix of each convolution layer to respond to the feature information, the normalization layer including a batch normalization layer, and the activation layer including a ReLU activation layer.

[0089] Step S202: obtaining the image feature separation function GS of the subordinate object, the image feature separation function of the subordinate object including a convolution layer, an activation function and a normalization layer, wherein the image feature separation function of the subordinate object includes three convolution layers, respectively obtaining three different size feature information of the image feature of the subject object, according to the size of the feature information from large to small, responding to the feature information through the weight matrix of the corresponding size, outputting the convolution result of the previous convolution layer through the normalization layer and the activation layer between two convolution layers, the normalization layer including an instance normalization layer, and the activation layer including a LeakyReLU activation layer.

[0090] Step S203: obtaining the separation feature M of the image feature of the subject object raw t ,

[0091] M raw t = GM(F t ), obtaining the separation feature S of the image feature of the subordinate object raw t , Sraw t = GS (F t ) ;

[0092] Step S204: obtaining the weight A of the subject object in the target frame M (F t ),

[0093] wherein AvgPool (F t ) represents average pooling of F t , MaxPool (F t ) represents maximum pooling of F t ;

[0094] C M 1 and C M 2 are weight matrices, wherein C M 1 represents a weight matrix of image features of the subject object in image weights in the feature frame, and C M 2 represents a weight matrix of the corresponding region of the subject object on all regions of the feature frame;

[0095] Step S205: obtaining the weight A of the dependent object in the target frame S (F t ),

[0096] C S 1 and C S 2 are weight matrices, wherein C S 1 represents a weight matrix of image features of the dependent object in the feature frame, and C S 2 represents a weight matrix of the corresponding region of the dependent object on all regions of the feature frame;

[0097] Step S206: calculating the spatially weighted image features M t of the subject object = M raw t ⊙ A M (F t ) and the spatially weighted image features S t of the dependent object = S raw t ⊙ A S (F t ), wherein ⊙ represents Hadamard product;

[0098] Step S207: collecting the spatially weighted image features of the subject object and the spatially weighted image features of the dependent object in the target frame to form a feature pair (M t , S t ).

[0099] wherein W1 M , W2 M and W3 M represent independent convolution kernel weight matrices, the size of W3 M , W2 M and W1 M are the same, for example, the size of each layer is 3x3;

[0100] wherein represents a dilated convolution, to expand the field of view to protect the overall characteristics of the subject object, in the embodiment, the dilated rate is set to 2, and the size of the receptive field after the field of view is expanded is 7x7.

[0101] wherein V1 S , V2 S and V3 S represent independent convolution kernel weight matrices, the size of V3 S is smaller than the size of V2 S , the size of V2 S is smaller than the size of V1 S , V1 S captures large-scale background and motion patterns, such as the background of the sky and the road, V2 S captures medium-sized backgrounds, such as a fluttering flag and a moving vehicle, and V1 S captures cross-channel interaction features, such as changes in texture features caused by changes in light and shadow or motion blur.

[0102] In the embodiment, in order to avoid gradient disappearance, the slope of the linear activation function is set to 0.2, the size of V1 S may be selected as 5x5, the size of V2 S may be selected as 3x3, and the size of V3 S may be selected as 1x1

[0103] Step S300: Extract the image features, position information of the subject object in each video frame, and the image features and position information of the dependent object and the position change of the previous frame to form a memory feature group. Combine the frame feature change rule and the time sequence feature change rule of a plurality of memory feature groups to obtain the spatio-temporal change features of the plurality of memory feature groups.

[0104] wherein step S300 comprises:

[0105] Step S301: Merge the target frame and all the previous k video frames of the target frame into a target frame sequence FR, and obtain a feature pair of each video frame in the target frame sequence to form a frame feature sequence Y, Y=(M t-k , S t-k ), (M t-k+1 , St-k+1 ), (M t-k+2 , S t-k+2 ), …, (M t , S t ), wherein (M t-k , S t-k ) represents a feature pair corresponding to the kth video frame before the target frame, and (M t , S t ) represents a feature pair corresponding to the target frame.

[0106] Step S302: obtaining a feature pair (M i , S i ) corresponding to the ith video frame in the target sequence, obtaining a position vector p i of the subject object in the ith video frame, and a position vector p i-1 of the subject object in the video frame corresponding to the i-1th feature pair, calculating a displacement vector Δp i of the ith feature pair, Δp i = p i - p i-1 .

[0107] Step S303: calculating a memory feature group m i corresponding to the feature pair (M i , S i ) corresponding to the ith video frame, m i = (ROIAlign (M i , p i ), Hist (S i ), Δp i ), wherein Hist (S i ) represents the distribution probability of all colors in S i , and ROIAlign represents a region of interest alignment method.

[0108] Step S304: collecting the memory feature groups of the target frame and all the k video frames before the target frame to form a memory feature sequence ME, ME: (m t-k , m t-k+1 , m t-k+2 , …, m t ), wherein m t-k represents a memory feature group corresponding to the kth video frame before the target frame, m t represents a memory feature group corresponding to the target frame, and a global dependency feature h TCN of the target frame is obtained through a dilated convolution network. t TCN t = Dilated-Conv (ME), Dilated-Conv represents a dilated convolution network.

[0109] Step S305: obtaining the time sequence feature of the memory feature group through the long short-term memory network, h LSTM t =LSTM(h t-1 , m t );

[0110] Step S306: performing feature fusion on the global dependency feature and the time sequence feature through a normalization operation to obtain the spatio-temporal change feature H t , H t =LayerNorm(Y1·h TCN t +Y2·h LSTM t ), wherein LayerNorm represents the normalization operation, and wherein Y1 and Y2 represent weight matrices respectively.

[0111] Step S400: taking a frame in the video sequence as a target frame, obtaining the image feature of the subject object in the target frame and the feature of the subject object in the previous frame of the target frame, obtaining the historical correlation of the image feature and the subordinate feature of the subject object, and calculating a tracking contribution factor of the image feature of the subject object;

[0112] In the step S400, the following steps are included:

[0113] Step S401: obtaining the spatially weighted image feature M t of the subject object in the target frame and the spatially weighted image feature S t of the subordinate object, and obtaining the spatially weighted image feature M t-1 of the subject object in the previous video frame of the target frame.

[0114] Step S402: calculating the similarity between M t and M t-1 , denoted as sim t , and normalizing the similarity sim t to the interval (0, 1) to obtain the subject object occlusion confidence O t in the target frame, wherein when sim t tends to 0, O t tends to 1, and when sim t tends to 1, O t tends to 0.

[0115] In the embodiment, the feature of the subject object in the target frame is represented by a matrix, the similarity is calculated in a first-order norm manner to improve the calculation efficiency of the similarity, and the similarity is quantified through a ratio relationship, wherein ||·||1 represents a first-order norm operation.

[0116] When M t t-1 When the similarity is high, M t The value of M t-1 is close to 0, O t is close to 0, in order to prevent the part of the subject object from being unable to be directly observed, such as occlusion or the part of the subject object leaving the field of view causing the subject object tracking to be lost, therefore, the first order norm of the local feature is selected to evaluate the feature change of the subject object, and when the similarity is calculated by using the first order norm, the complexity of the calculation is reduced

[0117] Step S403: establishing the correlation vector g of the image feature of the spatially weighted subject object of the target frame and the image feature of the spatially weighted subject object of the previous frame of the target frame t as an input vector of the gated recurrent unit network, g t =[AvgPool(M t )⊕MaxPool(S t )⊕h t-1 ⊕O t ], wherein ⊕ represents splicing operation, AvgPool(M t ) represents performing average pooling on M t , and MaxPool(S t ) represents performing maximum pooling on St;

[0118] Step S404: obtaining a gated recurrent unit network model, establishing an update gate z t , z t =σ(W z ·g t +b z ), wherein W z represents an update gate weight matrix, b z represents an update gate bias term, and σ represents a Sigmoid activation function, establishing a reset gate r t , r t =σ(W r ·g t +b r ), wherein W r represents a reset gate weight matrix, b r represents a reset gate bias term, and σ represents a Sigmoid activation function, and a candidate hidden state wherein W h represents a candidate state weight matrix, U h represents a candidate state input weight matrix, and tanh represents a tanh activation function, establishing a hidden state output gate h t , wherein ⊙ represents Hadamard product operation;​

[0119] Step S405: calculating the tracking contribution factor β t , β t = σ(W β · h t + b β ), where W β represents a contribution factor weight matrix, and b β represents a contribution factor output bias term.

[0120] In an embodiment, W z , W r , W h , and U h are initialized by Xavier normal distribution, b z and b r are initialized by all-zero, W β is initialized by uniform distribution, and all gating parameters are updated by gradient learning.

[0121] Step S500: calculating the contribution degree of the image feature of the subordinate object in the target frame to the target frame feature change, collecting the image features of the main object in all video frames before the target frame, and strengthening the image features of the main object in the target frame;

[0122] Step S500 includes:

[0123] Step S501: calculating a query vector Q of the image feature of the subordinate object in the target frame, Q = W q · S t , where W q represents a query weight matrix, calculating a key vector K t-j corresponding to the jth video frame before the target frame, K t-j = W k · H t-j , where W k represents a key weight matrix, and H t-j represents the spatiotemporal change feature corresponding to the jth video frame before the target frame, calculating a value vector R t-j , R t-j = W v · H t-j , where W v represents a value weight matrix; and Step S502: calculating an attention weight a t-j ,

[0124] Step S503: calculating an attention Attn(St, Ht) of the image feature of the subordinate object and the spatiotemporal change feature,

[0125] Step S504: Collect the image features of the jth video frame before the target frame, compensate the features of the subject object in the target frame, and obtain the optimized features of the subject object in the target frame

[0126] A scene adaptive video tracking system for fused images, the system comprising: a spatio-temporal feature management module, a feature pair management module, a spatio-temporal change feature management module, a tracking contribution factor management module and a feature enhancement module.

[0127] The spatio-temporal feature management module is configured to collect records of video frames and manage spatio-temporal features of each video frame, and the spatio-temporal feature management module comprises a residual neural network feature extraction unit, a three-dimensional convolution layer extraction unit and a spatio-temporal feature splicing unit, wherein the residual neural network feature extraction unit is configured to extract subject object features of a target frame through a residual neural network, the three-dimensional convolution layer extraction unit is configured to extract time sequence information of a target frame associated sequence through a three-dimensional convolution layer, and the spatio-temporal feature splicing unit is configured to splice the features of the subject object to obtain spatio-temporal features of the target frame.

[0128] The feature pair management module is configured to manage feature pairs of the subject image features and the feature groups of the subordinate image features in each video frame, and the feature pair management module comprises a subject object feature separation unit, a subordinate object feature separation unit, a weight calculation unit, a feature weighting unit and a feature pair management unit, wherein the subject object feature separation unit is configured to separate the image features of the subject object in the video frame through a subject object image feature separation function, the subordinate object feature separation unit is configured to separate the image features of the subordinate object in the video frame through a subordinate object image feature separation function, the weight calculation unit is configured to calculate weight values of the image features of the subject object and the image features of the subordinate object in the video frame through a weight function, respectively, the feature weighting unit is configured to weight the image features of the subject object and the image features of the subordinate object, and the feature pair management unit is configured to manage the feature pairs corresponding to each video frame.

[0129] The spatio-temporal change feature management module is configured to manage spatio-temporal change features of the memory feature group, and the spatio-temporal change feature management module comprises a frame feature sequence management unit, a memory feature group management unit, a memory feature sequence management unit, and a spatio-temporal change feature management unit.

[0130] The tracking contribution factor management module is configured to calculate tracking contribution factors of image features of the tracked subject object in the video frame sequence, and the tracking contribution factor management module comprises a similarity quantization unit, an input vector management unit, a gated recurrent unit network management unit, and a tracking contribution factor management unit.

[0131] The feature enhancement module is configured to enhance features of the subject object in the target frame, and the feature enhancement module comprises an attention weight management unit, an attention management unit, and an optimized output unit.

[0132] It is apparent for a person skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced in the present application. Any reference signs in the claims should not be considered as limiting the claims involved.

Claims

1. A scene-adaptive video tracking method that fuses images, characterized in that: Step S100: Obtain the video frame sequence, extract the static features and the continuous change features of the local range in each video frame, and concatenate the static features and the continuous change features to obtain the spatiotemporal features of the video frame. Step S200: Mark the main object and subordinate objects in the historical acquisition records of the video frames. Obtain the weights of the image features of the main object and subordinate objects in the spatiotemporal features from the image features of the main object and subordinate objects in the video frames. Perform feature enhancement on the acquired original main image features and subordinate image features through weighted calculation. Combine the enhanced main image features and subordinate image features in each video frame to form the feature pair corresponding to the video frame. Step S300: Extract the image features and position information of the main object in each video frame, and the image features and position information of the subordinate objects, and combine them with the position changes of the previous frame to form a memory feature group. Combine the frame feature change rules and temporal feature change rules of several memory feature groups to obtain the spatiotemporal change features of several memory feature groups. Step S300 includes: Step S301: Merge the target frame and all its first k video frames into the target frame sequence FR, and obtain the feature pairs of each video frame in the target frame sequence to form the frame feature sequence Y, Y = (M t-k S t-k (M) t-k+1 S t-k+1 (M) t-k+2 S t-k+2 ), ..., (M t S t ), of which (M t-k S t-k ) represents the feature pair corresponding to the k-th video frame preceding the target frame, (M t S t () represents the feature pair corresponding to the target frame; Step S302: Obtain the feature pair (M) corresponding to the i-th video frame in the target sequence. i S i ), obtain the position vector p of the main object in the i-th video frame. i , and the position vector p of the main object in the corresponding video frame of the (i-1)th feature pair i-1 Calculate the displacement vector Δp ​​of the i-th feature pair. i , Δp i =p i - p i-1 ; Step S303: Calculate the feature pair (M) corresponding to the i-th video frame. i S i The corresponding memory feature group m i m i =(ROIAlign(M i p i Hist (S) i ), Δp i ), where Hist (S i ) represents S i The probability distribution of all colors in the region, where ROIAlign represents the region of interest alignment method; Step S304: Collect the memory feature groups of the target frame and all the previous k video frames of the target frame to form a memory feature sequence ME, ME: (m t-k m t-k+1 m t-k+2 , ..., m t ), where m t-k m represents the memory feature set corresponding to the k-th video frame preceding the target frame. t This represents the memory feature set corresponding to the target frame, and the global dependency feature h of the target frame is obtained through a dilated convolutional network. TCN t h TCN t =Dilated-Conv(ME), where Dilated-Conv represents a dilated convolutional network; Step S305: Obtain the temporal features of the memory feature group through the Long Short-Term Memory Network, h LSTM t =LSTM(h) t-1 m t ); Step S306: Perform feature fusion on global dependency features and temporal features through normalization operation to obtain spatiotemporal variation features H. t H t =LayerNorm(Y1·h TCN t +Y2·h LSTM t ), where LayerNorm represents the normalization operation, and Y1 and Y2 represent the weight matrices respectively; Step S400: Take a frame in the video sequence as the target frame, obtain the image features of the main object in the target frame and the features of the main object in the previous frame of the target frame, obtain the historical correlation between the image features and subordinate features of the main object, and calculate the tracking contribution factor of the image features of the main object. Step S500: Calculate the contribution of the image features of subordinate objects to the feature changes in the target frame, collect the image features of the main objects in all video frames before the target frame, and enhance the image features of the main objects in the target frame.

2. The scene adaptive video tracking method based on fused images according to claim 1, characterized in that: Step S100 includes: Step S101: Obtain the historical acquisition sequence V of video frames, V = I1, I2, I3, ..., I T Among them, I1, I2, I3, ... and I T These represent the 1st, 2nd, 3rd, ..., and Tth video frames, respectively, and the main object in each video frame is identified. Step S102: Take the t-th video frame I in the current video sequence t As the target frame, features are extracted from the target frame using a residual neural network to obtain a ResNet(I) algorithm. t ), where ResNet represents a residual neural network; Step S103: Form the target frame association sequence L from the a1 video frames before the target frame to the a2 video frames after the target frame. t Conv3D(L) obtains the temporal information of the target frame associated sequence through convolutional kernels. t ), where Conv3D represents a three-dimensional convolutional layer; Step S104: Combine ResNet (I t ) and Conv3D (L t The spatiotemporal features F of the target frame are obtained by concatenating channels. t .

3. The scene adaptive video tracking method based on fused images according to claim 2, characterized in that: Step S200 includes: Step S201: Obtain the image feature separation function GM of the main object. The image feature separation function of the main object includes a convolutional layer, an activation function, and a normalization layer. The image feature separation function of the main object includes three convolutional layers of the same size, which respectively obtain the image features of the main object of three different sizes of feature information. Set the weight matrix of each convolutional layer to respond to the feature information. The normalization layer includes a batch normalization layer, and the activation layer includes a ReLU activation layer. Step S202: Obtain the image feature separation function GS of the subordinate object. The image feature separation function of the subordinate object includes a convolutional layer, an activation function, and a normalization layer. The image feature separation function of the subordinate object includes three convolutional layers, which respectively obtain the image features of the main object at three different sizes. According to the size of the feature information from large to small, the feature information is responded to by a weight matrix of the corresponding size. Between two convolutional layers, a normalization layer and an activation layer are used to output the convolution result of the previous convolutional layer. The normalization layer includes an instance normalization layer, and the activation layer includes a LeakyReLU activation layer. Step S203: Obtain the separation feature M of the image features of the main object. raw t , M raw t =GM(F t ), to obtain the separation feature S of the image features of the subordinate object. raw t S raw t =GS(F t ); Step S204: Obtain the weight A of the main object in the target frame. M (F) t ), Where AvgPool(Ft) represents average pooling of Ft, MaxPool(Ft) represents max pooling of Ft, σ represents the Sigmoid activation function, and ⊕ represents concatenation operation; C M 1 and C M 2 is the weight matrix, where C M 1 represents the weight matrix of the image features of the main object in the feature frame, and C represents the image weights of the main object. M 2 represents the weight matrix of the corresponding region of the main object across all regions of the feature frame; Step S205: Obtain the weight A of the subordinate object in the target frame. S (F) t ), C S 1 and C S 2 is the weight matrix, where C S 1 represents the weight matrix of the image features of the subordinate object in the feature frame, C S 2 represents the weight matrix of the corresponding region of the subordinate object across all regions of the feature frame, σ represents the Sigmoid activation function, and ⊕ represents the concatenation operation; Step S206: Calculate the spatially weighted image features M of the main object. t =M raw t ⊙A M (F) t ), and spatially weighted image features S of subordinate objects t =S raw t ⊙A S (F) t ), where ⊙ represents the Hadamard volume; Step S207: Collect the spatially weighted image features of the main object and the spatially weighted image features of the subordinate objects in the target frame to form a feature pair (M). t S t ).

4. The scene adaptive video tracking method based on fused images according to claim 3, characterized in that: Step S400 includes: Step S401: Obtain the spatially weighted image features M of the main object in the target frame. t Image features S of spatially weighted subordinate objects t Obtain the spatially weighted image features M of the main object in the video frame preceding the target frame. t-1 ; Step S402: Calculate M t and M t-1 The similarity is denoted as sim t sim similarity t Normalized to the (0,1) interval, the occlusion confidence O of the main object in the target frame is obtained. t When sim t When O approaches 0 t tending towards 1, when sim t When O approaches 1 t It tends towards 0; Step S403: Establish the correlation vector gt between the spatially weighted image features of the target frame and the spatially weighted image features of the subject object in the previous frame, and use it as the input vector of the gated recurrent unit network. t =[AvgPool(M t )⊕MaxPool(S t )⊕h t-1 ⊕O t ], where ⊕ represents the concatenation operation, AvgPool(M t ) indicates that for M t Perform average pooling, MaxPool(S) t This indicates that max pooling is performed on St; Step S404: Obtain the gated recurrent unit network model and establish the update gate z. t , z t =σ(W z ·g t +b z ), where W z This indicates updating the gate weight matrix, b z The update gate bias term is labeled, σ represents the Sigmoid activation function, and a reset gate r is constructed. t r t =σ(W r ·g t +b r ), where W r Denotes the reset gate weight matrix, b r This represents the reset gate bias term, σ represents the Sigmoid activation function, and the candidate hidden state. , W h U represents the candidate state weight matrix. h Let denot be the candidate state input weight matrix, tanh be the tanh activation function, and h be the hidden state output gate. t , , where ⊙ represents the Hadamard product operation; Step S405: Calculate the tracking contribution factor β t ,β t =σ(W β ·h t +b β ), where W β Let b represent the contribution factor weight matrix. β Contribution factor output bias term.

5. The scene adaptive video tracking method based on fused images according to claim 4, characterized in that: Step S500 includes: Step S501: Calculate the query vector Q of the image features of the subordinate objects in the target frame, Q=W q ·S t W q This represents the query weight matrix, calculating the key vector K corresponding to the j-th video frame preceding the target frame. t-j K t-j =W k ·H t-j W k H represents the key weight matrix. t-j Representing the spatiotemporal variation features corresponding to the j-th preceding video frame of the target frame, the value vector R is calculated. t-j R t-j =W v ·H t-j W v Represents the value weight matrix; Step S502: Calculate the attention weight α of the j-th video frame preceding the target frame. t-j , Where T represents the matrix transpose and d represents the dimension of the image features of the spatially weighted subordinate objects; Step S503: Calculate the attention Attn(St, Ht) between the image features and spatiotemporal change features of the subordinate object. , Step S504: Collect the image features of the main object from the j-th video frame preceding the target frame, compensate for the features of the main object in the target frame, and obtain the optimized features of the main object in the target frame. , .

6. A scene-adaptive video tracking system for fused images, used to execute the scene-adaptive video tracking method for fused images as described in any one of claims 1-5, characterized in that: The system includes: Spatiotemporal feature management module, feature pair management module, spatiotemporal change feature management module, tracking contribution factor management module, and feature enhancement module; The system includes a spatiotemporal feature management module for collecting and recording video frames and managing the spatiotemporal features of each video frame; a feature pair management module for managing the enhanced main image features and subordinate image feature groups in each video frame; a spatiotemporal change feature management module for managing the spatiotemporal change features of memory feature groups; a tracking contribution factor management module for calculating the tracking contribution factor of the image features of the tracking main object in the video frame sequence; and a feature enhancement module for enhancing the features of the main object in the target frame.

7. A scene-adaptive video tracking system based on fused images according to claim 6, characterized in that: The spatiotemporal feature management module includes: a residual neural network feature extraction unit, a three-dimensional convolutional layer extraction unit, and a spatiotemporal feature stitching unit; Among them, the residual neural network feature extraction unit is used to extract the main object features of the target frame through the residual neural network, the three-dimensional convolutional layer extraction unit is used to extract the temporal information of the associated sequence of the target frame through the three-dimensional convolutional layer, and the spatiotemporal feature splicing unit is used to splice the features of the main object to obtain the spatiotemporal features of the target frame. The feature pair management module includes: a main object feature separation unit, a subordinate object feature separation unit, a weight calculation unit, a feature weighting unit, and a feature pair management unit; The main object feature separation unit is used to separate the image features of the main object in the video frame using the image feature separation function of the main object; the subordinate object feature separation unit is used to separate the image features of the subordinate object in the video frame using the image feature separation function of the subordinate object; the weight calculation unit is used to calculate the weight values ​​of the image features of the main object and the image features of the subordinate object in the video frame respectively using the weight function; the feature weighting unit is used to weight the image features of the main object and the image features of the subordinate object; and the feature pair management unit is used to manage the feature pairs corresponding to each video frame. The spatiotemporal variation feature management module includes: a frame feature sequence management unit, a memory feature group management unit, a memory feature sequence management unit, and a spatiotemporal variation feature management unit; Among them, the frame feature sequence management unit is used to manage the frame feature sequence, the memory feature group management unit is used to manage the memory feature group corresponding to each video frame, the memory feature sequence management unit is used to collect the memory feature groups to obtain the memory feature sequence, and the spatiotemporal change feature management unit is used to extract the global dependency features and temporal features of the memory feature sequence to obtain the spatiotemporal change features of the memory feature sequence.

8. A scene-adaptive video tracking system based on fused images according to claim 6, characterized in that: The tracking contribution factor management module includes: a similarity quantification unit, an input vector management unit, a gated recurrent unit network management unit, and a tracking contribution factor management unit. The similarity quantification unit is used to compare the similarity of image features of the main object in the target frame and the previous frame, and to quantify the similarity. The input vector management unit is used to manage the input vector of the gated recurrent unit network. The gated recurrent unit network management unit is used to manage the parameters of the gated recurrent unit network model. The tracking contribution factor management unit is used to calculate the tracking contribution factor.

9. A scene-adaptive video tracking system based on fused images according to claim 6, characterized in that: The feature enhancement module includes: an attention weight management unit, an attention management unit, and an output optimization unit; The attention weight management unit manages the attention weights of the video frames preceding the target frame, the attention management unit manages the attention of the image features and spatiotemporal change features of the subordinate objects, and the optimization output unit outputs the optimized features of the subject object when the image features of the subject object in the target frame are lost.

Images