A refined optical flow estimation system integrating multi-scale kernel selection and super-resolution technology

Abstract

A fine optical flow estimation system integrating multi-scale kernel selection and super-resolution technology. It belongs to the field of image processing technology, and specifically to the field of fine optical flow estimation technology. It solves the problem of reduced estimation accuracy of RAFT network. The system includes a feature extraction network module, a super-resolution iterative update module and a context network module; the feature extraction network module is used to extract features of two adjacent frames of images, and input the extracted related features into the super-resolution iterative update module; the context network module is used to extract context features of the previous frame of two adjacent frames of images, and input the extracted context features into the super-resolution iterative update module; the super-resolution iterative update module fuses the related features and the context features to obtain an optical flow vector diagram output with super-resolution.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular to the technical field of fine optical flow estimation. Background Art

[0002] As a basic computer vision task, the core of optical flow estimation is to estimate motion based on the changes of objects and backgrounds in the temporal dimension between adjacent frames. Optical flow estimation has a wide range of applications in fields such as surveillance video and autonomous driving.

[0003] The success of early optical flow estimation is attributed to the end-to-end U-Net network and the coarse-to-fine pyramid network. The encoder and decoder structure in the U-Net network estimates motion vectors through skip connections and inverse convolution. The coarse-to-fine pyramid network estimates residual co-motion at each scale through a coarse-to-fine iterative method, and compensates for the motion estimated in the first frame by warping the second frame to the first frame.

[0004] The RAFT network captures the similarity information between image pairs by using GRU to continuously optimize the optical flow estimation. This method breaks the dominance of the U-Net network and the coarse-to-fine pyramid network estimation method. The RAFT network achieves high accuracy and efficiency in optical flow estimation, providing strong support for computer vision tasks. The AnyFlow network combines the RAFT update network with the coarse-to-fine estimation to obtain more refined optical flow values.

[0005] In pursuit of high-precision and efficient optical flow estimation, many methods use various optimization methods for different scenarios to reduce the endpoint error (EPE) of the overall optical flow estimation. For example, multi-scale feature fusion and recursive optimization techniques are used to gradually refine the optical flow estimation from coarse to fine, ensuring accuracy while improving computational efficiency. Existing methods enhance the perception ability of the model by introducing attention mechanisms and contextual information, enabling it to more accurately capture motion details in complex scenes, but this affects timeliness.

[0006] Since the RAFT network is updated and iterated at a fixed resolution scale, the receptive field of the convolution kernel cannot increase with iteration, resulting in insufficient capture of fine-grained features and poor adaptability to weak texture areas. Summary of the invention

[0007] In order to solve the problem that in the updating and iterative process of the previous RAFT network, the estimation accuracy is reduced due to the gradual compression of image features and insufficient capture of fine-grained features, the present invention provides a fine optical flow estimation system integrating multi-scale kernel selection and super-resolution technology.

[0008] The system includes a feature extraction network module, a super-resolution iterative update module and a context network module;

[0009] The feature extraction network module is used to extract features of two adjacent frames of images, and input the extracted relevant features into the super-resolution iterative update module;

[0010] The context network module is used to extract the context features of the previous frame of the image between two adjacent frames of the image, and input the extracted context features into the super-resolution iterative update module;

[0011] The super-resolution iterative update module fuses the relevant features and the context features to obtain an optical flow vector map output with super-resolution.

[0012] Furthermore, the feature extraction network module is composed of a feature extraction network and a visual similarity calculation network in the baseline network RAFT. The feature extraction network performs feature extraction on two input images, and inputs the extracted features into the visual similarity calculation network. In the visual similarity calculation network, dot product calculation is performed on the features extracted from the two images to finally obtain relevant features.

[0013] Furthermore, the context network module is independently designed, and includes a residual network, a multi-scale kernel selective network, a convolutional layer and a residual connection layer from input to output. The residual connection layer adds the output of the convolutional layer to the output of the residual network to obtain context features.

[0014] Furthermore, the residual network includes a 7*7 convolutional layer, six residual units and a 3*3 convolutional layer from input to output, the number of channels of the six residual units are 64, 64, 96, 96, 128 and 128 respectively, and the six residual units are named residual unit No. 1, residual unit No. 2, residual unit No. 3, residual unit No. 4, residual unit No. 5 and residual unit No. 6 respectively.

[0015] Further, the multi-scale kernel selective network includes a Fuse part and a Select part;

[0016] The fuse part receives and combines the feature information from residual unit 2, residual unit 4, residual unit 6, and 3*3 convolutional layer to obtain multi-scale features, performs global average pooling and maximum average pooling on the multi-scale features in the spatial dimension to obtain combined features, performs convolution kernel and activation operations on the combined features to obtain compact features, and inputs the compact features into four parallel convolutional layers to obtain four descriptors corresponding to residual unit 2, residual unit 4, residual unit 6, and 3*3 convolutional layer. V 1. V 2. V 3 and V 4;

[0017] Select part V 1. V2. V 3 and V 4 Perform activation operation and obtain activation coefficients respectively s 1. s 2. s 3 and s 4. Use activation coefficients to V 1. V 2. V 3 and V 4 is calibrated, and then the calibrated descriptors are fused to obtain the context features processed by the kernel selection mechanism.

[0018] Furthermore, the super-resolution iterative update module is an independent design. The super-resolution iterative update module is based on the iterative update module in the baseline network RAFT and improves it. The specific improvement is: a parameter-free Swift 3D weighted attention mechanism is added to the path before the context feature input gated activation unit.

[0019] Furthermore, the parameter-free Swift 3D weighted attention mechanism is specifically: the features extracted by the 3D weight module H i With input O i-1 Perform residual connection output U i , and then H i After an activation function symmetric to the origin Get attention V i ,at last V i and U i Multiply elements by element to get the final output O i .

[0020] The beneficial effects of the system of the present invention are:

[0021] The RAFT network is updated and iterated at a fixed resolution scale, and the receptive field of the convolution kernel cannot increase with the iteration, resulting in insufficient capture of fine-grained features and poor adaptability to weak texture areas. To alleviate this problem, the present invention proposes a fine optical flow estimation system that integrates multi-scale kernel selection and super-resolution technology. In order to capture fine-grained features more accurately, a super-resolution module based on the parameter-free Swift 3D weight attention mechanism is constructed in the super-resolution iterative update module to generate a fine optical flow map. The 3D weights generated by the neuron energy function are connected through residual connections and symmetric activation functions to generate parameter-free Swift 3D weight attention feature maps, and the importance of different regions in the feature map is dynamically adjusted without increasing the number of parameters. In order to enrich the feature representation ability of the GRU hidden layer, a context network module with multi-scale kernel selectivity is designed. The network module retains the original feature information of each spatial resolution by dynamically combining feature information of different depths in the context network, expands the receptive field of the network, and enhances the adaptability to weak texture areas. Experimental results show that the system described in the present invention effectively alleviates the problem of gradual compression of image features and insufficient capture of fine-grained features in the recursive process of the baseline network. The proposed module is lightweight and flexible, demonstrating its practicality and potential application value, especially in resource-constrained scenarios.

[0022] The system of the present invention can be applied in fields such as surveillance video and autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a structural diagram of a fine optical flow estimation system integrating multi-scale kernel selection and super-resolution technology described in a specific embodiment of the present invention;

[0024] Figure 2 is a structural diagram of a multi-scale kernel selectivity network (MKS) described in a specific embodiment of the present invention;

[0025] Figure 3 It is a structural diagram of the parameter-free Swift 3D weighted attention mechanism described in a specific embodiment of the present invention;

[0026] Figure 4 Schematic diagram of input and output of the gate activation unit described in a specific embodiment of the present invention;

[0027] Figure 5 This is a comparison diagram of optical flow estimation results obtained by using different optical flow estimation methods described in a specific embodiment of the present invention and using the fine optical flow estimation system described in the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0029] Embodiment 1,

[0030] This embodiment provides a fine optical flow estimation system that integrates multi-scale kernel selection and super-resolution technology. Figure 1 As shown, the system includes a feature extraction network module, a super-resolution iterative update module and a context network module;

[0031] The feature extraction network module is used to extract the features of two adjacent frames of images, and input the extracted relevant features into the super-resolution iterative update module; the context network module is used to extract the context features of the previous frame of the two adjacent frames of images, and input the extracted context features into the super-resolution iterative update module; the super-resolution iterative update module fuses the relevant features and the context features to obtain a high-resolution feature map output.

[0032] Embodiment 2,

[0033] This embodiment further limits the embodiment 1 and further limits the feature extraction network module.

[0034] The feature extraction network module is composed of a feature extraction network (Feature Network) and a visual similarity calculation network (Correlation volume) in the benchmark network RAFT. The feature extraction network performs feature extraction on two input images, and inputs the extracted features into the visual similarity calculation network. In the visual similarity calculation network, dot product calculation is performed on the features extracted from the two images to finally obtain the correlation features.

[0035] Embodiment 3,

[0036] This embodiment further limits the embodiment 1 and further limits the context network module.

[0037] The context network module (Context Network) is independently designed, such as Figure 1 As shown in , from input to output, it includes a residual network, a multi-scale kernel selectivity network (MKS), a convolutional layer and a residual connection layer. The residual connection layer adds the output of the convolutional layer to the output of the residual network to obtain context features.

[0038] The residual network includes a 7*7 convolutional layer, six residual units and a 3*3 convolutional layer from input to output. The number of channels of the six residual units is 64, 64, 96, 96, 128 and 128 respectively. The six residual units are named residual unit No. 1, residual unit No. 2, residual unit No. 3, residual unit No. 4, residual unit No. 5 and residual unit No. 6 respectively.

[0039] like Figure 2 The figure shows the structure of the multi-scale kernel selective network (MKS). The multi-scale kernel selective network can enrich the feature information of the hidden layer of the gated activation unit (ConvGRU) in the super-resolution iterative update module, and comprehensively capture the motion information at different scales by fusing the features of different scales in the context network, thereby enhancing the adaptability to complex motion patterns.

[0040] The multi-scale kernel selective network includes a Fuse part and a Select part;

[0041] The fuse part receives and combines the feature information from residual unit 2, residual unit 4, residual unit 6, and 3*3 convolutional layer to obtain multi-scale features, performs global average pooling and maximum average pooling on the multi-scale features in the spatial dimension to obtain combined features, performs convolution kernel and activation operations on the combined features to obtain compact features, and inputs the compact features into four parallel convolutional layers to obtain four descriptors corresponding to residual unit 2, residual unit 4, residual unit 6, and 3*3 convolutional layer. V 1. V 2. V 3 and V 4;

[0042] The Fuse part generates the descriptor of the global and maximum features by fusing feature information of different resolutions.

[0043] The input is the parallel feature information of four scales from the residual network. L1, L2, L3, and L4 are combined by element-by-element summation to obtain the multi-scale feature L∈R C×W×H :

[0044] ;

[0045] exist L ∈R C×W×H Perform global average pooling and maximum average pooling on the spatial dimension of S ∈R C×1×1 :

[0046] ;

[0047] In the formula, GAP is the global average pooling, MAPis the maximum average pooling. GAP and MAP The fusion of the obtained global and local feature information is enriched.

[0048] Compact feature information Z∈ is generated through a 3×3 convolution and ReLU operation r×1×1 ,

[0049] Output Z:

[0050] ;

[0051] Z flows through four parallel convolutions to obtain L 1. L 2. L 3. L 4 Corresponding descriptors V 1. V 2. V 3. V 4. V ∈R C ×1×1 .

[0052] Select part V 1. V 2. V 3 and V 4 Perform activation operation and obtain activation coefficients respectively s 1. s 2. s 3 and s 4. Use activation coefficients to V 1. V 2. V 3 and V 4 is calibrated, and then the calibrated descriptors are fused to obtain the context features processed by the kernel selection mechanism.

[0053] The Select section will fuse the descriptor obtained V 1. V 2. V 3. V 4After a softmax Function, get the activation coefficient s 1. s 2. s 3. s 4. Use the activation coefficients to calibrate the four scale feature information respectively, and then fuse them to obtain U:

[0054]

[0055] Embodiment 4,

[0056] This embodiment further limits the embodiment 1, and further limits the super-resolution iterative update module.

[0057] The super-resolution iterative update module is independently designed, such as Figure 1 As shown in the figure, the super-resolution iterative update module is based on the iterative update module (Interative Update Block) in the baseline network RAFT, and improves it. The specific improvement is: adding a parameter-free Swift 3D weighted attention mechanism (PS3D) in the path before the context feature input gated activation unit (ConvGRU).

[0058] like Figure 3 The figure shows the structure of the parameter-free Swift 3D weighted attention mechanism. The parameter-free Swift 3D weighted attention mechanism is specifically: the features extracted by the 3D weight module (3D weights) H i With input O i-1 Perform residual connection output U i , and then H i After an activation function symmetric to the origin Get attention V i ,at last V i and U i Multiply elements by element to get the final output O i .

[0059] The 3D weight module dynamically adjusts the weight of each neuron and the importance of different regions in the feature map to generate high-resolution feature maps and more accurately capture fine-grained features. Specifically:

[0060] (1) Define the energy function

[0061] Calculate the energy function between neurons, identify important feature areas in the image, and define the energy function of each neuron based on the linear separability between the target neuron and other neurons :

[0062] ;

[0063] In the formula, and yes t → x i The linear transformation of t Yes Input Oi-1 ∈R C×W×H The target neurons, x i Yes Input O i-1 ∈R C×W×H Other neurons in a single channel, i is the spatial dimension index, w t and b t are weights and biases:

[0064] , Indicates the reference value, here we take = −1 ensures that the target neuron 𝑡 is as close to the value 1 as possible, while the non-target neurons The output of approaches the value -1 and separates from the target neuron.

[0065] Will y t and y 0 is binarized, we get:

[0066] ;

[0067] Perform regularization operation to obtain the final energy function:

[0068] ;

[0069] in represents the regularization strength;

[0070] In each channel there is H × W Energy function, solving for weights w t and bias b t :

[0071] ;

[0072]

[0073] The above formula is solved in the case of a single channel.

[0074] To reduce the number of and Iterative calculation of and Represent the mean and variance respectively, and get the minimum energy:

[0075] ;

[0076] The formula shows that energy F t The lower the neuron t The greater the difference from surrounding neurons, the more important it is for visual processing.

[0077] (2) Feature refinement process

[0078] Use the sigmoid function to compress the input real value to between 0 and 1 to achieve feature refinement:

[0079]

[0080] In the formula, E All F t Grouping on channel and spatial dimensions.

[0081] In the parameter-free Swift 3D Weighted Attention Mechanism (PS3D):

[0082] The features extracted by each 3D weight module H i With input O i-1 Perform residual connection output U i ,Will H i After an activation function symmetric to the origin Get attention V i .

[0083] at last V i and U i Multiply elements by element to get the final output O i .

[0084] The final output of the PS3D module is represented as O i :

[0085] ; , ; ;

[0086] In the formula, is the element-wise multiplication of features, is the element-wise sum of features, F C,W,H It is a 3D weight module.

[0087] Embodiment 5,

[0088] This embodiment further limits the embodiment 1 and further introduces the data processing in the super-resolution iterative update module.

[0089] like Figure 1 As shown, the GRU in the figure represents the gated activation unit (ConvGRU) containing 3 inputs ( , , ), 2 outputs ( , ).

[0090] Recursively update the gated activation unit module to obtain the updated feature map , updated hidden state , updated intermediate feature map for generating super-resolution feature map .

[0091] like Figure 1 and 4 As shown, the update operator of the gated activation unit (ConvGRU) changes from Start estimating optical flow values , and output the final optical flow estimation map through N iterations.

[0092] Figure 4 As shown, the following network module (CNMKS) is composed of the proposed multi-scale kernel selective network (MKS) and a convolution (3×3) to process information (Context), and then a residual module is used to generate context features ( ), which is used to obtain the multi-scale hidden states of the gated activation unit (ConvGRU).

[0093] like Figure 1 As shown in the figure, the super-resolution iterative update module takes the last optical flow value (Flow) as input, and runs two branches in parallel. One branch generates an intermediate feature map through the parameter-free Swift 3D weighted attention mechanism (PS3D). , used as the input of the updated intermediate features and related features (Corr) before the gated activation unit (ConvGRU); one branch is combined with the related features after two convolutions (7×7, 3×3), and after the combination, it undergoes a cost space lookup operation ( Figure 1 "L" is used to indicate this operation); Figure 1 The symbol “C” in represents the Concat operation.

[0094] like Figure 4 As shown, the correlation feature (Corr) is processed by two convolutions (1×1, 3×3). After each convolution layer, the feature information from the previous optical flow value is injected and the feature information is output. .

[0095] like Figure 5 The figure shows a comparison of the optical flow estimation results obtained by using different optical flow estimation methods and the fine optical flow estimation system of the present invention, where Ours represents the optical flow estimation result obtained by the fine optical flow estimation system of the present invention. It can be seen from the figure that the optical flow estimation result obtained by the fine optical flow estimation system of the present invention is lower than the optical flow estimation result obtained by the benchmark method and the endpoint error value is lower than that of the benchmark method RAFT. The specific experimental data is a reduction of 14.93%. The optical flow estimation result obtained by the fine optical flow estimation system of the present invention is close to the true optical flow value (GT). The benchmark method RAFT misestimates the motion trend and cannot clearly show the change of optical flow. The optical flow estimation result obtained by the fine optical flow estimation system of the present invention retains better clear texture at the end of the tail and the overall edge. The fine optical flow estimation system of the present invention is more prominent in capturing and processing details.

Claims

1. A fine optical flow estimation system integrating multi-scale kernel selection and super-resolution technology, characterized by: The system includes a feature extraction network module, a super-resolution iterative update module and a context network module; The feature extraction network module is used to extract features of two adjacent frames of images, and input the extracted relevant features into the super-resolution iterative update module; The context network module is used to extract the context features of the previous frame of the image between two adjacent frames of the image, and input the extracted context features into the super-resolution iterative update module; The context network module includes a residual network, a multi-scale kernel selective network, a convolution layer and a residual connection layer from input to output. The residual connection layer adds the output of the convolution layer to the output of the residual network to obtain context features. The residual network includes a 7*7 convolution layer, six residual units and a 3*3 convolution layer from input to output. The number of channels of the six residual units is 64, 64, 96, 96, 128 and 128 respectively. The multi-scale kernel selective network includes a Fuse part and a Select part; The fuse part receives and combines the feature information from the second residual unit, the fourth residual unit, the sixth residual unit, and the 3*3 convolution layer to obtain multi-scale features, performs global average pooling and maximum average pooling on the multi-scale features in the spatial dimension to obtain combined features, performs convolution kernel and activation operations on the combined features to obtain compact features, and inputs the compact features into four parallel convolution layers to obtain four descriptors corresponding to the second residual unit, the fourth residual unit, the sixth residual unit, and the 3*3 convolution layer. V 1. V 2. V 3 and V 4; Select part V 1. V 2. V 3 and V 4 Perform activation operation and obtain activation coefficients respectively s 1. s 2. s 3 and s 4. Use activation coefficients to V 1. V 2. V 3 and V 4. Calibrate and then fuse the calibrated descriptors to obtain context features processed by the kernel selection mechanism; The super-resolution iterative update module is based on the iterative update module in the baseline network RAFT, and adds a parameter-free Swift 3D weighted attention mechanism in the path before the context feature input gated activation unit. The super-resolution iterative update module fuses the relevant features and context features to obtain an optical flow vector map output with super-resolution.

2. The fine optical flow estimation system integrating multi-scale kernel selection and super-resolution technology according to claim 1, characterized in that: The feature extraction network module is composed of the feature extraction network and the visual similarity calculation network in the benchmark network RAFT. The feature extraction network performs feature extraction on two input images, and inputs the extracted features into the visual similarity calculation network. In the visual similarity calculation network, dot product calculation is performed on the features extracted from the two images to finally obtain relevant features.

3. The fine optical flow estimation system integrating multi-scale kernel selection and super-resolution technology according to claim 2, characterized in that: The parameter-free Swift 3D weighted attention mechanism is specifically: the features extracted by the 3D weight module H i With input O i-1 Perform residual connection output U i , and then H i After an activation function symmetric to the origin Get attention V i ,at last V i and U i Multiply elements by element to get the final output O i .

Images