Optical flow estimation model training method, optical flow estimation method, device and equipment

Abstract

The invention discloses an optical flow estimation model training method, an optical flow estimation method, an optical flow estimation device and optical flow estimation equipment. The method comprises the following steps of: respectively extracting a plurality of front frame feature images with different scales of a front frame training image in a training image pair and a plurality of rear frame feature images with different scales of a rear frame training image in the training image pair by utilizing an optical flow estimation model to be trained; extracting a plurality of attention feature maps of different scales of the previous frame of training image; according to the plurality of front frame feature images with different scales, the plurality of rear frame feature images with different scales and the plurality of attention feature images with different scales, predicting to obtain a plurality of predicted optical flow graphs with different scales; and constructing a target loss function according to the plurality of predicted optical flow diagrams of different scales and the real optical flow field, and performing model training on a to-be-trained optical flow estimation model by using the target loss function to obtain a trained optical flow estimation model. The technical problem that errors exist when a traditional optical flow estimation algorithm processes a complex scene or an image which moves violently is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optical flow estimation model training method, an optical flow estimation method, a device and equipment. Background Art

[0002] Optical flow estimation is an important task in the field of computer vision, which aims to analyze the motion trajectory of objects in image sequences. The optical flow field represents the speed and direction of each pixel in the image between two frames. Traditional optical flow estimation algorithms mainly include region-based methods and matching-based methods, which often have certain errors when processing complex scenes or images with intense motion. Summary of the invention

[0003] The main technical problem solved by the present invention is that the traditional optical flow estimation algorithm has errors when processing complex scenes or images with violent motion.

[0004] According to the first aspect, an embodiment provides an optical flow estimation model training method, including:

[0005] Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images;

[0006] Using a feature extraction network in a preset optical flow estimation model to be trained, respectively extract a plurality of front frame feature images of different scales of the front frame training image in the training image pair and a plurality of rear frame feature images of different scales of the rear frame training image in the training image pair;

[0007] Extracting a plurality of attention feature maps of different scales of the previous frame training image using the attention module in the optical flow estimation model to be trained;

[0008] Predicting a plurality of predicted optical flow maps of different scales according to the plurality of front frame feature images of different scales, the plurality of rear frame feature images of different scales, and the plurality of attention feature maps of different scales;

[0009] A target loss function is constructed according to the multiple predicted optical flow maps of different scales and the preset real optical flow field, and the optical flow estimation model to be trained is trained using the target loss function to obtain a trained optical flow estimation model.

[0010] In some embodiments, the plurality of front frame feature images of different scales include N front frame feature images of different scales, the plurality of rear frame feature images of different scales include N rear frame feature images of different scales, and the plurality of attention feature maps of different scales include N attention feature maps of different scales, wherein the scale decreases from the front frame feature image of the Nth scale to the front frame feature image of the 1st scale, the front frame feature image of the nth scale, the rear frame feature image of the nth scale, and the attention feature map of the nth scale are the same in scale, and n=1,...,N, where N is an integer not less than 2;

[0011] The step of predicting a plurality of predicted optical flow maps of different scales based on the plurality of front frame feature images of different scales, the plurality of rear frame feature images of different scales, and the plurality of attention feature maps of different scales comprises:

[0012] Generate a related feature map according to the previous frame feature image of the first scale and the next frame feature image of the first scale, and perform optical flow prediction according to the related feature map to generate a predicted optical flow map of the first scale;

[0013] For the k-th scale front frame feature image and the k-th scale rear frame feature image, upsample the k-1-th scale predicted optical flow map to the k-th scale to obtain the k-th scale upsampled optical flow map, and obtain the k-th scale predicted front frame feature image according to the k-th scale upsampled optical flow map and the k-th scale rear frame feature image, calculate the reprojection error between the k-th scale predicted front frame feature image and the k-th scale front frame feature image, and obtain the k-th scale error optical flow estimation; wherein, k = 2, ..., N;

[0014] The error optical flow estimation of the k-th scale, the attention feature map of the k-th scale, the upsampled optical flow map of the k-th scale and the previous frame feature image of the k-th scale are fused to obtain a predicted optical flow map of the k-th scale.

[0015] In some embodiments, the objective loss function is expressed as:

[0016]

[0017] Among them, loss represents the target loss function, N represents the number of scales, n represents the index of the scale, α represents the hyperparameter, L1 represents the L1 loss function, and flow n Represents the predicted optical flow map of the nth scale, GT n Represents the preset real optical flow field corresponding to the nth scale.

[0018] In some embodiments, obtaining the predicted previous frame feature image at the kth scale according to the upsampled optical flow map at the kth scale and the subsequent frame feature image at the kth scale includes:

[0019] Reframe1[:,i,j]=frame2[:,i+u,j+v]

[0020] where u=flow[0,i,j],v=flow[1,i,j]

[0021] Among them, Reframe1[:,i,j] represents the predicted previous frame feature image of the kth scale, frame2 represents the subsequent frame feature image of the kth scale, i represents the position of the pixel in the subsequent frame feature image of the kth scale on the x-axis, j represents the position of the pixel in the subsequent frame feature image of the kth scale on the y-axis, u and flow[0,i,j] represent the optical flow of the pixel at position (i,j) on the x-axis, and v and flow[1,i,j] represent the optical flow of the pixel at position (i,j) on the y-axis.

[0022] In some embodiments, the expression of the predicted optical flow map at the first scale is:

[0023]

[0024] Wherein, flow0 represents the predicted optical flow map of the first scale, C flow represents the relevant feature map, u1 represents the optical flow range of the pixels in the relevant feature map in the x direction, v1 represents the optical flow range of the pixels in the relevant feature map in the y direction, H represents the height of the relevant feature map, W represents the width of the relevant feature map, R H×W×2 Represents the dimension of the predicted optical flow map of the first scale.

[0025] In some embodiments, the expression of the relevant feature map is:

[0026]

[0027] Among them, C flow represents the related feature map, F1 represents the position of any pixel in the feature image of the previous frame of the first scale, F2 represents the position of any pixel in the feature image of the next frame of the first scale, T represents the transposition symbol, H represents the height of the related feature map, W represents the width of the related feature map, R H×W×H×W Represents the dimension of the related feature map.

[0028] According to the second aspect, an embodiment provides an optical flow estimation method, including:

[0029] Acquire a pair of images to be processed of a target object in a preset motion scene, wherein the pair of images to be processed includes two adjacent frames of images to be processed;

[0030] The image pair to be processed is input into an optical flow estimation model trained by an optical flow estimation model training method to obtain optical flow estimation information of the target object.

[0031] According to the third aspect, an embodiment provides an optical flow estimation device, including:

[0032] An image acquisition module, used to acquire a pair of images to be processed of a target object in a preset motion scene, wherein the pair of images to be processed includes two adjacent frames of images to be processed;

[0033] The optical flow estimation module is used to input the image pair to be processed into the optical flow estimation model trained by the optical flow estimation model training method to obtain the optical flow estimation information of the target object.

[0034] According to a fourth aspect, an embodiment provides an optical flow estimation model training device, including:

[0035] Memory, used to store programs;

[0036] A processor is used to implement an optical flow estimation model training method by executing a program stored in the memory.

[0037] According to the fifth aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement an optical flow estimation model training method.

[0038] According to the optical flow estimation model training method, optical flow estimation method, optical flow estimation device, optical flow estimation model training equipment and computer-readable storage medium of the above-mentioned embodiments, since the attention module in the optical flow estimation model to be trained is used to extract multiple attention feature maps of different scales of the previous frame training image, the model improves the degree of attention to the target area and overcomes the limitations of the traditional optical flow estimation algorithm. The optical flow estimation model to be trained is trained according to the predicted multiple predicted optical flow maps of different scales. The predicted optical flow maps of different scales can realize the role of gradually correcting errors during the model training process, thereby obtaining a trained optical flow estimation model. The predicted optical flow maps of different scales can assist the model in predicting the optical flow position from different scales, thereby providing a more accurate and more robust optical flow estimation result, and improving the robustness and accuracy of subsequent optical flow estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of an optical flow estimation model training method according to an embodiment of the present application;

[0040] Figure 2 A flowchart of an embodiment of predicting a plurality of predicted optical flow maps of different scales based on a plurality of front frame feature images of different scales, a plurality of rear frame feature images of different scales, and a plurality of attention feature maps of different scales;

[0041] Figure 3 A structural block diagram of an embodiment of predicting a plurality of predicted optical flow maps of different scales based on a plurality of front frame feature images of different scales, a plurality of rear frame feature images of different scales, and a plurality of attention feature maps of different scales;

[0042] Figure 4 A flowchart of an optical flow estimation method according to an embodiment;

[0043] Figure 5 A schematic diagram of the structure of an optical flow estimation device according to an embodiment. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0045] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0046] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0047] Please refer to Figure 1In an embodiment of the present invention, a method for training an optical flow estimation model is provided, including steps S10 to S50, which are described in detail below.

[0048] Step S10: Obtain a training image sequence, and obtain a training image pair from the training image sequence.

[0049] In some embodiments, the training image sequence includes a plurality of training image pairs, and the training image pair includes two adjacent training image frames, namely, a preceding training image frame and a succeeding training image frame.

[0050] Step S20: using the feature extraction network in the preset optical flow estimation model to be trained, respectively extracting a plurality of front frame feature images of different scales of the front frame training image in the training image pair and a plurality of rear frame feature images of different scales of the rear frame training image in the training image pair.

[0051] In some embodiments, for the front frame training image and the back frame training image in the training image pair, a plurality of front frame feature images and back frame feature images of different scales are respectively extracted through a feature extraction network with shared weights, wherein the feature extraction network may be a convolutional neural network, and the shared weights mean that the network weights of the feature extraction network remain unchanged when the front frame feature image of the front frame training image and the back frame feature image of the back frame training image are respectively extracted. Therefore, a front frame feature image of 1 / 4 scale of the front frame training image, a front frame feature image of 1 / 8 scale of the front frame training image, and a front frame feature image of 1 / 16 scale of the front frame training image, as well as a back frame feature image of 1 / 4 scale of the back frame training image, a back frame feature image of 1 / 8 scale of the back frame training image, and a back frame feature image of 1 / 16 scale of the back frame training image can be obtained.

[0052] Step S30: Utilize the attention module in the optical flow estimation model to be trained to extract multiple attention feature maps of different scales of the previous frame training image.

[0053] In some embodiments, the attention module in the trained optical flow estimation model is used to extract a 1 / 4 scale attention feature map, a 1 / 8 scale attention feature map, and a 1 / 16 scale attention feature map of the previous frame training image.

[0054] In some embodiments, global maximum pooling, convolution, and activation functions can be used on the channel dimension of the previous frame training image to output multiple attention feature maps of different scales, so that the model can pay more attention to areas with intense motion or larger gradients, which can effectively help the model pay more attention to important information in the image, thereby improving the robustness of optical flow estimation.

[0055] Step S40: Predicting a plurality of predicted optical flow maps of different scales based on a plurality of front frame feature images of different scales, a plurality of rear frame feature images of different scales, and a plurality of attention feature maps of different scales.

[0056] In some embodiments, the multiple front-frame feature images of different scales include N front-frame feature images of different scales, the multiple back-frame feature images of different scales include N back-frame feature images of different scales, and the multiple attention feature maps of different scales include N attention feature maps of different scales, wherein the scale decreases from the front-frame feature image of the Nth scale to the front-frame feature image of the 1st scale, the scales of the front-frame feature image of the nth scale, the back-frame feature image of the nth scale, and the attention feature map of the nth scale are the same, and n=1,...,N, where N is an integer not less than 2.

[0057] For example, when N=3, the multiple front frame feature images of different scales include three front frame feature images of different scales, namely, a front frame feature image of 1 / 16 scale, a front frame feature image of 1 / 8 scale, and a front frame feature image of 1 / 4 scale; the multiple back frame feature images of different scales include three back frame feature images of different scales, namely, a back frame feature image of 1 / 16 scale, a back frame feature image of 1 / 8 scale, and a back frame feature image of 1 / 4 scale; the multiple attention feature maps of different scales include three attention feature maps of different scales, namely, an attention feature map of 1 / 16 scale, an attention feature map of 1 / 8 scale, and an attention feature map of 1 / 4 scale.

[0058] Please refer to Figure 2 and Figure 3 In some embodiments, step S40 predicts multiple predicted optical flow maps of different scales based on multiple front frame feature images of different scales, multiple back frame feature images of different scales, and multiple attention feature maps of different scales, including steps S41 to S43, which are described in detail below.

[0059] Step S41: generating a related feature map according to the previous frame feature image of the first scale and the next frame feature image of the first scale, and performing optical flow prediction according to the related feature map to generate a predicted optical flow map of the first scale.

[0060] In some embodiments, the expression of the predicted optical flow map at the first scale is:

[0061]

[0062] Among them, flow0 represents the predicted optical flow map of the first scale, C flowrepresents the relevant feature map, u1 represents the optical flow range of the pixels in the relevant feature map in the x direction, v1 represents the optical flow range of the pixels in the relevant feature map in the y direction, H represents the height of the relevant feature map, W represents the width of the relevant feature map, and R H×W×2 Indicates the dimension of the predicted optical flow map at the first scale.

[0063] In some embodiments, the expression of the relevant feature map is:

[0064]

[0065] Among them, C flow represents the relevant feature map, F1 represents the position of any pixel in the feature image of the previous frame of the first scale, F2 represents the position of any pixel in the feature image of the next frame of the first scale, T represents the transposition symbol, H represents the height of the relevant feature map, W represents the width of the relevant feature map, and R H×W×H×W Represents the dimension of the relevant feature map.

[0066] In some embodiments, the above method of generating the predicted optical flow map at the first scale is differentiable, which means that backpropagation can be performed in an end-to-end training process.

[0067] Step S42: For the front frame feature image of the k-th scale and the rear frame feature image of the k-th scale, upsample the predicted optical flow map of the k-1-th scale to the k-th scale to obtain the upsampled optical flow map of the k-th scale, and obtain the predicted front frame feature image of the k-th scale according to the upsampled optical flow map of the k-th scale and the rear frame feature image of the k-th scale, calculate the reprojection error between the predicted front frame feature image of the k-th scale and the front frame feature image of the k-th scale, and obtain the error optical flow estimation of the k-th scale.

[0068] In some embodiments, obtaining a predicted previous frame feature image at the kth scale according to the upsampled optical flow map at the kth scale and the subsequent frame feature image at the kth scale includes:

[0069] Reframe1[:,i,j]=frame2[:,i+u,j+v]

[0070] where u=flow[0,i,j],v=flow[1,i,j]

[0071] Among them, Reframe1[:,i,j] represents the predicted previous frame feature image of the kth scale, frame2 represents the next frame feature image of the kth scale, i represents the position of the pixel on the x-axis in the next frame feature image of the kth scale, j represents the position of the pixel on the y-axis in the next frame feature image of the kth scale, u and flow[0,i,j] represent the optical flow of the pixel at position (i,j) on the x-axis, and v and flow[1,i,j] represent the optical flow of the pixel at position (i,j) on the y-axis.

[0072] In some embodiments, the reprojection error between the predicted previous frame feature image at the k-th scale and the previous frame feature image at the k-th scale is calculated by the following formula to obtain the error optical flow estimate at the k-th scale:

[0073] Error = frame1-Reframe1

[0074] Among them, Error represents the error optical flow estimation of the k-th scale, Reframe1 represents the predicted previous frame feature image of the k-th scale, and frame1 represents the previous frame feature image of the k-th scale.

[0075] In some embodiments, k=2, ..., N. Upsampling the predicted optical flow map at the k-1th scale to the kth scale can continuously obtain refined optical flow during the process of optical flow propagation.

[0076] Step S43: Fusing the error optical flow estimation of the k-th scale, the attention feature map of the k-th scale, the upsampled optical flow map of the k-th scale, and the previous frame feature image of the k-th scale to obtain a predicted optical flow map of the k-th scale.

[0077] In some embodiments, the predicted optical flow map at the kth scale is obtained by the following formula:

[0078] F k =concat(frame1,Error,F k atten ,flow uk )

[0079] Among them, F k represents the predicted optical flow map of the k-th scale, frame1 represents the feature image of the previous frame of the k-th scale, Error represents the error optical flow estimation of the k-th scale, and F k atten represents the attention feature map of the kth scale, flow uk Represents the upsampled optical flow map at the kth scale.

[0080] Step S50: construct a target loss function according to a plurality of predicted optical flow images of different scales and a preset real optical flow field, and use the target loss function to perform model training on the optical flow estimation model to be trained to obtain a trained optical flow estimation model.

[0081] In some embodiments, the objective loss function is expressed as:

[0082]

[0083] Among them, loss represents the target loss function, N represents the number of scales, n represents the index of the scale, α represents the hyperparameter, L1 represents the L1 loss function, and flow n Represents the predicted optical flow map of the nth scale, GT n Represents the preset real optical flow field corresponding to the nth scale.

[0084] In some embodiments, the target loss function constructed by multiple predicted optical flow maps of different scales can timely correct the optical flow offsets at different scales during the training process to improve the accuracy of optical flow estimation.

[0085] Please refer to Figure 4 In some embodiments, the present application provides an optical flow estimation method, including steps S60 to S70, which are described in detail below.

[0086] Step S60: obtaining a pair of to-be-processed images of the target object in a preset motion scene.

[0087] In some embodiments, the pair of images to be processed includes two adjacent frames of images to be processed.

[0088] Step S70: inputting the image pair to be processed into the optical flow estimation model trained by the optical flow estimation model training method to obtain the optical flow estimation information of the target object.

[0089] In some embodiments, the specific implementation of the optical flow estimation model training method is as follows:

[0090] Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images;

[0091] A feature extraction network in a preset optical flow estimation model to be trained is used to respectively extract a plurality of front frame feature images of different scales of a front frame training image in a training image pair and a plurality of rear frame feature images of different scales of a rear frame training image in a training image pair;

[0092] The attention module in the optical flow estimation model to be trained is used to extract multiple attention feature maps of different scales of the previous frame training image;

[0093] According to multiple front frame feature images of different scales, multiple back frame feature images of different scales, and multiple attention feature maps of different scales, multiple predicted optical flow maps of different scales are predicted;

[0094] A target loss function is constructed according to multiple predicted optical flow maps of different scales and a preset real optical flow field. The target loss function is used to train the optical flow estimation model to be trained to obtain a trained optical flow estimation model.

[0095] Please refer to Figure 5 The present application provides an optical flow estimation device, including an image acquisition module 10 and an optical flow estimation module 20, which are described in detail below.

[0096] The image acquisition module 10 is used to acquire a pair of images to be processed of a target object in a preset motion scene.

[0097] In some embodiments, the pair of images to be processed includes two adjacent frames of images to be processed.

[0098] The optical flow estimation module 20 is used to input the image pair to be processed into the optical flow estimation model trained by the optical flow estimation model training method to obtain the optical flow estimation information of the target object.

[0099] In some embodiments, the specific implementation of the optical flow estimation model training method is as follows:

[0100] Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images;

[0101] A feature extraction network in a preset optical flow estimation model to be trained is used to respectively extract a plurality of front frame feature images of different scales of a front frame training image in a training image pair and a plurality of rear frame feature images of different scales of a rear frame training image in a training image pair;

[0102] The attention module in the optical flow estimation model to be trained is used to extract multiple attention feature maps of different scales of the previous frame training image;

[0103] According to multiple front frame feature images of different scales, multiple back frame feature images of different scales, and multiple attention feature maps of different scales, multiple predicted optical flow maps of different scales are predicted;

[0104] A target loss function is constructed according to multiple predicted optical flow maps of different scales and a preset real optical flow field. The target loss function is used to train the optical flow estimation model to be trained to obtain a trained optical flow estimation model.

[0105] In some embodiments, the present application provides an optical flow estimation model training device, including: a memory for storing programs; and a processor for implementing an optical flow estimation model training method by executing the programs stored in the memory.

[0106] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0107] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A method for training an optical flow estimation model, characterized in that: include: Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images; Using a feature extraction network in a preset optical flow estimation model to be trained, respectively extract a plurality of front frame feature images of different scales of the front frame training image in the training image pair and a plurality of rear frame feature images of different scales of the rear frame training image in the training image pair; Extracting a plurality of attention feature maps of different scales of the previous frame training image using the attention module in the optical flow estimation model to be trained; Predicting a plurality of predicted optical flow maps of different scales according to the plurality of front frame feature images of different scales, the plurality of rear frame feature images of different scales, and the plurality of attention feature maps of different scales; A target loss function is constructed according to the multiple predicted optical flow maps of different scales and the preset real optical flow field, and the optical flow estimation model to be trained is trained using the target loss function to obtain a trained optical flow estimation model.

2. The optical flow estimation model training method according to claim 1, characterized in that: The plurality of front frame feature images of different scales include N front frame feature images of different scales, the plurality of rear frame feature images of different scales include N rear frame feature images of different scales, and the plurality of attention feature maps of different scales include N attention feature maps of different scales, wherein the scales of the front frame feature image of the Nth scale to the front frame feature image of the 1st scale decrease, the front frame feature image of the nth scale, the rear frame feature image of the nth scale, and the attention feature map of the nth scale are the same, n=1,...,N, and N is an integer not less than 2; The step of predicting a plurality of predicted optical flow maps of different scales based on the plurality of front frame feature images of different scales, the plurality of rear frame feature images of different scales, and the plurality of attention feature maps of different scales comprises: Generate a related feature map according to the previous frame feature image of the first scale and the next frame feature image of the first scale, and perform optical flow prediction according to the related feature map to generate a predicted optical flow map of the first scale; For the k-th scale front frame feature image and the k-th scale rear frame feature image, upsample the k-1-th scale predicted optical flow map to the k-th scale to obtain the k-th scale upsampled optical flow map, and obtain the k-th scale predicted front frame feature image according to the k-th scale upsampled optical flow map and the k-th scale rear frame feature image, calculate the reprojection error between the k-th scale predicted front frame feature image and the k-th scale front frame feature image, and obtain the k-th scale error optical flow estimation; wherein, k = 2, ..., N; The error optical flow estimation of the k-th scale, the attention feature map of the k-th scale, the upsampled optical flow map of the k-th scale and the previous frame feature image of the k-th scale are fused to obtain a predicted optical flow map of the k-th scale.

3. The optical flow estimation model training method according to claim 2, characterized in that: The expression of the objective loss function is: Among them, loss represents the target loss function, N represents the number of scales, n represents the index of the scale, α represents the hyperparameter, L1 represents the L1 loss function, and flow n Represents the predicted optical flow map of the nth scale, GT n Represents the preset real optical flow field corresponding to the nth scale.

4. The optical flow estimation model training method according to claim 2, characterized in that: The step of obtaining a predicted front frame feature image at the kth scale according to the upsampled optical flow map at the kth scale and the rear frame feature image at the kth scale comprises: Reframe1[:,i,j]=frame2[:,i+u,j+v] where u=flow[0,i,j],v=flow[1,i,j] Among them, Reframe1[:,i,j] represents the predicted previous frame feature image of the kth scale, frame2 represents the subsequent frame feature image of the kth scale, i represents the position of the pixel in the subsequent frame feature image of the kth scale on the x-axis, j represents the position of the pixel in the subsequent frame feature image of the kth scale on the y-axis, u and flow[0,i,j] represent the optical flow of the pixel at position (i,j) on the x-axis, and v and flow[1,i,j] represent the optical flow of the pixel at position (i,j) on the y-axis.

5. The optical flow estimation model training method according to claim 2, characterized in that: The expression of the predicted optical flow map of the first scale is: Wherein, flow0 represents the predicted optical flow map of the first scale, C flow represents the relevant feature map, u1 represents the optical flow range of the pixels in the relevant feature map in the x direction, v1 represents the optical flow range of the pixels in the relevant feature map in the y direction, H represents the height of the relevant feature map, W represents the width of the relevant feature map, R H×W×2 Represents the dimension of the predicted optical flow map of the first scale.

6. The optical flow estimation model training method according to claim 2, characterized in that: The expression of the relevant feature map is: Among them, C flow represents the related feature map, F1 represents the position of any pixel in the feature image of the previous frame of the first scale, F2 represents the position of any pixel in the feature image of the next frame of the first scale, T represents the transposition symbol, H represents the height of the related feature map, W represents the width of the related feature map, R H×W×H×W Represents the dimension of the related feature map.

7. An optical flow estimation method, characterized in that: include: Acquire a pair of images to be processed of a target object in a preset motion scene, wherein the pair of images to be processed includes two adjacent frames of images to be processed; The image pair to be processed is input into an optical flow estimation model trained by the optical flow estimation model training method according to any one of claims 1 to 6 to obtain optical flow estimation information of the target object.

8. An optical flow estimation device, characterized in that: include: An image acquisition module, used to acquire a pair of images to be processed of a target object in a preset motion scene, wherein the pair of images to be processed includes two adjacent frames of images to be processed; The optical flow estimation module is used to input the image pair to be processed into the optical flow estimation model trained by the optical flow estimation model training method according to any one of claims 1 to 6 to obtain the optical flow estimation information of the target object.

9. An optical flow estimation model training device, characterized in that: include: Memory, used to store programs; A processor, configured to implement the optical flow estimation model training method according to any one of claims 1 to 6 by executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which can be executed by a processor to implement the optical flow estimation model training method according to any one of claims 1 to 6.

Images