Optical flow estimation model training method, optical flow estimation method, device and equipment

Abstract

A method for training an optical flow estimation model, an optical flow estimation method, an apparatus, and an apparatus, comprising: using the optical flow estimation model to be trained to extract multiple front-frame feature images of different scales from the front-frame training image in a training image pair and multiple back-frame feature images of different scales from the back-frame training image in the training image pair; extracting multiple attention feature maps of different scales from the front-frame training image; predicting multiple predicted optical flow maps of different scales based on the multiple front-frame feature images of different scales, the multiple back-frame feature images of different scales, and the multiple attention feature maps of different scales; constructing a target loss function based on the multiple predicted optical flow maps of different scales and the true optical flow field, and using the target loss function to train the optical flow estimation model to be trained to obtain a trained optical flow estimation model. This method solves the technical problem that traditional optical flow estimation algorithms have errors when processing complex scenes or images with intense motion.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optical flow estimation model training method, an optical flow estimation method, a device and equipment. Background Art

[0002] Optical flow estimation is a key task in computer vision, aiming to analyze the motion trajectory of objects in image sequences. The optical flow field represents the speed and direction of each pixel's movement between two frames. Traditional optical flow estimation algorithms, including region-based and matching-based methods, often suffer from errors when processing complex scenes or images with intense motion. Summary of the Invention

[0003] The main technical problem solved by the present invention is that traditional optical flow estimation algorithms have errors when processing complex scenes or images with intense motion.

[0004] According to the first aspect, an embodiment provides an optical flow estimation model training method, including:

[0005] Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images;

[0006] Extracting a plurality of front-frame feature images of different scales of the front-frame training image in the training image pair and a plurality of rear-frame feature images of different scales of the rear-frame training image in the training image pair using a feature extraction network in a preset optical flow estimation model to be trained;

[0007] Extracting a plurality of attention feature maps of different scales of the previous frame training image using the attention module in the optical flow estimation model to be trained;

[0008] Predicting a plurality of predicted optical flow maps of different scales based on the plurality of front frame feature images of different scales, the plurality of rear frame feature images of different scales, and the plurality of attention feature maps of different scales;

[0009] A target loss function is constructed according to the multiple predicted optical flow maps of different scales and the preset real optical flow field, and the optical flow estimation model to be trained is trained using the target loss function to obtain a trained optical flow estimation model.

[0010] In some embodiments, the plurality of front frame feature images of different scales include N front frame feature images of different scales, the plurality of rear frame feature images of different scales include N rear frame feature images of different scales, and the plurality of attention feature maps of different scales include N attention feature maps of different scales, wherein the scale decreases from the front frame feature image of the Nth scale to the front frame feature image of the 1st scale, and the scale of the front frame feature image of the nth scale, the rear frame feature image of the nth scale, and the attention feature map of the nth scale are the same, and n=1,...,N, where N is an integer not less than 2;

[0011] The step of predicting a plurality of predicted optical flow maps of different scales based on the plurality of front frame feature images of different scales, the plurality of rear frame feature images of different scales, and the plurality of attention feature maps of different scales includes:

[0012] Generating a correlation feature map according to the first scale of the previous frame feature image and the first scale of the next frame feature image, and performing optical flow prediction according to the correlation feature map to generate a predicted optical flow map of the first scale;

[0013] For the k-th scale front frame feature image and the k-th scale back frame feature image, upsample the k-1-th scale predicted optical flow map to the k-th scale to obtain the k-th scale upsampled optical flow map, and obtain the k-th scale predicted front frame feature image based on the k-th scale upsampled optical flow map and the k-th scale back frame feature image, calculate the reprojection error between the k-th scale predicted front frame feature image and the k-th scale front frame feature image, and obtain the k-th scale error optical flow estimate; wherein k = 2, ..., N;

[0014] The error optical flow estimation of the k-th scale, the attention feature map of the k-th scale, the upsampled optical flow map of the k-th scale and the previous frame feature image of the k-th scale are fused to obtain a predicted optical flow map of the k-th scale.

[0015] In some embodiments, the objective loss function is expressed as:

[0016]

[0017] Among them, loss represents the target loss function, N represents the number of scales, n represents the index of the scale, α represents the hyperparameter, L1 represents the L1 loss function, flow n Represents the predicted optical flow map of the nth scale, GT n Represents the preset real optical flow field corresponding to the nth scale.

[0018] In some embodiments, obtaining the predicted previous frame feature image at the kth scale based on the upsampled optical flow map at the kth scale and the subsequent frame feature image at the kth scale includes:

[0019] Reframe1[:,i,j]=frame2[:,i+u,j+v]

[0020] where u=flow[0,i,j],v=flow[1,i,j]

[0021] Among them, Reframe1[:,i,j] represents the predicted previous frame feature image of the kth scale, frame2 represents the post-frame feature image of the kth scale, i represents the position of the pixel in the post-frame feature image of the kth scale on the x-axis, j represents the position of the pixel in the post-frame feature image of the kth scale on the y-axis, u and flow[0,i,j] represent the optical flow of the pixel at position (i,j) on the x-axis, and v and flow[1,i,j] represent the optical flow of the pixel at position (i,j) on the y-axis.

[0022] In some embodiments, the expression of the predicted optical flow map at the first scale is:

[0023]

[0024] Among them, flow0 represents the predicted optical flow map of the first scale, C flow represents the related feature map, u1 represents the optical flow range of the pixels in the related feature map in the x direction, v1 represents the optical flow range of the pixels in the related feature map in the y direction, H represents the height of the related feature map, W represents the width of the related feature map, R H×W×2 Indicates the dimension of the predicted optical flow map of the first scale.

[0025] In some embodiments, the expression of the correlation feature map is:

[0026]

[0027] Among them, C flow represents the related feature map, F1 represents the position of any pixel in the feature image of the previous frame of the first scale, F2 represents the position of any pixel in the feature image of the next frame of the first scale, T represents the transposition symbol, H represents the height of the related feature map, W represents the width of the related feature map, R H×W×H×W Represents the dimension of the related feature map.

[0028] According to the second aspect, an embodiment provides an optical flow estimation method, including:

[0029] Acquire a pair of to-be-processed images of a target object in a preset motion scene, wherein the pair of to-be-processed images includes two adjacent frames of to-be-processed images;

[0030] The image pair to be processed is input into an optical flow estimation model trained by an optical flow estimation model training method to obtain optical flow estimation information of the target object.

[0031] According to the third aspect, an embodiment provides an optical flow estimation device, including:

[0032] An image acquisition module is used to acquire a pair of images to be processed of a target object in a preset motion scene, wherein the pair of images to be processed includes two adjacent frames of images to be processed;

[0033] The optical flow estimation module is used to input the image pair to be processed into the optical flow estimation model trained by the optical flow estimation model training method to obtain the optical flow estimation information of the target object.

[0034] According to a fourth aspect, an embodiment provides an optical flow estimation model training device, including:

[0035] Memory, used to store programs;

[0036] A processor is used to implement an optical flow estimation model training method by executing a program stored in the memory.

[0037] According to the fifth aspect, an embodiment provides a computer-readable storage medium, on which a program is stored. The program can be executed by a processor to implement an optical flow estimation model training method.

[0038] According to the optical flow estimation model training method, optical flow estimation method, optical flow estimation device, optical flow estimation model training equipment and computer-readable storage medium of the above-mentioned embodiments, since the attention module in the optical flow estimation model to be trained extracts multiple attention feature maps of different scales of the previous frame training image, the model improves the degree of attention to the target area, overcoming the limitations of traditional optical flow estimation algorithms. The optical flow estimation model to be trained is trained based on the predicted multiple predicted optical flow maps of different scales. The multiple predicted optical flow maps of different scales can gradually correct errors during the model training process, thereby obtaining a trained optical flow estimation model. The predicted optical flow maps of different scales can assist the model in predicting optical flow positions at different scales, thereby providing more accurate and robust optical flow estimation results, thereby improving the robustness and accuracy of subsequent optical flow estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of a method for training an optical flow estimation model according to an embodiment of the present application;

[0040] Figure 2 A flowchart of an embodiment of the present invention for predicting multiple predicted optical flow maps of different scales based on multiple previous frame feature images of different scales, multiple subsequent frame feature images of different scales, and multiple attention feature maps of different scales;

[0041] Figure 3 A structural block diagram of an embodiment of predicting multiple predicted optical flow maps of different scales based on multiple previous frame feature images of different scales, multiple subsequent frame feature images of different scales, and multiple attention feature maps of different scales;

[0042] Figure 4 is a flowchart of an optical flow estimation method according to an embodiment;

[0043] Figure 5 FIG. 4 is a structural diagram of an optical flow estimation device according to an embodiment. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0045] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0046] The serial numbers assigned to components herein, such as "first," "second," etc., are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0047] Please refer to Figure 1In an embodiment of the present invention, a method for training an optical flow estimation model is provided, including steps S10 to S50, which are described in detail below.

[0048] Step S10: Acquire a training image sequence, and acquire a training image pair from the training image sequence.

[0049] In some embodiments, the training image sequence includes a plurality of training image pairs, and a training image pair includes two adjacent training image frames, namely, a preceding training image frame and a succeeding training image frame.

[0050] Step S20: using a feature extraction network in a preset optical flow estimation model to be trained, extracting a plurality of front-frame feature images of different scales of the front-frame training image in the training image pair and a plurality of rear-frame feature images of different scales of the rear-frame training image in the training image pair.

[0051] In some embodiments, for the front-frame training image and the back-frame training image in the training image pair, a plurality of front-frame feature images and back-frame feature images of different scales are respectively extracted through a feature extraction network with shared weights, wherein the feature extraction network can be a convolutional neural network, and the shared weights mean that the network weights of the feature extraction network remain unchanged when the front-frame feature image of the front-frame training image and the back-frame feature image of the back-frame training image are respectively extracted. Therefore, a front-frame feature image of 1 / 4 scale of the front-frame training image, a front-frame feature image of 1 / 8 scale of the front-frame training image, and a front-frame feature image of 1 / 16 scale of the front-frame training image, as well as a back-frame feature image of 1 / 4 scale of the back-frame training image, a back-frame feature image of 1 / 8 scale of the back-frame training image, and a back-frame feature image of 1 / 16 scale of the back-frame training image can be obtained.

[0052] Step S30: Utilize the attention module in the optical flow estimation model to be trained to extract multiple attention feature maps of different scales of the previous frame training image.

[0053] In some embodiments, the attention module in the trained optical flow estimation model is used to extract the 1 / 4 scale attention feature map, the 1 / 8 scale attention feature map, and the 1 / 16 scale attention feature map of the previous frame training image.

[0054] In some embodiments, global maximum pooling, convolution, and activation functions can be used on the channel dimension of the previous frame training image to output multiple attention feature maps of different scales, so that the model can pay more attention to areas with intense motion or larger gradients, which can effectively help the model better focus on important information in the image, thereby improving the robustness of optical flow estimation.

[0055] Step S40: Based on a plurality of front frame feature images of different scales, a plurality of back frame feature images of different scales, and a plurality of attention feature maps of different scales, a plurality of predicted optical flow maps of different scales are predicted.

[0056] In some embodiments, the multiple front-frame feature images of different scales include N front-frame feature images of different scales, the multiple back-frame feature images of different scales include N back-frame feature images of different scales, and the multiple attention feature maps of different scales include N attention feature maps of different scales, wherein the scale decreases from the front-frame feature image of the Nth scale to the front-frame feature image of the 1st scale, the scales of the front-frame feature image of the nth scale, the back-frame feature image of the nth scale, and the attention feature map of the nth scale are the same, n=1,...,N, where N is an integer not less than 2.

[0057] For example, when N=3, the multiple front frame feature images of different scales include three front frame feature images of different scales, namely, a front frame feature image of 1 / 16 scale, a front frame feature image of 1 / 8 scale, and a front frame feature image of 1 / 4 scale; the multiple back frame feature images of different scales include three back frame feature images of different scales, namely, a back frame feature image of 1 / 16 scale, a back frame feature image of 1 / 8 scale, and a back frame feature image of 1 / 4 scale; the multiple attention feature maps of different scales include three attention feature maps of different scales, namely, an attention feature map of 1 / 16 scale, an attention feature map of 1 / 8 scale, and an attention feature map of 1 / 4 scale.

[0058] Please refer to Figure 2 and Figure 3 In some embodiments, step S40 predicts multiple predicted optical flow maps of different scales based on multiple front frame feature images of different scales, multiple back frame feature images of different scales, and multiple attention feature maps of different scales, including steps S41 to S43, which are described in detail below.

[0059] Step S41: generating a correlation feature map according to the first scale front frame feature image and the first scale back frame feature image, and performing optical flow prediction according to the correlation feature map to generate a predicted optical flow map of the first scale.

[0060] In some embodiments, the expression of the predicted optical flow map at the first scale is:

[0061]

[0062] Among them, flow0 represents the predicted optical flow map of the first scale, C flowRepresents the relevant feature map, u1 represents the optical flow range of the pixels in the relevant feature map in the x direction, v1 represents the optical flow range of the pixels in the relevant feature map in the y direction, H represents the height of the relevant feature map, W represents the width of the relevant feature map, R H×W×2 Indicates the dimension of the predicted optical flow map at the first scale.

[0063] In some embodiments, the expression of the relevant feature map is:

[0064]

[0065] Among them, C flow Represents the relevant feature map, F1 represents the position of any pixel in the feature image of the previous frame of the first scale, F2 represents the position of any pixel in the feature image of the next frame of the first scale, T represents the transpose symbol, H represents the height of the relevant feature map, W represents the width of the relevant feature map, R H×W×H×W Indicates the dimension of the relevant feature map.

[0066] In some embodiments, the above method for generating the predicted optical flow map at the first scale is differentiable, which means that backpropagation can be performed in an end-to-end training process.

[0067] Step S42: For the previous frame feature image of the kth scale and the subsequent frame feature image of the kth scale, upsample the predicted optical flow map of the k-1th scale to the kth scale to obtain the upsampled optical flow map of the kth scale, and obtain the predicted previous frame feature image of the kth scale based on the upsampled optical flow map of the kth scale and the subsequent frame feature image of the kth scale, calculate the reprojection error between the predicted previous frame feature image of the kth scale and the previous frame feature image of the kth scale, and obtain the error optical flow estimation of the kth scale.

[0068] In some embodiments, obtaining a predicted previous frame feature image at the kth scale based on the upsampled optical flow map at the kth scale and the subsequent frame feature image at the kth scale includes:

[0069] Reframe1[:,i,j]=frame2[:,i+u,j+v]

[0070] where u=flow[0,i,j],v=flow[1,i,j]

[0071] Among them, Reframe1[:,i,j] represents the predicted previous frame feature image of the kth scale, frame2 represents the post-frame feature image of the kth scale, i represents the position of the pixel on the x-axis in the post-frame feature image of the kth scale, j represents the position of the pixel on the y-axis in the post-frame feature image of the kth scale, u and flow[0,i,j] represent the optical flow of the pixel at position (i,j) on the x-axis, and v and flow[1,i,j] represent the optical flow of the pixel at position (i,j) on the y-axis.

[0072] In some embodiments, the reprojection error between the predicted previous frame feature image at the k-th scale and the previous frame feature image at the k-th scale is calculated using the following formula to obtain the error optical flow estimate at the k-th scale:

[0073] Error=frame1-Reframe1

[0074] Among them, Error represents the error optical flow estimation of the k-th scale, Reframe1 represents the predicted previous frame feature image of the k-th scale, and frame1 represents the previous frame feature image of the k-th scale.

[0075] In some embodiments, k = 2, ..., N. By upsampling the predicted optical flow map at the k-1th scale to the kth scale, a refined optical flow can be continuously obtained during the optical flow propagation process.

[0076] Step S43: Fusing the error optical flow estimation of the k-th scale, the attention feature map of the k-th scale, the upsampled optical flow map of the k-th scale, and the previous frame feature image of the k-th scale to obtain a predicted optical flow map of the k-th scale.

[0077] In some embodiments, the predicted optical flow map at the kth scale is obtained by the following formula:

[0078] F k =concat(frame1,Error,F k atten ,flow uk )

[0079] Among them, F k represents the predicted optical flow map of the kth scale, frame1 represents the feature image of the previous frame of the kth scale, Error represents the error optical flow estimation of the kth scale, F k atten Represents the attention feature map of the k-th scale, flow uk Represents the upsampled optical flow map at the kth scale.

[0080] Step S50: constructing a target loss function according to a plurality of predicted optical flow maps of different scales and a preset real optical flow field, and using the target loss function to train the optical flow estimation model to obtain a trained optical flow estimation model.

[0081] In some embodiments, the objective loss function is expressed as:

[0082]

[0083] Among them, loss represents the target loss function, N represents the number of scales, n represents the index of the scale, α represents the hyperparameter, L1 represents the L1 loss function, flow n Represents the predicted optical flow map of the nth scale, GT n Represents the preset real optical flow field corresponding to the nth scale.

[0084] In some embodiments, the target loss function constructed by multiple predicted optical flow maps of different scales can timely correct the optical flow offset at different scales during the training process to improve the accuracy of optical flow estimation.

[0085] Please refer to Figure 4 In some embodiments, the present application provides an optical flow estimation method, including steps S60 to S70, which are described in detail below.

[0086] Step S60: obtaining a pair of to-be-processed images of the target object in a preset motion scene.

[0087] In some embodiments, the pair of images to be processed includes two adjacent frames of images to be processed.

[0088] Step S70: inputting the image pair to be processed into the optical flow estimation model trained by the optical flow estimation model training method to obtain optical flow estimation information of the target object.

[0089] In some embodiments, the optical flow estimation model training method is specifically implemented as follows:

[0090] Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images;

[0091] A feature extraction network in a preset optical flow estimation model to be trained is used to extract a plurality of front-frame feature images of different scales of a front-frame training image in a training image pair and a plurality of rear-frame feature images of different scales of a rear-frame training image in a training image pair;

[0092] Utilize the attention module in the optical flow estimation model to be trained to extract multiple attention feature maps of different scales of the previous frame training image;

[0093] Based on multiple front-frame feature images of different scales, multiple back-frame feature images of different scales, and multiple attention feature maps of different scales, multiple predicted optical flow maps of different scales are predicted;

[0094] A target loss function is constructed based on multiple predicted optical flow maps of different scales and the preset real optical flow field. The target loss function is used to train the optical flow estimation model to be trained to obtain a trained optical flow estimation model.

[0095] Please refer to Figure 5 The present application provides an optical flow estimation device, including an image acquisition module 10 and an optical flow estimation module 20, which are described in detail below.

[0096] The image acquisition module 10 is used to acquire a pair of images to be processed of a target object in a preset motion scene.

[0097] In some embodiments, the pair of images to be processed includes two adjacent frames of images to be processed.

[0098] The optical flow estimation module 20 is used to input the image pair to be processed into the optical flow estimation model trained by the optical flow estimation model training method to obtain optical flow estimation information of the target object.

[0099] In some embodiments, the optical flow estimation model training method is specifically implemented as follows:

[0100] Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images;

[0101] A feature extraction network in a preset optical flow estimation model to be trained is used to extract a plurality of front-frame feature images of different scales of a front-frame training image in a training image pair and a plurality of rear-frame feature images of different scales of a rear-frame training image in a training image pair;

[0102] Utilize the attention module in the optical flow estimation model to be trained to extract multiple attention feature maps of different scales of the previous frame training image;

[0103] Based on multiple front-frame feature images of different scales, multiple back-frame feature images of different scales, and multiple attention feature maps of different scales, multiple predicted optical flow maps of different scales are predicted;

[0104] A target loss function is constructed based on multiple predicted optical flow maps of different scales and the preset real optical flow field. The target loss function is used to train the optical flow estimation model to be trained to obtain a trained optical flow estimation model.

[0105] In some embodiments, the present application provides an optical flow estimation model training device, including: a memory for storing a program; and a processor for implementing an optical flow estimation model training method by executing the program stored in the memory.

[0106] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0107] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A method for training an optical flow estimation model, characterized in that: include: Acquire a training image sequence, and acquire a training image pair from the training image sequence, wherein the training image pair includes two adjacent frames of training images; A feature extraction network in a preset optical flow estimation model to be trained is used to extract a plurality of front-frame feature images of different scales of the front-frame training image in the training image pair and a plurality of rear-frame feature images of different scales of the rear-frame training image in the training image pair; the plurality of front-frame feature images of different scales include N front-frame feature images of different scales, and the plurality of rear-frame feature images of different scales include N rear-frame feature images of different scales; Extracting a plurality of attention feature maps of different scales from the previous frame training image using an attention module in the optical flow estimation model to be trained; the plurality of attention feature maps of different scales include N attention feature maps of different scales, wherein the scale decreases from the previous frame feature image of the Nth scale to the previous frame feature image of the 1st scale, and the scale of the previous frame feature image of the nth scale, the subsequent frame feature image of the nth scale, and the attention feature map of the nth scale are the same, where n=1,...,N, and N is an integer not less than 2; Generating a correlation feature map according to the first scale of the previous frame feature image and the first scale of the next frame feature image, and performing optical flow prediction according to the correlation feature map to generate a predicted optical flow map of the first scale; For the k-th scale front frame feature image and the k-th scale back frame feature image, upsample the k-1-th scale predicted optical flow map to the k-th scale to obtain the k-th scale upsampled optical flow map, and obtain the k-th scale predicted front frame feature image based on the k-th scale upsampled optical flow map and the k-th scale back frame feature image, calculate the reprojection error between the k-th scale predicted front frame feature image and the k-th scale front frame feature image, and obtain the k-th scale error optical flow estimate; wherein k=2,...,N; fusing the error optical flow estimate of the k-th scale, the attention feature map of the k-th scale, the upsampled optical flow map of the k-th scale, and the previous frame feature image of the k-th scale to obtain a predicted optical flow map of the k-th scale; A target loss function is constructed according to the multiple predicted optical flow maps of different scales and the preset real optical flow field, and the optical flow estimation model to be trained is trained using the target loss function to obtain a trained optical flow estimation model.

2. The optical flow estimation model training method according to claim 1, wherein: The expression of the objective loss function is: Among them, loss represents the target loss function, N represents the number of scales, and n represents the index of the scale. represents the hyperparameter, express loss function, represents the predicted optical flow map of the nth scale, Represents the preset real optical flow field corresponding to the nth scale.

3. The optical flow estimation model training method according to claim 1, wherein: The obtaining of a predicted previous frame feature image at the kth scale according to the upsampled optical flow map at the kth scale and the subsequent frame feature image at the kth scale includes: in, represents the predicted previous frame feature image of the k-th scale, represents the post-frame feature image of the k-th scale, i represents the position of the pixel on the x-axis in the feature image of the subsequent frame at the k-th scale, j represents the position of the pixel on the y-axis in the feature image of the k-th scale, u and Indicates the location The optical flow of pixels on the x-axis, v and Indicates the location The optical flow of the pixel on the y-axis.

4. The optical flow estimation model training method according to claim 1, wherein: The expression of the predicted optical flow map of the first scale is: in, Represents the predicted optical flow map of the first scale, represents the related feature map, represents the optical flow range of the pixels in the relevant feature map in the x direction, represents the optical flow range of the pixels in the relevant feature map in the y direction, H represents the height of the relevant feature map, and W represents the width of the relevant feature map. Indicates the dimension of the predicted optical flow map of the first scale.

5. The optical flow estimation model training method according to claim 1, wherein: The expression of the related feature map is: in, represents the related feature map, represents the position of any pixel in the previous frame feature image of the first scale, represents the position of any pixel in the feature image of the subsequent frame at the first scale, T represents the transposition symbol, H represents the height of the relevant feature map, and W represents the width of the relevant feature map. Represents the dimension of the related feature map.

6. A method for optical flow estimation, characterized in that: include: Acquire a pair of to-be-processed images of a target object in a preset motion scene, wherein the pair of to-be-processed images includes two adjacent frames of to-be-processed images; The image pair to be processed is input into an optical flow estimation model trained by the optical flow estimation model training method according to any one of claims 1 to 5 to obtain optical flow estimation information of the target object.

7. An optical flow estimation device, characterized in that: include: An image acquisition module is used to acquire a pair of images to be processed of a target object in a preset motion scene, wherein the pair of images to be processed includes two adjacent frames of images to be processed; An optical flow estimation module is used to input the image pair to be processed into an optical flow estimation model trained by the optical flow estimation model training method according to any one of claims 1 to 5 to obtain optical flow estimation information of the target object.

8. An optical flow estimation model training device, characterized in that: include: Memory, used to store programs; A processor, configured to implement the optical flow estimation model training method according to any one of claims 1 to 5 by executing the program stored in the memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which can be executed by a processor to implement the optical flow estimation model training method according to any one of claims 1 to 5.

Images