Optical flow prediction method and device

Abstract

The present invention provides an optical flow prediction method and device, which relates to the field of computer vision technology, including: inputting two frames of continuous images to be predicted into an optical flow prediction network; obtaining predicted optical flow according to the optical flow prediction network, wherein the loss of the optical flow prediction network is determined in the following manner, including: cropping the input image according to a preset cropping rule to determine two frames of cropped images; inputting the two frames of cropped images into the optical flow prediction network to obtain forward optical flow and backward optical flow; performing boundary filling mapping according to the forward optical flow, the backward optical flow and the two frames of input images to determine the two frames of mapped images after mapping; determining the loss of the optical flow prediction network according to the cropped image and the mapped image. The present invention performs boundary filling mapping on the cropped image, so that the pixels that move beyond the boundary of the cropped image can still be replaced by the corresponding pixels of the input image after boundary filling mapping, thereby obtaining the correct image loss and improving the accuracy of optical flow prediction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an optical flow prediction and device. Background Art

[0002] With the development of computer technology, the development of computer vision is also quite rapid. Optical flow prediction is a basic task of computer vision and plays an important role in many fields such as autonomous driving, video processing, target detection, and behavior analysis. In recent years, due to the rise of deep learning, optical flow prediction methods based on deep learning have gradually become mainstream. Existing optical flow prediction methods based on deep learning include FlowNet, FlowNet-v2, PWC-Net, and IRR-PWCNet. Its main advantages are high accuracy, fast speed, fast deployment, and strong generalization. However, the optical flow prediction method based on deep learning has very high requirements for training data and requires accurate annotation of optical flow labels. It is very difficult to manually perform a large number of optical flow annotations. Therefore, people try to create virtual data, such as the FlyingChirs dataset, use virtual datasets to train optical flow models, and then apply them to real scenes, or use virtual datasets to train optical flow models and then use a small amount of manually annotated sparse optical flow labels for retraining. Since the illumination changes and occlusion changes in real scenes are very different from those in virtual datasets, such methods often perform poorly in real applications.

[0003] Existing optical flow prediction methods based on deep learning are highly dependent on labeled data, which limits their application in real scenarios. To solve this problem, researchers proposed unsupervised optical flow methods. Unsupervised optical flow methods no longer require optical flow annotations and can train optical flow models using only image pairs. The basic approach is to: Figure 1 and Figure 2 Input the neural network, output the optical flow, and then use the output optical flow to Figure 2 Map to Figure 1 , by comparing the mapping results and Figure 1 If the neural network predicts the optical flow correctly, the resulting image will be the same as Figure 1 Basically consistent. Since the unsupervised optical flow method can train the optical flow model using image pairs, no manual annotation is required. Therefore, the cost of the unsupervised optical flow method is very low and it is easy to apply to various real scenes.

[0004] However, when the object moves, for example, when the object moves beyond the image boundary, the above-mentioned mapping calculation loss method will result in the inability to learn the correct optical flow, and may even have a serious impact on the entire learning process. To address this problem, existing methods first perform bidirectional optical flow prediction, detect occluded areas based on the bidirectional optical flow, and then exclude occluded areas from the training loss to avoid the adverse effects of occluded areas. This method can significantly improve the accuracy of optical flow in non-occluded areas, but it cannot solve the learning problem of occluded areas, resulting in poor optical flow estimation in occluded areas. Summary of the invention

[0005] The present invention aims to solve the technical problems in the related art at least to a certain extent. To achieve the above object, in a first aspect, the present invention provides an optical flow prediction method, which comprises:

[0006] Input two consecutive frames of images to be predicted into the optical flow prediction network;

[0007] The predicted optical flow is obtained according to the optical flow prediction network, wherein the loss of the optical flow prediction network is determined by:

[0008] Cropping the input image according to a preset cropping rule to determine two frames of cropped images;

[0009] Inputting the two frames of the cropped images into the optical flow prediction network to obtain forward optical flow and backward optical flow;

[0010] Performing boundary filling mapping according to the forward optical flow, the backward optical flow and the two frames of the input image to determine two frames of mapped images after mapping, wherein the boundary filling mapping is used to provide optical flow loss for pixels that move beyond the boundary of the cropped image;

[0011] The loss of the optical flow prediction network is determined based on the cropped image and the mapped image.

[0012] Furthermore, the cropping according to a preset rule includes: cropping the input image with a first number of pixel widths from a length boundary of the input image and a second number of pixel widths from a width boundary of the input image.

[0013] Further, performing boundary filling mapping according to the forward optical flow, the backward optical flow and the two frames of the input image to determine the two frames of mapped images after mapping includes:

[0014] Perform the boundary filling mapping according to the forward optical flow and the second input image in the two frames of the input images to determine a first mapped image after mapping;

[0015] The boundary filling mapping is performed according to the backward optical flow and a first input image of the two frames of input images to determine a mapped second mapping image.

[0016] Further, performing the boundary filling mapping according to the forward optical flow and the second input image in the two frames of the input images to determine the mapped first mapping image includes:

[0017] Determine, according to the forward optical flow and the second input image, coordinates corresponding to each pixel point in the first cropped image corresponding to the first input image in the second input image;

[0018] The pixel points corresponding to the coordinates in the second input image are used to replace each pixel point in the first cropped image to perform the boundary filling mapping, and the first mapping image is determined.

[0019] Further, determining, according to the forward optical flow and the second input image, the coordinates corresponding to each pixel point in the first cropped image corresponding to the first input image in the second input image includes:

[0020] determining the coordinates of a vertex of the first cropped image;

[0021] The coordinates corresponding to each pixel point in the second input image are determined according to the coordinates of the vertices, the coordinates of each pixel point in the first cropped image, and the value of the forward optical flow at each pixel point.

[0022] Further, performing the boundary filling mapping according to the backward optical flow and the first input image of the two frames of the input images to determine the mapped second mapping image includes:

[0023] Determine, according to the backward optical flow and the first input image, coordinates corresponding to each pixel point in the second cropped image corresponding to the second input image in the first input image;

[0024] The pixel points corresponding to the coordinates in the first input image are used to replace each pixel point in the second cropped image to perform the boundary filling mapping, and the second mapping image is determined.

[0025] Further, determining, according to the backward optical flow and the first input image, the coordinates corresponding to each pixel point in the second cropped image corresponding to the second input image in the first input image includes:

[0026] determining the coordinates of a vertex of the second cropped image;

[0027] The coordinates corresponding to each pixel point in the first input image are determined according to the coordinates of the vertices, the coordinates of each pixel point in the second cropped image, and the value of the backward optical flow at each pixel point.

[0028] Further, determining the loss of the optical flow prediction network according to the cropped image and the mapped image includes:

[0029] Determine an optical flow error loss based on the two frames of the cropped images and the two frames of the mapped images;

[0030] Determining a smoothing loss according to the forward optical flow and the backward optical flow;

[0031] The loss of the optical flow prediction network is determined according to the optical flow error loss and the smoothness loss.

[0032] Further, determining the optical flow error loss according to the two frames of the cropped images and the two frames of the mapped images includes:

[0033] Determine a first error loss according to a difference between a first cropped image in the two frames of cropped images and a first mapped image determined by the boundary filling mapping;

[0034] Determine a second error loss according to a difference between a second cropped image in the two frames of cropped images and a second mapped image determined by the boundary filling mapping;

[0035] The optical flow error loss is determined according to the first error loss and the second error loss.

[0036] Furthermore, the optical flow prediction network includes a convolutional neural network and a plurality of decoding networks, wherein:

[0037] The convolutional neural network is used to extract a plurality of feature pairs of different scales according to the two frames of the cropped images, and the plurality of feature pairs of different scales correspond one-to-one to the plurality of the decoding networks;

[0038] The plurality of decoding networks respectively obtain a plurality of sets of bidirectional optical flow information according to the feature pairs of different scales,

[0039] Among them, the bidirectional optical flow information corresponding to each scale is input into the decoding network corresponding to the adjacent large-scale feature pair after bilinear upsampling operation, and the bidirectional optical flow information determined by the decoding network corresponding to the feature pair with the largest scale is output as the optical flow prediction network after upsampling operation.

[0040] To achieve the above object, in a second aspect, the present invention provides an optical flow prediction device, comprising:

[0041] An input module, used to input two consecutive frames of images to be predicted into the optical flow prediction network;

[0042] A prediction module is used to obtain a predicted optical flow according to the optical flow prediction network, wherein the loss of the optical flow prediction network is determined in the following manner, including: cropping the input image according to a preset cropping rule to determine two frames of cropped images; inputting the two frames of the cropped images into the optical flow prediction network to obtain a forward optical flow and a backward optical flow; and further used to perform boundary filling mapping according to the forward optical flow, the backward optical flow and the two frames of the input images to determine two frames of mapped images after mapping, wherein the boundary filling mapping is used to provide an optical flow loss for pixels that move beyond the boundary of the cropped image; and further used to determine the loss of the optical flow prediction network according to the cropped image and the mapped image; and further used to perform optical flow prediction according to the loss of the optical flow prediction network to determine the predicted optical flow.

[0043] Using the optical flow prediction method or device of the present invention, after performing certain regular cropping on the input image, a cropped image is determined and input into an optical flow network for bidirectional optical flow prediction, and then boundary filling mapping is performed based on the predicted optical flow and the uncropped input image, so that pixels that move beyond the boundary of the cropped image can still obtain the correct image loss through the corresponding pixels of the input image after boundary filling mapping, thereby improving the accuracy of determining the image loss, so that the obtained optical flow prediction network can obtain accurate optical flow prediction results, thereby improving the accuracy of the overall optical flow prediction.

[0044] To achieve the above-mentioned purpose, in a third aspect, the present invention provides a non-temporary computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the optical flow prediction method according to the first aspect of the present invention is implemented.

[0045] To achieve the above-mentioned purpose, in a fourth aspect, the present invention provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the optical flow prediction method according to the first aspect of the present invention is implemented.

[0046] The non-transitory computer-readable storage medium and computing device according to the present invention have similar beneficial effects as the optical flow prediction method according to the first aspect of the present invention, which will not be described in detail herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of an optical flow prediction method according to an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a process of determining optical flow prediction network loss according to an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of cropping an image according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the black edge problem caused by the traditional mapping method;

[0051] Figure 5 is a schematic diagram of a training image according to an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of the structure of an optical flow prediction network according to an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of a process of determining a mapping image according to an embodiment of the present invention;

[0054] Figure 8 A schematic diagram of the principle of performing border filling mapping according to an embodiment of the present invention;

[0055] Fig. 9 A schematic diagram of a process of determining a first mapping image according to an embodiment of the present invention;

[0056] Fig.10 A schematic diagram of a process of determining a second mapping image according to an embodiment of the present invention;

[0057] Fig.11 A schematic diagram of a process for calculating the loss of an optical flow prediction network according to an embodiment of the present invention;

[0058] Fig.12 Schematic diagram of test results based on the KITTI2012 test set according to an embodiment of the present invention;

[0059] Fig.13 Schematic diagram of test results based on the KITTI2015 test set according to an embodiment of the present invention;

[0060] Fig.14 Schematic diagram of the structure of an optical flow prediction device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. When describing the drawings, unless otherwise indicated, the same reference numerals in different drawings represent the same or similar elements. It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments of the present invention. They are only examples of devices and methods consistent with some aspects disclosed in the present invention as detailed in the claims, and the scope of the present invention is not limited to this. The features of the various embodiments of the present invention can be combined with each other without contradiction.

[0062] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0063] Images are the most commonly used information carriers in the human world. When the human eye observes a moving object, a series of continuously changing images will be formed on the retina. This information "flows" through the retina in the form of light flow, so it is called optical flow. The simple explanation of the role of optical flow is that it can record the movement of objects in the image scene. When observing two adjacent frames of images, people will find that there is a time correlation and position correlation between the objects in the image. Using this correlation to obtain motion information is the core of the optical flow method. Strictly speaking, optical flow expresses the instantaneous speed of objects in the three-dimensional world on the two-dimensional imaging plane. By gathering the optical flows of all pixels in the image, the optical flow field of the entire image can be obtained.

[0064] Optical flow prediction is a classic research problem in the field of computer vision. In computer vision, this task defines the movement of objects in an image, which can be caused by camera movement or object movement. Specifically, it refers to the movement of pixels representing the same object (object) in one frame of a video image to the next frame, represented by a two-dimensional vector (representing movement in the x and y directions, respectively). In computer vision, optical flow contains potential motion information, so it has very important applications in video target detection, target segmentation, behavior recognition, target tracking, robot navigation, and shape information recovery.

[0065] The development of deep learning has brought about a breakthrough in the field of optical flow prediction: on the one hand, by using supervised learning algorithms and constructing data sets through virtual acquisition, the optical flow prediction network based on deep learning has surpassed the traditional optical flow algorithm in terms of speed and accuracy, which has greatly promoted the development of this direction; on the other hand, by using unsupervised learning algorithms, the accuracy of the optical flow prediction network has reached that of the traditional optical flow algorithm, and its speed is also far higher than that of the traditional algorithm. The unsupervised optical flow learning method greatly reduces the requirements for the data set because it does not require manual annotation of the data set, and is more suitable for applications in various real scenarios.

[0066] However, existing unsupervised optical flow learning methods have the following main technical problems: (1) The optical flow of the occluded area in the image cannot be learned, resulting in incorrect optical flow estimation in the occluded area; (2) All pixels in the occluded area are excluded from the image loss, resulting in some learnable areas being blocked and unable to be learned, affecting the optical flow estimation performance.

[0067] The present invention crops an input image to obtain a cropped image, performs bidirectional optical flow prediction on the cropped image for an input optical flow network, and then performs boundary filling mapping on the predicted optical flow and the uncropped input image, determines the movement amount of pixel points in the cropped image by predicting the optical flow, and determines corresponding pixels in the original uncropped image for replacement, so that pixels that move beyond the boundary of the cropped image can still be replaced by corresponding pixels of the input image after boundary filling mapping, and obtains correct image loss, thereby improving the accuracy of optical flow prediction and solving the problems existing in the prior art.

[0068] Figure 1 FIG. 1 is a flow chart of an optical flow prediction method according to an embodiment of the present invention, including steps S1 to S2.

[0069] In step S1, two consecutive frames of images to be predicted are input into an optical flow prediction network. In an embodiment of the present invention, a video for which optical flow prediction is required is obtained, and two consecutive frames of images to be predicted are input into an optical flow prediction network for optical flow prediction.

[0070] In step S2, a predicted optical flow is obtained according to the optical flow prediction network, wherein the loss of the optical flow prediction network is determined as follows: Figure 2 It is a schematic diagram of a process of determining optical flow prediction network loss according to an embodiment of the present invention, including steps S21 to S24.

[0071] In step S21, the input image is cropped according to a preset cropping rule to determine two cropped images. In the embodiment of the present invention, since no optical flow annotation is required, the input data only needs to extract video frames from the video. Two adjacent continuous frames of images are used as input images to determine the subsequent image loss.

[0072] In an embodiment of the present invention, the preset cropping rule includes: cropping the input image with a first number of pixel widths from the length boundary of the input image and a second number of pixel widths from the width boundary of the input image. Specifically, the rule for cropping the input image (source image) is 16 pixels away from the boundary (that is, the first number and the second number are both 16), the cropping size is fixed, and the input image is cropped. For example, when training in the flyingchairs dataset, the source image size is 384*480, then it is cropped to 320*320 for training; when training in the Sintel dataset, the source image size is 436*1024, then it is cropped to 320*896 for training. It can be understood that the cropping rules can be adjusted accordingly according to actual needs, and the present invention is not limited to this, but it is necessary to ensure that the entire cropping frame is within the source image.

[0073] Figure 3FIG. 1 is a schematic diagram of cropping an image according to an embodiment of the present invention, wherein two consecutive input image frames are obtained, namely, the first input image and the second input image , the input image is cropped according to the preset cropping rule to obtain the cropped first cropped image I a and the second cropped image I b It can be understood that a cropping frame of a fixed size can be determined according to a preset rule, and the input image is cropped to obtain a cropped image. Figure 4 The diagram shows the black edge problem caused by the traditional mapping method. Figure 4 As shown, taking the forward optical flow as an example, according to the second cropped image I b and the predicted forward optical flow V f After mapping using a traditional mapping method, when the pixel movement exceeds the boundary of the image, the pixels in the area in the traditional mapping method cannot find the corresponding pixels and will be automatically filled with 0, resulting in the inability to provide the correct optical flow loss to learn the corresponding relationship, forming a black edge. In an embodiment of the present invention, boundary filling mapping is performed based on the bidirectional predicted optical flow and the input image before cropping, so that when the pixel movement exceeds the image boundary, the correct optical flow loss can still be provided for learning, thereby avoiding the black edge problem caused by the pixel movement exceeding the image boundary in the traditional mapping method.

[0074] In step S22, the two frames of the cropped images are input into the optical flow prediction network to obtain the forward optical flow and the backward optical flow. In the embodiment of the present invention, the existing KITTI dataset can be used to train the optical flow network. Figure 5 The diagram is a schematic diagram of a training image according to an embodiment of the present invention, where 13372 pairs of video frames are extracted from a road shooting video to form a training data set for optical flow network training. It can be understood that the images in the training data set are all cropped images.

[0075] In an embodiment of the present invention, after a training data set is selected, it is input into a convolutional neural network for optical flow prediction. Figure 6 FIG. 1 is a schematic diagram of the structure of an optical flow prediction network according to an embodiment of the present invention. In the embodiment of the present invention, the optical flow prediction network is used as shown in FIG. Figure 6 The optical flow prediction network shown in the figure performs optical flow prediction. The optical flow prediction network includes 1 convolutional neural network and 5 decoding networks. The convolutional neural network is used to extract multiple feature pairs of different scales based on the two frames of cropped images. The multiple feature pairs of different scales correspond to the multiple decoding networks one by one. The multiple decoding networks are respectively used to obtain multiple sets of bidirectional optical flow information based on the feature pairs of different scales. The input of the optical flow prediction network is the previous and next two frames of cropped images: the first cropped image I a and the second cropped image I b, and 5 feature pairs of different scales are extracted through the same convolutional neural network, namely Figure 4 f1 in i and f2 i , i∈{0,1,2,3,4}. Among them, i=0 has the smallest scale, which is 1 / 32 of the cropped image, and i=5 has the largest scale, which is 1 / 2 of the cropped image.

[0076] In the embodiment of the present invention, starting from the scale i=0, the features are passed through a decoding network ( Figure 6 Decoder in the code) obtains bidirectional optical flow information, that is, forward optical flow and backward optical flow Bidirectional optical flow information (such as forward optical flow and backward optical flow ) After bilinear upsampling, it is input into the decoding network corresponding to the next large-scale feature pair, and iterates until the decoding network corresponding to the feature pair with the largest scale obtains the bidirectional optical flow information. and Directly upsample by two times to get and As the final output of the optical flow prediction network. In the embodiment of the present invention, the decoding networks used at each scale share parameters, so the optical flow network has fewer parameters. It is understandable that the optical flow prediction network may also adopt network models of other structures, and the present invention is not limited thereto.

[0077] In an embodiment of the present invention, the optical flow prediction network sets multiple different scales to perform optical flow estimation from coarse to fine. Because a larger displacement on the original image does not result in a larger displacement on a small scale, estimating from a small scale to a large scale is very beneficial for the case of large displacement and can improve the accuracy of the overall optical flow prediction.

[0078] In step S23, boundary filling mapping is performed based on the forward optical flow, the backward optical flow and the two frames of the input image to determine the two frames of mapped images after mapping, wherein the boundary filling mapping is used to provide optical flow loss for pixels that move beyond the boundary of the cropped image. In an embodiment of the present invention, the cropped image after cropping is input into an optical flow prediction network to obtain predicted forward optical flow and backward optical flow, and boundary filling mapping is performed based on the predicted forward optical flow and backward optical flow, and the optical flow loss is calculated. In an embodiment of the present invention, boundary filling mapping is performed using the input image before cropping so that the mapped image has no black edges. Figure 7 It is a schematic diagram of a process of determining a mapping image according to an embodiment of the present invention, including steps S231 to S232.

[0079] In step S231, the boundary filling mapping is performed according to the forward optical flow and the second input image of the two frames of input images to determine a mapped first mapping image. Figure 8 FIG. 1 is a schematic diagram showing the principle of performing boundary filling mapping according to an embodiment of the present invention. Taking the forward optical flow as an example, according to the second input image before clipping, and the predicted forward optical flow V f Perform boundary filling mapping so that the mapped image has no black edges. Fig. 9 It is a schematic diagram of a process of determining a first mapping image according to an embodiment of the present invention, including steps S2311 to S2312.

[0080] In step S2311, the coordinates of each pixel in the first cropped image corresponding to the first input image in the second input image are determined according to the forward optical flow and the second input image. In an embodiment of the present invention, the coordinates of a vertex of the first cropped image are determined; and the coordinates of each pixel in the second input image are determined according to the coordinates of the vertex, the coordinates of each pixel in the first cropped image, and the value of the forward optical flow at each pixel. Specifically, for example, the second input image I before cropping is selected. b r The coordinate system is established with the upper left corner of the image as the origin, and the coordinates of each pixel can be determined. At this time, the second cropped image I b The upper left corner vertex is p0, then for any pixel point p, determine the forward optical flow V f The value at point p is V f (p). It can be understood that according to p0+p+V f (p) The first cropped image I can be obtained a Each pixel in the image is moved to the second input image The position of the pixel in the second input image can be obtained by The corresponding coordinates in .

[0081] In step S2312, the pixel points corresponding to the coordinates in the second input image are used to replace each pixel point in the first cropped image to perform the boundary filling mapping to determine the first mapping image. In the embodiment of the present invention, the boundary filling mapping method can be expressed as:

[0082]

[0083] in, represents the mapping result, which is used to calculate the image loss, p represents the pixel coordinates, and p0 represents the coordinates of the upper left corner vertex of the cropped image. It can be understood that in this embodiment of the present invention, according to the first cropped image I aThe movement of each pixel in the second input image determines its and use the second input image The corresponding pixel points in the image are replaced by the corresponding pixel points in the image, and the boundary filling mapping is performed so that the movement exceeds the first cropped image I a The pixels at the border can still be in the uncropped second input image. Find the corresponding pixels and replace them to get a new mapped image. Then, compare the new mapped image with the cropped image. When the optical flow prediction is more accurate, the image loss will be smaller.

[0084] In step S232, the boundary filling mapping is performed according to the backward optical flow and the first input image of the two frames of input images to determine a mapped second mapping image. Fig.10 It is a schematic diagram of a process of determining a second mapping image according to an embodiment of the present invention, including steps S2321 to S2322.

[0085] In step S2321, the coordinates of each pixel in the second cropped image corresponding to the second input image in the first input image are determined based on the backward optical flow and the first input image. In an embodiment of the present invention, it includes determining the coordinates of a vertex of the second cropped image; and determining the coordinates of each pixel in the first input image based on the coordinates of the vertex, the coordinates of each pixel in the second cropped image, and the value of the backward optical flow at each pixel. It can be understood that in an embodiment of the present invention, the method of performing boundary filling mapping based on the backward optical flow and the first input image to obtain the mapped second mapped image is the same as the mapping method of the forward optical flow described above, and will not be repeated here.

[0086] In step S2322, the pixel points corresponding to the coordinates in the first input image are replaced with each pixel point in the second cropped image to perform the boundary filling mapping to determine the second mapped image. In the embodiment of the present invention, a second mapped image based on the boundary filling mapping of the first input image is also obtained. It can be understood that by comparing the second mapped image with the second cropped image, the optical flow loss of the backward optical flow can be obtained, which is used to determine the overall image loss.

[0087] In step S24, the loss of the optical flow prediction network is determined based on the cropped image and the mapped image. Fig.11 It is a schematic diagram of a process of calculating the loss of an optical flow prediction network according to an embodiment of the present invention, including steps S241 to S243.

[0088] In step S241, the optical flow error loss is determined based on the two frames of the cropped images and the two frames of the mapped images. In an embodiment of the present invention, the first error loss is determined based on the difference between the first cropped image in the cropped images and the first mapped image determined by the boundary filling mapping. In an embodiment of the present invention, the first mapped image determined by the boundary filling mapping is compared with the first cropped image, and the difference between them is used as the first error loss. It can be understood that the difference can be calculated according to, for example, an L1 function or an L2 function. In an embodiment of the present invention, a robust loss function is used for calculation, and the robust loss function is shown in the following formula:

[0089] ψ(x)=(|x|+ε) q ,

[0090] Among them, ψ represents the robust loss function, the coefficient ε is set to 0.01, and the coefficient q is set to 0.4.

[0091] In the embodiment of the present invention, the first error loss is expressed as Among them, w Ω Represents the border fill map.

[0092] In the embodiment of the present invention, the second error loss is determined according to the difference between the second cropped image in the cropped image and the second mapped image determined by the boundary filling mapping. It can be understood that in the embodiment of the present invention, the second error loss is determined according to the method as described above.

[0093] In step S242, a smoothing loss is determined based on the forward optical flow and the backward optical flow. In the embodiment of the present invention, a smoothing constraint is added to the optical flow predicted by the optical flow network, that is, the optical flows of adjacent pixels need to be similar. Specifically, the difference between adjacent predicted optical flows in the x and y directions is directly used as the smoothing loss:

[0094] L s =|▽V f |+|▽V b |.

[0095] In step S243, the loss of the optical flow prediction network is determined according to the optical flow error loss and the smoothing loss. In an embodiment of the present invention, the optical flow error loss is jointly determined according to the difference between the two groups of mapped images and the cropped image:

[0096]

[0097] In the embodiment of the present invention, the final image loss is:

[0098] L=λ p Lp +λ s L s ,

[0099] Among them, λ p , s are weight terms of optical flow error loss and smoothing loss, respectively. Since a large smoothing loss weight will cause the optical flow output by the optical flow prediction network to be a constant value, in this embodiment of the present invention, a smaller λ is set. s =0.01, and set λ p =1.

[0100] In the embodiment of the present invention, the predicted optical flow result can be obtained according to the above steps, thereby achieving accurate optical flow prediction.

[0101] Fig.12 The figure shows a test result diagram of the unsupervised optical flow prediction method based on the KITTI2012 test set. It shows the first frame image (reference) of the input, the true optical flow (GroundTruth), the optical flow results and the error spectrum using and not using the boundary filling mapping method of the embodiment of the present invention. It is worth noting that the true optical flow annotation is not used in network training, but is only used to measure the accuracy of the predicted optical flow. In the error spectrum, the brighter the area, the larger the error, and the darker the area, the smaller the error. It can be seen that the optical flow results obtained by the unsupervised optical flow prediction method of the embodiment of the present invention are very accurate, especially in the edge area of ​​the image. Fig.13 The figure shows a schematic diagram of the test results of using the above unsupervised optical flow prediction method based on the KITTI2015 test set. It can also be seen that the unsupervised optical flow prediction method using the embodiment of the present invention can learn accurate optical flow prediction results without the need for any manual annotation of optical flow.

[0102] The optical flow prediction method of unsupervised learning of the embodiment of the present invention is adopted. After the input image is cropped according to certain rules, the cropped image is determined, and the cropped image is input into the optical flow network for bidirectional optical flow prediction. Then, the predicted optical flow and the uncropped input image are used for boundary filling mapping. The movement amount of the pixel points in the cropped image is determined by predicting the optical flow, and the corresponding pixels are determined in the original uncropped image for replacement, so that the pixels that move beyond the boundary of the cropped image can still be replaced by the corresponding pixels of the input image after boundary filling mapping, and the correct image loss is obtained, so that the partial occluded area can be converted into a learnable area, and the accuracy of determining the image loss in the occluded area is significantly improved. The present invention performs unsupervised training based on the image loss determined by the mapped image and the cropped image, avoids the problem that the pixels that move beyond the image boundary cannot be learned, improves the accuracy of unsupervised learning optical flow prediction, and makes the learned optical flow model more accurate and has better performance.

[0103] The embodiment of the second aspect of the present invention further provides an optical flow prediction device. Fig.14 FIG. 1 is a schematic diagram of the structure of an optical flow prediction device 1400 for unsupervised learning according to an embodiment of the present invention, which includes an input module 1401 and a prediction module 1402 .

[0104] The input module 1401 is used to input two consecutive frames of input images into the optical flow prediction network;

[0105] The prediction module 1402 is used to obtain the predicted optical flow according to the optical flow prediction network, wherein the loss of the optical flow prediction network is determined in the following manner, including: cropping the input image according to a preset cropping rule to determine two frames of cropped images; determining the forward optical flow and the backward optical flow according to the two frames of the cropped images; and performing boundary filling mapping according to the forward optical flow, the backward optical flow and the two frames of the input images to determine the two frames of mapped images after mapping, wherein the boundary filling mapping is used to provide optical flow loss for pixels that move beyond the boundary of the cropped image; and determining the loss of the optical flow prediction network according to the cropped image and the mapped image.

[0106] In an embodiment of the present invention, the prediction module 1402 is further used to perform the boundary filling mapping according to the forward optical flow and the second input image in the two frames of the input image to determine the first mapped image after mapping; and is also used to perform the boundary filling mapping according to the backward optical flow and the first input image in the two frames of the input image to determine the second mapped image after mapping.

[0107] The more specific implementation of each module of the optical flow prediction device 1400 can refer to the description of the optical flow prediction method of the present invention, and has similar beneficial effects, which will not be repeated here.

[0108] An embodiment of the third aspect of the present invention provides a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the optical flow prediction method according to the first aspect of the present invention is implemented.

[0109] Generally speaking, the computer instructions for implementing the method of the present invention can be carried by any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media can include any computer-readable media except for the signal itself that is temporarily propagating.

[0110] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0111] Computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0112] The embodiment of the fourth aspect of the present invention provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the optical flow prediction method according to the first aspect of the present invention is implemented. It is understandable that the computing device of the present invention can be a server or a terminal device with limited computing power.

[0113] The non-temporary computer-readable storage medium and computing device according to the third and fourth aspects of the present invention can be implemented with reference to the contents specifically described in the embodiment according to the first aspect of the present invention, and have similar beneficial effects as the optical flow prediction method according to the embodiment of the first aspect of the present invention, which will not be described in detail here.

[0114] Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An optical flow prediction method, characterized in that: include: Input two consecutive frames of images to be predicted into the optical flow prediction network; The predicted optical flow is obtained according to the optical flow prediction network, wherein the loss of the optical flow prediction network is determined by: Cropping the input image according to a preset cropping rule to determine two frames of cropped images; wherein the cropping according to the preset cropping rule includes: cropping the input image with a first number of pixel widths from a length boundary of the input image and a second number of pixel widths from a width boundary of the input image; Inputting the two frames of the cropped images into the optical flow prediction network to obtain forward optical flow and backward optical flow; Performing boundary filling mapping according to the forward optical flow, the backward optical flow and the two frames of the input image to determine two frames of mapped images after mapping, wherein the boundary filling mapping is used to provide optical flow loss for pixels that move beyond the boundary of the cropped image; Specifically comprising: performing the boundary filling mapping according to the forward optical flow and the second input image in the two frames of the input image, and determining the mapped first mapping image; S231 specifically comprises: determining the coordinates corresponding to each pixel point in the first cropped image corresponding to the first input image in the second input image according to the forward optical flow and the second input image; Replace each pixel in the first cropped image with a pixel corresponding to the coordinates in the second input image to perform the boundary filling mapping, and determine the first mapping image; The boundary filling mapping is performed according to the backward optical flow and the first input image of the two frames of the input image to determine the mapped second mapping image; specifically comprising: determining the coordinates corresponding to each pixel point in the second cropped image corresponding to the second input image in the first input image according to the backward optical flow and the first input image; Replace each pixel in the second cropped image with a pixel corresponding to the coordinates in the first input image to perform the boundary filling mapping, and determine the second mapping image; The loss of the optical flow prediction network is determined based on the cropped image and the mapped image.

2. The optical flow prediction method according to claim 1, characterized in that: The determining, according to the forward optical flow and the second input image, the coordinates corresponding to each pixel point in the first cropped image corresponding to the first input image in the second input image comprises: determining the coordinates of a vertex of the first cropped image; The coordinates corresponding to each pixel point in the second input image are determined according to the coordinates of the vertices, the coordinates of each pixel point in the first cropped image, and the value of the forward optical flow at each pixel point.

3. The optical flow prediction method according to claim 1, characterized in that: The determining, according to the backward optical flow and the first input image, the coordinates corresponding to each pixel point in the second cropped image corresponding to the second input image in the first input image comprises: determining the coordinates of a vertex of the second cropped image; The coordinates corresponding to each pixel point in the first input image are determined according to the coordinates of the vertices, the coordinates of each pixel point in the second cropped image, and the value of the backward optical flow at each pixel point.

4. The optical flow prediction method according to claim 1, characterized in that: Determining the loss of the optical flow prediction network according to the cropped image and the mapped image comprises: Determine an optical flow error loss based on the two frames of the cropped images and the two frames of the mapped images; Determining a smoothing loss according to the forward optical flow and the backward optical flow; The loss of the optical flow prediction network is determined according to the optical flow error loss and the smoothness loss.

5. The optical flow prediction method according to claim 4, characterized in that: Determining the optical flow error loss according to the two frames of the cropped images and the two frames of the mapped images comprises: Determine a first error loss according to a difference between a first cropped image in the two frames of cropped images and a first mapped image determined by the boundary filling mapping; Determine a second error loss according to a difference between a second cropped image in the two frames of cropped images and a second mapped image determined by the boundary filling mapping; The optical flow error loss is determined according to the first error loss and the second error loss.

6. The optical flow prediction method according to any one of claims 1 to 5, characterized in that: The optical flow prediction network includes a convolutional neural network and a plurality of decoding networks, wherein: The convolutional neural network is used to extract a plurality of feature pairs of different scales according to the two frames of the cropped images, and the plurality of feature pairs of different scales correspond one-to-one to the plurality of the decoding networks; The plurality of decoding networks respectively obtain a plurality of sets of bidirectional optical flow information according to the feature pairs of different scales, Among them, the bidirectional optical flow information corresponding to each scale is input into the decoding network corresponding to the adjacent large-scale feature pair after bilinear upsampling operation, and the bidirectional optical flow information determined by the decoding network corresponding to the feature pair with the largest scale is output as the optical flow prediction network after upsampling operation.

7. An optical flow prediction device, characterized in that: Implementing the optical flow prediction method according to any one of claims 1 to 6, comprising: An input module, used to input two consecutive frames of images to be predicted into the optical flow prediction network; A prediction module is used to obtain a predicted optical flow according to the optical flow prediction network, wherein the loss of the optical flow prediction network is determined in the following manner, including: cropping the input image according to a preset cropping rule to determine two frames of cropped images; inputting the two frames of the cropped images into the optical flow prediction network to obtain a forward optical flow and a backward optical flow; and performing boundary filling mapping according to the forward optical flow, the backward optical flow and the two frames of the input images to determine two frames of mapped images after mapping, wherein the boundary filling mapping is used to provide an optical flow loss for pixels that move beyond the boundary of the cropped image; and is also used to determine the loss of the optical flow prediction network according to the cropped image and the mapped image.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the optical flow prediction method according to any one of claims 1 to 6 is implemented.

9. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the optical flow prediction method according to any one of claims 1-6 is implemented.

Images