Video tracking method and device based on optical flow estimation, equipment and storage medium

Abstract

The application provides a video tracking method and device based on optical flow estimation, equipment and storage medium, wherein the method comprises: first inputting the images of two adjacent frames in a video, i.e. a first frame image and a second frame image, into a dense optical flow prediction network to obtain a first predicted image of the first frame image, and inputting the first predicted image, the first frame image and the second frame image into a sparse optical flow prediction network to obtain a target predicted image as a tracking result of the second frame image. The dense optical flow prediction network can track the obvious changes of the tracking object in the first frame image, and the sparse optical flow prediction network can finely adjust and correct the first predicted image, thereby improving the accuracy of optical flow prediction. Moreover, the tracking mark on the first frame image is deformed according to the optical flow predicted by the dense optical flow prediction network and the sparse optical flow prediction network, thereby improving the accuracy and reliability of video tracking.

Description

Technical Field

[0001] This application relates to the field of video tracking technology, and more specifically, to a video tracking method, apparatus, device, and storage medium based on optical flow estimation. Background Technology

[0002] Optical flow estimation utilizes pixel changes in an image sequence and the correlation between adjacent frames to find the correspondence between pixels in the previous and current frames, thereby calculating the motion information of pixels between adjacent frames. Optical flow estimation has many applications, such as action recognition, autonomous driving, video editing, and video tracking.

[0003] In recent years, Convolutional Neural Networks (CNNs) have greatly promoted the development of the field of computer vision. For the problem of optical flow estimation, a variety of optical flow CNN models have been proposed. These methods can directly predict optical flow by training the network.

[0004] However, when using existing optical flow CNN models for optical flow estimation, the accuracy of optical flow prediction for fast-moving objects is low, so the reliability of video tracking based on optical flow is difficult to guarantee. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a video tracking method, apparatus, device, and storage medium based on optical flow estimation, so as to solve the problem that the reliability of video tracking is difficult to guarantee in the prior art.

[0006] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0007] In a first aspect, this application provides a video tracking method based on optical flow estimation, the method comprising:

[0008] Acquire a first frame image and a second frame image, wherein the first frame image is the frame image preceding the second frame image, and the first frame image includes: a tracking object and a tracking identifier attached to the tracking object;

[0009] The first frame image and the second frame image are input into a dense optical flow prediction network to predict the dense optical flow information of the first frame image and the second frame image. Based on the dense optical flow information, a first predicted image of the first frame image is obtained. The first predicted image includes: the tracking object and a first deformation identifier of the tracking identifier attached to the tracking object.

[0010] The first predicted image, the first frame image, and the second frame image are input into a sparse optical flow prediction network to predict the sparse optical flow information of the first frame image and the second frame image. Based on the sparse optical flow information and the first predicted image, a target predicted image of the first frame image is predicted. The target predicted image is used as the tracking result of the second frame image. The target predicted image includes the tracking object and the target deformation identifier of the tracking identifier attached to the tracking object.

[0011] Optionally, the dense optical flow prediction network includes: a plurality of dense optical flow prediction sub-networks connected in sequence;

[0012] The step of inputting the first frame image and the second frame image into a dense optical flow prediction network, predicting the dense optical flow information of the first frame image and the second frame image, and obtaining a first predicted image of the first frame image based on the dense optical flow information includes:

[0013] The first frame image and the second frame image are input into the first dense optical flow prediction sub-network to predict the initial dense optical flow information of the first frame image and the second frame image;

[0014] Based on the initial dense optical flow information and the first frame image, the initial prediction image of the first dense optical flow prediction sub-network is obtained;

[0015] The initial predicted image and the second frame image are input into the next dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork, and each dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork performs prediction processing in sequence to obtain the dense optical flow information of the first frame image and the second frame image.

[0016] The first predicted image of the first frame image is obtained based on the dense optical flow information.

[0017] Optionally, obtaining the initial prediction image of the first dense optical flow prediction sub-network based on the initial dense optical flow information and the first frame image includes:

[0018] Interpolation is performed on the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction sub-network.

[0019] Optionally, the step of interpolating the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction sub-network includes:

[0020] Based on the initial dense optical flow information, interpolation calculations are performed on the first frame image to determine the deformed position of each pixel in the first frame image;

[0021] Each pixel is moved to its corresponding deformed position to obtain the initial predicted image.

[0022] Optionally, the step of inputting the first predicted image, the first frame image, and the second frame image into a sparse optical flow prediction network, predicting the sparse optical flow information of the first frame image and the second frame image, and predicting the target predicted image of the first frame image based on the sparse optical flow information and the first predicted image includes:

[0023] The first predicted image, the first frame image, and the second frame image are input into the sparse optical flow prediction network, and the sparse optical flow prediction network predicts the sparse optical flow information based on the first frame image and the second frame image.

[0024] The first predicted image is interpolated based on the sparse optical flow information to obtain the target predicted image of the first frame image.

[0025] Optionally, the step of predicting the sparse optical flow information by the sparse optical flow prediction network based on the first frame image and the second frame image includes:

[0026] The sparse optical flow prediction network performs feature point detection on the first frame image and the second frame image to obtain multiple first feature points in the first frame image and the corresponding second feature points in the second frame image. The sparse optical flow information is then determined based on the distance between each first feature point and each second feature point.

[0027] Optionally, before inputting the first frame image and the second frame image into the dense optical flow prediction network, the following steps are included:

[0028] Obtain a pre-labeled first sample image and a pre-labeled second sample image, wherein the first sample image and the second sample image are respectively labeled with the tracking object and the corresponding tracking identifier;

[0029] The first sample image and the second sample image are input into the initial dense optical flow prediction network to obtain the first predicted sample image of the first sample image;

[0030] The first predicted sample image, the first sample image and the second sample image are input into the initial sparse optical flow prediction network to obtain the target predicted sample image of the sample image;

[0031] The loss value is determined based on the target prediction sample image and the second sample image, and the initial dense optical flow prediction network and the initial sparse optical flow prediction network are iteratively corrected based on the loss value to obtain the dense optical flow prediction network and the sparse optical flow prediction network.

[0032] Secondly, this application provides a video tracking device based on optical flow estimation, the device comprising:

[0033] The acquisition module is used to: acquire a first frame image and a second frame image, wherein the first frame image is the frame image preceding the second frame image, and the first frame image includes: a tracking object and a tracking identifier attached to the tracking object;

[0034] A dense optical flow prediction module is used to: input the first frame image and the second frame image into a dense optical flow prediction network, predict the dense optical flow information of the first frame image and the second frame image, and obtain a first predicted image of the first frame image based on the dense optical flow information. The first predicted image includes: the tracking object and a first deformation identifier of the tracking identifier attached to the tracking object.

[0035] A sparse optical flow prediction module is configured to: input the first predicted image, the first frame image, and the second frame image into a sparse optical flow prediction network; predict the sparse optical flow information of the first frame image and the second frame image; predict the target prediction image of the first frame image based on the sparse optical flow information and the first predicted image; and use the target prediction image as the tracking result of the second frame image. The target prediction image includes the tracking object and the target deformation identifier of the tracking identifier attached to the tracking object.

[0036] Optionally, the dense optical flow prediction network includes: a plurality of dense optical flow prediction sub-networks connected in sequence;

[0037] The dense optical flow prediction module is also used for:

[0038] The first frame image and the second frame image are input into the first dense optical flow prediction sub-network to predict the initial dense optical flow information of the first frame image and the second frame image;

[0039] Based on the initial dense optical flow information and the first frame image, the initial prediction image of the first dense optical flow prediction sub-network is obtained;

[0040] The initial predicted image and the second frame image are input into the next dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork, and each dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork performs prediction processing in sequence to obtain the dense optical flow information of the first frame image and the second frame image.

[0041] The first predicted image of the first frame image is obtained based on the dense optical flow information.

[0042] Optionally, the dense optical flow prediction module is further used for:

[0043] Interpolation is performed on the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction sub-network.

[0044] Optionally, the dense optical flow prediction module is further used for:

[0045] Based on the initial dense optical flow information, interpolation calculations are performed on the first frame image to determine the deformed position of each pixel in the first frame image;

[0046] Each pixel is moved to its corresponding deformed position to obtain the initial predicted image.

[0047] Optionally, the sparse optical flow prediction module is further used for:

[0048] The first predicted image, the first frame image, and the second frame image are input into the sparse optical flow prediction network, and the sparse optical flow prediction network predicts the sparse optical flow information based on the first frame image and the second frame image.

[0049] The first predicted image is interpolated based on the sparse optical flow information to obtain the target predicted image of the first frame image.

[0050] Optionally, the sparse optical flow prediction module is further used for:

[0051] The sparse optical flow prediction network performs feature point detection on the first frame image and the second frame image to obtain multiple first feature points in the first frame image and the corresponding second feature points in the second frame image. The sparse optical flow information is then determined based on the distance between each first feature point and each second feature point.

[0052] Thirdly, this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the video tracking method based on optical flow estimation as described above.

[0053] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the video tracking method based on optical flow estimation described above.

[0054] The beneficial effects of this application are: by using a dense optical flow prediction network, significant changes (such as large motion displacement or large deformation) of the tracked object in the first frame image can be tracked; by using a sparse optical flow prediction network, the first prediction image can be finely adjusted and corrected, thereby improving the accuracy of optical flow prediction; and by using the optical flow predicted by the dense and sparse optical flow prediction networks to perform corresponding deformation on the tracking markers in the first frame image, the accuracy and reliability of video tracking can be improved. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this application is shown;

[0057] Figure 2 A flowchart of a video tracking method based on optical flow estimation provided in an embodiment of this application is shown;

[0058] Figure 3 This document illustrates a flowchart of a method for determining a first predicted image, as provided in an embodiment of this application.

[0059] Figure 4 A schematic diagram of the structure of a dense optical flow prediction network provided in an embodiment of this application is shown;

[0060] Figure 5 This document illustrates a flowchart of a method for determining an initial predicted image, as provided in an embodiment of this application.

[0061] Figure 6 This document illustrates a flowchart of a method for determining a target prediction image according to an embodiment of this application.

[0062] Figure 7 The flowcharts illustrating a training method for a dense optical flow prediction network and a sparse optical flow prediction network provided in embodiments of this application are shown.

[0063] Figure 8 The flowchart of another method for training a dense optical flow prediction network and a sparse optical flow prediction network provided in the embodiments of this application is shown;

[0064] Figure 9 This illustration shows a schematic diagram of a video tracking device based on optical flow estimation provided in an embodiment of this application;

[0065] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0067] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0068] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0069] In recent years, convolutional neural networks have greatly promoted the development of the field of computer vision. Inspired by the success of deep learning in advanced vision tasks, researchers in this field have proposed a variety of optical flow CNN models. These models can directly predict the optical flow between a pair of frames, avoiding the optimization problem of traditional optical flow methods (energy minimization optimization function).

[0070] However, when using existing optical flow CNN models for optical flow estimation, such as when estimating the optical flow of objects in two adjacent frames of a video, each pixel in the previous frame can be reasonably paired with thousands of points in the other frame, which is computationally intensive and has low accuracy for optical flow prediction of fast-moving objects.

[0071] Therefore, when performing video tracking based on the optical flow predicted by the above method, the low accuracy of the optical flow leads to unreliable video tracking results.

[0072] To address the aforementioned issues, this application proposes a video tracking method based on optical flow estimation. By predicting optical flow from coarse to fine and detecting feature points to fine-tune and optimize the optical flow, this method can fully utilize the information between two adjacent frames to obtain a high-precision, smooth optical flow field, and achieve reliable video tracking results based on the optical flow field.

[0073] Figure 1 This is a schematic diagram illustrating an application scenario of the method provided in this application, such as... Figure 1 As shown, two adjacent frames of a fitness video are input into an electronic device. The back of person A is selected as the tracking object in the previous frame, and a texture is added to the tracking object. The electronic device can then add a texture to the back of person A in the next frame based on the method of this application, and the texture will change accordingly with the movement or deformation of person A's back.

[0074] Next, combine Figure 2 The video tracking method based on optical flow estimation in this application will be further described. The subject executing this method can be an electronic device, such as... Figure 2 As shown, the method includes:

[0075] S201: Acquire the first frame image and the second frame image. The first frame image is the frame image preceding the second frame image. The first frame image includes: the tracking object and the tracking identifier attached to the tracking object.

[0076] Optionally, the first frame image can be an image sampled from any position in the video, and the second frame image can be an image in the video that is adjacent to and follows the first frame image.

[0077] Optionally, the object to be tracked can be an object in the first frame of the image that the user wishes to track. For example, suppose the user wants to track the movement information of person A's back in a fitness video, referring to... Figure 1 Therefore, the tracking object here can be the back of A, and the tracking marker can be the "texture marker" added to the back of A in the first frame image.

[0078] Optionally, the tracking identifier can be text, stickers, or other identifiers that can visually represent changes in the position, shape, and size of the tracked object in the video. For example, if the tracked object gradually shrinks, the tracking identifier can also shrink proportionally along with the tracked object. This application does not limit the specific form of the tracking identifier.

[0079] For example, an electronic device can sample images from two adjacent frames of a video and perform preprocessing such as photometric enhancement, spatial enhancement, and occlusion enhancement to obtain a first frame image and a second frame image, and add a tracking marker to the tracking object in the first frame image. Specifically, photometric enhancement can be achieved by randomly adjusting the brightness, contrast, saturation, and hue of the image with a probability of 0.2; spatial enhancement can be achieved by randomly changing and stretching the image scale with a probability of 0.8; and occlusion enhancement can be achieved by randomly erasing rectangular areas of the image with a probability of 0.5 to simulate occlusion.

[0080] S202: Input the first frame image and the second frame image into the dense optical flow prediction network, predict the dense optical flow information of the first frame image and the second frame image, and obtain the first predicted image of the first frame image based on the dense optical flow information. The first predicted image includes: the tracking object and the first deformation mark of the tracking mark attached to the tracking object.

[0081] Optionally, the dense optical flow prediction network can be a network or model that performs dense optical flow prediction on the first and second input images.

[0082] The dense optical flow prediction network may include a decoder and an encoder for extracting features from a first frame image and a second frame image, predicting at least one dense optical flow from the first frame image and the second frame image, and performing deformation processing on the first frame image based on the dense optical flow to obtain a first predicted image of the first frame image.

[0083] Optionally, the dense optical flow information can be an optical flow field composed of the offsets of all points on the first and second frame images. Point-to-point, pixel-level image registration can be performed based on the dense optical flow information.

[0084] Optionally, the first predicted image may be an image obtained by deforming the tracking marker in the first frame image according to dense optical flow information.

[0085] It is worth noting that the shape, size, and position of the tracking marker in the first frame should change to the first deformation marker as the tracked object changes in the first frame.

[0086] S203: Input the first predicted image, the first frame image, and the second frame image into the sparse optical flow prediction network, predict the sparse optical flow information of the first frame image and the second frame image, and predict the target prediction image of the first frame image based on the sparse optical flow information and the first predicted image. Use the target prediction image as the tracking result of the second frame image. The target prediction image includes: the tracking object and the target deformation mark of the tracking mark attached to the tracking object.

[0087] Optionally, the sparse optical flow prediction network can be a network or model that performs sparse optical flow prediction on the first and second input images.

[0088] Optionally, sparse optical flow information can be used to achieve image registration based on corresponding key points in the image. The tracking markers in the first predicted image can then be deformed again based on the sparse optical flow information to obtain the target predicted image.

[0089] Optionally, the target prediction image can be an image obtained by deforming the first deformation identifier in the first prediction image according to sparse optical flow information.

[0090] As one possible implementation, using the target prediction image as the tracking result of the second frame image can be achieved by copying the tracking identifier in the target prediction image to the corresponding position in the second frame image.

[0091] It is worth noting that the above steps of predicting the image based on dense and sparse optical flow information can be understood as performing deformation processing on the tracking marker based on sparse and dense optical flow information to obtain the final target deformation marker, thus completing the video tracking.

[0092] In this application, the images of two adjacent frames in the video, namely the first frame and the second frame, are first input into a dense optical flow prediction network to obtain a first predicted image of the first frame. Then, the first predicted image, the first frame, and the second frame are input into a sparse optical flow prediction network to obtain a target predicted image, which serves as the tracking result for the second frame. The dense optical flow prediction network can track significant changes in the tracked object in the first frame (such as large displacements or deformations). The sparse optical flow prediction network then performs fine-tuning and correction on the first predicted image, improving the accuracy of optical flow prediction. Furthermore, based on the optical flow predicted by both the dense and sparse optical flow prediction networks, the tracking markers on the first frame are deformed accordingly, further enhancing the accuracy and reliability of video tracking.

[0093] The dense optical flow prediction network in this application will be described next. The dense optical flow prediction network may include multiple dense optical flow prediction sub-networks connected in sequence.

[0094] like Figure 3 As shown, in step S202 above, the first frame image and the second frame image are input into the dense optical flow prediction network to predict the dense optical flow information of the first frame image and the second frame image, and the first predicted image of the first frame image is obtained based on the dense optical flow information, including:

[0095] S301: Input the first frame image and the second frame image into the first dense optical flow prediction sub-network to predict the initial dense optical flow information of the first frame image and the second frame image.

[0096] Optionally, the dense optical flow prediction network may include multiple dense optical flow prediction sub-networks connected in sequence, for example, Figure 4 A schematic diagram of a dense optical flow prediction network including three dense optical flow prediction sub-networks is given. The first dense optical flow pre-network can output initial dense optical flow information based on the input first frame image and second frame image.

[0097] Optionally, the initial dense optical flow information can be dense optical flow information with lower precision, which can characterize the optical flow field with lower precision between the first frame image and the second frame image.

[0098] S302: Based on the initial dense optical flow information and the first frame image, obtain the initial prediction image of the first dense optical flow prediction subnetwork.

[0099] Optional, continue to refer to Figure 4 Based on the initial dense optical flow information and the first frame image, the initial prediction image of the first dense optical flow prediction sub-network can be obtained. The initial prediction image can be understood as the image obtained by deforming the tracking marker in the first frame image based on the initial dense optical flow information.

[0100] S303: Input the initial predicted image and the second frame image into the next dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork, and have each dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork perform prediction processing in sequence to obtain the dense optical flow information of the first frame image and the second frame image.

[0101] It is worth noting that after the first dense optical flow prediction subnetwork, each dense optical flow prediction subnetwork can take the dense optical flow information output by the previous dense optical flow prediction subnetwork and the second frame image as input, and finally obtain the dense optical flow information of the first frame image and the second frame image.

[0102] Optionally, the accuracy of the dense optical flow information output by each dense optical flow prediction sub-network can be increased sequentially, that is, the accuracy of the dense optical flow information output by the later dense optical flow prediction sub-network is higher.

[0103] For example, refer to Figure 4 After inputting the first and second frame images into the first dense optical flow prediction subnetwork, a dense optical flow information 1 can be obtained. Based on the dense optical flow information 1, the first frame image is predicted to obtain the initial prediction image, which is... Figure 4 The predicted image is 1. Using the predicted image 1 and the second frame image as input to the second dense optical flow prediction sub-network, dense optical flow information 2 can be obtained. Next, the predicted image 1 is predicted based on the dense optical flow information 2 to obtain the predicted image 2. Then, the predicted image 2 and the second frame image are used as input to the third dense optical flow prediction sub-network to obtain dense optical flow information 3. Finally, the predicted image 2 is predicted based on the dense optical flow information 3 to obtain the predicted image 3 (i.e., the first predicted image).

[0104] It should be noted that, Figure 4 The accuracy of dense optical flow information 3 is greater than that of dense optical flow information 2, and the accuracy of dense optical flow information 2 is greater than that of dense optical flow information 1. This allows for iterative refinement of the optical flow from coarse to fine, resulting in more accurate and reliable predicted images.

[0105] S304: Obtain the first predicted image of the first frame image based on the dense optical flow information.

[0106] For example, Figure 4 Based on the dense optical flow information 3 and the prediction image 2, the first prediction image of the first frame image, namely the prediction image 3, can be obtained.

[0107] It is worth noting that the dense optical flow prediction subnetworks in the dense optical flow prediction network of this application do not need to share parameters. Except for the first dense optical flow prediction subnetwork, the other dense optical flow prediction subnetworks can be fine-tuned and optimized based on the parameters of the previous dense optical flow prediction subnetwork, thereby accelerating the convergence speed.

[0108] In this embodiment, the first frame image and the second frame image are predicted by a dense optical flow prediction sub-network. The resulting first predicted image can quickly capture the obvious displacement and shape changes of the tracked object, thus achieving rapid tracking object detection.

[0109] The following describes the steps for obtaining the initial prediction image of the first dense optical flow prediction sub-network based on the initial dense optical flow information and the first frame image. Step S302 includes:

[0110] Interpolation is performed on the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction subnetwork.

[0111] Optionally, the interpolation operation can be nearest neighbor interpolation, bicubic interpolation, or bilinear interpolation, with bilinear interpolation being a preferred method.

[0112] Reference Figure 4 The initial dense optical flow information can be the dense optical flow information 1 in the figure. By performing interpolation operation on the first frame image based on the dense optical flow information 1, the predicted image 1, i.e. the initial predicted image, can be obtained.

[0113] like Figure 5 The diagram illustrates the steps involved in interpolating the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction subnetwork. These steps include:

[0114] S501: Perform interpolation calculations on the first frame image based on the initial dense optical flow information to determine the deformed position of each pixel in the first frame image.

[0115] Optionally, interpolation calculations can be performed on each pixel in the first frame image based on the initial dense optical flow information. It should be noted that since the calculated values ​​may be floating-point numbers, while the pixel positions in the image are required to be integers, an interpolation algorithm, such as bilinear interpolation, can be used to perform the interpolation calculations to determine the deformed positions of each pixel in the first frame image.

[0116] S502: Move each pixel to its corresponding deformed position to obtain the initial predicted image.

[0117] Optionally, after obtaining the deformed positions of each pixel in the first frame image, each pixel in the first frame image can be moved to its corresponding deformed position to obtain the initial prediction image.

[0118] For example, assuming that the deformed position of point A in the first frame image is A' after interpolation calculation, then point A in the first frame image can be moved to A', and this operation can be performed on all points in the first frame image to obtain the initial predicted image.

[0119] It is worth noting that, Figure 4 The prediction of the image based on dense optical flow information can be achieved through the steps S501-S502 described above. Figure 4 The prediction image 1 can be interpolated based on the dense optical flow information 2 to obtain the prediction image 2. Then, the prediction image 2 can be interpolated based on the dense optical flow information 3 to obtain the prediction image 3, which is the aforementioned first prediction image.

[0120] After obtaining the first predicted image through the dense optical flow network, this application can input the first predicted image, the first frame image, and the second frame image into the sparse optical flow prediction network to predict the sparse optical flow information of the first frame image and the second frame image, and predict the target predicted image of the first frame image based on the sparse optical flow information and the first predicted image, such as... Figure 6 As shown, the above step S203 includes:

[0121] S601: Input the first predicted image, the first frame image, and the second frame image into the sparse optical flow prediction network, and the sparse optical flow prediction network predicts the sparse optical flow information based on the first frame image and the second frame image.

[0122] Optionally, the sparse optical flow information can be the optical flow composed of the offsets of corresponding points in the first and second frame images. For example, refer to... Figure 1 The corresponding points in the first and second frame images can be key points on the tracked object (A's back).

[0123] S602: Interpolate the first predicted image based on the sparse optical flow information to obtain the target predicted image of the first frame image.

[0124] Optionally, the step of interpolating the first predicted image based on the sparse optical flow information can refer to steps S501-S502 above to obtain the target predicted image of the first frame image, i.e. Figure 1 The specific process of tracking the next frame image is not detailed here.

[0125] The following is a further explanation of the steps described above, in which the sparse optical flow prediction network predicts sparse optical flow information based on the first and second frame images. Step S601 includes:

[0126] The sparse optical flow prediction network performs feature point detection on the first frame image and the second frame image to obtain multiple first feature points in the first frame image and the corresponding second feature points in the second frame image. The sparse optical flow information is determined based on the distance between each first feature point and each second feature point.

[0127] Optionally, a feature point detection algorithm can be used to detect feature points in the first frame image and the second frame image to obtain multiple first feature points in the first frame image and the corresponding second feature points in the second frame image.

[0128] For example, the feature point detection algorithm can be the Speeded Up Robust Feature (SURF) algorithm, the Scale Invariant Feature Transform (SIFT) algorithm, the Feature from Accelerated Segment Test (FAST) algorithm, etc., and this application does not limit it.

[0129] Before performing video tracking based on optical flow estimation as described above, the dense optical flow prediction network and sparse optical flow prediction network of this application can be trained first, such as... Figure 7 The following is a detailed explanation of the training process:

[0130] S701: Obtain a pre-labeled first sample image and a pre-labeled second sample image, wherein the first sample image and the second sample image are respectively labeled with the tracking object and the corresponding tracking identifier.

[0131] Optionally, the first sample image and the second sample image can be images from two adjacent frames. The first sample image includes the labeled tracking object and the tracking markers attached to the tracking object, while the second sample image may have the tracking object pre-labeled and the tracking markers pre-labeled on the tracking object.

[0132] S702: Input the first sample image and the second sample image into the initial dense optical flow prediction network to obtain the first predicted sample image of the first sample image.

[0133] Optionally, the initial dense optical flow prediction network may include multiple initial dense optical flow prediction sub-networks connected in sequence, used to predict dense optical flow information based on the input image.

[0134] Optionally, the first predicted sample image may be a deformed image obtained by interpolating the first sample image based on dense optical flow information.

[0135] S703: Input the first predicted sample image, the first sample image, and the second sample image into the initial sparse optical flow prediction network to obtain the target predicted sample image of the sample image.

[0136] Optionally, the initial sparse optical flow prediction network can be used to generate sparse optical flow information based on the first sample image and the second sample image, and to interpolate the first prediction sample image based on the sparse optical flow information to obtain the final deformed target prediction sample image.

[0137] S704: Determine the loss value based on the first predicted sample image, the target predicted sample image, and the second sample image, and iteratively correct the initial dense optical flow prediction network and the initial sparse optical flow prediction network based on the loss value to obtain the dense optical flow prediction network and the sparse optical flow prediction network.

[0138] Optionally, based on the tracking identifier predicted from the first prediction sample image and the tracking identifier predicted from the target prediction sample image, the loss value of the initial sparse optical flow prediction network can be determined, and the initial sparse optical flow prediction network can be iteratively corrected based on the loss value to obtain the sparse optical flow prediction network.

[0139] Optionally, based on the pre-annotated tracking markers in the second sample image and the tracking markers predicted from the first prediction sample image, the loss value of the initial dense optical flow prediction network can be determined, and the initial dense optical flow prediction network can be iteratively corrected based on the loss value to obtain the dense optical flow prediction network.

[0140] Reference Figure 8 The loss function can consist of two parts: first, image similarity, which ensures the accuracy of predicted optical flow by maximizing the correlation between the first and second predicted sample images and the target predicted sample image and the second predicted sample image; second, constrained optical flow smoothness, which applies regularization terms to sparse and dense optical flow information. The formula is as follows:

[0141] L ncc =ncc(frame1',frame2)+ncc(frame1”,frame2)

[0142] L smooth =l1(Flow) l )+l1(Flow m )+l1(Flow h )+l1(Flow s )

[0143] L = L ncc +ɑL smooth

[0144] Where α is a parameter, for example, α can be set to 1000. frame1' can be the first predicted sample image, frame2 can be the second sample image, and frame1' can be the target predicted sample image. Flow l Flow m Flow h These can be categorized as low-precision dense optical flow information, medium-precision dense optical flow information, and high-precision dense optical flow information, respectively. s It can be sparse optical flow information.

[0145] Based on the same inventive concept, this application also provides a video tracking device based on optical flow estimation, which corresponds to the video tracking method based on optical flow estimation. Since the principle of the device in this application is similar to the video tracking method based on optical flow estimation described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0146] Reference Figure 9 The diagram shown is a schematic of a video tracking device based on optical flow estimation provided in an embodiment of this application. The device includes: an acquisition module 901, a dense optical flow prediction module 902, and a sparse optical flow prediction module 903, wherein:

[0147] The acquisition module 901 is used to: acquire a first frame image and a second frame image, wherein the first frame image is the frame image preceding the second frame image, and the first frame image includes: the tracking object and the tracking identifier attached to the tracking object;

[0148] The dense optical flow prediction module 902 is used to: input a first frame image and a second frame image into a dense optical flow prediction network, predict the dense optical flow information of the first frame image and the second frame image, and obtain a first predicted image of the first frame image based on the dense optical flow information. The first predicted image includes: a tracking object and a first deformation identifier of a tracking identifier attached to the tracking object.

[0149] The sparse optical flow prediction module 903 is used to: input a first predicted image, a first frame image, and a second frame image into a sparse optical flow prediction network, predict the sparse optical flow information of the first frame image and the second frame image, and predict the target prediction image of the first frame image based on the sparse optical flow information and the first predicted image, and use the target prediction image as the tracking result of the second frame image. The target prediction image includes: the tracking object and the target deformation mark of the tracking mark attached to the tracking object.

[0150] Optionally, the dense optical flow prediction network includes: multiple dense optical flow prediction sub-networks connected in sequence;

[0151] The dense optical flow prediction module 902 is also used for:

[0152] The first and second frame images are input into the first dense optical flow prediction sub-network to predict the initial dense optical flow information of the first and second frame images;

[0153] Based on the initial dense optical flow information and the first frame image, the initial prediction image of the first dense optical flow prediction subnetwork is obtained;

[0154] The initial predicted image and the second frame image are input into the next dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork, and each dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork performs prediction processing in sequence to obtain the dense optical flow information of the first frame image and the second frame image.

[0155] The first predicted image of the first frame is obtained based on the dense optical flow information.

[0156] Optionally, the dense optical flow prediction module 902 is also used for:

[0157] Interpolation is performed on the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction subnetwork.

[0158] Optionally, the dense optical flow prediction module 902 is also used for:

[0159] Interpolation calculations are performed on the first frame image based on the initial dense optical flow information to determine the deformed position of each pixel in the first frame image;

[0160] Each pixel is moved to its corresponding deformed position to obtain the initial predicted image.

[0161] Optionally, the sparse optical flow prediction module 903 is also used for:

[0162] The first predicted image, the first frame image, and the second frame image are input into the sparse optical flow prediction network, and the sparse optical flow prediction network predicts sparse optical flow information based on the first frame image and the second frame image.

[0163] The first predicted image is interpolated based on the sparse optical flow information to obtain the target predicted image of the first frame image.

[0164] Optionally, the sparse optical flow prediction module 903 is also used for:

[0165] The sparse optical flow prediction network performs feature point detection on the first frame image and the second frame image to obtain multiple first feature points in the first frame image and the corresponding second feature points in the second frame image. The sparse optical flow information is determined based on the distance between each first feature point and each second feature point.

[0166] Optionally, the apparatus of this application may further include a training module for:

[0167] Obtain a pre-labeled first sample image and a pre-labeled second sample image, wherein the first sample image and the second sample image are respectively labeled with the tracking object and the corresponding tracking identifier;

[0168] The first sample image and the second sample image are input into the initial dense optical flow prediction network to obtain the first predicted sample image of the first sample image;

[0169] The first predicted sample image, the first sample image, and the second sample image are input into the initial sparse optical flow prediction network to obtain the target predicted sample image of the sample image;

[0170] The loss value is determined based on the target prediction sample image and the second sample image. The initial dense optical flow prediction network and the initial sparse optical flow prediction network are then iteratively corrected based on the loss value to obtain the dense optical flow prediction network and the sparse optical flow prediction network.

[0171] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0172] This application embodiment uses a dense optical flow prediction network to track significant changes (such as large motion displacements or large deformations) of the tracked object in the first frame image. Then, a sparse optical flow prediction network is used to finely adjust and correct the first prediction image, improving the accuracy of optical flow prediction. Furthermore, based on the optical flow predicted by the dense and sparse optical flow prediction networks, the tracking markers on the first frame image are deformed accordingly, improving the accuracy and reliability of video tracking.

[0173] This application also provides an electronic device, such as... Figure 10 The diagram shown is a schematic representation of an electronic device structure provided in an embodiment of this application, including: a processor 1001, a memory 1002, and a bus. The memory 1002 stores machine-readable instructions executable by the processor 1001 (e.g., ...). Figure 9 The device acquires the execution instructions corresponding to the module 901, dense optical flow prediction module 902, and sparse optical flow prediction module 903. When the computer device is running, the processor 1001 communicates with the memory 1002 via a bus. When the machine-readable instructions are executed by the processor 1001, the above-mentioned video tracking method based on optical flow estimation is performed.

[0174] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the video tracking method based on optical flow estimation described above.

[0175] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A video tracking method based on optical flow estimation, characterized in that, include: Acquire a first frame image and a second frame image, wherein the first frame image is the frame image preceding the second frame image, and the first frame image includes: a tracking object and a tracking identifier attached to the tracking object; The first frame image and the second frame image are input into a dense optical flow prediction network to predict the dense optical flow information of the first frame image and the second frame image. Based on the dense optical flow information, a first predicted image of the first frame image is obtained. The first predicted image includes: the tracking object and a first deformation identifier of the tracking identifier attached to the tracking object. The first predicted image, the first frame image, and the second frame image are input into a sparse optical flow prediction network to predict the sparse optical flow information of the first frame image and the second frame image. Based on the sparse optical flow information and the first predicted image, a target predicted image of the first frame image is predicted. The target predicted image is used as the tracking result of the second frame image. The target predicted image includes: the tracking object and the target deformation identifier of the tracking identifier attached to the tracking object. The step of inputting the first predicted image, the first frame image, and the second frame image into a sparse optical flow prediction network, predicting the sparse optical flow information of the first frame image and the second frame image, and predicting the target predicted image of the first frame image based on the sparse optical flow information and the first predicted image includes: The first predicted image, the first frame image, and the second frame image are input into a sparse optical flow prediction network. The sparse optical flow prediction network performs feature point detection on the first frame image and the second frame image to obtain multiple first feature points in the first frame image and the second feature points corresponding to each first feature point in the second frame image. The sparse optical flow information is determined based on the distance between each first feature point and each second feature point. The first predicted image is interpolated based on the sparse optical flow information to obtain the target predicted image of the first frame image.

2. The method according to claim 1, characterized in that, The dense optical flow prediction network includes: multiple dense optical flow prediction sub-networks connected in sequence; The step of inputting the first frame image and the second frame image into a dense optical flow prediction network, predicting the dense optical flow information of the first frame image and the second frame image, and obtaining a first predicted image of the first frame image based on the dense optical flow information includes: The first frame image and the second frame image are input into the first dense optical flow prediction sub-network to predict the initial dense optical flow information of the first frame image and the second frame image; Based on the initial dense optical flow information and the first frame image, the initial prediction image of the first dense optical flow prediction sub-network is obtained; The initial predicted image and the second frame image are input into the next dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork, and each dense optical flow prediction subnetwork after the first dense optical flow prediction subnetwork performs prediction processing in sequence to obtain the dense optical flow information of the first frame image and the second frame image. The first predicted image of the first frame image is obtained based on the dense optical flow information.

3. The method according to claim 2, characterized in that, The step of obtaining the initial prediction image of the first dense optical flow prediction sub-network based on the initial dense optical flow information and the first frame image includes: Interpolation is performed on the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction sub-network.

4. The method according to claim 3, characterized in that, The step of interpolating the initial dense optical flow information and the first frame image to obtain the initial prediction image of the first dense optical flow prediction sub-network includes: Based on the initial dense optical flow information, interpolation calculations are performed on the first frame image to determine the deformed position of each pixel in the first frame image; Each pixel is moved to its corresponding deformed position to obtain the initial predicted image.

5. The method according to any one of claims 1-4, characterized in that, Before inputting the first frame image and the second frame image into the dense optical flow prediction network, the following steps are included: Obtain a pre-labeled first sample image and a pre-labeled second sample image, wherein the first sample image and the second sample image are respectively labeled with the tracking object and the corresponding tracking identifier; The first sample image and the second sample image are input into the initial dense optical flow prediction network to obtain the first predicted sample image of the first sample image; The first predicted sample image, the first sample image and the second sample image are input into the initial sparse optical flow prediction network to obtain the target predicted sample image of the sample image; The loss value is determined based on the target prediction sample image and the second sample image, and the initial dense optical flow prediction network and the initial sparse optical flow prediction network are iteratively corrected based on the loss value to obtain the dense optical flow prediction network and the sparse optical flow prediction network.

6. A video tracking device based on optical flow estimation, characterized in that, include: The acquisition module is used to: acquire a first frame image and a second frame image, wherein the first frame image is the frame image preceding the second frame image, and the first frame image includes: a tracking object and a tracking identifier attached to the tracking object; A dense optical flow prediction module is used to: input the first frame image and the second frame image into a dense optical flow prediction network, predict the dense optical flow information of the first frame image and the second frame image, and obtain a first predicted image of the first frame image based on the dense optical flow information. The first predicted image includes: the tracking object and a first deformation identifier of the tracking identifier attached to the tracking object. A sparse optical flow prediction module is used to: input the first predicted image, the first frame image, and the second frame image into a sparse optical flow prediction network, predict the sparse optical flow information of the first frame image and the second frame image, and predict the target prediction image of the first frame image based on the sparse optical flow information and the first predicted image, and use the target prediction image as the tracking result of the second frame image. The target prediction image includes: the tracking object and the target deformation identifier of the tracking identifier attached to the tracking object. The sparse optical flow prediction module is used for: The first predicted image, the first frame image, and the second frame image are input into a sparse optical flow prediction network. The sparse optical flow prediction network performs feature point detection on the first frame image and the second frame image to obtain multiple first feature points in the first frame image and the second feature points corresponding to each first feature point in the second frame image. The sparse optical flow information is determined based on the distance between each first feature point and each second feature point. The first predicted image is interpolated based on the sparse optical flow information to obtain the target predicted image of the first frame image.

7. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus. The processor executes the program instructions to perform the steps of the video tracking method based on optical flow estimation as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the video tracking method based on optical flow estimation as described in any one of claims 1 to 5.

Images