Optical flow estimation method and device, electronic equipment and storage medium

Abstract

The invention provides an optical flow estimation method and device, electronic equipment and a storage medium, and belongs to the field of computer vision. The method comprises the steps of determining cost body information between pixels in a first video frame and pixels in a second video frame based on a first feature map and a second feature map; performing super-pixel segmentation on the first feature map to obtain a super-pixel segmentation result of the first feature map; and on the basis of the cost body information, the superpixel segmentation result and the first feature map, updating initial optical flow information to obtain target optical flow information between the first video frame and the second video frame. According to the scheme, the optical flow information between the video frames can be determined by using the cost body information between the video frames, information compensation can be carried out on ill-conditioned areas such as occlusion and weak texture in the video frames according to the clustering information of the multiple pixels, the optical flow estimation difficulty of the areas is reduced, and the accuracy of optical flow estimation is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to an optical flow estimation method, device, electronic device and storage medium. Background Art

[0002] In the field of computer vision, the goal of optical flow estimation is to determine the motion information between two consecutive frames in a video. Specifically, it is to predict the corresponding pixel in the second frame for each pixel in the first frame. Since the optical flow estimation results contain rich scene motion information, they can be widely used in scenes such as inter-frame alignment, video encoding and decoding, and scene motion estimation. Therefore, how to perform optical flow estimation on video frames and determine the optical flow information of video frames is a technical problem that needs to be solved.

[0003] In the related art, the optical flow estimation method is mainly implemented based on convolutional neural networks. The method first extracts features from two consecutive video frames based on the convolutional neural network. Then, the distance between the feature vectors of each pixel in the two video frames is calculated based on the extracted image features, and the similarity between each pixel is determined based on the distance. Finally, based on the similarity between each pixel, the optical flow estimation results of the consecutive video frames are determined, that is, the matching relationship between the pixels is determined.

[0004] However, the above scheme only determines the optical flow information of the video frame based on the displacement relationship of the pixels in the previous and next frames. For weak texture areas and occluded areas in the video frame, the difference in the feature vectors of the pixels is not obvious, so it is difficult to determine the matching relationship between the pixels, resulting in low accuracy of optical flow estimation. Summary of the invention

[0005] The embodiments of the present application provide an optical flow estimation method, device, electronic device and storage medium, which can not only determine the optical flow information between video frames using the cost volume information between video frames, but also compensate for the occlusion and weak texture of the video frame according to the clustering information of multiple pixels, thereby reducing the difficulty of optical flow estimation in these areas and improving the accuracy of optical flow estimation. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a method for optical flow estimation is provided, the method comprising:

[0007] Determining cost volume information between pixels in a first video frame and pixels in a second video frame based on a first feature map and a second feature map, wherein the first feature map is obtained by extracting features of the first video frame based on a feature extraction network, and the second feature map is obtained by extracting features of the second video frame based on the feature extraction network, the first video frame is a video frame adjacent to and preceding the second video frame, and the cost volume information is used to indicate a degree of similarity between pixels in the first video frame and pixels in the second video frame;

[0008] Performing superpixel segmentation on the first feature map to obtain a superpixel segmentation result of the first feature map, wherein the superpixel segmentation is used to divide pixels having similar pixel features in the first feature map into the same pixel region, and the superpixel segmentation result is used to indicate that the pixels having similar pixel features in the first feature map are located in the same pixel region;

[0009] Based on the cost volume information, the superpixel segmentation result and the first feature map, the initial optical flow information is updated to obtain the target optical flow information between the first video frame and the second video frame, the initial optical flow information is a vector field of all zeros, and the target optical flow information is used to indicate the position matching relationship between the pixels in the first video frame and the pixels in the second video frame.

[0010] According to another aspect of an embodiment of the present application, an optical flow estimation device is provided, the device comprising:

[0011] a determination module, configured to determine cost volume information between pixels in a first video frame and pixels in a second video frame based on a first feature map and a second feature map, wherein the first feature map is obtained by extracting features of the first video frame based on a feature extraction network, and the second feature map is obtained by extracting features of the second video frame based on the feature extraction network, the first video frame is a video frame adjacent to and preceding the second video frame, and the cost volume information is used to indicate a degree of similarity between pixels in the first video frame and pixels in the second video frame;

[0012] a segmentation module, configured to perform superpixel segmentation on the first feature map to obtain a superpixel segmentation result of the first feature map, wherein the superpixel segmentation is used to divide pixels having similar pixel features in the first feature map into the same pixel region, and the superpixel segmentation result is used to indicate that pixels having similar pixel features in the first feature map are located in the same pixel region;

[0013] An updating module is used to update the initial optical flow information based on the cost volume information, the superpixel segmentation result and the first feature map to obtain the target optical flow information between the first video frame and the second video frame, wherein the initial optical flow information is a vector field of all zeros, and the target optical flow information is used to indicate the position matching relationship between the pixels in the first video frame and the pixels in the second video frame.

[0014] In some embodiments, the cost volume information includes a plurality of cost volume sub-information, and the cost volume sub-information is used to indicate the similarity between any pixel in the first video frame and each pixel in the second video frame;

[0015] The determining module comprises:

[0016] A first determining unit, configured to determine a pixel feature of each pixel in the first video frame based on the first feature map;

[0017] A second determining unit, configured to determine a pixel feature of each pixel in the second video frame based on the second feature map;

[0018] The third determining unit is configured to determine, for any pixel in the first video frame, cost volume sub-information of the pixel based on a pixel feature of the pixel and a pixel feature of each pixel in the second video frame.

[0019] In some embodiments, the segmentation module includes:

[0020] A first division unit, used for randomly dividing the first feature map into a plurality of mutually independent pixel regions;

[0021] A fourth determining unit is configured to determine, for a first pixel region among the plurality of pixel regions, a plurality of second pixel regions adjacent to the first pixel region from among the plurality of pixel regions, wherein the first pixel region is any pixel region among the plurality of pixel regions;

[0022] a fifth determining unit, configured to determine, for any pixel in the first pixel area, a target pixel area from the first pixel area and the plurality of second pixel areas based on a pixel feature of the pixel, a pixel feature of each pixel in the first pixel area except the pixel, and a pixel feature of each pixel in the plurality of second pixel areas, the target pixel area being an area where a target pixel is located and a pixel feature of the target pixel having the highest degree of similarity to a pixel feature of the pixel;

[0023] The second division unit is used to divide the pixels into the target pixel area.

[0024] In some embodiments, the fifth determination unit is used to determine, for any pixel in the first pixel area, a pixel type of the pixel based on the position of the pixel in the first pixel area; when the pixel type is an edge pixel, determine the target pixel area from the first pixel area and the multiple second pixel areas based on the pixel features of the pixel, the pixel features of each pixel except the pixel in the first pixel area, and the pixel features of each pixel in the multiple second pixel areas; when the pixel type is not the edge pixel, determine the first pixel area as the target pixel area.

[0025] In some embodiments, the update module includes:

[0026] an acquisition unit, configured to acquire a plurality of initial homography matrices, wherein the initial homography matrix is ​​a unit matrix, and the initial homography matrix is ​​used to indicate a mapping relationship between pixels in a third pixel region and pixels in a fourth pixel region, wherein the third pixel region is any pixel region in the first video frame, and the fourth pixel region is a pixel region in the second video frame having the same region identifier as the third pixel region;

[0027] an information encoding unit, configured to encode the initial optical flow information based on the multiple initial homography matrices to obtain motion encoding information, wherein the motion encoding information is used to indicate a motion relationship between pixels in the first video frame and the second video frame;

[0028] An information processing unit, configured to sample and encode the cost volume information based on the initial optical flow information to obtain cost volume encoding information;

[0029] An information updating unit is used to update the initial optical flow information based on the motion coding information, the cost volume coding information, the first feature map and the superpixel segmentation result to obtain the target optical flow information.

[0030] In some embodiments, the information processing unit is used to determine, for any pixel in the first video frame, a matching pixel having a position matching relationship with the pixel from the second video frame based on the initial optical flow information; sample the cost body information based on the pixel, the matching pixel and a plurality of pixels adjacent to the matching pixel to obtain cost body sampling information, wherein the cost body sampling information is used to indicate the degree of similarity between the pixel and the matching pixel and the degree of similarity between the pixel and a plurality of pixels adjacent to the matching pixel; and encode the cost body sampling information to obtain the cost body encoding information.

[0031] In some embodiments, the information updating unit is used to encode the motion coding information, the cost body coding information and the first feature map to obtain gated recurrent network GRU coding information, and the GRU coding information is used to indicate the difference between the motion coding information and the cost body coding information; based on the GRU coding information and the superpixel segmentation result, an optical flow update result of the initial optical flow information is determined; based on the optical flow update result, the initial optical flow information is updated to obtain the target optical flow information.

[0032] In some embodiments, the information updating unit is also used to determine, from the second video frame, matching pixels that have a position matching relationship with each pixel in the first video frame based on the target optical flow information; determine, based on the superpixel segmentation result, a pixel region where each pixel in the first video frame is located, the pixel region including multiple pixels with similar pixel features; determine, based on the matching pixels and the pixel region where each pixel in the first video frame is located, a matching pixel region corresponding to the pixel region, the matching pixel region being the pixel region where the matching pixel is located in the second video frame, and the matching pixel region having the same region identifier as the pixel region; and update the initial homography matrix based on a mapping relationship between pixels in the pixel region and matching pixels in the matching pixel region.

[0033] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory; the memory stores at least one program code, and the at least one program code is used to be executed by the processor to implement the optical flow estimation method as described in the above aspect.

[0034] According to another aspect of an embodiment of the present application, a chip is provided, which includes a programmable logic circuit and / or program instructions. When the chip runs on an electronic device, it is used to implement the optical flow estimation method described in the above aspects.

[0035] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores at least one program code, and the at least one program code is used to be executed by a processor to implement the optical flow estimation method as described in the above aspects.

[0036] According to another aspect of an embodiment of the present application, a computer program product is provided. The computer program product stores at least one program code, and the at least one program code is used to be executed by a processor to implement the optical flow estimation method described in the above aspect.

[0037] The embodiment of the present application provides an optical flow estimation method, which can determine the cost volume information indicating the similarity between pixels in the first video frame and the second video frame by analyzing the first feature map and the second feature map corresponding to the adjacent first video frame and the second video frame. Then, by performing superpixel segmentation on the first feature map, the pixel area where pixels with similar pixel features are located in the first feature map, that is, the clustering information of multiple pixels, can be determined based on the superpixel segmentation result of the first feature map. Compared with the traditional optical flow estimation method, the optical flow estimation method provided by the present application can not only determine the optical flow information between video frames using the cost volume information between video frames, but also perform information compensation for pathological areas such as occlusion and weak texture in the video frame based on the clustering information of multiple pixels, thereby reducing the difficulty of optical flow estimation in these areas and improving the accuracy of optical flow estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 Schematic diagram of an implementation environment of an optical flow estimation method provided in an embodiment of the present application;

[0040] Figure 2 is a flow chart of an optical flow estimation method provided by an embodiment of the present application;

[0041] Figure 3 is a flowchart of another optical flow estimation method provided by an embodiment of the present application;

[0042] Figure 4 It is an overall framework diagram of an optical flow estimation method provided in an embodiment of the present application;

[0043] Figure 5 is a schematic diagram of an optical flow update module provided in an embodiment of the present application;

[0044] Figure 6 is a structural schematic diagram of an optical flow estimation device provided in an embodiment of the present application;

[0045] Figure 7 is a schematic diagram of the structure of another optical flow estimation device provided in an embodiment of the present application;

[0046] Figure 8 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0048] In this article, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0049] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the first video frame and the second video frame involved in this application are both obtained with full authorization.

[0050] The optical flow method provided in the embodiment of the present application can be executed by an electronic device. In some embodiments, the electronic device is a terminal or a server. The following takes the electronic device as an example to introduce the implementation environment of the model quantization method provided in the embodiment of the present application. Figure 1 Schematic diagram of an implementation environment of an optical flow estimation method provided in an embodiment of the present application. Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 can be directly or indirectly connected to the server 102 through wired or wireless communication, which is not limited in this application.

[0051] In some embodiments, the terminal 101 may be a mobile phone, a desktop computer, a laptop computer, a tablet computer, an intelligent voice interaction device, a vehicle-mounted terminal, or other types of terminals. An application is installed and run on the terminal 101. The terminal 101 can perform optical flow estimation on any two consecutive video frames in a video through the application.

[0052] In some embodiments, the server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 102 can provide background services for the application.

[0053] In some embodiments, the server 102 undertakes the main computing work and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture to perform collaborative computing.

[0054] Figure 2 1 is a flow chart of an optical flow estimation method provided in an embodiment of the present application. The method is executed by an electronic device, which is the terminal 101 or the server 102 in the above embodiment. Figure 2 , the method comprising:

[0055] 201. Based on a first feature map and a second feature map, determine cost volume information between pixels in a first video frame and pixels in a second video frame, wherein the first feature map is obtained by extracting features of the first video frame based on a feature extraction network, and the second feature map is obtained by extracting features of the second video frame based on a feature extraction network, and the first video frame is a video frame adjacent to and in front of the second video frame, and the cost volume information is used to indicate a degree of similarity between pixels in the first video frame and pixels in the second video frame.

[0056] In an embodiment of the present application, the first video frame and the second video frame to be subjected to optical flow estimation are any two adjacent video frames in the target video. The electronic device extracts features from the input first video frame and the second video frame respectively through a feature extraction network, and can obtain a first feature map and a second feature map. The first feature map is used to represent the pixel features of the pixels in the first video frame, and the second feature map is used to represent the pixel features of the pixels in the second video frame. Based on the first feature map and the second feature map, the electronic device can determine the cost body information indicating the degree of similarity between the pixels in the first video frame and the pixels in the second video frame. Among them, the first video frame can be the first video frame in the target video, or it can be other video frames, and the second video frame is a video frame after the first video frame. The target video can be a video stored locally in the electronic device, or it can be a video currently shot by the electronic device. The embodiment of the present application does not limit the format and source of the target video. The feature extraction network can be a CNN (Convolutional Neural Network), such as ResNet (residual network), MobileNet (mobile network), etc., or a Transformer (self-attention) network, such as Swin (Shifted window), ViT (Vision Transformer, visual self-attention model), etc.

[0057] 202. Perform superpixel segmentation on the first feature map to obtain a superpixel segmentation result of the first feature map, where the superpixel segmentation is used to divide pixels with similar pixel features in the first feature map into the same pixel region, and the superpixel segmentation result is used to indicate that the pixels with similar pixel features in the first feature map are located in the same pixel region.

[0058] In an embodiment of the present application, superpixel segmentation is an image segmentation technology that can group pixels using the similarity between pixel features. Since the first feature map is used to represent the pixel features of pixels in the first video frame, the electronic device can divide the pixels with similar pixel features in the first feature map into the same pixel area during the process of superpixel segmentation of the first feature map, and obtain the superpixel segmentation result of the first feature map. Among them, the superpixel segmentation result is used to indicate that the pixels with similar pixel features in the first feature map are located in the same pixel area, and the first feature map includes multiple independent pixel areas, each of which includes multiple pixels with similar pixel features.

[0059] Optionally, the electronic device can input the extracted first feature map into a superpixel segmentation network to obtain a superpixel segmentation result of the first feature map. The embodiment of the present application can use a UNet network as a superpixel segmentation network to perform superpixel segmentation on the first feature map, and can also sample other networks as superpixel segmentation networks. The embodiment of the present application does not limit the network structure of the superpixel segmentation network.

[0060] 203. Based on the cost volume information, the superpixel segmentation result and the first feature map, the initial optical flow information is updated to obtain the target optical flow information between the first video frame and the second video frame. The initial optical flow information is a vector field of all zeros, and the target optical flow information is used to indicate the position matching relationship between the pixels in the first video frame and the pixels in the second video frame.

[0061] In an embodiment of the present application, in order to improve the accuracy of optical flow estimation, the electronic device can update the initial optical flow information multiple times. During the first update, the initial optical flow information is set to a vector field of all zeros, indicating that the motion displacement of the pixel between the first video frame and the second video frame is 0. Based on the cost body information, the electronic device can determine the similarity between the pixels in the first video frame and the pixels in the second video frame. The higher the similarity between the pixels, the higher the degree of matching between the pixels. Based on the superpixel segmentation result of the first feature map, the electronic device can determine the pixel area where the pixel in the first video frame is located, that is, the clustering relationship between any pixel in the first video frame and multiple adjacent pixels. Based on the first feature map, the electronic device can determine the pixel features of multiple pixels in the first video frame. Therefore, based on the cost body information, the superpixel segmentation result and the first feature map, the electronic device can update the initial optical flow information to obtain the target optical flow information.

[0062] The embodiment of the present application provides an optical flow estimation method, which can determine the cost volume information indicating the similarity between pixels in the first video frame and the second video frame by analyzing the first feature map and the second feature map corresponding to the adjacent first video frame and the second video frame. Then, by performing superpixel segmentation on the first feature map, the pixel area where pixels with similar pixel features are located in the first feature map, that is, the clustering information of multiple pixels, can be determined based on the superpixel segmentation result of the first feature map. Compared with the traditional optical flow estimation method, the optical flow estimation method provided by the present application can not only determine the optical flow information between video frames using the cost volume information between video frames, but also perform information compensation for pathological areas such as occlusion and weak texture in the video frame based on the clustering information of multiple pixels, thereby reducing the difficulty of optical flow estimation in these areas and improving the accuracy of optical flow estimation.

[0063] Figure 3 1 is a flowchart of another optical flow estimation method provided in an embodiment of the present application. The method is executed by an electronic device, which is the terminal 101 or the server 102 in the above embodiment. Figure 3 , the method comprising:

[0064] 301. Determine a pixel feature of each pixel in a first video frame based on a first feature map, where the first feature map is obtained by extracting features of the first video frame based on a feature extraction network.

[0065] In an embodiment of the present application, the electronic device extracts features from the input first video frame through a feature extraction network, and can obtain a first feature map. The first video frame can be any video frame in the target video, and the first feature map is used to represent the pixel features of the pixels in the first video frame. The electronic device can determine the pixel features of each pixel in the first video frame based on the first feature map. Among them, the target video can be a video stored locally by the electronic device, or it can be a video currently shot by the electronic device. The embodiment of the present application does not limit the format and source of the target video. The feature extraction network can be a CNN (Convolutional Neural Network), such as ResNet (residual network), MobileNet (mobile network), etc., and can also be a Transformer (self-attention) network, such as Swin (Shifted window), ViT (Vision Transformer, visual self-attention model), etc. Pixel features include color features, color features, texture features, shape features, etc.

[0066] 302. Determine pixel features of each pixel in the second video frame based on a second feature map, wherein the second feature map is obtained by extracting features of the second video frame based on a feature extraction network, and the first video frame is a video frame adjacent to and in front of the second video frame.

[0067] In an embodiment of the present application, the electronic device extracts features from the input second video frame through a feature extraction network, and can obtain a second feature map. The second video frame is a video frame adjacent to and behind the first video frame, and the second feature map is used to represent the pixel features of the pixels in the second video frame. The electronic device can determine the pixel features of each pixel in the second video frame based on the second feature map. Among them, the feature extraction network has the same structure as the feature extraction network in the above step 301, and will not be repeated here.

[0068] The following step 303 is a specific process for the electronic device to determine the cost volume information between the pixels in the first video frame and the pixels in the second video frame, and the following steps 304-307 are specific processes for the electronic device to perform superpixel segmentation on the first feature map. It should be noted that the electronic device may first perform the following step 303, or may first perform the following steps 304-307, or may perform steps 303 and steps 304-307 in parallel, and the embodiment of the present application does not limit the execution order of steps 303 and steps 304-307.

[0069] 303. Determine cost volume information between pixels in the first video frame and pixels in the second video frame based on pixel features of each pixel in the first video frame and pixel features of each pixel in the second video frame, wherein the cost volume information is used to indicate a degree of similarity between pixels in the first video frame and pixels in the second video frame.

[0070] In an embodiment of the present application, the electronic device can determine cost volume information indicating the degree of similarity between a pixel in the first video frame and a pixel in the second video frame based on the characteristics of each pixel in the first video frame and the characteristics of each pixel in the second video frame. The cost volume information includes cost volume sub-information of multiple pixels. For any pixel in the first video frame, the electronic device can determine the cost volume sub-information of the pixel based on the pixel characteristics of the pixel and the pixel characteristics of each pixel in the second video frame. The cost volume sub-information of the pixel is used to indicate the degree of similarity between the pixel and each pixel in the second video frame.

[0071] Optionally, the electronic device can determine the spatial distance between the pixel features of any pixel in the first video frame and the pixel features of each pixel in the second video frame as the cost volume sub-information of any pixel in the first video frame. The spatial distance is used to indicate the degree of similarity between any pixel in the first video frame and the pixel in the second video frame. The spatial distance is inversely correlated with the degree of similarity. The closer the spatial distance, the higher the degree of similarity between the pixel in the first video frame and the pixel in the second video frame. The embodiment of the present application uses an existing cost volume calculation scheme to perform cost volume calculation, including VCN (a cosine similarity calculation method), Local VCN, etc.

[0072] It should be noted that the electronic device can obtain a superpixel segmentation result of the first feature map by performing superpixel segmentation on the first feature map through the following steps 304-307. The superpixel segmentation is used to divide pixels with similar pixel features in the first feature map into the same pixel region, and the superpixel segmentation result is used to indicate that pixels with similar pixel features in the first feature map are located in the same pixel region.

[0073] 304. Randomly divide the first feature map into a plurality of mutually independent pixel regions.

[0074] In an embodiment of the present application, in the process of superpixel segmentation of the first feature map, the electronic device first randomly divides the first feature map, and can divide the first feature map into multiple independent pixel regions. Among them, the first feature map includes multiple non-overlapping pixel regions, and the sizes and shapes of the multiple pixel regions can be the same or different. The electronic device can randomly divide the first feature map into multiple pixel regions of the same size and shape, and can also randomly divide the first feature map into multiple pixel regions of unequal sizes and shapes. The embodiment of the present application does not limit the size and shape of the pixel region.

[0075] 305. For a first pixel region among a plurality of pixel regions, determine a plurality of second pixel regions adjacent to the first pixel region from the plurality of pixel regions, where the first pixel region is any pixel region among the plurality of pixel regions.

[0076] In the embodiment of the present application, the first pixel region is any pixel region among the plurality of pixel regions in the first feature map. For the first pixel region, the electronic device can determine a plurality of second pixel regions adjacent to the first pixel region from the plurality of pixel regions according to the first feature map after random division. The second pixel region can be located above, below, to the left, to the right, to the upper left, to the lower left, to the upper right, to the lower right, etc. of the first pixel region.

[0077] 306. For any pixel in the first pixel area, based on the pixel feature of the pixel, the pixel feature of each pixel except the pixel in the first pixel area, and the pixel feature of each pixel in the plurality of second pixel areas, determine a target pixel area from the first pixel area and the plurality of second pixel areas, the target pixel area being the area where the target pixel is located, and the pixel feature of the target pixel is most similar to the pixel feature of the pixel.

[0078] In an embodiment of the present application, the first pixel area includes a plurality of pixels. For the pixel features of any pixel, the electronic device can determine the target pixel area from the first pixel area and the plurality of second pixel areas by determining the degree of similarity between the pixel and the other pixels based on the pixel features of the pixel, the pixel features of other pixels in the first pixel area, and the pixel features of each pixel in the plurality of second pixel areas adjacent to the first pixel area. Among them, the target pixel area is the pixel area where the target pixel with the highest degree of similarity to the pixel features of the pixel is located. If the target pixel is located in the first pixel area, the target pixel area is the first pixel area. If the target pixel is located in the second pixel area, the target pixel area is the second pixel area.

[0079] In some embodiments, the electronic device can determine the area where the target pixel with the highest degree of similarity to the pixel feature of the pixel is located based on the pixel type of the pixel. Accordingly, for any pixel in the first pixel area, the pixel type of the pixel is determined based on the position of the pixel in the first pixel area; when the pixel type is an edge pixel, the target pixel area is determined from the first pixel area and the plurality of second pixel areas based on the pixel feature of the pixel, the pixel feature of each pixel in the first pixel area except the pixel, and the pixel feature of each pixel in the plurality of second pixel areas; when the pixel type is not an edge pixel, the first pixel area is determined as the target pixel area. The first pixel area includes a plurality of pixels, and the electronic device can determine the pixel type of the pixel based on the position of the pixel in the first pixel area. When the pixel type is an edge pixel, it indicates that the pixel is located near the region boundary of the first pixel area, and the degree of similarity between the pixel and other pixels in the first pixel area is likely to be lower than the degree of similarity between the pixel and the pixel in the second pixel area. Therefore, the electronic device can determine the area where the pixels with the highest degree of similarity to the pixel characteristics of the pixel are located from the first pixel area and the multiple second pixel areas based on the pixel characteristics of the pixel, the pixel characteristics of other pixels in the first pixel area, and the pixel characteristics of each pixel in the multiple second pixel areas adjacent to the first pixel area, that is, the target pixel area. In the case where the pixel type is not an edge type, it indicates that the pixel is located near the area center of the first pixel area, and the degree of similarity between the pixel and other pixels in the first pixel area is significantly higher than the degree of similarity between the pixel and the pixels in the second pixel area. Therefore, the electronic device can determine the first pixel area as the target pixel area. Optionally, in the case where the distance between the position of the pixel and the area boundary of the first pixel area is not greater than the distance threshold, the electronic device determines the pixel type of the pixel as an edge pixel.

[0080] 307. Divide the pixels into target pixel areas.

[0081] In an embodiment of the present application, for any pixel in the first pixel region, the electronic device divides the pixel into a pixel region where a target pixel having the highest degree of similarity to the pixel feature of the pixel is located, that is, a target pixel region. After each pixel in the first pixel region is divided, the electronic device can obtain a superpixel segmentation result of the first feature map. The superpixel segmentation result is used to indicate that pixels with similar pixel features in the first feature map are located in the same pixel region.

[0082] Optionally, the superpixel segmentation result is a superpixel segmentation probability map. For a first pixel region in a plurality of pixel regions, the superpixel segmentation probability map can be used to indicate the probability that the first pixel region and a plurality of second pixel regions are target pixel regions. The electronic device can determine the pixel region with the highest probability as the target pixel region. The higher the probability of a pixel region, the higher the similarity between the pixel features of the pixel in the pixel region and the pixel features of any pixel in the first pixel region.

[0083] 308. Obtain multiple initial homography matrices, where the initial homography matrix is ​​a unit matrix, and the initial homography matrix is ​​used to indicate a mapping relationship between pixels in a third pixel region and pixels in a fourth pixel region, where the third pixel region is any pixel region in the first video frame, and the fourth pixel region is a pixel region in the second video frame having the same region identifier as the third pixel region.

[0084] In an embodiment of the present application, the electronic device can determine the pixel area where multiple pixels are located in the first video frame based on the superpixel segmentation result of the first feature map. Since there is local consistency in the movement of pixels between the first video frame and the second video frame, and the pixel area of ​​the first video frame can be regarded as approximately coplanar in physical space, multiple pixels in any pixel area in the first video frame and multiple pixels in the pixel area with the same area identifier as the any pixel area in the second video frame should conform to the same mapping relationship. Among them, the mapping relationship can be represented by a homography matrix. For multiple pixel areas, the electronic device can obtain multiple initial homography matrices. Multiple initial homography matrices correspond one-to-one to multiple pixel areas. The initial homography matrix is ​​used to indicate the mapping relationship between pixels in the third pixel area and pixels in the fourth pixel area, the third pixel area is any pixel area in the first video frame, and the fourth pixel area is a pixel area in the second video frame with the same area identifier as the third pixel area.

[0085] 309. Encode the initial optical flow information based on multiple initial homography matrices to obtain motion coding information, where the initial optical flow information is a vector field of all zeros, and the motion coding information is used to indicate the motion relationship between pixels in the first video frame and the second video frame.

[0086] In an embodiment of the present application, since the initial optical flow information is used to indicate the position matching relationship between pixels in the first video frame and pixels in the second video frame, the initial homography matrix is ​​used to indicate the mapping relationship between multiple pixels in any pixel area in the first video frame and pixels in the second video frame. Therefore, the electronic device encodes the initial optical flow information based on multiple initial homography matrices and can compensate for the initial optical flow information. When a pixel is blocked or the feature of the pixel is not obvious, the electronic device can indirectly determine the matching pixels that have a position matching relationship with the pixel based on the matching pixels that have a position matching relationship with other pixels belonging to the same pixel area as the pixel.

[0087] For example, in a certain pixel area in the first video frame, pixel A is located above pixel B. When the matching pixel of pixel B in the second video frame is blocked, the electronic device can infer the position of the matching pixel of pixel B in the second video frame based on the matching pixel of pixel A in the second video frame and the relative position of pixel A and pixel B. Among them, the matching pixel of pixel A is also located above the matching pixel of pixel B.

[0088] 310. Based on the initial optical flow information, sample and encode the cost volume information to obtain cost volume encoding information.

[0089] In an embodiment of the present application, the electronic device can determine matching pixels in the second video frame that have a position matching relationship with pixels in the first video frame based on the initial optical flow information. Since the cost volume information is used to indicate the degree of similarity between each pixel in the first video frame and each pixel in the second video frame, the electronic device can sample the cost volume information based on the matching pixels of the pixels in the first video frame and the pixels in the second video frame. By encoding the sampled cost volume information, the electronic device can improve the encoding efficiency of the cost volume information and speed up the optical flow estimation of the first video frame and the second video frame.

[0090] In some embodiments, the electronic device can sample and encode the cost volume information between the pixels in the first video frame and the pixels in the second video frame based on the position matching relationship between the pixels in the first video frame and the pixels in the second video frame indicated by the initial optical flow information. Accordingly, for any pixel in the first video frame, based on the initial optical flow information, a matching pixel having a position matching relationship with the pixel is determined from the second video frame; based on the pixel, the matching pixel, and multiple pixels adjacent to the matching pixel, the cost volume information is sampled to obtain cost volume sampling information, and the cost volume sampling information is used to indicate the similarity between the pixel and the matching pixel and the similarity between the pixel and multiple pixels adjacent to the matching pixel; the cost volume sampling information is encoded to obtain cost volume encoding information. Wherein, for any pixel in the first video frame, the electronic device can determine the matching pixel having a position matching relationship with the pixel in the second video frame based on the position matching relationship between the pixel in the first video frame and the pixel in the second video frame indicated by the initial optical flow information. In the process of sampling the cost volume information, since the accuracy of the initial optical flow information is low, the electronic device can sample the cost volume information to obtain the cost volume information indicating the similarity between the pixel and the matching pixel and the pixel and the plurality of pixels adjacent to the matching pixel. By encoding the sampled cost volume information, the electronic device can improve the encoding efficiency of the cost volume information and speed up the optical flow estimation of the first video frame and the second video frame.

[0091] 311. Based on the motion coding information, the cost volume coding information, the first feature map and the superpixel segmentation result, the initial optical flow information is updated to obtain the target optical flow information. The superpixel segmentation result is used to indicate that pixels with similar pixel features in the first feature map are located in the same pixel area. The target optical flow information is used to indicate the position matching relationship between the pixels in the first video frame and the pixels in the second video frame.

[0092] In an embodiment of the present application, in order to improve the accuracy of optical flow estimation, the electronic device can update the initial optical flow information multiple times. During the first update, the initial optical flow information is set to a vector field of all zeros, indicating that the motion displacement of the pixel between the first video frame and the second video frame is 0. The electronic device can determine the motion relationship between the pixel in the first video frame and the second video frame based on the motion coding information. The electronic device can determine the similarity between the pixel in the first video frame and the pixel in the second video frame based on the cost body coding information. The higher the similarity between the pixels, the higher the matching degree between the pixels. The electronic device can determine the pixel area where multiple pixels in the first video frame are located based on the superpixel segmentation result, that is, the clustering relationship between any pixel in the first video frame and multiple adjacent pixels. The electronic device can determine the pixel features of multiple pixels in the first video frame based on the first feature map. Therefore, the electronic device updates the initial optical flow information based on the motion coding information, the cost body coding information, the superpixel segmentation result and the first feature map. The electronic device can determine the target optical flow information between the first video frame and the second video frame based on multiple information, thereby improving the accuracy of optical flow estimation.

[0093] In some embodiments, the electronic device determines the optical flow update result based on multiple information to update the initial optical flow information. Accordingly, the motion coding information, the cost body coding information and the first feature map are encoded to obtain the gated recurrent network GRU coding information, and the GRU coding information is used to indicate the difference between the motion coding information and the cost body coding information; based on the GRU coding information and the superpixel segmentation result, the optical flow update result of the initial optical flow information is determined; based on the optical flow update result, the initial optical flow information is updated to obtain the target optical flow information. Among them, since the motion coding information is the coding information obtained by comprehensively considering the initial optical flow information and the initial homography matrix to indicate the motion relationship between the pixels in the first video frame and the second video frame, the cost body coding information is the coding information of the similarity between the pixels in the first video frame and the pixels in the second video frame determined based on the pixel features. Therefore, in the process of first updating the initial optical flow information, the electronic device can determine the GRU coding information used to indicate the difference between the motion coding information and the cost body coding information by encoding the motion coding information, the cost body coding information and the first feature map. Based on the GRU encoding information, the electronic device can determine the optical flow update result, and based on the optical flow update result, update the initial optical flow information to obtain the target optical flow information, thereby improving the accuracy of the optical flow estimation.

[0094] In some embodiments, the electronic device can update the initial homography matrix based on the target optical flow information and the superpixel segmentation result. Accordingly, based on the target optical flow information, the matching pixels having a position matching relationship with each pixel in the first video frame are determined from the second video frame; based on the superpixel segmentation result, the pixel region where each pixel in the first video frame is located is determined, and the pixel region includes multiple pixels with similar pixel features; based on the matching pixel and the pixel region where each pixel in the first video frame is located, the matching pixel region corresponding to the pixel region is determined, and the matching pixel region is the pixel region where the matching pixel is located in the second video frame, and the matching pixel region is the same as the region identifier of the pixel region; based on the mapping relationship between the pixel in the pixel region and the matching pixel in the matching pixel region, the initial homography matrix is ​​updated. Wherein, based on the target optical flow information, the electronic device can determine the motion displacement of the pixel between the first video frame and the second video frame. Based on the motion displacement, the electronic device can determine the matching pixel having a position matching relationship with each pixel in the first video frame from the second video frame. The electronic device can determine the matching pixel area with the same region identifier as the pixel area, that is, the pixel area where the matching pixel is located, based on the pixel area indicated by the matching pixel and the superpixel segmentation result. For any pixel area, the electronic device can update the initial homography matrix of the pixel area based on the mapping relationship between the pixels in the pixel area and the matching pixels in the matching pixel area. The electronic device can encode the updated initial optical flow information again based on the updated initial homography matrix, that is, the electronic device can repeat the above steps 309-311, update the initial optical flow information multiple times, and obtain the final target optical flow information. In addition, the embodiment of the present application solves the homography matrix in SL3 (3D Special Linear Group, three-dimensional special linear group) space. Compared with the solution method using the least squares method in Euclidean space, this method can improve the estimation efficiency and accuracy of the homography matrix, thereby improving the estimation accuracy and speed of the optical flow information. The embodiment of the present application can obtain the update direction of the homography matrix through the Gaussian-Newton method, and update the homography matrix based on the update direction to obtain the updated homography matrix.

[0095] For example, Figure 4 is an overall framework diagram of an optical flow estimation method provided in an embodiment of the present application. Figure 4As shown, first, the electronic device inputs the first video frame and the second video frame to be subjected to optical flow estimation into the feature extraction network, and the feature extraction network extracts features from the first video frame and the second video frame respectively to obtain the first feature map and the second feature map. Then, the electronic device inputs the first feature map and the second feature map into the cost volume calculation module to obtain the cost volume information between the first video frame and the second video frame. The first feature map is input into the superpixel segmentation module to obtain the superpixel segmentation result of the first feature map. Finally, the electronic device inputs the cost volume information, the superpixel segmentation result, the first feature map and the initial optical flow information into the optical flow update module, and can update the initial optical flow information based on the cost volume information, the superpixel segmentation result and the first feature map to obtain the target optical flow information.

[0096] For example, Figure 5 is a schematic diagram of an optical flow update module provided in an embodiment of the present application. Figure 5 As shown, the optical flow update module is divided into four submodules, namely, the motion coding module, the cost volume coding module, the GRU module and the optical flow estimation module. In the process of first updating the initial optical flow information, the input of the motion coding module is the initial optical flow information and the initial homography matrix, and the input is the motion coding information. Among them, the initial homography matrix can be represented by Lie algebra. The input of the cost volume coding module is the cost volume sampling information obtained by sampling the cost volume information based on the initial optical flow information, and the output is the cost volume coding information. The input of the GRU module is the motion coding information, the cost volume coding information and the first feature map, and the output is the GRU coding information. The input of the optical flow estimation module is the GRU coding information and the superpixel segmentation result, and the output is the optical flow update result. Based on the optical flow update result, the electronic device updates the initial optical flow information, and can obtain the initial optical flow information after this update, that is, the target optical flow information. In addition, the electronic device can also update the initial homography matrix based on the updated initial optical flow information and the superpixel result. At this time, the input of the motion coding module is the updated initial optical flow information and the updated initial homography matrix. The electronic device can continuously update the initial optical flow information based on the optical flow update result, thereby improving the accuracy of the optical flow information.

[0097] The embodiment of the present application provides an optical flow estimation method, which can determine the cost volume information indicating the similarity between pixels in the first video frame and the second video frame by analyzing the first feature map and the second feature map corresponding to the adjacent first video frame and the second video frame. Then, by performing superpixel segmentation on the first feature map, the pixel area where pixels with similar pixel features are located in the first feature map, that is, the clustering information of multiple pixels, can be determined based on the superpixel segmentation result of the first feature map. Compared with the traditional optical flow estimation method, the optical flow estimation method provided by the present application can not only determine the optical flow information between video frames using the cost volume information between video frames, but also perform information compensation for pathological areas such as occlusion and weak texture in the video frame based on the clustering information of multiple pixels, thereby reducing the difficulty of optical flow estimation in these areas and improving the accuracy of optical flow estimation.

[0098] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0099] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0100] Figure 6 is a schematic diagram of the structure of an optical flow estimation device provided in an embodiment of the present application. Figure 6 , the device comprises:

[0101] A determination module 601 is used to determine cost volume information between pixels in a first video frame and pixels in a second video frame based on a first feature map and a second feature map, wherein the first feature map is obtained by extracting features of the first video frame based on a feature extraction network, and the second feature map is obtained by extracting features of the second video frame based on a feature extraction network, and the first video frame is a video frame adjacent to and in front of the second video frame, and the cost volume information is used to indicate a degree of similarity between pixels in the first video frame and pixels in the second video frame;

[0102] A segmentation module 602 is used to perform superpixel segmentation on the first feature map to obtain a superpixel segmentation result of the first feature map, wherein the superpixel segmentation is used to divide pixels having similar pixel features in the first feature map into the same pixel region, and the superpixel segmentation result is used to indicate that the pixels having similar pixel features in the first feature map are located in the same pixel region;

[0103] The updating module 603 is used to update the initial optical flow information based on the cost volume information, the superpixel segmentation result and the first feature map to obtain the target optical flow information between the first video frame and the second video frame. The initial optical flow information is a vector field of all zeros, and the target optical flow information is used to indicate the position matching relationship between the pixels in the first video frame and the pixels in the second video frame.

[0104] In some embodiments, the cost volume information includes a plurality of cost volume sub-information, and the cost volume sub-information is used to indicate the degree of similarity between any pixel in the first video frame and each pixel in the second video frame;

[0105] Figure 7 is a schematic diagram of the structure of another optical flow estimation device provided in an embodiment of the present application, see Figure 7 , determining module 601, including:

[0106] A first determining unit 701 is used to determine a pixel feature of each pixel in a first video frame based on a first feature map;

[0107] A second determining unit 702 is used to determine a pixel feature of each pixel in the second video frame based on the second feature map;

[0108] The third determining unit 703 is configured to determine, for any pixel in the first video frame, cost volume sub-information of the pixel based on the pixel feature of the pixel and the pixel feature of each pixel in the second video frame.

[0109] In some embodiments, see Figure 7 , the segmentation module 602 includes:

[0110] A first division unit 704, configured to randomly divide the first feature map into a plurality of mutually independent pixel regions;

[0111] A fourth determining unit 705 is configured to determine, for a first pixel region among the plurality of pixel regions, a plurality of second pixel regions adjacent to the first pixel region from the plurality of pixel regions, wherein the first pixel region is any pixel region among the plurality of pixel regions;

[0112] a fifth determining unit 706, configured to determine, for any pixel in the first pixel region, a target pixel region from the first pixel region and the plurality of second pixel regions based on the pixel feature of the pixel, the pixel feature of each pixel other than the pixel in the first pixel region, and the pixel feature of each pixel in the plurality of second pixel regions, the target pixel region being a region where the target pixel is located, and the pixel feature of the target pixel has the highest degree of similarity with the pixel feature of the pixel;

[0113] The second division unit 707 is used to divide the pixels into target pixel areas.

[0114] In some embodiments, the fifth determination unit 706 is used to determine the pixel type of the pixel for any pixel in the first pixel area based on the position of the pixel in the first pixel area; when the pixel type is an edge pixel, determine the target pixel area from the first pixel area and the multiple second pixel areas based on the pixel features of the pixel, the pixel features of each pixel except the pixel in the first pixel area, and the pixel features of each pixel in the multiple second pixel areas; when the pixel type is not an edge pixel, determine the first pixel area as the target pixel area.

[0115] In some embodiments, see Figure 7 , the update module 603 includes:

[0116] an acquisition unit 708, configured to acquire a plurality of initial homography matrices, where the initial homography matrix is ​​a unit matrix, and the initial homography matrix is ​​used to indicate a mapping relationship between pixels in a third pixel region and pixels in a fourth pixel region, where the third pixel region is any pixel region in the first video frame, and the fourth pixel region is a pixel region in the second video frame having the same region identifier as the third pixel region;

[0117] An information encoding unit 709, configured to encode the initial optical flow information based on the multiple initial homography matrices to obtain motion encoding information, where the motion encoding information is used to indicate a motion relationship between pixels in the first video frame and the second video frame;

[0118] An information processing unit 710 is used to sample and encode the cost volume information based on the initial optical flow information to obtain cost volume encoding information;

[0119] The information updating unit 711 is used to update the initial optical flow information based on the motion coding information, the cost volume coding information, the first feature map and the superpixel segmentation result to obtain the target optical flow information.

[0120] In some embodiments, the information processing unit 710 is used to determine, for any pixel in the first video frame, a matching pixel having a position matching relationship with the pixel from the second video frame based on the initial optical flow information; sample the cost volume information based on the pixel, the matching pixel, and a plurality of pixels adjacent to the matching pixel to obtain cost volume sampling information, wherein the cost volume sampling information is used to indicate the degree of similarity between the pixel and the matching pixel and the degree of similarity between the pixel and a plurality of pixels adjacent to the matching pixel; and encode the cost volume sampling information to obtain cost volume encoding information.

[0121] In some embodiments, the information updating unit 711 is used to encode the motion coding information, the cost body coding information and the first feature map to obtain the gated recurrent network GRU coding information, where the GRU coding information is used to indicate the difference between the motion coding information and the cost body coding information; based on the GRU coding information and the superpixel segmentation result, the optical flow update result of the initial optical flow information is determined; based on the optical flow update result, the initial optical flow information is updated to obtain the target optical flow information.

[0122] In some embodiments, the information updating unit 711 is also used to determine, from the second video frame based on the target optical flow information, matching pixels that have a position matching relationship with each pixel in the first video frame; based on the superpixel segmentation result, determine the pixel region where each pixel in the first video frame is located, the pixel region includes multiple pixels with similar pixel features; based on the matching pixels and the pixel region where each pixel in the first video frame is located, determine the matching pixel region corresponding to the pixel region, the matching pixel region is the pixel region where the matching pixel is located in the second video frame, and the matching pixel region is the same as the region identifier of the pixel region; based on the mapping relationship between the pixels in the pixel region and the matching pixels in the matching pixel region, update the initial homography matrix.

[0123] The embodiment of the present application provides an optical flow estimation device, which can determine the cost volume information indicating the similarity between pixels in the first video frame and the second video frame by analyzing the first feature map and the second feature map corresponding to the adjacent first video frame and the second video frame. Then, by performing superpixel segmentation on the first feature map, the pixel area where pixels with similar pixel features are located in the first feature map, that is, the clustering information of multiple pixels, can be determined based on the superpixel segmentation result of the first feature map. Compared with the traditional optical flow estimation method, the optical flow estimation method provided by the present application can not only determine the optical flow information between video frames using the cost volume information between video frames, but also perform information compensation for pathological areas such as occlusion and weak texture in the video frame based on the clustering information of multiple pixels, thereby reducing the difficulty of optical flow estimation in these areas and improving the accuracy of optical flow estimation.

[0124] It should be noted that the device provided in the above embodiment, when implementing its functions, only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0125] An embodiment of the present application provides an electronic device, which includes a processor and a memory; the memory stores at least one program code, and the at least one program code is used to be executed by the processor to implement the optical flow estimation method provided by the above-mentioned various method embodiments.

[0126] Figure 8 800 is a block diagram of a structure of an electronic device 800 provided according to an embodiment of the present application. In some embodiments, the electronic device 800 is a smart phone, a tablet computer, a computer with wireless communication function, or a wearable device. The electronic device 800 in the present application includes at least one or more of the following components: a processor 810, a memory 820, and at least two wireless links 830.

[0127] In some embodiments, the processor 810 includes one or more processing cores. The processor 810 uses various interfaces and lines to connect various parts of the entire electronic device 800, and executes various functions and processes data of the electronic device 800 by running or executing program codes stored in the memory 820, and calling data stored in the memory 820. In some embodiments, the processor 810 is implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 810 can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the display screen; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 810, but may be implemented by a separate chip.

[0128] In some embodiments, the processor 810 is used to control the working conditions of at least two wireless links 830. Accordingly, the processor 810 is a processor integrated with a wireless fidelity (Wireless Fidelity, Wi-Fi) chip. The Wi-Fi chip is a chip with dual Wi-Fi processing capabilities. For example, the Wi-Fi chip is a dual-band dual concurrent (Dual Band Dual Concurrent, DBDC) chip, or a dual-band synchronous (Dual Band Simultaneous, DBS) chip, etc.

[0129] In some embodiments, the memory 820 includes a random access memory (RAM), and in some embodiments, the memory 820 includes a read-only memory (ROM). In some embodiments, the memory 820 includes a non-transitory computer-readable storage medium. The memory 820 can be used to store program codes. The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data (such as audio data, a phone book) created according to the use of the electronic device 800, etc.

[0130] In some embodiments, the memory 820 stores reception schemes for receiving beacon frames of different wireless links 830, as well as identifiers of access nodes connected to different wireless links 830, identifiers of wireless links 830, and the like.

[0131] The at least two wireless links 830 are used to connect different access points (APs) to receive downlink data sent by the APs, wherein the different access points are access points in the same router or access points in different routers.

[0132] In some embodiments, the electronic device 800 also includes a display screen. The display screen is a display component for displaying a user interface. In some embodiments, the display screen is a display screen with a touch function, through which the user can use any suitable object such as a finger or a stylus to perform touch operations on the display screen. In some embodiments, the display screen is generally arranged on the front panel of the electronic device 800. In some embodiments, the display screen is designed to be a full screen, a curved screen, a special-shaped screen, a double-sided screen or a folding screen. In some embodiments, the display screen is also designed to be a combination of a full screen and a curved screen, a combination of a special-shaped screen and a curved screen, etc., which is not limited in this embodiment.

[0133] In addition, those skilled in the art will appreciate that the structure of the electronic device 800 shown in the above figures does not limit the electronic device 800, and the electronic device 800 includes more or fewer components than shown, or combines certain components, or arranges components differently. For example, the electronic device 800 also includes components such as a microphone, a speaker, an input unit, a sensor, an audio circuit, a module, a power supply, and a Bluetooth module, which will not be described in detail here.

[0134] The present application also provides a computer-readable storage medium, which stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the optical flow estimation method shown in the above embodiments.

[0135] The present application also provides a chip, which includes a programmable logic circuit and / or program instructions. When the chip runs on a terminal, it is used to implement the optical flow estimation method shown in the above embodiments.

[0136] The present application also provides a computer program product, which stores at least one program code, and the at least one program code is used to be executed by a processor to implement the optical flow estimation method shown in the above embodiments.

[0137] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0138] Those skilled in the art can understand that all or part of the steps in the optical flow estimation method of the above embodiment can be completed by hardware, or by a program to instruct the relevant hardware to complete, and the program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc. The above is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An optical flow estimation method, characterized in that: The method comprises: Determining cost volume information between pixels in a first video frame and pixels in a second video frame based on a first feature map and a second feature map, wherein the first feature map is obtained by extracting features of the first video frame based on a feature extraction network, and the second feature map is obtained by extracting features of the second video frame based on the feature extraction network, the first video frame is a video frame adjacent to and preceding the second video frame, and the cost volume information is used to indicate a degree of similarity between pixels in the first video frame and pixels in the second video frame; Performing superpixel segmentation on the first feature map to obtain a superpixel segmentation result of the first feature map, wherein the superpixel segmentation is used to divide pixels having similar pixel features in the first feature map into the same pixel region, and the superpixel segmentation result is used to indicate that the pixels having similar pixel features in the first feature map are located in the same pixel region; Based on the cost volume information, the superpixel segmentation result and the first feature map, the initial optical flow information is updated to obtain the target optical flow information between the first video frame and the second video frame, the initial optical flow information is a vector field of all zeros, and the target optical flow information is used to indicate the position matching relationship between the pixels in the first video frame and the pixels in the second video frame.

2. The method according to claim 1, characterized in that: The cost volume information includes a plurality of cost volume sub-information, wherein the cost volume sub-information is used to indicate the similarity between any pixel in the first video frame and each pixel in the second video frame; The determining, based on the first feature map and the second feature map, cost volume information between pixels in the first video frame and pixels in the second video frame includes: Determining a pixel feature of each pixel in the first video frame based on the first feature map; Determining a pixel feature of each pixel in the second video frame based on the second feature map; For any pixel in the first video frame, cost volume sub-information of the pixel is determined based on a pixel feature of the pixel and a pixel feature of each pixel in the second video frame.

3. The method according to claim 1, characterized in that: The performing superpixel segmentation on the first feature map to obtain a superpixel segmentation result of the first feature map includes: Randomly dividing the first feature map into a plurality of mutually independent pixel regions; For a first pixel region among the plurality of pixel regions, determining a plurality of second pixel regions adjacent to the first pixel region from the plurality of pixel regions, the first pixel region being any pixel region among the plurality of pixel regions; For any pixel in the first pixel area, based on the pixel feature of the pixel, the pixel feature of each pixel in the first pixel area except the pixel, and the pixel feature of each pixel in the plurality of second pixel areas, determine a target pixel area from the first pixel area and the plurality of second pixel areas, the target pixel area being an area where the target pixel is located, and the pixel feature of the target pixel has the highest degree of similarity with the pixel feature of the pixel; The pixels are divided into the target pixel area.

4. The method according to claim 3, characterized in that The method further comprises: determining, for any pixel in the first pixel region, a target pixel region from the first pixel region and the plurality of second pixel regions based on a pixel feature of the pixel, a pixel feature of each pixel in the first pixel region except the pixel, and a pixel feature of each pixel in the plurality of second pixel regions, comprising: For any pixel in the first pixel area, determining a pixel type of the pixel based on a position of the pixel in the first pixel area; In a case where the pixel type is an edge pixel, determining the target pixel region from the first pixel region and the plurality of second pixel regions based on a pixel feature of the pixel, a pixel feature of each pixel other than the pixel in the first pixel region, and a pixel feature of each pixel in the plurality of second pixel regions; In a case where the pixel type is not the edge pixel, the first pixel region is determined as the target pixel region.

5. The method according to claim 1, characterized in that The updating of the initial optical flow information based on the cost volume information, the superpixel segmentation result, and the first feature map to obtain target optical flow information between the first video frame and the second video frame includes: Acquire a plurality of initial homography matrices, where the initial homography matrices are unit matrices, and the initial homography matrices are used to indicate a mapping relationship between pixels in a third pixel region and pixels in a fourth pixel region, where the third pixel region is any pixel region in the first video frame, and the fourth pixel region is a pixel region in the second video frame that has the same region identifier as the third pixel region; Based on the multiple initial homography matrices, the initial optical flow information is encoded to obtain motion encoding information, where the motion encoding information is used to indicate a motion relationship between pixels in the first video frame and the second video frame; Based on the initial optical flow information, sampling and encoding the cost volume information to obtain cost volume encoding information; Based on the motion coding information, the cost volume coding information, the first feature map and the superpixel segmentation result, the initial optical flow information is updated to obtain the target optical flow information.

6. The method according to claim 5, characterized in that The step of sampling and encoding the cost volume information based on the initial optical flow information to obtain the cost volume encoding information includes: For any pixel in the first video frame, based on the initial optical flow information, determining a matching pixel having a position matching relationship with the pixel from the second video frame; Based on the pixel, the matching pixel, and a plurality of pixels adjacent to the matching pixel, sampling the cost volume information to obtain cost volume sampling information, wherein the cost volume sampling information is used to indicate a similarity between the pixel and the matching pixel and a similarity between the pixel and a plurality of pixels adjacent to the matching pixel; The cost body sampling information is encoded to obtain the cost body encoding information.

7. The method according to claim 5, characterized in that The updating of the initial optical flow information based on the motion coding information, the cost volume coding information, the first feature map and the superpixel segmentation result to obtain the target optical flow information includes: Encoding the motion coding information, the cost body coding information and the first feature map to obtain gated recurrent network GRU coding information, where the GRU coding information is used to indicate a difference between the motion coding information and the cost body coding information; Determine an optical flow update result of the initial optical flow information based on the GRU encoding information and the superpixel segmentation result; Based on the optical flow update result, the initial optical flow information is updated to obtain the target optical flow information.

8. The method according to claim 5, characterized in that The method further comprises: Based on the target optical flow information, determining, from the second video frame, matching pixels having a position matching relationship with each pixel in the first video frame; Based on the superpixel segmentation result, determine a pixel region where each pixel in the first video frame is located, where the pixel region includes a plurality of pixels having similar pixel features; Based on the matching pixel and the pixel region where each pixel in the first video frame is located, determining a matching pixel region corresponding to the pixel region, the matching pixel region being the pixel region where the matching pixel is located in the second video frame, and the matching pixel region has the same region identifier as the pixel region; The initial homography matrix is ​​updated based on a mapping relationship between pixels in the pixel region and matching pixels in the matching pixel region.

9. An optical flow estimation device, characterized in that: The device comprises: a determination module, configured to determine cost volume information between pixels in a first video frame and pixels in a second video frame based on a first feature map and a second feature map, wherein the first feature map is obtained by extracting features of the first video frame based on a feature extraction network, and the second feature map is obtained by extracting features of the second video frame based on the feature extraction network, the first video frame is a video frame adjacent to and preceding the second video frame, and the cost volume information is used to indicate a degree of similarity between pixels in the first video frame and pixels in the second video frame; a segmentation module, configured to perform superpixel segmentation on the first feature map to obtain a superpixel segmentation result of the first feature map, wherein the superpixel segmentation is used to divide pixels having similar pixel features in the first feature map into the same pixel region, and the superpixel segmentation result is used to indicate that pixels having similar pixel features in the first feature map are located in the same pixel region; An updating module is used to update the initial optical flow information based on the cost volume information, the superpixel segmentation result and the first feature map to obtain the target optical flow information between the first video frame and the second video frame, wherein the initial optical flow information is a vector field of all zeros, and the target optical flow information is used to indicate the position matching relationship between the pixels in the first video frame and the pixels in the second video frame.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory; the memory stores at least one program code, and the at least one program code is used to be executed by the processor to implement the optical flow estimation method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The storage medium stores at least one program code, and the at least one program code is used to be executed by a processor to implement the optical flow estimation method according to any one of claims 1 to 8.

12. A chip, characterized in that: The chip includes a programmable logic circuit and / or program instructions, and when the chip runs on an electronic device, it is used to implement the optical flow estimation method according to any one of claims 1 to 8.

13. A computer program product, comprising a computer program, characterized in that The computer program product stores at least one program code, and the at least one program code is used to be executed by a processor to implement the optical flow estimation method according to any one of claims 1 to 8.

Images