Methods for generating optical flow datasets, related methods, and related products

By generating optical flow datasets, multi-plane images are generated from the original images, and target viewpoint transformation and volume rendering are performed. This solves the domain difference problem in the deployment of optical flow models in real environments in existing technologies, and realizes efficient and low-cost optical flow dataset construction and model training.

CN117291954BActive Publication Date: 2024-11-19XIAOHONGSHU TECH CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202311227893.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-21
Publication Date
2024-11-19
Estimated Expiration
2043-09-21

AI Technical Summary

Technical Problem

Existing deep learning-based dense optical flow estimation methods are difficult to deploy in real-world environments, mainly because the images in the training dataset come from computer simulations or animations, resulting in large domain differences and making it difficult to generalize to complex scenes.

Method used

By generating an optical flow dataset, a multi-plane image (MPI) is generated from the original image, and an MPI from the target's perspective is generated based on the target transformation relationship. Combined with neural network processing and volume rendering technology, target images and optical flow are generated to construct a highly realistic optical flow dataset.

Benefits of technology

We have achieved the generation of high-quality, high-fidelity optical flow datasets based on single real images, which improves the estimation accuracy of optical flow models in real environments and reduces the cost of constructing datasets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117291954B_ABST
    Figure CN117291954B_ABST
Patent Text Reader

Abstract

The present application discloses a method for generating an optical flow data set, related methods and related products. The method includes: obtaining an original image, the original image is an image obtained by an imaging device shooting a target scene at an original perspective, and the posture of the original image shot by the imaging device is the original posture; generating a multi-plane image at the original perspective based on the original image; based on a target conversion relationship, converting the multi-plane image at the original perspective to obtain a multi-plane image at a target perspective, the target conversion relationship is a conversion relationship between the original posture and the target posture, and the target posture is a virtual posture when the imaging device shoots the target scene at the target perspective; based on the multi-plane image at the target perspective, generating a target image; based on the optical flow between the multi-plane image at the original perspective and the multi-plane image at the target perspective, generating a target optical flow between the original image and the target image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method for generating an optical flow data set, related methods, and related products. Background Art

[0002] Optical flow refers to the movement relationship of pixels between two consecutive frames. Optical flow data plays an important role in many computer vision applications, such as video compression, motion estimation, target tracking, and video analysis. Dense optical flow estimation is a core problem in the field of computer vision, and its main goal is to estimate the motion vector of each pixel in an image between two frames.

[0003] In recent years, with the development of deep learning, dense optical flow estimation methods based on deep learning have gradually become a research hotspot. These methods use optical flow datasets consisting of two consecutive frames of images and optical flow labels for training. In order to obtain optical flow labels, some methods use fluorescent markers to record motion in real scenes, but these methods cannot obtain dense optical flow. Some other methods use professional equipment such as lidar and camera settings to collect dense optical flow in real scenes. However, due to the limited number of dense optical flows collected by professional equipment and the limited real scenes collected, the models trained using deep supervised learning are difficult to generalize to more complex scenes. Based on this, the current leading technology uses optical flow datasets containing a large number of real optical flow labels for training, where most of the images in the optical flow datasets come from computer simulations or animations. However, there are domain differences between these images obtained through computer simulations or animations and real images obtained by shooting real scenes, which makes it difficult for models trained by this technology to be deployed in real environments. Therefore, it is of great significance to generate optical flow datasets based on a single real image. Summary of the invention

[0004] The present application provides a method for generating an optical flow data set, related methods and related products, wherein the related methods include: a training method for an optical flow model, a mobile robot navigation method, and the related devices include: a device for generating an optical flow data set, a training device for an optical flow model, a mobile robot, an electronic device, and a computer-readable storage medium.

[0005] In a first aspect, a method for generating an optical flow dataset is provided, the method comprising:

[0006] Acquire an original image, where the original image is an image obtained by photographing a target scene with an original viewing angle by an imaging device, and the posture of the original image photographed by the imaging device is the original posture;

[0007] Generating a multiplane image (MPI) at the original viewing angle based on the original image;

[0008] Based on a target conversion relationship, the MPI under the original perspective is converted to obtain the MPI under the target perspective, wherein the target conversion relationship is a conversion relationship between the original posture and the target posture, and the target posture is a virtual posture when the imaging device shoots the target scene at the target perspective;

[0009] Based on the MPI at the target viewing angle, a target image is generated, where the target image is an image obtained by photographing the target scene at the target viewing angle by the imaging device;

[0010] Based on the optical flow between the MPI at the original perspective and the MPI at the target perspective, a target optical flow between the original image and the target image is generated.

[0011] In combination with any embodiment of the present application, the generating a target optical flow between the original image and the target image based on the optical flow between the MPI at the original perspective and the MPI at the target perspective includes:

[0012] Determine a plurality of dense optical flows based on the optical flow between the MPI at the original perspective and the MPI at the target perspective;

[0013] Volume rendering is performed on the multiple dense optical flows to obtain the target optical flow.

[0014] In combination with any embodiment of the present application, the determining of multiple dense optical flows based on the optical flow between the MPI at the original perspective and the MPI at the target perspective includes:

[0015] Based on the target conversion relationship, determining corresponding pixel pairs in the MPI at the original viewing angle and the MPI at the target viewing angle;

[0016] The multiple dense optical flows are obtained by determining the optical flows between the pixel pairs.

[0017] In combination with any embodiment of the present application, generating the MPI at the original viewing angle based on the original image includes:

[0018] Processing the original image through a neural network to extract color values ​​and volume densities at multiple depths;

[0019] The original image is generated based on the color values ​​and volume densities at the multiple depths to generate the MPI at the original viewing angle.

[0020] In combination with any implementation manner of the present application, the converting the MPI under the original perspective based on the target conversion relationship to obtain the MPI under the target perspective includes:

[0021] Based on the target conversion relationship, the MPI under the original viewing angle is converted to obtain a plurality of images to be interpolated;

[0022] Bilinear interpolation is performed on the multiple images to be interpolated to obtain the MPI under the target viewing angle.

[0023] In combination with any embodiment of the present application, generating a target image based on the MPI at the target perspective includes:

[0024] Volume rendering is performed on the MPI under the target viewing angle to generate the target image.

[0025] In a second aspect, a method for training an optical flow model is provided, the method comprising:

[0026] Acquire the original image, target image and target optical flow in the first aspect of the present application and any one of its embodiments;

[0027] Using the target optical flow as a dense optical flow label of the original image and the target image;

[0028] Based on the original image, the target image and the dense optical flow label, the model to be trained is trained to obtain an optical flow model, and the optical flow model is used to estimate the dense optical flow.

[0029] In a third aspect, a mobile robot navigation method is provided, the mobile robot navigation method is applied to a mobile robot, the mobile robot includes an imaging device, and the method includes:

[0030] A first image to be processed and a second image to be processed are obtained by photographing the imaging device, wherein the photographing time of the first image to be processed and the photographing time of the second image to be processed are different, and the first image to be processed and the second image to be processed both include a target object, and the target object is an object in the environment where the mobile robot is located;

[0031] Obtaining an optical flow model obtained according to the method of the second aspect of the present application;

[0032] Determining an estimated optical flow between the first image to be processed and the second image to be processed using the optical flow model;

[0033] Determining motion information of the target object based on the estimated optical flow;

[0034] The mobile robot is navigated based on the motion information.

[0035] In a fourth aspect, a device for generating an optical flow data set is provided, the device comprising:

[0036] An acquisition unit, used for acquiring an original image, wherein the original image is an image obtained by photographing a target scene with an original viewing angle by an imaging device, and the posture of the original image photographed by the imaging device is the original posture;

[0037] A generating unit, configured to generate the MPI at the original viewing angle based on the original image;

[0038] A conversion unit, configured to convert the MPI at the original viewing angle based on a target conversion relationship to obtain the MPI at the target viewing angle, wherein the target conversion relationship is a conversion relationship between the original posture and the target posture, and the target posture is a virtual posture when the imaging device shoots the target scene at the target viewing angle;

[0039] The generating unit is configured to generate a target image based on the MPI at the target viewing angle, wherein the target image is an image obtained by photographing the target scene at the target viewing angle by the imaging device;

[0040] The generating unit is configured to generate a target optical flow between the original image and the target image based on the optical flow between the MPI at the original viewing angle and the MPI at the target viewing angle.

[0041] In combination with any implementation manner of the present application, the generating unit is used to:

[0042] Determine a plurality of dense optical flows based on the optical flow between the MPI at the original perspective and the MPI at the target perspective;

[0043] Volume rendering is performed on the multiple dense optical flows to obtain the target optical flow.

[0044] In combination with any implementation manner of the present application, the generating unit is used to:

[0045] Based on the target conversion relationship, determining corresponding pixel pairs in the MPI at the original viewing angle and the MPI at the target viewing angle;

[0046] The multiple dense optical flows are obtained by determining the optical flows between the pixel pairs.

[0047] In combination with any implementation manner of the present application, the generating unit is used to:

[0048] Processing the original image through a neural network to extract color values ​​and volume densities at multiple depths;

[0049] The original image is generated based on the color values ​​and volume densities at the multiple depths to generate the MPI at the original viewing angle.

[0050] In combination with any embodiment of the present application, the conversion unit is used to:

[0051] Based on the target conversion relationship, the MPI under the original viewing angle is converted to obtain a plurality of images to be interpolated;

[0052] Bilinear interpolation is performed on the multiple images to be interpolated to obtain the MPI under the target viewing angle.

[0053] In combination with any implementation manner of the present application, the generating unit is used to:

[0054] Volume rendering is performed on the MPI under the target viewing angle to generate the target image.

[0055] In a fifth aspect, a training device for an optical flow model is provided, the device comprising:

[0056] An acquisition unit, used to acquire the original image, target image and target optical flow in the first aspect of the present application and any embodiment thereof;

[0057] A processing unit, configured to use the target optical flow as a dense optical flow label of the original image and the target image;

[0058] A training unit is used to train a model to be trained based on the original image, the target image and the dense optical flow label to obtain an optical flow model, wherein the optical flow model is used to estimate the dense optical flow.

[0059] In a sixth aspect, a mobile robot is provided, the mobile robot comprising:

[0060] An imaging device, used for capturing a first image to be processed and a second image to be processed, wherein the first image to be processed and the second image to be processed are captured at different times, and both the first image to be processed and the second image to be processed include a target object, and the target object is an object in the environment where the mobile robot is located;

[0061] An acquisition unit, used to acquire an optical flow model obtained according to the method of the second aspect of the present application;

[0062] a determining unit, configured to determine an estimated optical flow between the first image to be processed and the second image to be processed by using the optical flow model;

[0063] The determining unit is used to determine the motion information of the target object based on the estimated optical flow;

[0064] A navigation unit is used to navigate the mobile robot based on the motion information.

[0065] In the seventh aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions; when the program instructions are executed by the processor, the processor is caused to execute the first aspect and any one of its embodiments; when the program instructions are executed by the processor, the processor is caused to execute the second aspect and any one of its embodiments; when the program instructions are executed by the processor, the processor is caused to execute the third aspect and any one of its embodiments.

[0066] In an eighth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the first aspect and any one of its embodiments; when the processor executes the computer instructions, the electronic device either executes the second aspect and any one of its embodiments; when the processor executes the computer instructions, the electronic device either executes the third aspect and any one of its embodiments.

[0067] In the ninth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions; when the program instructions are executed by a processor, the processor is caused to execute the first aspect and any implementation manner thereof; when the program instructions are executed by the processor, the processor is caused to execute the second aspect and any implementation manner thereof; when the program instructions are executed by the processor, the processor is caused to execute the third aspect and any implementation manner thereof.

[0068] In the tenth aspect, a computer program product is provided, comprising a computer program or instructions; when the computer program or instructions are run on a computer, the computer is caused to execute the above-mentioned first aspect and any one of its embodiments; when the program instructions are executed by a processor, the processor is caused to execute the above-mentioned second aspect and any one of its embodiments; when the program instructions are executed by a processor, the processor is caused to execute the above-mentioned third aspect and any one of its embodiments.

[0069] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0070] In the present application, after acquiring the original image obtained by the imaging device shooting the target scene at the original perspective, the generating device generates the MPI at the original perspective based on the original image. Then, based on the target conversion relationship between the original posture and the target posture, the MPI at the original perspective is converted to obtain the MPI at the target perspective. Since the color value and volume density in the MPI at the original perspective are determined based on the original image, the authenticity of the color value and volume density in the MPI at the target perspective obtained by converting the MPI at the original perspective is relatively high. Therefore, based on the MPI at the target perspective, the target image obtained by the imaging device shooting the target scene at the target perspective is generated, which can improve the authenticity of the target image. And based on the optical flow between the MPI at the original perspective and the MPI at the target perspective, the target optical flow between the original image and the target image is generated, which can also improve the accuracy of the target optical flow. In an embodiment of the present application, the generating device can generate the target image and the target optical flow based on the original image, thereby realizing the generation of an optical flow data set based on a single real image. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0072] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0073] Figure 1 A flowchart of a method for generating an optical flow data set provided in an embodiment of the present application;

[0074] Figure 2 A schematic diagram of the relationship between an original viewing angle and a target viewing angle provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of an MPI from a target perspective provided in an embodiment of the present application;

[0076] Figure 4 A method provided in an embodiment of the present application is to Figure 3 The schematic diagram of the effect of MPI volume rendering to merge the color values ​​of different images;

[0077] Figure 5 A schematic diagram of another process for generating an optical flow data set provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of a target image provided in an embodiment of the present application;

[0079] Figure 7A method provided in the embodiment of the present application Figure 6 Schematic diagram of target optical flow corresponding to the target image shown;

[0080] Figure 8a A schematic diagram of another target image provided in an embodiment of the present application;

[0081] Figure 8b A method provided in the embodiment of the present application Figure 8a A partial enlarged view of

[0082] Figure 8c Another embodiment of the present application provides Figure 8a A partial enlarged view of

[0083] Figure 9a A schematic diagram of another target image provided in an embodiment of the present application;

[0084] Figure 9b A method provided in the embodiment of the present application Figure 9a A partial enlarged view of

[0085] Fig.9c Another embodiment of the present application provides Figure 9a A partial enlarged view of

[0086] Fig.10a A target image generated based on the method of the present application provided in an embodiment of the present application;

[0087] Fig.10b A method provided in the embodiment of the present application Fig.10a A partial enlarged view of

[0088] Fig.10c Another embodiment of the present application provides Fig.10a A partial enlarged view of

[0089] Fig.11a A schematic diagram of an original image provided for implementation of this application;

[0090] Fig.11b A method for implementing this application Fig.11a A schematic diagram of a part of the corresponding target image;

[0091] Fig.11c Another method provided for the implementation of this application is Fig.11a A schematic diagram of a part of the corresponding target image;

[0092] Fig.11d Another method provided for implementing this application is Fig.11a A schematic diagram of a part of the corresponding target image;

[0093] Fig.12aA schematic diagram of another original image provided for implementation of the present application;

[0094] Figure 12b The method for this application is Fig.12a The target image generated by the original image shown;

[0095] Fig.12c for Fig.12a and Figure 12b The target optical flow between

[0096] Fig.13a A schematic diagram of another original image provided for the implementation of this application;

[0097] Fig.13b The method for this application is Fig.13a The target image generated by the original image shown;

[0098] Fig.13c for Fig.13a and Fig.13b The target optical flow between

[0099] Fig.14a A schematic diagram of another original image provided for the implementation of this application;

[0100] Fig.14b The method for this application is Fig.14a The target image generated by the original image shown;

[0101] Fig.14c for Fig.14a and Fig.14b The target optical flow between

[0102] Fig.15a A schematic diagram of another original image provided for the implementation of this application;

[0103] Fig.15b The method for this application is Fig.15a The target image generated by the original image shown;

[0104] Fig.15c for Fig.15a and Fig.15b The target optical flow between

[0105] Fig.16a A schematic diagram of another original image provided for the implementation of this application;

[0106] Fig.16b The method for this application is Fig.16a The target image generated by the original image shown;

[0107] Fig.16c for Fig.16a and Fig.16b The target optical flow between

[0108] Fig.17a A schematic diagram of another original image provided for the implementation of this application;

[0109] Fig.17b The method for this application is Fig.17a The target image generated by the original image shown;

[0110] Fig.17c for Fig.17a and Fig.17b The target optical flow between

[0111] Fig.18 A flowchart of a method for training an optical flow model provided in an embodiment of the present application;

[0112] Fig.19 A flowchart of a mobile robot navigation method provided in an embodiment of the present application;

[0113] Fig. 20 A schematic diagram of the structure of a device for generating an optical flow data set provided in an embodiment of the present application;

[0114] Fig.21 A schematic diagram of the structure of a training device for an optical flow model provided in an embodiment of the present application;

[0115] Fig. 22 A schematic diagram of the structure of a mobile robot provided in an embodiment of the present application;

[0116] Fig.23 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0117] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0118] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0119] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0120] In the embodiments of the present application, the execution subject of the technical solution disclosed in the embodiment of the method for generating an optical flow data set is a device for generating an optical flow data set (hereinafter referred to as a generating device), wherein the generating device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method for generating an optical flow data set. Optionally, the generating device can be one of the following: a computer, a server.

[0121] It should be understood that the embodiment of the method for generating an optical flow data set can also be implemented by a processor executing a computer program code. The embodiment of the present application is described below in conjunction with the accompanying drawings in the embodiment of the present application. Figure 1 , Figure 1 It is a flowchart of a method for generating an optical flow data set provided in an embodiment of the present application.

[0122] 101. Obtain the original image.

[0123] In the embodiment of the present application, the original image is an image obtained by photographing the target scene with the original viewing angle by the imaging device. The imaging device can be any device with an imaging function, for example, the imaging device can be a camera, the imaging device can also be a webcam, and the imaging device can also be a video camera. The original viewing angle can be any shooting angle, and the target scene can be any scene. The posture of the imaging device when photographing the target scene with the original viewing angle to obtain the original image is the original posture.

[0124] In an implementation method of acquiring the original image, a communication connection is provided between the generating device and the imaging device, and the generating device acquires the original image captured by the imaging device through the communication connection.

[0125] In another implementation of acquiring the original image, the generating device receives the original image input by the user through an input component to acquire the original image, wherein the input component includes: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0126] In another implementation of acquiring the original image, the generating device receives the original image sent by the terminal to acquire the original image, wherein the terminal includes: a mobile phone, a computer, a tablet computer, and a server.

[0127] 102. Generate an MPI under the original viewing angle based on the original image.

[0128] In an embodiment of the present application, the MPI includes multiple images, the depth of pixels in each image is the same, and the depth of pixels in different images is different. In one possible implementation, the generating device processes the original image through a neural network, extracts color values ​​and volume densities at multiple depths, and generates the MPI at the original perspective of the original image based on the color values ​​and volume densities at multiple depths. The above color values ​​can be values ​​in a color channel.

[0129] In another possible implementation, the generating device uses monocular depth estimation to obtain the depth of the original image and the RGBD image, wherein the RGBD image is a depth image including the original image. The neural network then extracts the color values ​​and volume density at multiple depths based on the context information of the RGBD image, and generates the MPI of the original image at the original perspective based on the color values ​​and volume density at multiple depths.

[0130] 103. Based on the target conversion relationship, the MPI under the original perspective is converted to obtain the MPI under the target perspective.

[0131] In the embodiment of the present application, the target conversion relationship is the conversion relationship between the original posture and the target posture, wherein the target posture is a virtual posture when the imaging device shoots the target scene at the target perspective. The above-mentioned target perspective can be any shooting perspective, and the target perspective is different from the original perspective. The posture of the imaging device when shooting the target scene at the target perspective is the target posture. It should be understood that the generating device can generate the optical flow between the original image and the target image only based on the original image in the absence of an image obtained by shooting the target scene at the target perspective by the imaging device, so the target posture can be a virtual posture.

[0132] The generating device can convert the MPI under the original perspective into the MPI under the target perspective through the target conversion relationship. Specifically, the generating device uses the target conversion relationship to convert each image in the MPI under the original perspective to obtain the MPI under the target perspective. In a possible implementation, the generating device converts the MPI under the original perspective based on the target conversion relationship to obtain multiple images to be interpolated, and performs bilinear interpolation on the multiple images to be interpolated to obtain the MPI under the target perspective.

[0133] Optional, use (x s ,y s ) represents the pixel in the MPI under the original viewing angle, and (x t ,y t ) represents the pixel in the MPI at the target viewing angle, then (x s ,y s )、(x t ,y t ) satisfies the following formula:

[0134]

[0135] Among them, R is the rotation matrix in the target transformation relationship, t is the translation matrix in the target transformation relationship, K is the intrinsic parameter matrix inherent in the imaging device, n = [0,0,1] is the normal vector, d n is the depth of the nth image in MPI under the original view.

[0136] See also Figure 2 , Figure 2 A schematic diagram of the relationship between an original viewing angle and a target viewing angle provided in an embodiment of the present application. Figure 2 The image in is MPI, the dotted line is the light emitted from the imaging device when the shooting angle of the imaging device is the original angle of view, and the solid line is the light emitted from the imaging device when the shooting angle of the imaging device is the target angle of view. Figure 2 As shown, the content of the image captured at the original viewing angle is different from the content of the image captured at the target viewing angle. Therefore, the MPI at the original viewing angle is different from the MPI at the target viewing angle.

[0137] 104. Generate a target image based on the MPI under the target perspective.

[0138] In the embodiment of the present application, the target image is an image obtained by an imaging device shooting a target scene at a target viewing angle. The original image and the target image are two consecutive frames of images obtained by an imaging device shooting a target scene. In a possible implementation, the generating device generates the target image by performing volume rendering on the MPI at a target viewing angle.

[0139] Optionally, the generating device performs volume rendering on the MPI under the target viewing angle, and generates a target image expressed by the following formula:

[0140]

[0141] Among them, I t is the target image, N is the number of images in the MPI at the original view, c′ n is the color value of the nth image in the MPI at the target perspective, α′ n =exp(-δ n σ′ n ), δ n is the distance between the nth image in the MPI under the original view and the (n+1)th image in the MPI under the original view, σ′ n is the volume density of the nth image in the MPI at the target view, α′ m =exp(-δ m σ′ m ), δ m is the distance between the mth image in the MPI under the original view and the (m+1)th image in the MPI under the original view, σ′ m is the volume density in the mth image in the MPI at the target view.

[0142] When the color values ​​and volume densities of pixels in the MPI at the target perspective are obtained based on the real color values ​​and real volume densities in the MPI at the original perspective, by performing volume rendering on the MPI at the target perspective, the color values ​​and volume densities of different images in the MPI can be weightedly fused to generate the target image, thereby improving the realism of the target image, thereby achieving the generation of a real image at a new perspective based on a single real image.

[0143] See also Figure 3 , Figure 3 A schematic diagram of MPI from a target perspective provided for the implementation of this application. Figure 3 As shown, the MPI under the target perspective includes 64 images, among which, Figure 3 The first image (i.e. Figure 3 (a) 1 st ), the 8th image (i.e. Figure 3 (b) 8 th ), the 16th image (i.e. Figure 3 (c) 16 th ), the 24th image (i.e. Figure 3 (d) 24 th ), the 32nd image (i.e. Figure 3 (e)32 th ), the 40th image (i.e. Figure 3 (f) 40 th ), the 48th image (i.e. Figure 3 (g) 48 th ), the 56th image (i.e. Figure 3 (h)56 th ), the 64th image (i.e. Figure 3 (i) 64 th ).

[0144] See also Figure 4 , Figure 4 A method provided in an embodiment of the present application is to Figure 3 The schematic diagram of the effect of MPI volume rendering to merge the color values ​​of different images is shown. Figure 4 As shown, Figure 4 The images in Figure 3 By performing volume rendering on MPI, the colors of the images in MPI can be accumulated continuously.

[0145] 105. Generate a target optical flow between the original image and the target image based on the optical flow between the MPI at the original perspective and the MPI at the target perspective.

[0146] In an embodiment of the present application, the images in the MPI under the original perspective correspond to the images in the MPI under the target perspective. For example, the MPI under the original perspective includes image a and image b, and the MPI under the target perspective includes image c and image d, wherein image c is obtained by transforming image a using the target transformation relationship, and image d is obtained by transforming image b using the target transformation relationship. At this time, image a corresponds to image c, and image b corresponds to image d. The optical flow between the MPI under the original perspective and the MPI under the above-mentioned target perspective includes a dense optical flow between image a and image c, and a dense optical flow between image b and image d.

[0147] In an embodiment of the present application, the target optical flow is a dense optical flow between an original image and a target image. In one possible implementation, the generating device sums the dense optical flow between the MPI at the original perspective and the corresponding image in the MPI at the target perspective to obtain the target optical flow. For example, the MPI at the original perspective includes image a and image b, and the MPI at the target perspective includes image c and image d, wherein image a corresponds to image c, and image b corresponds to image d. The dense optical flow between image a and image b is a dense optical flow e, and the dense optical flow between image a and image b is a dense optical flow f. The generating device sums the dense optical flow e and the dense optical flow f to obtain the target optical flow.

[0148] In another possible implementation, the generating device determines the weight of the dense optical flow corresponding to the image according to the depth of the image in the MPI under the original perspective, wherein the weight is positively correlated with the depth, and then performs weighted summation on the dense optical flow between the MPI under the original perspective and the corresponding image in the MPI under the target perspective based on the weight to obtain the target optical flow. For example, the MPI under the original perspective includes image a and image b, and the MPI under the target perspective includes image c and image d, wherein image a corresponds to image c, and image b corresponds to image d. The dense optical flow between image a and image b is dense optical flow e, and the dense optical flow between image a and image b is dense optical flow f. The generating device determines the weight of the dense optical flow e according to the depth of image a, determines the weight of the dense optical flow f according to the depth of image b, and then performs weighted summation on the dense optical flow e and the dense optical flow f based on the weight of the dense optical flow e and the weight of the dense optical flow f to obtain the target optical flow.

[0149] In an embodiment of the present application, after acquiring the original image obtained by the imaging device shooting the target scene at the original perspective, the generating device generates the MPI at the original perspective based on the original image. Then, based on the target conversion relationship between the original posture and the target posture, the MPI at the original perspective is converted to obtain the MPI at the target perspective. Since the color value and volume density in the MPI at the original perspective are determined based on the original image, the authenticity of the color value and volume density in the MPI at the target perspective obtained by converting the MPI at the original perspective is relatively high. Therefore, based on the MPI at the target perspective, the target image obtained by the imaging device shooting the target scene at the target perspective is generated, which can improve the authenticity of the target image. And based on the optical flow between the MPI at the original perspective and the MPI at the target perspective, the target optical flow between the original image and the target image is generated, which can also improve the accuracy of the target optical flow. In an embodiment of the present application, the generating device can generate the target image and the target optical flow based on the original image, thereby realizing the generation of an optical flow data set based on a single real image.

[0150] As an optional implementation manner, the generating device performs the following steps during the process of executing step 105:

[0151] 201. Determine a plurality of dense optical flows based on the optical flow between the MPI under the original perspective and the MPI under the target perspective.

[0152] As described in step 105, the generating device includes, based on the optical flow between the MPI at the original perspective and the MPI at the target perspective, dense optical flows between corresponding images in the MPI at the original perspective and the MPI at the target perspective, so the generating device determines the dense optical flows between corresponding images based on the optical flow between the MPI at the original perspective and the MPI at the target perspective. The above-mentioned multiple dense optical flows include dense optical flows between all corresponding images. For example, the MPI at the original perspective includes image a and image b, and the MPI at the target perspective includes image c and image d, wherein image a corresponds to image c, and image b corresponds to image d. Based on the optical flow between the MPI at the original perspective and the MPI at the target perspective, the multiple dense optical flows determined include: dense optical flow between image a and image c, and dense optical flow between image b and image d.

[0153] 202. Perform volume rendering on the multiple dense optical flows to obtain the target optical flow.

[0154] Optionally, the generating device performs volume rendering on multiple dense optical flows, and obtains a target optical flow expressed by the following formula:

[0155]

[0156] Among them, F s→t is the target optical flow, N is the number of images in the MPI under the target perspective, and f n is the nth dense optical flow among multiple dense optical flows, α n =exp(-δ′ n σ′ n ),δ′ n is the distance between the nth image in the MPI under the target view and the (n+1)th image in the MPI under the target view, σ′ n is the volume density of the nth image in the MPI at the target perspective, α m =exp(-δ′ m σ′ m ),δ′ m is the distance between the mth image in the MPI under the target view and the (m+1)th image in the MPI under the target view, σ′ m is the volume density in the mth image in the MPI at the target view.

[0157] In this embodiment, after the generating device determines multiple dense optical flows based on the optical flow between the MPI at the original perspective and the MPI at the target perspective, it can perform volume rendering on the multiple dense optical flows to weightedly fuse the multiple dense optical flows to generate a target optical flow, thereby improving the accuracy of the target optical flow.

[0158] As an optional implementation, the generating device performs the following steps during the execution of step 201:

[0159] 301. Based on the target conversion relationship, determine corresponding pixel pairs in the MPI at the original viewing angle and the MPI at the target viewing angle.

[0160] In the embodiment of the present application, the corresponding pixel pairs include a pixel in the MPI under the original perspective and a pixel in the MPI under the target perspective, and the pixel in the MPI under the original perspective is converted using the target conversion relationship to obtain the pixel in the MPI under the target perspective.

[0161] Optional, use (x s ,y s ) represents the pixel in the MPI under the original viewing angle, and (x t ,y t ) represents the pixel in the MPI at the target viewing angle, then (x s ,y s )、(x t ,y t ) satisfies the following formula:

[0162]

[0163] Among them, K is the intrinsic parameter matrix inherent in the imaging device, is the inverse of R in formula (1), is the inverse of t in formula (1), n ​​= [0, 0, 1] is the normal vector, d n is the depth of the nth image in MPI under the original view.

[0164] 302. Obtain the multiple dense optical flows by determining the optical flows between the pixel pairs.

[0165] Optional, use (x s ,y s ) represents the pixel in the MPI under the original viewing angle, and (x t ,y t ) represents the pixel in the MPI at the target view, the dense optical flow f between the nth image in the MPI at the original view and the nth image in the MPI at the target view n =(x t -x s ,y t -y s ).

[0166] In this embodiment, the generating device first determines the corresponding pixel pairs in the MPI under the original perspective and the MPI under the target perspective based on the target transformation relationship, and then obtains multiple dense optical flows by determining the optical flow between the pixel pairs.

[0167] By using the method for generating an optical flow dataset provided in the embodiment of the present application, a target image and a target optical flow can be generated based on the original image, and then the target optical flow can be used as the optical flow label of the original image and the target image to construct an optical flow dataset, wherein the optical flow dataset includes the original image, the target image and the optical flow label. Since the method for generating an optical flow dataset provided in the embodiment of the present application can realize the construction of an optical flow dataset based on a single real image, the method for generating an optical flow dataset provided in the embodiment of the present application can efficiently construct a large-scale optical flow dataset, and the image in the constructed optical flow dataset has high authenticity. In addition, since the target scene in the original image is a real scene, and there are many ways to obtain the original image, the method for generating an optical flow dataset provided in the embodiment of the present application can also construct an optical flow dataset including diversified real scenes at a relatively low cost.

[0168] See also Figure 5 , Figure 5 Another flow chart of generating an optical flow data set provided in an embodiment of the present application is shown below. Figure 5 As shown, after inputting the original image, the depth image is obtained by performing depth estimation on the original image. By inputting the original image into the neural network, the MPI at the original perspective is output. Based on the color value, volume density provided by the depth image, and the MPI at the original perspective, the MPI at the target perspective and multiple dense optical flows between the MPI at the original perspective and the MPI at the target perspective are obtained. By performing volume rendering on the MPI at the target perspective, the target image can be obtained. By performing volume rendering on multiple dense optical flows, the target optical flow can be obtained.

[0169] See also Figure 6 , Figure 6 A schematic diagram of a target image provided in an embodiment of the present application. Figure 6 The target image shown is obtained by generating an optical flow dataset. Figure 6 As shown, by zooming in on the car, it can be seen that the target image has high detail quality and authenticity, which also proves that the target image generated by the method of generating the optical flow dataset has high quality and high authenticity.

[0170] See also Figure 7 , Figure 7 A method provided in the embodiment of the present application Figure 6 Schematic diagram of the target optical flow corresponding to the target image shown. Figure 7 The target optical flow shown is also obtained by generating an optical flow dataset. Figure 7 As shown, in the target optical flow, the optical flow of the car has a high match with the car in the target image, which also proves that the method of generating an optical flow dataset can generate an optical flow dataset with only one real image.

[0171] By zooming in on the car, we can see that the target image has high detail quality and realism, which also proves that the target image generated by the method of generating optical flow datasets has high quality and high realism.

[0172] To further illustrate the high quality of the optical flow data set generated using the method of the present application, the embodiments of the present application also provide some schematic diagrams of target images and visualization images of the target optical flow.

[0173] See also Figure 8a , Figure 8b and Figure 8c , Figure 8a A schematic diagram of another target image provided for the implementation of the present application, wherein the target image is generated by learning optical flow from still images (Depthstill) method, Figure 8b and Figure 8c Both Figure 8a A partial enlarged view, specifically, Figure 8b and Figure 8c Both Figure 8a Magnified image of the boxed area in . See Figure 9a , Figure 9b and Fig.9c , Figure 9a A schematic diagram of another target image provided for the implementation of the present application, wherein the target image is generated by a real optical flow (RealFlow: Em based realistic optical flow dataset generation from videos, RealFlow) method, Figure 9b and Fig.9c Both Figure 9a A partial enlarged view, specifically, Figure 9b and Fig.9c Both Figure 9a Magnified image of the boxed area in . See Fig.10a , Fig.10b and Fig.10c , Fig.10a A target image generated based on the method of the present application is provided for the implementation of the present application, Fig.10b and Fig.10c Both Fig.10a A partial enlarged view, specifically, Fig.10b and Fig.10c Both Fig.10a Magnified image of the image area within the box.

[0174] Through Figure 8b , Figure 9b , Fig.10b to compare and Figure 8c , Fig.9c , Fig.10c By comparison, it can be seen that the target image generated by the method of the present application has higher detail quality than the target images generated by the other two methods.

[0175] See also Fig.11a , Fig.11b , Fig.11c and Fig.11d , Fig.11a A schematic diagram of an original image provided for the implementation of this application. Fig.11b A method for implementing this application Fig.11a A schematic diagram of a portion of the corresponding target image, which is generated by the Depthstill method. Fig.11c Another method provided for the implementation of this application is Fig.11a A schematic diagram of a portion of a corresponding target image, which is generated by the RealFlow method. Fig.11d Another method provided for implementing this application is Fig.11a A schematic diagram of a portion of a corresponding target image, the target image being generated by the method of the present application. Fig.11b , Fig.11c , Fig.11d By comparison, it can be seen that the target image generated by the method of the present application has higher detail quality than the target images generated by the other two methods.

[0176] See also Fig.12a , Figure 12b and Fig.12c , Fig.12a A schematic diagram of another original image provided for the implementation of this application, Figure 12b The method for this application is Fig.12a The target image generated by the original image shown is Fig.12c for Fig.12a and Figure 12b The target optical flow between .

[0177] See also Fig.13a , Fig.13b and Fig.13c , Fig.13a A schematic diagram of another original image provided for the implementation of this application, Fig.13b The method for this application is Fig.13a The target image generated by the original image shown is Fig.13c for Fig.13a and Fig.13b The target optical flow between .

[0178] See also Fig.14a , Fig.14b and Fig.14c , Fig.14a A schematic diagram of another original image provided for the implementation of this application, Fig.14b The method for this application is Fig.14a The target image generated by the original image shown is Fig.14c for Fig.14a and Fig.14b The target optical flow between .

[0179] Optional, Fig.12a , Fig.13a , Fig.14a All are from the DAVIS dataset.

[0180] See also Fig.15a , Fig.15b and Fig.15c , Fig.15a A schematic diagram of another original image provided for the implementation of this application, Fig.15b The method for this application is Fig.15a The target image generated by the original image shown is Fig.15c for Fig.15a and Fig.15b The target optical flow between .

[0181] See also Fig.16a , Fig.16b and Fig.16c , Fig.16a A schematic diagram of another original image provided for the implementation of this application, Fig.16b The method for this application is Fig.16a The target image generated by the original image shown is Fig.16c for Fig.16a and Fig.16b The target optical flow between .

[0182] See also Fig.17a , Fig.17b and Fig.17c , Fig.17a A schematic diagram of another original image provided for the implementation of this application, Fig.17b The method for this application is Fig.17a The target image generated by the original image shown is Fig.17c for Fig.17a and Fig.17b The target optical flow between .

[0183] Optional, Fig.15a , Fig.16a , Fig.17a All are from the KITTI dataset.

[0184] The embodiment of the present application also provides a method for training an optical flow model based on the optical flow data set constructed above (hereinafter referred to as the training method of the optical flow model), wherein the execution subject of the technical solution disclosed in the embodiment of the training method of the optical flow model is a training device for the optical flow model (hereinafter referred to as the training device). The training device can be any electronic device that can execute the technical solution disclosed in the embodiment of the training method of the optical flow model. Optionally, the training device can be one of the following: a computer, a server.

[0185] It should be understood that the embodiment of the optical flow model training method can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the accompanying drawings in the embodiment of the present application. Fig.18 , Fig.18 It is a flowchart of a training method for an optical flow model provided in an embodiment of the present application.

[0186] 1801. Obtain an original image, a target image, and a target optical flow in a method for generating an optical flow dataset.

[0187] The original image in this step is the original image in step 101, the target image in this step is the target image generated by the method for generating an optical flow dataset described above, and the target optical flow in this step is the target optical flow generated by the method for generating an optical flow dataset described above.

[0188] 1802. Use the target optical flow as a dense optical flow label of the original image and the target image.

[0189] In the embodiment of the present application, the dense optical flow label represents the dense optical flow between the original image and the target image. The dense optical flow label can be used as the ground truth (GT) of the dense optical flow between the original image and the target image.

[0190] 1803. Based on the original image, the target image and the dense optical flow label, the model to be trained is trained to obtain an optical flow model.

[0191] In an embodiment of the present application, the model to be trained can be any model with a structure for estimating dense optical flow. The optical flow model obtained by training the model to be trained is used to estimate dense optical flow. In one possible implementation method, after the training device inputs the original image and the target image into the model to be trained, the model to be trained extracts the low-level features of the original image and the low-level features of the target image respectively, and then obtains the high-dimensional features of the original image and the target image by stacking all the low-level features together. After obtaining the high-dimensional features, the model to be trained generates four-dimensional association information of size H×W×H×W for all pixels in the original image and the target image through the feature association layer, and uses the confluence layer to generate lower resolution association information, wherein the height of the original image and the height of the target image are both H, and the width of the original image and the frame of the target image are both W. Then, a cyclic update operator based on a gated cyclic unit is used to find the four-dimensional association information, and then the optical flow field is iteratively updated, wherein the initial value of the optical flow field is 0. Finally, the training device calculates the loss of the iterative optical flow field and the dense optical flow label, and updates the parameters of each layer through the back propagation algorithm until the loss converges, completing the training of the model to be trained and obtaining the optical flow model.

[0192] In the embodiment of the present application, the original image is a real image, and the target image has a high degree of authenticity. Therefore, the training device uses the original image, the target image, and the target optical flow to train the model to be trained to obtain an optical flow model, which can improve the estimation accuracy of the optical flow model for the dense optical flow of the real image. Moreover, as mentioned above, by using the method for generating an optical flow data set, an optical flow data set including diversified real scenes can be constructed at a relatively low cost. Therefore, by using the method for generating an optical flow data set, an optical flow data set for training the model to be trained is constructed, and the optical flow data set is used to train the model to be trained, which can improve the training effect at a relatively low cost.

[0193] To further illustrate that using the optical flow data set generated by the method provided in the embodiment of the present application to train an optical flow model can improve the training effect, this embodiment will compare the optical flow models trained by various methods under the same experimental conditions.

[0194] Table 1 Comparison of other dense optical flow estimation schemes on the Sintel benchmark

[0195]

[0196] Table 2 Comparison of other dense optical flow estimation schemes on the KITTI test benchmark

[0197]

[0198] It can be seen from the results in Tables 1 and 2 that the optical flow model trained based on the method disclosed in this application can achieve very good dense optical flow prediction effects, both in the simulated Sintel dataset and in the KITTI dataset containing real test images. Tables 1 and 2 give the evaluation results of the average endpoint error (EPE) and two error rates, which are the percentages of pixels with an absolute error greater than 3 (>3) or an absolute error and relative error greater than 3 and 5% (Fl), respectively. These three indicators are model evaluation indicators widely used in video denoising tasks. Compared with other methods, this application can achieve better results in these three indicators. The results in Tables 1 and 2 demonstrate the superiority of this application for dense optical flow estimation tasks on a variety of datasets, which is significantly better than other methods.

[0199] The embodiment of the present application also provides a mobile robot navigation method, wherein the execution subject of the technical solution disclosed in the embodiment of the mobile robot navigation method is a mobile robot, and the mobile robot includes an imaging device. Optionally, the mobile robot can be one of the following: a drone, a sweeping robot.

[0200] It should be understood that the embodiment of the mobile robot navigation method can also be implemented by a processor executing a computer program code. The embodiment of the present application is described below in conjunction with the drawings in the embodiment of the present application. Fig.19 , Fig.19 It is a flow chart of a mobile robot navigation method provided in an embodiment of the present application.

[0201] 1901. Obtain a first image to be processed and a second image to be processed by photographing with an imaging device.

[0202] In the implementation of the present application, the shooting time of the first image to be processed and the shooting time of the second image to be processed are different, and the first image to be processed and the second image to be processed both include a target object, wherein the target object is an object in the environment in which the mobile robot is located, for example, the target object may be a person in the environment in which the mobile robot is located, the target object may also be a tree in the environment in which the mobile robot is located, or the target object may also be the ground in the environment in which the mobile robot is located.

[0203] In a possible implementation, the mobile robot is a drone, and the drone continuously photographs the environment during flight to obtain the first image to be processed and the second image to be processed. For example, during the flight of the drone, the drone continuously photographs the environment in which the drone is located through the camera, and the first image to be processed including the environment and the second image to be processed including the environment can be obtained.

[0204] In another possible implementation, the mobile robot is a sweeping robot, and the sweeping robot obtains the first image to be processed and the second image to be processed by continuously photographing the environment during driving. For example, during driving of the sweeping robot, the sweeping robot continuously photographs the environment in which the sweeping robot is located through the camera, and obtains the first image to be processed including the environment and the second image to be processed including the environment.

[0205] 1902. Obtain an optical flow model obtained according to the optical flow model training method.

[0206] 1903. Determine an estimated optical flow between the first image to be processed and the second image to be processed by using the optical flow model.

[0207] The mobile robot uses the optical flow model to estimate the dense optical flow between the first image to be processed and the second image to be processed, and then obtains the estimated optical flow.

[0208] 1904. Based on the estimated optical flow, determine the motion information of the target object.

[0209] In the embodiment of the present application, the motion information is information used to describe the motion of the target object. Optionally, the motion information includes at least one of the following: motion speed and motion direction.

[0210] 1905. Based on the above motion information, navigate the above mobile robot.

[0211] Specifically, the mobile robot can determine its own motion information based on the motion information of the target object, and then navigate the mobile robot according to the motion information of the mobile robot itself and the motion information of the target object.

[0212] In the embodiment of the present application, after acquiring the first image to be processed, the second image to be processed and the optical flow model, the mobile robot uses the optical flow model to determine the estimated optical flow between the first image to be processed and the second image to be processed, which can improve the accuracy of the estimated optical flow. Then, based on the estimated optical flow, the motion information of the target object in the first image to be processed and the second image to be processed is determined, which can improve the accuracy of the motion information of the target object. Therefore, based on the motion information of the target object, the navigation of the mobile robot is achieved, which can improve the navigation effect of the mobile robot.

[0213] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0214] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0215] See also Fig. 20 , Fig. 20 A schematic diagram of a structure of a device for generating an optical flow data set provided in an embodiment of the present application, wherein the device 1 for generating an optical flow data set includes: an acquisition unit 11, a generation unit 12, and a conversion unit 13, specifically:

[0216] An acquisition unit 11 is used to acquire an original image, wherein the original image is an image obtained by photographing a target scene with an original viewing angle by an imaging device, and the posture of the original image photographed by the imaging device is the original posture;

[0217] A generating unit 12, configured to generate the MPI at the original viewing angle based on the original image;

[0218] A conversion unit 13 is used to convert the MPI under the original perspective based on a target conversion relationship to obtain the MPI under the target perspective, wherein the target conversion relationship is a conversion relationship between the original posture and the target posture, and the target posture is a virtual posture when the imaging device shoots the target scene at the target perspective;

[0219] The generating unit 12 is configured to generate a target image based on the MPI at the target viewing angle, wherein the target image is an image obtained by photographing the target scene at the target viewing angle by the imaging device;

[0220] The generating unit 12 is configured to generate a target optical flow between the original image and the target image based on the optical flow between the MPI at the original viewing angle and the MPI at the target viewing angle.

[0221] In combination with any embodiment of the present application, the generating unit 12 is used to:

[0222] Determine a plurality of dense optical flows based on the optical flow between the MPI at the original perspective and the MPI at the target perspective;

[0223] Volume rendering is performed on the multiple dense optical flows to obtain the target optical flow.

[0224] In combination with any embodiment of the present application, the generating unit 12 is used to:

[0225] Based on the target conversion relationship, determining corresponding pixel pairs in the MPI at the original viewing angle and the MPI at the target viewing angle;

[0226] The multiple dense optical flows are obtained by determining the optical flows between the pixel pairs.

[0227] In combination with any embodiment of the present application, the generating unit 12 is used to:

[0228] Processing the original image through a neural network to extract color values ​​and volume densities at multiple depths;

[0229] The original image is generated based on the color values ​​and volume densities at the multiple depths to generate the MPI at the original viewing angle.

[0230] In combination with any embodiment of the present application, the conversion unit 13 is used to:

[0231] Based on the target conversion relationship, the MPI under the original viewing angle is converted to obtain a plurality of images to be interpolated;

[0232] Bilinear interpolation is performed on the multiple images to be interpolated to obtain the MPI under the target viewing angle.

[0233] In combination with any embodiment of the present application, the generating unit 12 is used to:

[0234] Volume rendering is performed on the MPI under the target viewing angle to generate the target image.

[0235] In an embodiment of the present application, after acquiring the original image obtained by the imaging device shooting the target scene at the original perspective, the generating device generates the MPI at the original perspective based on the original image. Then, based on the target conversion relationship between the original posture and the target posture, the MPI at the original perspective is converted to obtain the MPI at the target perspective. Since the color value and volume density in the MPI at the original perspective are determined based on the original image, the authenticity of the color value and volume density in the MPI at the target perspective obtained by converting the MPI at the original perspective is relatively high. Therefore, based on the MPI at the target perspective, the target image obtained by the imaging device shooting the target scene at the target perspective is generated, which can improve the authenticity of the target image. And based on the optical flow between the MPI at the original perspective and the MPI at the target perspective, the target optical flow between the original image and the target image is generated, which can also improve the accuracy of the target optical flow. In an embodiment of the present application, the generating device can generate the target image and the target optical flow based on the original image, thereby realizing the generation of an optical flow data set based on a single real image.

[0236] See also Fig.21 , Fig.21 A structural diagram of a training device for an optical flow model provided in an embodiment of the present application, wherein the training device 2 for the optical flow model comprises: an acquisition unit 21, a processing unit 22, and a training unit 23, specifically:

[0237] An acquisition unit 21, used to acquire an original image, a target image, and a target optical flow in the first aspect of the present application and any embodiment thereof;

[0238] A processing unit 22, configured to use the target optical flow as a dense optical flow label of the original image and the target image;

[0239] The training unit 23 is used to train the model to be trained based on the original image, the target image and the dense optical flow label to obtain an optical flow model, where the optical flow model is used to estimate the dense optical flow.

[0240] In the embodiment of the present application, the original image is a real image, and the target image has a high degree of authenticity. Therefore, the training device uses the original image, the target image, and the target optical flow to train the model to be trained to obtain an optical flow model, which can improve the estimation accuracy of the optical flow model for the dense optical flow of the real image. Moreover, as mentioned above, by using the method for generating an optical flow data set, an optical flow data set including diversified real scenes can be constructed at a relatively low cost. Therefore, by using the method for generating an optical flow data set, an optical flow data set for training the model to be trained is constructed, and the optical flow data set is used to train the model to be trained, which can improve the training effect at a relatively low cost.

[0241] See also Fig. 22 , Fig. 22 The structural diagram of a mobile robot provided in an embodiment of the present application is as follows. The mobile robot 3 includes: an imaging device 31, an acquisition unit 32, a determination unit 33, and a navigation unit 34. Specifically:

[0242] An imaging device 31 is used to capture a first image to be processed and a second image to be processed, wherein the first image to be processed and the second image to be processed are captured at different times, and both the first image to be processed and the second image to be processed include a target object, which is an object in the environment of the mobile robot;

[0243] An acquisition unit 32, used to acquire an optical flow model obtained according to the method of the second aspect of the present application;

[0244] A determining unit 33, configured to determine an estimated optical flow between the first image to be processed and the second image to be processed by using the optical flow model;

[0245] The determining unit 33 is used to determine the motion information of the target object based on the estimated optical flow;

[0246] The navigation unit 34 is used to navigate the mobile robot based on the motion information.

[0247] In the embodiment of the present application, after acquiring the first image to be processed, the second image to be processed and the optical flow model, the mobile robot uses the optical flow model to determine the estimated optical flow between the first image to be processed and the second image to be processed, which can improve the accuracy of the estimated optical flow. Then, based on the estimated optical flow, the motion information of the target object in the first image to be processed and the second image to be processed is determined, which can improve the accuracy of the motion information of the target object. Therefore, based on the motion information of the target object, the navigation of the mobile robot is achieved, which can improve the navigation effect of the mobile robot.

[0248] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0249] Fig.23 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 4 includes a processor 41 and a memory 42. Optionally, the electronic device 4 also includes an input device 43 and an output device 44. The processor 41, the memory 42, the input device 43 and the output device 44 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0250] The processor 41 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 41 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other through one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0251] The memory 42 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0252] The input device 43 is used to input data and / or signals, and the output device 44 is used to output data and / or signals. The input device 43 and the output device 44 can be independent devices or an integrated device.

[0253] It can be understood that in the embodiment of the present application, the memory 42 can be used not only to store relevant instructions, but also to store relevant data. The embodiment of the present application does not limit the specific data stored in the memory.

[0254] Understandably, Fig.23 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0255] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0256] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0257] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0260] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0261] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A method for generating an optical flow dataset, characterized in that: The method comprises: Acquire an original image, where the original image is an image obtained by photographing a target scene with an original viewing angle by an imaging device, and the posture of the original image photographed by the imaging device is the original posture; generating a multi-plane image at the original viewing angle based on the original image; Based on a target conversion relationship, the multi-plane image under the original perspective is converted to obtain a multi-plane image under the target perspective, wherein the target conversion relationship is a conversion relationship between the original posture and the target posture, and the target posture is a virtual posture when the imaging device shoots the target scene at the target perspective; Based on the multi-plane images under the target viewing angle, generating a target image, wherein the target image is an image obtained by photographing the target scene under the target viewing angle by the imaging device; generating a target optical flow between the original image and the target image based on the optical flow between the multi-plane image at the original perspective and the multi-plane image at the target perspective; The method of generating a target optical flow between the original image and the target image based on the optical flow between the multi-plane image at the original perspective and the multi-plane image at the target perspective comprises: determining a plurality of dense optical flows based on the optical flow between the multi-plane image at the original perspective and the multi-plane image at the target perspective; and performing volume rendering on the plurality of dense optical flows to obtain the target optical flow.

2. The method according to claim 1, characterized in that The determining of a plurality of dense optical flows based on the optical flows between the multi-plane image at the original perspective and the multi-plane image at the target perspective comprises: Based on the target conversion relationship, determining corresponding pixel pairs in the multi-plane image at the original viewing angle and the multi-plane image at the target viewing angle; The multiple dense optical flows are obtained by determining the optical flows between the pixel pairs.

3. The method according to claim 1 or 2, characterized in that: The generating the multi-plane image under the original viewing angle based on the original image comprises: Processing the original image through a neural network to extract color values ​​and volume densities at multiple depths; The original image is generated based on the color values ​​and volume densities at the multiple depths to generate a multi-plane image at the original viewing angle.

4. The method according to claim 1 or 2, characterized in that: The step of converting the multi-plane image under the original viewing angle based on the target conversion relationship to obtain the multi-plane image under the target viewing angle includes: Based on the target conversion relationship, the multi-plane image under the original viewing angle is converted to obtain a plurality of images to be interpolated; Bilinear interpolation is performed on the multiple images to be interpolated to obtain a multi-plane image under the target viewing angle.

5. The method according to claim 1 or 2, characterized in that: The generating a target image based on the multi-plane image under the target viewing angle includes: Volume rendering is performed on the multi-plane image under the target viewing angle to generate the target image.

6. A method for training an optical flow model, characterized in that: The method comprises: Obtaining the original image, target image and target optical flow in any one of the methods of claims 1 to 5; Using the target optical flow as a dense optical flow label of the original image and the target image; Based on the original image, the target image and the dense optical flow label, the model to be trained is trained to obtain an optical flow model, and the optical flow model is used to estimate the dense optical flow.

7. A mobile robot navigation method, characterized in that: The mobile robot navigation method is applied to a mobile robot, the mobile robot includes an imaging device, and the method includes: A first image to be processed and a second image to be processed are obtained by photographing the imaging device, wherein the photographing time of the first image to be processed and the photographing time of the second image to be processed are different, and the first image to be processed and the second image to be processed both include a target object, and the target object is an object in the environment where the mobile robot is located; Obtain an optical flow model obtained according to the method of claim 6; Determining an estimated optical flow between the first image to be processed and the second image to be processed using the optical flow model; Determining motion information of the target object based on the estimated optical flow; The mobile robot is navigated based on the motion information.

8. A device for generating an optical flow data set, characterized in that: The device comprises: An acquisition unit, used for acquiring an original image, wherein the original image is an image obtained by photographing a target scene with an original viewing angle by an imaging device, and the posture of the original image photographed by the imaging device is the original posture; A generating unit, configured to generate a multi-plane image at the original viewing angle based on the original image; A conversion unit, configured to convert the multi-plane image under the original viewing angle based on a target conversion relationship to obtain a multi-plane image under a target viewing angle, wherein the target conversion relationship is a conversion relationship between the original posture and the target posture, and the target posture is a virtual posture when the imaging device shoots the target scene at the target viewing angle; The generating unit is configured to generate a target image based on the multi-plane image at the target viewing angle, wherein the target image is an image obtained by the imaging device shooting the target scene at the target viewing angle; The generating unit is configured to generate a target optical flow between the original image and the target image based on the optical flow between the multi-plane image at the original viewing angle and the multi-plane image at the target viewing angle; The method of generating a target optical flow between the original image and the target image based on the optical flow between the multi-plane image at the original perspective and the multi-plane image at the target perspective comprises: determining a plurality of dense optical flows based on the optical flow between the multi-plane image at the original perspective and the multi-plane image at the target perspective; and performing volume rendering on the plurality of dense optical flows to obtain the target optical flow.

9. A training device for an optical flow model, characterized in that: The device comprises: An acquisition unit, used to acquire the original image, the target image and the target optical flow in any one of the methods of claims 1 to 5; A processing unit, configured to use the target optical flow as a dense optical flow label of the original image and the target image; A training unit is used to train a model to be trained based on the original image, the target image and the dense optical flow label to obtain an optical flow model, wherein the optical flow model is used to estimate the dense optical flow.

10. A mobile robot navigation method, characterized in that: The mobile robot comprises: An imaging device, used for capturing a first image to be processed and a second image to be processed, wherein the first image to be processed and the second image to be processed are captured at different times, and both the first image to be processed and the second image to be processed include a target object, and the target object is an object in the environment where the mobile robot is located; An acquisition unit, used to acquire the optical flow model obtained according to the method of claim 6; a determining unit, configured to determine an estimated optical flow between the first image to be processed and the second image to be processed by using the optical flow model; The determining unit is used to determine the motion information of the target object based on the estimated optical flow; A navigation unit is used to navigate the mobile robot based on the motion information.

11. An electronic device, characterized in that: include: a processor and a memory, the memory being configured to store computer program code, the computer program code comprising computer instructions; When the processor executes the computer instructions, the electronic device performs the method according to any one of claims 1 to 5; When the processor executes the computer instructions, the electronic device may perform the method according to claim 6; When the processor executes the computer instructions, the electronic device may perform the method as claimed in claim 7.

12. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, wherein the computer program includes program instructions; When the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5; When the program instructions are executed by a processor, or the processor is caused to execute the method according to claim 6; When the program instructions are executed by a processor, the processor is caused to execute the method according to claim 7.

Citation Information

Patent Citations

  • Ceiling image positioning method of robot and ceiling image positioning system thereof

    CN106708048A

  • Multi-view image fusion method and device, computer equipment, and storage medium

    CN108833785A

  • Method and device for training optical flow network and processing image

    CN110310299A

  • Parts recognition method and related equipment and device

    CN111814589A

  • Image splicing method and device, computer equipment and storage medium

    CN111915483A