Method for determining occlusion relationship, related method and related product

Abstract

The application discloses a method for determining an occlusion relationship, a related method and related products. The method comprises: obtaining an original image and depth information of the original image, the original image being an image obtained by a camera shooting a target scene at an original view angle, the target scene comprising a background and a moving target; processing the original image and the depth information by using a neural network to obtain a plurality of original depth images of a multi-plane image at the original view angle; generating a first depth image and a second depth image based on the plurality of original depth images; and determining a target occlusion relationship based on the first depth image and the second depth image, the target occlusion relationship being an occlusion relationship between the background and the moving target in a case where the camera shoots the target scene at a new view angle.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, related methods and related products for determining occlusion relationships. Background Technology

[0002] With the rapid development of computer vision technology, mature techniques have emerged in recent years that can generate images from a new perspective based on an original image captured from the original viewpoint. However, when the original image includes moving targets and a stationary background, determining the occlusion relationship between the moving targets and the background from the new perspective is crucial for improving the image quality. Summary of the Invention

[0003] This application provides a method for determining occlusion relationships, related methods, and related products. The related methods include: a method for training an optical flow model and a path planning method. The related devices include: a device for determining occlusion relationships, a device for training an optical flow model, a vehicle, electronic equipment, and a computer-readable storage medium.

[0004] Firstly, a method for determining occlusion relationships is provided, the method comprising:

[0005] Acquire the original image and its depth information. The original image is an image captured by the camera from the original viewpoint of the target scene, which includes the background and moving targets.

[0006] The original image and the depth information are processed using a neural network to obtain multiple original depth images of multiplane images (MPI) from the original viewpoint.

[0007] Based on the multiple original depth images, a first depth image and a second depth image are generated; the first depth image includes the depth information of the background when the camera captures the background from a new perspective, and the second depth image includes the depth information of the moving target when the camera captures the moving target from a new perspective.

[0008] Based on the first depth image and the second depth image, a target occlusion relationship is determined. The target occlusion relationship is the occlusion relationship between the background and the moving target when the camera captures the target scene from the new perspective.

[0009] In conjunction with any embodiment of this application, the method further includes:

[0010] The original image and the depth information are processed using a neural network to obtain the MPI;

[0011] Based on the MPI, a new image is generated from the new perspective, which is an image of the target scene captured by the camera from the new perspective;

[0012] For the new image, the occlusion relationship between the background and the moving target is corrected using the target occlusion relationship to obtain the target image from the new perspective.

[0013] In any embodiment of this application, determining the target occlusion relationship based on the first depth image and the second depth image includes:

[0014] Based on the original image, a background reference image is generated by the camera capturing the target scene in the first pose, and a moving target reference image is generated by the camera capturing the moving target in the second pose; the first pose is the pose of the camera capturing the background from the new perspective, and the second pose is the pose of the camera capturing the moving target from the new perspective.

[0015] Determine a first position range of the background in the background reference image, and a second position range of the moving target in the moving target reference image;

[0016] The location where the first location range and the second location range intersect is determined as the third location range;

[0017] Based on the first depth image and the second depth image, the occlusion relationship between the background and the moving target within the third position range is determined as the target occlusion relationship.

[0018] In any embodiment of this application, determining the occlusion relationship between the background and the moving target within the third position range based on the first depth image and the second depth image, as the target occlusion relationship, includes:

[0019] Based on the first depth image, a first reference depth of the background within the range of the third position is determined;

[0020] Based on the second depth image, a second reference depth of the moving target within the range of the third position is determined;

[0021] When the first reference depth is less than the second reference depth, the target occlusion relationship is determined to be that the moving target is occluded by the background;

[0022] If the first reference depth is greater than or equal to the second reference depth, the target occlusion relationship is determined to be that the background is occluded by the moving target.

[0023] In any embodiment of this application, the pose of the original image captured by the camera is the original pose;

[0024] Determining the first position range of the background in the background reference image and the second position range of the moving target in the moving target reference image includes:

[0025] Obtain the original mask, first transformation relationship, and second transformation relationship of the moving target in the original image. The first transformation relationship is the transformation relationship between the original pose and the first pose, and the second transformation relationship is the transformation relationship between the original pose and the second pose.

[0026] Based on the first transformation relationship, the original mask is transformed to obtain the first reference mask of the moving target in the background reference image;

[0027] Based on the second transformation relationship, the original mask is transformed to obtain the second reference mask of the moving target in the moving target reference image;

[0028] Based on the first reference mask, determine the first location range;

[0029] The second position range is determined based on the second reference mask.

[0030] In any embodiment of this application, generating a first depth image and a second depth image based on the plurality of original depth images includes:

[0031] Based on the first conversion relationship, the multiple original depth images are converted into multiple background depth images from the new perspective;

[0032] Based on the second conversion relationship, the multiple original depth images are converted into multiple moving target depth images from the new perspective;

[0033] The first depth image is obtained by performing volume rendering on the multiple background depth images;

[0034] The second depth image is obtained by performing volume rendering on the multiple moving target depth images.

[0035] Secondly, a training method for an optical flow model is provided, the method comprising:

[0036] Obtain the original image and the target image in the first aspect of the implementation;

[0037] Obtain the dense optical flow between the original image and the target image;

[0038] The dense optical flow is used as a label for the original image and the target image;

[0039] Based on the original image, the target image, and the label, the training model is trained to obtain an optical flow model, which is used to estimate dense optical flow.

[0040] Thirdly, a path planning method is provided, the path planning method being applied to a vehicle, the vehicle including a camera, the method comprising:

[0041] During the vehicle's operation, the camera continuously captures images of the vehicle's environment to obtain a first image to be processed and a second image to be processed. Both the first image to be processed and the second image to be processed include the target object.

[0042] Obtain the optical flow model obtained according to the method in the second aspect;

[0043] Using the optical flow model, the estimated optical flow between the first image to be processed and the second image to be processed is determined;

[0044] Based on the estimated optical flow, the first travel path of the target object is predicted;

[0045] Based on the first driving path, a second driving path for the vehicle is planned.

[0046] Fourthly, an apparatus for determining occlusion relationships is provided, the apparatus comprising:

[0047] An acquisition unit is used to acquire an original image and the depth information of the original image. The original image is an image obtained by a camera capturing a target scene from an original perspective. The target scene includes a background and moving targets.

[0048] The processing unit is used to process the original image and the depth information using a neural network to obtain multiple original depth images of MPI under the original viewpoint;

[0049] The generation unit is configured to generate a first depth image and a second depth image based on the plurality of original depth images; the first depth image includes: depth information of the background when the camera captures the background from a new perspective; the second depth image includes: depth information of the moving target when the camera captures the moving target from a new perspective.

[0050] The determining unit is configured to determine a target occlusion relationship based on the first depth image and the second depth image, wherein the target occlusion relationship is the occlusion relationship between the background and the moving target when the camera captures the target scene from the new perspective.

[0051] In any embodiment of this application, the processing unit is further configured to process the original image and the depth information using a neural network to obtain the MPI;

[0052] The generation unit is further configured to generate a new image from the new perspective based on the MPI, wherein the new image is an image captured by the camera of the target scene from the new perspective;

[0053] The generation unit is further configured to correct the occlusion relationship between the background and the moving target using the target occlusion relationship for the new image, thereby obtaining the target image from the new perspective.

[0054] In conjunction with any embodiment of this application, the determining unit is configured to:

[0055] Based on the original image, a background reference image is generated by the camera capturing the target scene in the first pose, and a moving target reference image is generated by the camera capturing the moving target in the second pose; the first pose is the pose of the camera capturing the background from the new perspective, and the second pose is the pose of the camera capturing the moving target from the new perspective.

[0056] Determine a first position range of the background in the background reference image, and a second position range of the moving target in the moving target reference image;

[0057] The location where the first location range and the second location range intersect is determined as the third location range;

[0058] Based on the first depth image and the second depth image, the occlusion relationship between the background and the moving target within the third position range is determined as the target occlusion relationship.

[0059] In conjunction with any embodiment of this application, the determining unit is configured to:

[0060] Based on the first depth image, a first reference depth of the background within the range of the third position is determined;

[0061] Based on the second depth image, a second reference depth of the moving target within the range of the third position is determined;

[0062] When the first reference depth is less than the second reference depth, the target occlusion relationship is determined to be that the moving target is occluded by the background;

[0063] If the first reference depth is greater than or equal to the second reference depth, the target occlusion relationship is determined to be that the background is occluded by the moving target.

[0064] In any embodiment of this application, the pose of the original image captured by the camera is the original pose;

[0065] The determining unit is used for:

[0066] Obtain the original mask, first transformation relationship, and second transformation relationship of the moving target in the original image. The first transformation relationship is the transformation relationship between the original pose and the first pose, and the second transformation relationship is the transformation relationship between the original pose and the second pose.

[0067] Based on the first transformation relationship, the original mask is transformed to obtain the first reference mask of the moving target in the background reference image;

[0068] Based on the second transformation relationship, the original mask is transformed to obtain the second reference mask of the moving target in the moving target reference image;

[0069] Based on the first reference mask, determine the first location range;

[0070] The second position range is determined based on the second reference mask.

[0071] In conjunction with any embodiment of this application, the generation unit is used for:

[0072] Based on the first conversion relationship, the multiple original depth images are converted into multiple background depth images from the new perspective;

[0073] Based on the second conversion relationship, the multiple original depth images are converted into multiple moving target depth images from the new perspective;

[0074] The first depth image is obtained by performing volume rendering on the multiple background depth images;

[0075] The second depth image is obtained by performing volume rendering on the multiple moving target depth images.

[0076] Fifthly, a training device for an optical flow model is provided, the device comprising:

[0077] The acquisition unit is used to acquire the original image and the target image in the first aspect of the implementation.

[0078] The acquisition unit is used to acquire the dense optical flow between the original image and the target image;

[0079] A processing unit is used to use the dense optical flow as a label for the original image and the target image;

[0080] The training unit is used to train the model to be trained based on the original image, the target image and the label to obtain an optical flow model, which is used to estimate dense optical flow.

[0081] Sixthly, a vehicle, said vehicle comprising:

[0082] A camera is used to continuously capture images of the environment in which the vehicle is located while the vehicle is in motion, to obtain a first image to be processed and a second image to be processed, both of which include a target object.

[0083] The acquisition unit is used to acquire the optical flow model obtained according to the method of the second aspect;

[0084] The determining unit is used to determine the estimated optical flow between the first image to be processed and the second image to be processed using the optical flow model;

[0085] The estimation unit is used to estimate the first travel path of the target object based on the estimated optical flow;

[0086] The planning unit is used to plan a second driving path for the vehicle based on the first driving path.

[0087] A seventh aspect provides an electronic device comprising: a processor and a memory, the memory being configured to store computer program code, the computer program code including computer instructions; wherein, when the program instructions are executed by the processor, the processor is configured to perform the first aspect and any embodiment thereof described above; wherein, when the program instructions are executed by the processor, the processor is configured to perform the second aspect and any embodiment thereof described above; wherein, when the program instructions are executed by the processor, the processor is configured to perform the third aspect and any embodiment thereof described above.

[0088] Eighthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs as described in the first aspect and any embodiment thereof; when the processor executes the computer instructions, the electronic device either performs as described in the second aspect and any embodiment thereof; when the processor executes the computer instructions, the electronic device either performs as described in the third aspect and any embodiment thereof.

[0089] A ninth aspect provides a computer-readable storage medium storing a computer program, the computer program including program instructions; when the program instructions are executed by a processor, causing the processor to perform the first aspect and any embodiment thereof described above; when the program instructions are executed by a processor, causing the processor to perform the second aspect and any embodiment thereof described above; when the program instructions are executed by a processor, causing the processor to perform the third aspect and any embodiment thereof described above.

[0090] In a tenth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions; when the computer program or instructions are executed on a computer, causing the computer to perform the first aspect and any embodiment thereof described above; when the program instructions are executed by a processor, causing the processor to perform the second aspect and any embodiment thereof described above; when the program instructions are executed by a processor, causing the processor to perform the third aspect and any embodiment thereof described above.

[0091] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0092] In this application, the original image is an image obtained by the camera from the original viewpoint of the target scene. After the generating device obtains the original image and the depth information of the original image, it can use a neural network to process the original image and the depth information to obtain multiple original depth images of MPI from the original viewpoint.

[0093] Since the target scene includes both a background and a moving target, during the process of changing the shooting perspective of the target scene from the original perspective to a new perspective, not only is there camera movement, but also movement of the moving target. Therefore, the change in depth of the background from the original perspective to the new perspective is caused by camera movement, while the change in depth of the moving target from the original perspective to the new perspective is caused by both camera movement and the movement of the moving target. Therefore, the generation device generates a first depth image based on multiple original depth images, including the depth information of the background when shooting the background from the new perspective, and a second depth image including the depth information of the moving target when shooting the moving target from the new perspective. In other words, the depth of the background can be determined based on the first depth image when the camera shoots the target scene from the new perspective, and the depth of the moving target can be determined based on the second depth image. Therefore, the generation device can determine the occlusion relationship between the background and the moving target when the camera shoots the target scene from the new perspective, based on the first and second depth images. Attached Figure Description

[0094] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0095] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0096] Figure 1 A flowchart illustrating a method for determining occlusion relationships provided in an embodiment of this application;

[0097] Figure 2 A flowchart illustrating a training method for an optical flow model provided in an embodiment of this application;

[0098] Figure 3 A schematic diagram of an architecture for generating optical flow datasets based on raw images, provided for embodiments of this application;

[0099] Figure 4a A schematic diagram of an original image provided for an embodiment of this application;

[0100] Figure 4b A method based on the embodiments of this application is provided. Figure 4a A schematic diagram of the generated target image;

[0101] Figure 4c An embodiment provided in this application Figure 4a and Figure 4b A schematic diagram of the dense optical flow between them;

[0102] Figure 5a A schematic diagram of an original image provided for an embodiment of this application;

[0103] Figure 5b A method based on the embodiments of this application is provided. Figure 5a The generated target image;

[0104] Figure 5c Another method based on the embodiments of this application is provided. Figure 5a The generated target image;

[0105] Figure 5d Another method based on the embodiments of this application is provided. Figure 5a The generated target image;

[0106] Figure 6 A flowchart illustrating a path planning method provided in an embodiment of this application;

[0107] Figure 7 A schematic diagram of the structure of a device for determining occlusion relationships provided in an embodiment of this application;

[0108] Figure 8 A schematic diagram of the structure of a training device for an optical flow model provided in an embodiment of this application;

[0109] Figure 9 A schematic diagram of a vehicle structure provided in an embodiment of this application;

[0110] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0111] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0112] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0113] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0114] The execution subject of this application embodiment is a device for determining occlusion relationships (hereinafter referred to as a generation device), wherein the generation device can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the generation device can be one of the following: a computer, a server.

[0115] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a method for determining occlusion relationships provided in an embodiment of this application.

[0116] 101. Obtain the original image and the depth information of the original image.

[0117] In this embodiment, the original image is an image captured by a camera from the original perspective of the target scene. The camera can be any device with imaging capabilities; for example, a camera can be a mobile phone with imaging capabilities, or a video camera. The original perspective can be any shooting perspective, and the target scene can be any scene.

[0118] The target scene includes a background and moving targets. The background includes stationary objects in the target scene, while the moving targets are objects in motion within the target scene. For example, a moving target could be a motor vehicle, a non-motorized vehicle, or a pedestrian.

[0119] In one implementation of acquiring raw images, the generating device has a communication connection with the camera, and the generating device acquires the raw images captured by the camera through this communication connection.

[0120] In another implementation of acquiring the original image, the generating device receives the original image input by the user through input components, wherein the input components include: a keyboard, a mouse, a touch screen, a touchpad, and an audio input device.

[0121] In another implementation of acquiring the original image, the generating device receives the original image sent by the terminal, which includes: a mobile phone, a computer, a tablet computer, or a server.

[0122] In this embodiment, the depth information of each pixel in the original image can be determined based on the depth information of the original image. Optionally, the depth information of the original image is a depth image of the original image. In one implementation of obtaining the depth information of the original image, the generating device estimates the depth image of the original image through a monocular depth estimation network and uses this depth image as the depth information of the original image. In another implementation of obtaining the depth information of the original image, the generating device receives the depth information of the original image input by the user through an input component.

[0123] It should be understood that the steps of acquiring the original image and acquiring the depth information of the original image can be performed simultaneously or separately, and this application does not limit this.

[0124] 102. Using a neural network to process the above original image and the above depth information, multiple original depth images of MPI under the above original viewpoint are obtained.

[0125] In this embodiment, the MPI includes multiple images, each containing pixels with the same depth, but the pixel depths differ between images. The original depth image is the depth image of the images in the MPI, and there is a one-to-one correspondence between the original depth image and the images in the MPI. Since the images in the MPI are based on the original images, the generation device uses a neural network to process the original image, extracting color values ​​and volume densities at multiple depths. Based on the color values ​​and volume densities at multiple depths, the original image can be generated to create the MPI at the original viewpoint, and the original depth image can be generated based on the depth of the images in the MPI. The aforementioned color values ​​can be values ​​from color channels.

[0126] Optionally, the generation device uses monocular depth estimation to obtain the depth of the original image and an RGBD image, where the RGBD image is a depth image including the original image. The neural network then extracts color values ​​and volume density at multiple depths based on the contextual information of the RGBD image, and generates the original image based on the color values ​​and volume density at multiple depths to generate the MPI at the original viewpoint, and then generates the original depth image based on the depth of the image in the MPI.

[0127] 103. Based on the above multiple original depth images, generate a first depth image and a second depth image.

[0128] In this application, the first depth image includes: depth information of the background when the camera captures the background from a new perspective. The new perspective can be any shooting perspective different from the original perspective. The second depth image includes: depth information of the moving target when the camera captures the moving target from a new perspective.

[0129] It should be understood that during the process of changing the shooting perspective of the target scene from the original perspective to the new perspective, not only is there camera movement, but also movement of the moving target. Therefore, the change in depth of the background from the original perspective to the new perspective is caused by camera movement, while the change in depth of the moving target from the original perspective to the new perspective is caused by both camera movement and the movement of the moving target. Thus, the depth information of the background in the first depth image is determined based on the camera movement and the depth information of the background in the original perspective, and the depth information of the moving target in the second depth image is determined based on the camera movement, the movement of the moving target, and the depth information of the background in the original perspective.

[0130] In one possible implementation, the generating device determines the depth information of the background and the depth information of the moving target in the multiple original depth images based on the depth information of the original depth images. A first depth image is generated based on a first motion amount and the background depth information in the multiple original depth images, where the first motion amount is the motion of the camera when the shooting angle changes from the original viewpoint to a new viewpoint. A second depth image is generated based on the first motion amount, a second motion amount, and the depth information of the moving target in the multiple original depth images, where the second motion amount is the motion of the moving target when the shooting angle changes from the original viewpoint to a new viewpoint.

[0131] 104. Based on the first depth image and the second depth image mentioned above, determine the target occlusion relationship.

[0132] In this embodiment, the target occlusion relationship refers to the occlusion relationship between the background and the moving target when the camera captures the target scene from a new perspective. Since the first depth image includes the depth information of the background when the camera captures the background from a new perspective, and the second depth image includes the depth information of the moving target when the camera captures the moving target from a new perspective, the generating device can determine the depth of the background and the depth of the moving target when the camera captures the target scene from a new perspective based on the first depth image and the second depth image, and thus determine the target occlusion relationship based on the depth of the background and the depth of the moving target.

[0133] In one possible implementation, the generating device determines the range of the background's position within the shooting range when the camera captures from a new perspective, based on the background's position in a first depth image. Based on the moving target's position in a second depth image, it determines the range of the moving target's position within the shooting range when the camera captures from a new perspective. The intersection of the background's and the moving target's positions within the shooting range is determined as the overlapping range of the background and the moving target. For the range outside the overlapping range within the shooting range, the target occlusion relationship is determined to be that the moving target is occluded by the background, meaning that within this range, the camera captures the background. For the overlapping range, the generating device determines the background depth based on the first depth image and the moving target depth based on the second depth image, then determines the occlusion relationship between the background and the moving target by comparing the depths of the background and the moving target. Specifically, if the background depth is less than the moving target depth, the generating device determines the target occlusion relationship to be that the moving target is occluded by the background, meaning that the camera captures the background; if the background depth is greater than or equal to the moving target depth, the generating device determines the target occlusion relationship to be that the background is occluded by the moving target, meaning that the camera captures the moving target.

[0134] In this embodiment of the application, the original image is an image obtained by the camera from the original viewpoint of the target scene. After the generating device obtains the original image and the depth information of the original image, it can use a neural network to process the original image and the depth information to obtain multiple original depth images of MPI under the original viewpoint.

[0135] Since the target scene includes both a background and a moving target, during the process of changing the shooting perspective of the target scene from the original perspective to a new perspective, not only is there camera movement, but also movement of the moving target. Therefore, the change in depth of the background from the original perspective to the new perspective is caused by camera movement, while the change in depth of the moving target from the original perspective to the new perspective is caused by both camera movement and the movement of the moving target. Therefore, the generation device generates a first depth image based on multiple original depth images, including the depth information of the background when shooting the background from the new perspective, and a second depth image including the depth information of the moving target when shooting the moving target from the new perspective. In other words, the depth of the background can be determined based on the first depth image when the camera shoots the target scene from the new perspective, and the depth of the moving target can be determined based on the second depth image. Therefore, the generation device can determine the occlusion relationship between the background and the moving target when the camera shoots the target scene from the new perspective, based on the first and second depth images.

[0136] As an optional implementation, the generating apparatus also performs the following steps:

[0137] 2001. The original image and depth information were processed using a neural network to obtain the MPI under the original viewpoint.

[0138] For details on how to implement this step, please refer to step 102. It will not be repeated here.

[0139] 2002. Based on the MPI under the original perspective, generate a new image under the new perspective.

[0140] In this embodiment, the new image is an image obtained by the camera capturing the target scene from a new perspective. In one possible implementation, the pose of the original image captured by the camera from the original perspective is the original pose, and the pose of the camera capturing the target scene from the new perspective is the first pose. The generation device determines a first transformation relationship between the original pose and the first pose based on the original pose and the first pose.

[0141] Based on the first transformation relation, the MPI under the original viewpoint is transformed to obtain the MPI under the new viewpoint. Specifically, the generating device uses the first transformation relation to transform each image in the MPI under the original viewpoint to obtain the MPI under the new viewpoint. In one possible implementation, the generating device transforms the MPI under the original viewpoint based on the first transformation relation to obtain multiple images to be interpolated, and performs bilinear interpolation on the multiple images to be interpolated to obtain the MPI under the new viewpoint.

[0142] Optional, use Representing pixels in the MPI from the original viewpoint, using To represent pixels in MPI from a new perspective, then , Satisfy the following formula:

[0143] …Formula (1)

[0144] in, Let be the rotation matrix in the first transformation relation. This is the translation matrix in the first transformation relation. This is the camera's inherent intrinsic parameter matrix. It is the normal vector. The first in MPI from the original perspective The depth of the image.

[0145] After obtaining the MPI from the new perspective, the generation device performs volume rendering on the MPI to obtain a new image. Optionally, the generation device performs volume rendering on the MPI from the new perspective, and the generated new image is expressed by the following formula:

[0146] …Formula (2)

[0147] in, For the new image, N is the number of images in the MPI from the original viewpoint. The color value of the nth image in the MPI from the new perspective. , The first in MPI from the original perspective The ( ) image and the MPI from the original viewpoint The distance between the images The first in MPI from a new perspective Volume density of the image , The first in MPI from the original perspective The ( ) image and the MPI from the original viewpoint The distance between the images The first in MPI from a new perspective Volume density in an image.

[0148] Given that the color values ​​and volume density of pixels in the MPI under the new perspective are based on the real color values ​​and volume density of the MPI under the original perspective, by performing volume rendering on the MPI under the new perspective, the color values ​​and volume densities of different images in the MPI can be weighted and fused to generate a new image, thereby improving the realism of the new image. This enables the generation of a real image under the new perspective based on a single real image.

[0149] 2003. For the above new image, the occlusion relationship between the background and the moving target is corrected by using the above target occlusion relationship to obtain the target image under the above new perspective.

[0150] Because generating a new image based on the original viewpoint using MPI only considers camera motion when the camera's shooting angle changes from the original viewpoint to the new viewpoint, and does not consider the motion of moving targets, there are errors in the occlusion relationship between the moving target and the background in the new image. However, the target occlusion relationship is determined based on the first and second depth images, thus considering not only camera motion but also the motion of the moving target. Therefore, the target occlusion relationship can be used to correct the occlusion relationship between the background and the moving target in the new image. Specifically, by correcting the new image using the target occlusion relationship, a target image captured by the camera from the new viewpoint can be obtained, where the occlusion relationship between the moving target and the background in the target image is the target occlusion relationship.

[0151] In this embodiment, the generation device uses a neural network to process the original image and depth information to obtain the MPI from the original viewpoint. Based on the multi-plane image, a new image from a new viewpoint is generated. The new image is then corrected using target occlusion relationships, resulting in a target image captured by the camera from a new viewpoint, which improves the quality of the target image.

[0152] As an optional implementation, the generating apparatus performs the following steps during step 104:

[0153] 3001. Based on the original image, generate a background reference image obtained by the camera capturing the target scene in the first pose, and a moving target reference image obtained by the camera capturing the target scene in the second pose.

[0154] In this embodiment, the first pose is the pose of the camera when shooting the background from a new perspective. It should be understood that when the camera's shooting perspective changes from the original perspective to the new perspective, the camera moves, but the background does not. Therefore, in step 2002, the first pose being the pose of the camera when shooting the target scene from a new perspective means that, without considering the movement of the moving target, the pose of the camera when shooting the target scene from a new perspective is the same as the pose of the camera when shooting the background from a new perspective. The second pose is the pose of the camera when shooting the moving target from a new perspective.

[0155] The camera's pose when shooting the background from its original viewpoint and when shooting a moving target from its original viewpoint are both considered the original pose. The change in the background's shooting perspective from the original viewpoint to a new perspective is caused by the camera's movement. Therefore, the change from the original pose to the original pose can characterize the amount of camera movement during the process of changing the background's shooting perspective from the original viewpoint to a new perspective.

[0156] Furthermore, since the moving target is in motion, it will also move during the process of changing the shooting perspective of the background from the original perspective to the new perspective. Therefore, changing the shooting perspective of the moving target from the original perspective to the new perspective involves not only the movement of the camera but also the movement of the moving target. The change from the original pose to the second pose can be used to characterize the amount of camera movement and the amount of moving target movement during the process of changing the shooting perspective of the moving target from the original perspective to the new perspective.

[0157] In this embodiment, the background reference image is an image obtained by capturing the background with the camera after the shooting perspective is changed from the original perspective to a new perspective, based on the camera's motion. The moving target reference image is an image obtained by capturing the moving target with the camera after the shooting perspective is changed from the original perspective to a new perspective, based on both the camera's motion and the moving target's motion.

[0158] In one possible implementation, the generating device acquires an image generation neural network. The original image, the original pose, and the first pose are input into the image generation neural network to generate a background reference image. The original image, the original pose, and the second pose are input into the image generation neural network to generate a moving target reference image.

[0159] In another possible implementation, the generation device is based on MPI technology, using the original image, the original pose, and the first pose to generate a background reference image. Based on MPI technology, a moving target reference image is generated using the original image, the original pose, and the second pose.

[0160] 3002. Determine the first position range of the background in the background reference image and the second position range of the moving target in the moving target reference image.

[0161] Since the background reference image is an image obtained by the camera capturing the background from a new perspective, the first position range is the position range of the background within the shooting range when the camera captures the target scene from a new perspective. Since the moving target reference image is an image obtained by the camera capturing the moving target from a new perspective, the second position range is the position range of the moving target within the shooting range when the camera captures the target scene from a new perspective.

[0162] 3003. The location at the intersection of the first position range and the second position range is determined as the third position range.

[0163] In this embodiment of the application, the third position range is the overlapping position range of the background and the moving target within the shooting range when the camera shoots the target scene from a new perspective.

[0164] In one possible implementation, the generating apparatus acquires an original mask, a first transformation relation, and a second transformation relation for a moving target in the original image. Based on the first transformation relation, the original mask is transformed to obtain a first reference mask for the moving target in the background reference image. Based on the second transformation relation, the original mask is transformed to obtain a second reference mask for the moving target in the moving target reference image. A first position range is determined based on the first reference mask. A second position range is determined based on the second reference mask.

[0165] 3004. Based on the first depth image and the second depth image, determine the occlusion relationship between the background and the moving target within the third position range, and use it as the target occlusion relationship.

[0166] When the camera is shooting a moving target from a new perspective, the overlapping position of the background and the moving target is within the third position range. The background and the moving target may be occluded within the third position range, so it is necessary to determine the occlusion relationship between the background and the moving target within the third position range.

[0167] Since the first depth image includes depth information of the background when the camera captures the background from a new perspective, and the second depth image includes depth information of the moving target when the camera captures the moving target from a new perspective, the generation device can determine the depth of the background and the depth of the moving target within a third position range when the camera captures the target scene from a new perspective, based on the first and second depth images. Furthermore, based on the depths of the background and the moving target within the third position range, the device can determine the occlusion relationship between the background and the moving target within the third position range, which is then used as the target occlusion relationship.

[0168] In one possible implementation, the generating device determines a first reference depth of the background within the third position range based on a first depth image. Based on a second depth image, it determines a second reference depth of the moving target within the third position range. If the first reference depth is less than the second reference depth, the target occlusion relationship is determined to be that the moving target is occluded by the background. If the first reference depth is greater than or equal to the second reference depth, the target occlusion relationship is determined to be that the background passively occludes the target.

[0169] Optionally, after acquiring the first reference mask, the depth image of the target image, the second reference mask, and the depth image of the moving target reference image, the generating device may determine the following formula:

[0170] …Formula (3)

[0171] in, This represents the area in the target image where a moving target is obscured by the background. Indicates the first reference mask. Indicates the second reference mask. The depth image representing the target image. A depth image representing a reference image of a moving target.

[0172] In this embodiment, the first pose is the camera's pose for capturing the background from a new perspective, and the second pose is the camera's pose for capturing the moving target from a new perspective. Since the change in the background's shooting perspective from the original viewpoint to the new viewpoint is caused by camera movement, the transition from the original pose to the first pose represents the amount of camera movement during this process. Similarly, the change in the moving target's shooting perspective from the original viewpoint to the new viewpoint is caused by both camera movement and the moving target's movement, therefore, the transition from the original pose to the second pose represents the amount of camera movement and the amount of moving target movement during this process.

[0173] The generation device generates a background reference image obtained by the camera capturing the moving target in a first pose, and a moving target reference image obtained by the camera capturing the moving target in a second pose, based on the original image. This allows the background in the background reference image to be the background from a new perspective determined by the camera's motion, and the moving target reference image to be the moving target from a new perspective determined by both the camera's motion and the moving target's motion.

[0174] Therefore, the generating device determines a first position range of the background in the background reference image, which is the position range of the background within the shooting range when the camera captures the target scene from a new perspective. The generating device determines a second position range of the moving target in the moving target reference image, which is the position range of the moving target within the shooting range when the camera captures the target scene from a new perspective. Then, the intersection of the first and second position ranges is determined as a third position range, where the third position range is the overlapping position range of the background and the moving target within the shooting range when the camera captures the target scene from a new perspective.

[0175] When the camera captures a moving target from a new perspective, the overlap between the background and the moving target falls within a third position range, and occlusion may exist between them within this range. Based on the first depth image, the depth of the background within this third position range can be determined when the camera captures the target scene from a new perspective. Similarly, based on the second depth image, the depth of the moving target within this third position range can be determined. Therefore, the generation device can determine the occlusion relationship between the background and the moving target within this third position range based on the first and second depth images, and use this as the target occlusion relationship.

[0176] As an optional implementation, the generating apparatus performs the following steps during step 103:

[0177] 4001. Based on the first transformation relationship described above, the above multiple original depth images are converted into multiple background depth images under the new perspective.

[0178] The generating device transforms each of the multiple original depth images from the original viewpoint to the new viewpoint through the first transformation relationship, thereby obtaining multiple background depth images. Among them, the multiple background depth images are the MPI depth images of the background reference image under the new viewpoint.

[0179] 4002. Based on the second transformation relationship described above, the original depth images are converted into multiple moving target depth images from the new perspective.

[0180] The generating device transforms each of the multiple original depth images from the original viewpoint to the new viewpoint through the second transformation relationship, thereby obtaining multiple moving target depth images. Among them, the multiple moving target depth images are the MPI depth images of the moving target reference image under the new viewpoint.

[0181] 4003. Perform volume rendering on the above multiple background depth images to obtain the above first depth image.

[0182] Since the background depth image is the MPI depth image of the background reference image under a new perspective, by performing volume rendering on multiple background depth images, the multiple background depth images can be weighted and fused to obtain the depth image of the background reference image (i.e., the first depth image mentioned above).

[0183] 4004. Perform volume rendering on the above multiple moving target depth images to obtain the above second depth image.

[0184] Since the moving target depth image is the MPI depth image of the moving target reference image under a new perspective, by performing volume rendering on multiple moving target depth images, the multiple moving target depth images can be weighted and fused to obtain the depth image of the moving target reference image (i.e., the second depth image mentioned above).

[0185] In this embodiment, the generation device converts multiple original depth images into multiple background depth images from a new perspective based on a first transformation relationship, thereby obtaining a depth image of the background reference image's MPI from the new perspective. Based on a second transformation relationship, the multiple original depth images are converted into multiple moving target depth images from the new perspective, thereby obtaining a depth image of the moving target reference image's MPI from the new perspective. Then, by performing volume rendering on the multiple background depth images, a first depth image of the background reference image is obtained; and by performing volume rendering on the multiple moving target depth images, a second depth image of the moving target reference image is obtained.

[0186] The method for determining occlusion relationships provided in this application can generate a target image based on the original image. Furthermore, by acquiring the dense optical flow between the original and target images, the dense optical flow can be used as an optical flow label for both images to construct an optical flow dataset. This dataset can then be used to train a model, enabling it to estimate dense optical flow. Based on this, this application also provides a method for training an optical flow model.

[0187] The subject of execution of the technical solution disclosed in the embodiments of the optical flow model training method is an optical flow model training device (hereinafter referred to as the training device). The training device can be any electronic device capable of executing the technical solution disclosed in the embodiments of the optical flow model training method. Optionally, the training device can be one of the following: a computer or a server.

[0188] It should be understood that embodiments of the optical flow model training method can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a training method for an optical flow model provided in an embodiment of this application.

[0189] 201. Obtain the original image and target image in the method for determining occlusion relationships.

[0190] The original image in this step is the original image in step 101, and the target image in this step is the target image generated through steps 2001 to 2003.

[0191] 202. Obtain the dense optical flow between the original image and the target image.

[0192] In one possible implementation, the generating apparatus generates an MPI (Multi-View Image) from an original viewpoint based on the original image. Using a first transformation relation, the MPI from the original viewpoint is converted to an MPI from a new viewpoint. Multiple reference optical flows are obtained by calculating the optical flow between the MPI from the original viewpoint and the MPI from the new viewpoint. For example, the MPI from the original viewpoint includes images a and b, and the MPI from the new viewpoint includes images c and d, where image c is the image obtained by transforming image a using the first transformation relation, and image d is the image obtained by transforming image b using the first transformation relation. In this case, the multiple reference optical flows include: a dense optical flow between image a and image c, and a dense optical flow between image b and image d. By performing volume rendering on the multiple reference optical flows, a dense optical flow between the original image and the target image is obtained.

[0193] 203. Use the above-mentioned dense optical flow as the label for the above-mentioned original image and the above-mentioned target image.

[0194] In this embodiment, the dense optical flow label represents the dense optical flow between the original image and the target image. The dense optical flow label can serve as the ground truth (GT) of the dense optical flow between the original image and the target image.

[0195] 204. Based on the above original image, the above target image and the above labels, train the model to be trained to obtain the optical flow model.

[0196] The model to be trained can be any model with a structure for estimating dense optical flow. The optical flow model obtained by training the model to be trained is used to estimate the dense optical flow. In one possible implementation, the model to be trained estimates the predicted optical flow between the original and target images by processing the original and target images. The loss of the model to be trained is determined based on the difference between the predicted optical flow and the label, where the difference is positively correlated with the loss. The gradient of the backpropagation of the model to be trained is determined based on this loss, and the parameters of the model to be trained are updated based on this gradient until the loss converges, resulting in the optical flow model.

[0197] In this embodiment, the original image is a real image, and the target image based on the original image has high realism and quality. Therefore, the training device uses the original image, the target image, and the dense optical flow between the original image and the target image to train the model to obtain an optical flow model, which can improve the estimation accuracy of the optical flow model for the dense optical flow of real images. Moreover, by determining the occlusion relationship, an optical flow dataset including diverse real scenes can be constructed at a lower cost. Therefore, by determining the occlusion relationship, generating the target image, and constructing an optical flow dataset for training the model based on the original image and the target image, and using the optical flow dataset to train the model, the training effect can be improved at a lower cost.

[0198] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating an architecture for generating optical flow datasets based on raw images, provided as an embodiment of this application. Figure 3 As shown, the architecture comprises two parts: 1. Optical Flow from MPI, and 2. Depth-Aware Inpainting. The input is the original image (i.e.,...) Figure 3 In After that, the depth image of the original image is obtained by performing depth estimation on the original image (i.e., Figure 3 In By inputting the original image into a neural network, the output is the MPI from the original viewpoint.

[0199] Based on the color values, volume density, and MPI from the original viewpoint provided by the depth image, volume rendering, first pose, second pose, and volume rendering are performed to generate the first depth image (i.e., Figure 3 In ), second depth image (i.e. Figure 3 In The depth-aware filling module determines the target occlusion relationship (i.e., based on the first and second depth images) and new images from a new perspective. Figure 3 middle The depth-sensing filling module then corrects the occlusion relationship between the background and the moving target in the new image using the target occlusion relationship, thus obtaining the target image from the new perspective (i.e., Figure 3 In ).

[0200] Furthermore, based on the color values, volume density, and MPI provided by the depth image, volume rendering, first pose, second pose, and volume rendering can be performed, and dense optical flow (i.e., between the original image and the target image) can also be generated. Figure 3 In The implementation of this process can be found in step 202 below. Finally, the output consists of the original image, the target image, and the dense optical flow between the original image and the target image as an optical flow dataset.

[0201] Please see Figure 4a and Figure 4b , Figure 4a This is a schematic diagram of an original image provided in an embodiment of this application. Figure 4b A method based on the embodiments of this application is provided. Figure 4a A schematic diagram of the generated target image, wherein, Figure 4b The target image shown was obtained using the method described in this application. Figure 4b As shown, magnifying the car reveals that the target image exhibits high detail quality and realism, proving the high quality and realism of the target image generated by the method of this application. Furthermore, through comparison... Figure 4a and Figure 4b It can be seen that the car not only underwent relative motion with respect to the background, but also Figure 4b The occlusion relationship between the car and the traffic sign is correctly shown, which also shows that the target image generated by the method of this application can accurately determine the target occlusion relationship between the moving target and the background from a new perspective when there is a moving target in the target scene.

[0202] Please see Figure 4c , Figure 4c An embodiment provided in this application Figure 4a and Figure 4b A schematic diagram of the dense optical flow between them. Figure 4c The dense optical flow shown is obtained through step 202. For example... Figure 4c As shown, the dense optical flow has a high matching degree with the target image, which also proves that the method of this application can generate an optical flow dataset with only one real image.

[0203] To further illustrate the high quality of the optical flow dataset generated using the method of this application, embodiments of this application also provide some comparative schematic diagrams of the method of this application with other methods.

[0204] Please see Figure 5a , Figure 5b , Figure 5c and Figure 5d , Figure 5a A schematic diagram of an original image provided for the implementation of this application. Figure 5b , Figure 5c and Figure 5d All are based on Figure 5a The generated target image, where, Figure 5aIt is generated by the method of learning optical flow from still images (Depthstill). Figure 5c It was generated using the RealFlow (Em based realistic optical flow dataset generation from videos) method. Figure 5d It was generated using the method described in this application. (By comparison...) Figure 5b , Figure 5c , Figure 5d As can be seen from the magnified view, the target image generated by the method of this application has higher detail quality compared to the target images generated by the other two methods.

[0205] This application also provides a path planning method, wherein the subject of the technical solution disclosed in the embodiment of the path planning method is a vehicle, which includes a camera and includes motor vehicles and non-motor vehicles.

[0206] It should be understood that embodiments of the path planning method can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 6 , Figure 6 This is a flowchart illustrating a path planning method provided in an embodiment of this application.

[0207] 601. During the driving of the vehicle, the environment in which the vehicle is located is continuously photographed by the camera to obtain a first image to be processed and a second image to be processed.

[0208] In this application, the shooting time of the first image to be processed and the shooting time of the second image to be processed are different, and both the first image to be processed and the second image to be processed include a target object. The target object is an object in the environment where the mobile robot is located. For example, the target object can be a person in the environment where the mobile robot is located, or it can be other vehicles in the environment where the mobile robot is located.

[0209] 602. Obtain the optical flow model obtained from the training method of the optical flow model.

[0210] 603. Using the above optical flow model, determine the estimated optical flow between the first image to be processed and the second image to be processed.

[0211] By using an optical flow model, the dense optical flow between the first image to be processed and the second image to be processed can be estimated, thereby obtaining the estimated optical flow.

[0212] 604. Based on the above estimated optical flow, predict the first travel path of the target object.

[0213] In one possible implementation, the vehicle determines the motion information of the target object based on estimated optical flow, whereby the motion information includes: motion speed and motion direction. The vehicle can then predict the future driving path of the target object based on this motion information, which is the aforementioned first driving path.

[0214] 605. Based on the first driving path described above, plan the second driving path for the vehicle.

[0215] Based on the first driving path, the car plans its future driving path to avoid collisions with the target object. The second driving path mentioned above is the planned future driving path of the car.

[0216] In this embodiment, the vehicle captures a first image and a second image to be processed via a camera while in motion. Then, after obtaining an optical flow model, the estimated optical flow between the first and second images to be processed is determined using the optical flow model, which improves the accuracy of the estimated optical flow. Based on the estimated optical flow, a second driving path for the target object in the first and second images to be processed is predicted, which improves the accuracy of the second driving path. Finally, based on the first driving path, the vehicle's second driving path is planned, which further improves the accuracy of the second driving path.

[0217] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0218] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0219] Please see Figure 7 , Figure 7 This is a schematic diagram of a device for determining occlusion relationships provided in an embodiment of this application. The device 1 for determining occlusion relationships includes: an acquisition unit 11, a processing unit 12, a generation unit 13, and a determination unit 14. Specifically:

[0220] The acquisition unit 11 is used to acquire an original image and the depth information of the original image. The original image is an image obtained by the camera from the original viewpoint of the target scene. The target scene includes the background and moving targets.

[0221] Processing unit 12 is used to process the original image and the depth information using a neural network to obtain multiple original depth images of MPI under the original viewpoint;

[0222] The generation unit 13 is configured to generate a first depth image and a second depth image based on the plurality of original depth images; the first depth image includes: depth information of the background when the camera takes a picture of the background from a new perspective; the second depth image includes: depth information of the moving target when the camera takes a picture of the moving target from a new perspective.

[0223] The determining unit 14 is used to determine the target occlusion relationship based on the first depth image and the second depth image. The target occlusion relationship is the occlusion relationship between the background and the moving target when the camera takes a picture of the target scene from the new perspective.

[0224] In any embodiment of this application, the processing unit 12 is further configured to process the original image and the depth information using a neural network to obtain the MPI;

[0225] The generation unit 13 is further configured to generate a new image from the new perspective based on the MPI, wherein the new image is an image captured by the camera of the target scene from the new perspective;

[0226] The generation unit 13 is further configured to correct the occlusion relationship between the background and the moving target using the target occlusion relationship for the new image, thereby obtaining the target image from the new perspective.

[0227] In any embodiment of this application, the determining unit 14 is configured to:

[0228] Based on the original image, a background reference image is generated by the camera capturing the target scene in the first pose, and a moving target reference image is generated by the camera capturing the moving target in the second pose; the first pose is the pose of the camera capturing the background from the new perspective, and the second pose is the pose of the camera capturing the moving target from the new perspective.

[0229] Determine a first position range of the background in the background reference image, and a second position range of the moving target in the moving target reference image;

[0230] The location where the first location range and the second location range intersect is determined as the third location range;

[0231] Based on the first depth image and the second depth image, the occlusion relationship between the background and the moving target within the third position range is determined as the target occlusion relationship.

[0232] In any embodiment of this application, the determining unit 14 is configured to:

[0233] Based on the first depth image, a first reference depth of the background within the range of the third position is determined;

[0234] Based on the second depth image, a second reference depth of the moving target within the range of the third position is determined;

[0235] When the first reference depth is less than the second reference depth, the target occlusion relationship is determined to be that the moving target is occluded by the background;

[0236] If the first reference depth is greater than or equal to the second reference depth, the target occlusion relationship is determined to be that the background is occluded by the moving target.

[0237] In any embodiment of this application, the pose of the original image captured by the camera is the original pose;

[0238] The determining unit 14 is used for:

[0239] Obtain the original mask, first transformation relationship, and second transformation relationship of the moving target in the original image. The first transformation relationship is the transformation relationship between the original pose and the first pose, and the second transformation relationship is the transformation relationship between the original pose and the second pose.

[0240] Based on the first transformation relationship, the original mask is transformed to obtain the first reference mask of the moving target in the background reference image;

[0241] Based on the second transformation relationship, the original mask is transformed to obtain the second reference mask of the moving target in the moving target reference image;

[0242] Based on the first reference mask, determine the first location range;

[0243] The second position range is determined based on the second reference mask.

[0244] In conjunction with any embodiment of this application, the generation unit 13 is used for:

[0245] Based on the first conversion relationship, the multiple original depth images are converted into multiple background depth images from the new perspective;

[0246] Based on the second conversion relationship, the multiple original depth images are converted into multiple moving target depth images from the new perspective;

[0247] The first depth image is obtained by performing volume rendering on the multiple background depth images;

[0248] The second depth image is obtained by performing volume rendering on the multiple moving target depth images.

[0249] In this embodiment of the application, the original image is an image obtained by the camera from the original viewpoint of the target scene. After the generating device obtains the original image and the depth information of the original image, it can use a neural network to process the original image and the depth information to obtain multiple original depth images of MPI under the original viewpoint.

[0250] Since the target scene includes both a background and a moving target, during the process of changing the shooting perspective of the target scene from the original perspective to a new perspective, not only is there camera movement, but also movement of the moving target. Therefore, the change in depth of the background from the original perspective to the new perspective is caused by camera movement, while the change in depth of the moving target from the original perspective to the new perspective is caused by both camera movement and the movement of the moving target. Therefore, the generation device generates a first depth image based on multiple original depth images, including the depth information of the background when shooting the background from the new perspective, and a second depth image including the depth information of the moving target when shooting the moving target from the new perspective. In other words, the depth of the background can be determined based on the first depth image when the camera shoots the target scene from the new perspective, and the depth of the moving target can be determined based on the second depth image. Therefore, the generation device can determine the occlusion relationship between the background and the moving target when the camera shoots the target scene from the new perspective, based on the first and second depth images.

[0251] Please see Figure 8 , Figure 8 This is a schematic diagram of a training device for an optical flow model provided in an embodiment of this application. The training device 2 for the optical flow model includes: an acquisition unit 21, a processing unit 22, and a training unit 23. Specifically:

[0252] Acquisition unit 21 is used to acquire the original image and the target image in the first aspect embodiment;

[0253] The acquisition unit 21 is used to acquire the dense optical flow between the original image and the target image;

[0254] Processing unit 22 is used to use the dense optical flow as a label for the original image and the target image;

[0255] Training unit 23 is used to train the model to be trained based on the original image, the target image and the label to obtain an optical flow model, which is used to estimate dense optical flow.

[0256] In this embodiment, the original image is a real image, and the target image based on the original image has high realism and quality. Therefore, the training device uses the original image, the target image, and the dense optical flow between the original image and the target image to train the model to obtain an optical flow model, which can improve the estimation accuracy of the optical flow model for the dense optical flow of real images. Moreover, by determining the occlusion relationship, an optical flow dataset including diverse real scenes can be constructed at a lower cost. Therefore, by determining the occlusion relationship, generating the target image, and constructing an optical flow dataset for training the model based on the original image and the target image, and using the optical flow dataset to train the model, the training effect can be improved at a lower cost.

[0257] Please see Figure 9 , Figure 9 This application provides a schematic diagram of the structure of a vehicle 3, which includes: a camera 31, an acquisition unit 32, a determination unit 33, an estimation unit 34, and a planning unit 35. Specifically:

[0258] Camera 31 is used to continuously capture images of the environment in which the vehicle is located during the vehicle's movement, and to obtain a first image to be processed and a second image to be processed, both of which include the target object.

[0259] Acquisition unit 32 is used to acquire the optical flow model obtained according to the method of the second aspect;

[0260] Determining unit 33 is used to determine the estimated optical flow between the first image to be processed and the second image to be processed using the optical flow model;

[0261] The estimation unit 34 is used to estimate the first travel path of the target object based on the estimated optical flow;

[0262] Planning unit 35 is used to plan a second driving path for the vehicle based on the first driving path.

[0263] In this embodiment, the vehicle captures a first image and a second image to be processed via a camera while in motion. Then, after obtaining an optical flow model, the estimated optical flow between the first and second images to be processed is determined using the optical flow model, which improves the accuracy of the estimated optical flow. Based on the estimated optical flow, a second driving path for the target object in the first and second images to be processed is predicted, which improves the accuracy of the second driving path. Finally, based on the first driving path, the vehicle's second driving path is planned, which further improves the accuracy of the second driving path.

[0264] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0265] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 4 includes a processor 41 and a memory 42. Optionally, the electronic device 4 also includes an input device 43 and an output device 44. The processor 41, memory 42, input device 43, and output device 44 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0266] Processor 41 may include one or more processors, such as one or more central processing units (CPUs). If the processor is a CPU, it may be a single-core CPU or a multi-core CPU. Optionally, processor 41 may be a processor group consisting of multiple CPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in this embodiment.

[0267] The memory 42 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0268] Input device 43 is used to input data and / or signals, and output device 44 is used to output data and / or signals. Input device 43 and output device 44 can be independent devices or an integrated device.

[0269] It is understood that in this embodiment of the application, the memory 42 can be used not only to store related instructions, but also to store related data. This embodiment of the application does not limit the specific data stored in the memory.

[0270] Understandable, Figure 10 This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0271] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0272] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0273] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0274] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0275] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0276] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0277] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for determining occlusion relationships, characterized in that, The method includes: Acquire the original image and its depth information. The original image is an image captured by the camera from the original viewpoint of the target scene, which includes the background and moving targets. The original image and the depth information are processed using a neural network to obtain multiple original depth images of the multi-plane image under the original viewpoint; Based on the multiple original depth images, a first depth image and a second depth image are generated; the first depth image includes: the depth information of the background when the camera captures the background from a new perspective, and the second depth image includes: the depth information of the moving target when the camera captures the moving target from a new perspective; the first depth image is obtained based on a first motion amount and the depth information of the background in the multiple original depth images, and the second depth image is obtained based on the first motion amount, the second motion amount, and the depth information of the moving target in the multiple original depth images, wherein the first motion amount is the motion amount of the camera when the camera's shooting angle changes from the original perspective to the new perspective, and the second motion amount is the motion amount of the moving target when the camera's shooting angle changes from the original perspective to the new perspective; Based on the first depth image and the second depth image, a target occlusion relationship is determined. The target occlusion relationship is the occlusion relationship between the background and the moving target when the camera captures the target scene from the new perspective.

2. The method according to claim 1, characterized in that, The method further includes: The original image and the depth information are processed using a neural network to obtain the multi-planar image; Based on the multi-plane image, a new image is generated from the new perspective, which is an image of the target scene captured by the camera from the new perspective; For the new image, the occlusion relationship between the background and the moving target is corrected using the target occlusion relationship to obtain the target image from the new perspective.

3. The method according to claim 1 or 2, characterized in that, Determining the target occlusion relationship based on the first depth image and the second depth image includes: Based on the original image, a background reference image is generated by the camera capturing the target scene in a first pose, and a moving target reference image is generated by the camera capturing the moving target in a second pose; the first pose is the pose of the camera capturing the background from the new perspective, and the second pose is the pose of the camera capturing the moving target from the new perspective. Determine a first position range of the background in the background reference image, and a second position range of the moving target in the moving target reference image; The location where the first location range and the second location range intersect is determined as the third location range; Based on the first depth image and the second depth image, the occlusion relationship between the background and the moving target within the third position range is determined as the target occlusion relationship.

4. The method according to claim 3, characterized in that, The step of determining the occlusion relationship between the background and the moving target within the third position range based on the first depth image and the second depth image, as the target occlusion relationship, includes: Based on the first depth image, a first reference depth of the background within the range of the third position is determined; Based on the second depth image, a second reference depth of the moving target within the range of the third position is determined; When the first reference depth is less than the second reference depth, the target occlusion relationship is determined to be that the moving target is occluded by the background; If the first reference depth is greater than or equal to the second reference depth, the target occlusion relationship is determined to be that the background is occluded by the moving target.

5. The method according to claim 3, characterized in that, The pose of the original image captured by the camera is the original pose; Determining the first position range of the background in the background reference image and the second position range of the moving target in the moving target reference image includes: Obtain the original mask, first transformation relationship, and second transformation relationship of the moving target in the original image. The first transformation relationship is the transformation relationship between the original pose and the first pose, and the second transformation relationship is the transformation relationship between the original pose and the second pose. Based on the first transformation relationship, the original mask is transformed to obtain the first reference mask of the moving target in the background reference image; Based on the second transformation relationship, the original mask is transformed to obtain the second reference mask of the moving target in the moving target reference image; Based on the first reference mask, determine the first location range; The second position range is determined based on the second reference mask.

6. The method according to claim 5, characterized in that, The step of generating a first depth image and a second depth image based on the multiple original depth images includes: Based on the first conversion relationship, the multiple original depth images are converted into multiple background depth images from the new perspective; Based on the second conversion relationship, the multiple original depth images are converted into multiple moving target depth images from the new perspective; The first depth image is obtained by performing volume rendering on the multiple background depth images; The second depth image is obtained by performing volume rendering on the multiple moving target depth images.

7. A training method for an optical flow model, characterized in that, The method includes: Obtain the original image and the target image as described in claim 2; Obtain the dense optical flow between the original image and the target image; The dense optical flow is used as a label for the original image and the target image; Based on the original image, the target image, and the label, the training model is trained to obtain an optical flow model, which is used to estimate dense optical flow.

8. A path planning method, characterized in that, The path planning method is applied to a vehicle, the vehicle including a camera, and the method includes: During the vehicle's operation, the camera continuously captures images of the vehicle's environment to obtain a first image to be processed and a second image to be processed. Both the first image to be processed and the second image to be processed include the target object. Obtain the optical flow model obtained by the method according to claim 7; Using the optical flow model, the estimated optical flow between the first image to be processed and the second image to be processed is determined; Based on the estimated optical flow, the first travel path of the target object is predicted; Based on the first driving path, a second driving path for the vehicle is planned.

9. A device for determining occlusion relationships, characterized in that, The device includes: An acquisition unit is used to acquire an original image and the depth information of the original image. The original image is an image obtained by a camera capturing a target scene from an original perspective. The target scene includes a background and moving targets. The processing unit is used to process the original image and the depth information using a neural network to obtain multiple original depth images of the multi-plane image under the original viewpoint; A generation unit is configured to generate a first depth image and a second depth image based on the plurality of original depth images; the first depth image includes depth information of the background when the camera captures the background from a new perspective, and the second depth image includes depth information of the moving target when the camera captures the moving target from a new perspective; the first depth image is obtained based on a first motion amount and the depth information of the background in the plurality of original depth images, and the second depth image is obtained based on the first motion amount, the second motion amount, and the depth information of the moving target in the plurality of original depth images; the first motion amount is the motion amount of the camera when the camera's shooting angle changes from the original perspective to the new perspective, and the second motion amount is the motion amount of the moving target when the camera's shooting angle changes from the original perspective to the new perspective; The determining unit is configured to determine a target occlusion relationship based on the first depth image and the second depth image, wherein the target occlusion relationship is the occlusion relationship between the background and the moving target when the camera captures the target scene from the new perspective.

10. A training device for an optical flow model, characterized in that, The device includes: The acquisition unit is used to acquire the original image and the target image as described in claim 2; The acquisition unit is used to acquire the dense optical flow between the original image and the target image; A processing unit is used to use the dense optical flow as a label for the original image and the target image; The training unit is used to train the model to be trained based on the original image, the target image and the label to obtain an optical flow model, which is used to estimate dense optical flow.

11. A vehicle, characterized in that, The vehicle includes: A camera is used to continuously capture images of the environment in which the vehicle is located while the vehicle is in motion, to obtain a first image to be processed and a second image to be processed, both of which include a target object. The acquisition unit is used to acquire the optical flow model obtained by the method according to claim 7; The determining unit is used to determine the estimated optical flow between the first image to be processed and the second image to be processed using the optical flow model; The estimation unit is used to estimate the first travel path of the target object based on the estimated optical flow; The planning unit is used to plan a second driving path for the vehicle based on the first driving path.

12. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions; When the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 6, or the method as described in claim 7, or the method as described in claim 8.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions; When the program instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 6, or the processor is caused to perform the method according to claim 7, or the processor is caused to perform the method according to claim 8.

Images