Device and method for improving the process of determining depth maps, relative poses, or semantic segmentation

Abstract

The present invention relates to the fields of advanced driver assistance systems (ADAS), computer vision and machine learning (ML). The present invention provides an ML-based method for training a neural network based on a synthetic image (generated from a real image or a virtual image) to improve the process of determining a depth map, relative pose or semantic segmentation. Therefore, the present invention provides a device (100) for determining a depth map (101), a relative pose (102) or a semantic segmentation (103). The device (100) comprises: a neural network (104) for determining the depth map (101), the relative pose (102) or the semantic segmentation (103) based on an input image (105) during an inference phase; a generator (106) for generating a synthetic image (107) based on a real image (108) or a virtual image (109) and according to a loss function (110) during a training phase; the neural network (104) is trained based on the synthetic image (107); wherein the loss function (110) includes a semantic edge function (111).

Description

Technical Field

[0001] The present invention relates to the fields of advanced driver assistance systems (ADAS), computer vision, and machine learning (ML). The present invention provides an ML-based method for training a neural network based on synthetic images (generated from real or virtual images) to improve the process of determining depth maps, relative poses, or semantic segmentation (e.g., which can be used to train an autopilot for an autonomous vehicle). Furthermore, the present invention relates to domain adaptation, for example, solving computer vision problems by training a neural network on virtual images and testing it on real images. Background Art

[0002] Estimating relative pose, depth map, or semantic segmentation based on sensor input is an important task for robotics, ADAS, or autonomous driving systems. For example, the sensor input for performing the estimation comes from a visual sensor (e.g., an input image captured by a digital camera). The information estimated from the input image (depth map, semantic segmentation, or relative pose) can be used (together with the input image) for further training of the robotics, ADAS, or autonomous driving system. For example, a vehicle's autopilot can be trained based on the input image and the corresponding relative pose, depth map, or semantic segmentation.

[0003] With the rapid development of neural networks, estimation techniques such as stereo matching or self-supervised deep learning methods have made progress in solving computer vision problems. However, these methods require a large number of real, high-quality stereo images or real image sequences as input images for estimation training. Even though real stereo images or real image sequences are easy to generate, it is difficult to create accurate depth maps, relative poses, or semantic segmentation corresponding to these real images (which can be used as ground truth for further training).

[0004] It is possible to create virtual images (not captured by a camera, but computer-generated) with accurate information about depth maps, relative poses, or semantic segmentation. In this way, large amounts of training data can be created (for example, for training robots, ADAS, or autonomous driving systems).

[0005] However, a domain gap can exist between virtual and real images, for example, because the image texture or color intensity of the virtual image is not as good as in the real image. This domain gap also degrades the quality of the estimated depth map, relative pose estimation, or semantic segmentation based on the input image, if the neural network performing the estimation is trained only on virtual images.

[0006] Traditional approaches to this problem use deep neural networks to learn domain-invariant features or representations, or to push the two domain distributions closer together. However, the results of these methods are not very effective. In other words, traditional methods cannot effectively reduce the domain gap between virtual and real images. Summary of the Invention

[0007] In view of the above problems, an object of embodiments of the present invention is to improve domain adaptation between different image domains (eg, between virtual images and real images).

[0008] This object and other objects are achieved by the embodiments of the invention described in the accompanying independent claims. Advantageous implementations of the embodiments of the invention are further defined in the dependent claims.

[0009] A first aspect of the present invention provides a device for determining a depth map, relative pose or semantic segmentation, wherein the device includes: a neural network for determining a depth map, relative pose or semantic segmentation based on an input image in an inference phase; a generator for generating a synthetic image based on a real image or a virtual image and based on a loss function in a training phase, and training the neural network based on the synthetic image, wherein the loss function includes a semantic edge function.

[0010] This ensures that the domain gap between virtual and real images can be effectively reduced. Thus, the reliance on creating real images for training neural networks can be avoided, and virtual images can be used instead, for which labels (e.g., indicating segmentation, pose, or depth) can be automatically created. Specifically, the semantic edge function ensures that the domain gap is effectively reduced.

[0011] Specifically, the input image can be a real image or a virtual image input to the device. Specifically, the real image or the virtual image is input to the device for generating training data for the vehicle autopilot.

[0012] Specifically, the real image is a two-dimensional image (eg, an RGB or chrominance-luminance image) acquired in a real environment using a stereo camera. For example, the real image is acquired based on at least one of the following datasets: KITTI, CITYSCAPES, and AEV.

[0013] Specifically, the virtual image is a two-dimensional image (eg, an RGB image or a chroma-luminance image) or a photo-realistic dataset acquired using a stereo camera in a virtual environment such as Carla.

[0014] Specifically, a synthetic image is a two-dimensional image (such as an RGB image or a chrominance-luminance image) generated by a generative adversarial network (GAN) based on a real image or a virtual image.

[0015] Specifically, a depth map is a 2D image or matrix in which each pixel or element describes the depth of a corresponding 3D point in a scene relative to a camera. Specifically, a scene is a predefined region of interest in the real world captured by a camera.

[0016] Specifically, the x-coordinate difference of a 3D point in the scene projected onto a stereo camera image pair (left, right) is calculated according to the following formula: I_left(x+d)=I_right(x), where d is the disparity.

[0017] Specifically, the disparity map is a 2D image or matrix, in which each pixel or element describes the disparity of the pixel or element.

[0018] Specifically, the relative pose is a 6D vector consisting of 3D position coordinates (eg, x, y, z) and 3 angles for vehicle orientation (eg, yaw, pitch, roll).

[0019] Specifically, semantic segmentation involves classifying each pixel of an input image as representing the type of object (e.g., at least one of a car, vegetation, building, sky, and road).

[0020] Specifically, the semantic edge function considers at least one edge in a real image or a virtual image. Specifically, the edge includes a boundary of an object in the image, wherein the boundary includes a significant contrast change.

[0021] In an implementation manner of the first aspect, the semantic edge function is used to maintain semantic gradient information and / or edge information in the synthesized image.

[0022] This ensures that the domain gap between real and synthesized images is reduced according to the semantic gradient information and / or edge information in the synthesized image.

[0023] In another implementation of the first aspect, the device further includes a first discriminator and a second discriminator, wherein the generator is further configured to provide the synthesized image to the first discriminator or the second discriminator during a training phase to train the neural network.

[0024] This ensures that the texture or color intensity in the synthesized image can be enhanced and the domain gap between the synthesized image and the real image can be reduced.

[0025] Specifically, texture is an area of ​​an image that depicts content with significant variations in color intensity.

[0026] In another implementation of the first aspect, the generator is further configured to train the neural network according to a determination result of the discriminator providing the synthesized image during a training phase.

[0027] This ensures that the texture or color intensity in the synthesized image can be further enhanced and the domain gap between the synthesized image and the real image can be reduced.

[0028] In another implementation of the first aspect, the generator is further configured to randomly provide the synthesized image to the first discriminator or the second discriminator during a training phase.

[0029] This ensures that the texture or color intensity in the synthesized image can be further enhanced and the domain gap between the synthesized image and the real image can be reduced.

[0030] In another implementation of the first aspect, the first discriminator is also used to determine during the training phase that a synthetic image generated by the generator based on a virtual image is a false image, and to determine that the virtual image is an original image; the second discriminator is also used to determine during the training phase that a synthetic image generated by the generator based on a real image is a false image, and to determine that the real image is an original image.

[0031] This ensures that the texture or color intensity in the synthesized image can be further enhanced and the domain gap between the synthesized image and the real image can be reduced.

[0032] In another implementation of the first aspect, the device is further configured to train a neural network based on the synthesized image to determine at least one of the following: a depth map, a relative pose, and a semantic segmentation.

[0033] This ensures that the process of determining at least one of a depth map, relative pose, and semantic segmentation can be improved based on the synthesized image.

[0034] In another implementation of the first aspect, the device is further configured to generate a learnable mask based on the synthesized image during a training phase; and train the neural network based on the learnable mask.

[0035] This ensures that the domain gap can also be reduced based on the learnable mask.

[0036] Specifically, the learnable mask is a region of interest in the synthesized image. Specifically, the learnable mask enables determining whether a pixel of the synthesized image can be used for view reconstruction.

[0037] In another implementation of the first aspect, the learnable mask is a semantic inner mask.

[0038] This ensures that the domain gap can also be reduced based on the semantic intra-level masks.

[0039] Specifically, the semantic inner mask is a region of interest in the synthesized image. Specifically, the semantic inner mask supports determining whether pixels of the synthesized image can be used for view reconstruction. Specifically, the semantic inner mask includes semantic segmentation information. The semantic segmentation information can be used to determine whether pixels of the synthesized image can be used for view reconstruction. Specifically, the semantic inner mask is generated by a neural network based on the semantic segmentation information.

[0040] In another implementation of the first aspect, the device is further configured to train a neural network based on the learnable mask to determine at least one of the following: a depth map, a relative pose, and a semantic segmentation.

[0041] This ensures that the process of determining at least one of the depth map, relative pose, semantic segmentation can be improved based on the learnable mask.

[0042] Specifically, a neural network is trained based on semantic inner layer masks to determine the depth map.

[0043] In another implementation of the first aspect, the device is further configured to determine segmentation information based on the synthesized image during a training phase, and generate a learnable mask based on the segmentation information.

[0044] This ensures that the learnable mask can take segmentation information into account.

[0045] In another implementation of the first aspect, the device is further configured to determine posture information based on the synthesized image during a training phase, and generate a learnable mask based on the posture information.

[0046] This ensures that the learnable mask can take pose information into account.

[0047] In another implementation of the first aspect, the device is further configured to determine an inner mask according to the synthesized image during a training phase, and generate a learnable mask according to the inner mask.

[0048] This ensures that the learnable mask can take inner masks into account.

[0049] Specifically, the inner layer mask is a region of interest in the synthesized image. Specifically, the inner layer mask supports determining whether a pixel of the synthesized image can be used for view reconstruction.

[0050] Specifically, the device is further configured to: during a training phase, if the synthetic image is generated based on the real image, apply self-supervised training to the neural network based on the synthetic image.

[0051] Specifically, the device is further configured to: during a training phase, if the synthetic image is generated based on the virtual image, apply supervised training to the neural network based on the synthetic image and / or the ground truth label corresponding to the synthetic image.

[0052] Specifically, the device uses ground truth labels including depth maps, relative poses, or semantic segmentation.

[0053] Specifically, the device further includes a third discriminator and a fourth discriminator, wherein the device is further configured to train the neural network according to a determination result of the third discriminator and / or according to a determination result of the fourth discriminator.

[0054] Specifically, the apparatus is configured to train a neural network for determining a depth map based on a third discriminator.

[0055] Specifically, the device is used to train a neural network for determining semantic segmentation based on a fourth discriminator.

[0056] A second aspect of the present invention provides a method for determining a depth map, relative pose or semantic segmentation, the method comprising the following steps: in an inference phase, the neural network of the device determines a depth map, relative pose or semantic segmentation based on an input image; in a training phase, the generator of the device generates a synthetic image based on a real image or a virtual image and according to a loss function, and the generator trains the neural network based on the synthetic image, wherein the loss function includes a semantic edge function.

[0057] In an implementation of the second aspect, the semantic edge function maintains semantic gradient information and / or edge information in the synthesized image.

[0058] In another implementation of the second aspect, the method further includes: in a training phase, the generator provides the synthesized image to the first discriminator of the device or the second discriminator of the device for training the neural network.

[0059] In another implementation of the second aspect, the method further includes: in a training phase, the generator trains a neural network based on a determination result of the discriminator that provides the synthesized image.

[0060] In another implementation of the second aspect, the method further includes: during a training phase, the generator randomly provides a synthetic image to the first discriminator or the second discriminator.

[0061] In another implementation of the second aspect, the method further includes: in a training phase, the first discriminator determines that the synthetic image generated by the generator based on the virtual image is a false image, and the first discriminator determines that the virtual image is the original image; in a training phase, the second discriminator determines that the synthetic image generated by the generator based on the real image is a false image, and determines that the real image is the original image.

[0062] In another implementation of the second aspect, the method further includes the device training a neural network based on the synthesized image to determine at least one of the following: a depth map, a relative pose, and a semantic segmentation.

[0063] In another implementation of the second aspect, the method further includes: in a training phase, the device generates a learnable mask based on the synthesized image; and the device trains a neural network based on the learnable mask.

[0064] In another implementation of the second aspect, the learnable mask is a semantic inner mask.

[0065] In another implementation of the second aspect, the method further includes the device training a neural network based on the learnable mask to determine at least one of the following: a depth map, a relative pose, and a semantic segmentation.

[0066] In another implementation of the second aspect, the method further includes: in a training phase, the device determines segmentation information based on the synthesized image, and the device generates a learnable mask based on the segmentation information.

[0067] In another implementation of the second aspect, the method further includes: in a training phase, the device determines posture information based on the synthesized image, and generates a learnable mask based on the posture information.

[0068] In another implementation of the second aspect, the method further includes: in a training phase, the device determines an inner mask according to the synthesized image, and the device generates a learnable mask according to the inner mask.

[0069] The second aspect and its implementations include the same advantages as the first aspect and its corresponding implementations.

[0070] A third aspect of the present invention provides a computer program product comprising instructions, which, when a computer executes the program, causes the computer to perform the steps of the method provided by the second aspect or any implementation thereof.

[0071] The third aspect and its implementations include the same advantages as the second aspect and its corresponding implementations.

[0072] It should be noted that all devices, elements, units and modules described in this application can be implemented by software or hardware elements or any type of combination thereof. All steps performed by the various entities described in this application and the functions described to be performed by the various entities are intended to indicate that the corresponding entities are used to perform the corresponding steps and functions. Although in the description of the following specific embodiments, the specific functions or steps performed by the external entity are not reflected in the description of the specific detailed elements of the entity that performs the specific steps or functions, it should be clear to the technician that these methods and functions can be implemented by corresponding hardware or software elements or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The following description of specific embodiments, in conjunction with the accompanying drawings, illustrates various aspects and implementations of the present invention, wherein:

[0074] Figure 1 A schematic diagram of a device provided by an embodiment of the present invention is shown;

[0075] Figure 2 A schematic diagram of a device provided by an embodiment of the present invention is shown in detail;

[0076] Figure 3 A schematic diagram showing an operation scenario provided by the present invention is shown;

[0077] Figure 4 A schematic diagram showing a depth map provided by the present invention is shown;

[0078] Figure 5 A schematic diagram showing an operation scenario provided by the present invention is shown;

[0079] Figure 6 A schematic diagram showing an operation scenario provided by the present invention is shown;

[0080] Figure 7 A schematic diagram showing an operation scenario provided by the present invention is shown;

[0081] Figure 8 A schematic diagram showing an operation scenario provided by the present invention is shown;

[0082] Figure 9 A schematic diagram of a method provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0083] Figure 1 A schematic diagram of an apparatus 100 according to an embodiment of the present invention is shown. Apparatus 100 is configured to determine a depth map 101, relative pose 102, or semantic segmentation 103 based on an input image 105. Furthermore, the determination process is enhanced using an ML-based approach. To this end, apparatus 100 includes a neural network 104 and a generator 106. Specifically, the determination process is enhanced based on neural network 104, which is trained based on a synthetic image 107.

[0084] Therefore, the device 100 distinguishes between an inference phase and a training phase. In the inference phase, the trained neural network 104 is applied to the input image 105 to determine the depth map 101, the relative pose 102, or the semantic segmentation 103 based on the input image 105. In the training phase, the training data (i.e., the real image 108 or the virtual image 109) is provided to the neural network 104. Therefore, the generator 106 is used to generate the synthetic image 107 based on the real image 108 or the virtual image 109. For the generation of the synthetic image 107, the generator also uses a loss function 110. Specifically, the loss function 110 includes a semantic edge function 111. Once the synthetic image 107 is generated, the generator 106 trains the neural network 104 based on the synthetic image 107.

[0085] Optionally, the semantic edge function 111 may maintain semantic gradient information and / or edge information in the synthesized image 107 .

[0086] In other words, the device 100 can use the virtual image 109 and the ground truth labels (such as depth map, semantic segmentation or relative pose, which are generated specifically for the virtual image 109) to train the neural network 104 and test it on the real image 108, etc.

[0087] From real images 109 and virtual images 108 corresponding to two different domains, a synthetic image 107 corresponding to an intermediate domain can be created. Training a neural network 104 based on the synthetic image 107 of the intermediate domain can produce robust and accurate depth maps 101, relative poses 102, or semantic segmentations 103, independent of domain-specific texture features or image distribution.

[0088] The device 100 may include a processor or processing circuit (not shown) for executing, performing, or initiating the various operations of the device 100 described herein. The processing circuit may include hardware and / or the processing circuit may be controlled by software. The hardware may include analog circuits or digital circuits, or both. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a multi-purpose processor.

[0089] Device 100 may also include memory circuitry that stores one or more instructions that can be executed by a processor or processing circuitry (particularly, executed under the control of software). For example, the memory circuitry may include a non-transitory storage medium that stores executable software code that, when executed by the processor or processing circuitry, causes device 100 to perform various operations.

[0090] In one embodiment, the processing circuit includes one or more processors and non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code that, when executed by the one or more processors, causes the device 100 to perform, proceed, or initiate the operations or methods described herein.

[0091] Figure 2 A schematic diagram of the device 100 provided by an embodiment of the present invention is shown in detail. Figure 2 The device 100 shown includes Figure 1 All features and functions of network device 100, as well as the following optional features.

[0092] like Figure 2 As shown, the device 100 optionally includes a first discriminator 201 and a second discriminator 202. During the training phase, the generator 106 may optionally provide the synthetic image 107 to the first discriminator 201 or the second discriminator 202 to train the neural network 104. The first discriminator 201 and the second discriminator 202 enable the generator 106 to create a synthetic image 107 with a texture similar to the domain using the virtual image 109 and the real image 108.

[0093] More specifically, the generator 106 may randomly provide the synthetic image 107 to the first discriminator 201 or the second discriminator 202. Then, the generator 106 trains the neural network 104 according to the determination results of the discriminators 201, 202 to which the synthetic image 107 is provided.

[0094] In other words, the generator 106 and the first and second discriminators 201 and 202 generate the synthetic image 107 for training the neural network 104 according to the adversarial loss function (i.e., the loss function 110) that mainly considers common features such as semantic edges and semantic information, patterns, scene structures, and artifacts such as textures, color filters, noise, lighting effects, shadows, and reflections generated by the camera sensor.

[0095] In other words, for the loss function 110, for example, semantic gradient or edge-based reconstruction (possibly combined with a robust loss function) can be used instead of using a mean square error (MSE) loss to reconstruct the input image 105. This enables the generator 106 to reconstruct the synthetic image 107 by preserving the scene structure and geometric properties, and to blend or create texture, intensity, or quality between the virtual image 109 and the real image 108.

[0096] Optionally, the loss function 110 is a semantic edge function 111, the purpose of which may be to maintain semantic gradient information and / or semantic edge information to generate the synthetic image 107. In addition, by using the first discriminator 201 and the second discriminator 202, the generator 106 can generate image textures similar to the real image domain without losing geometric structure.

[0097] The synthesized images 107 generated by the generator 106, the first discriminator 201, and the second discriminator 202 can be used to further train the neural network 104, for example, to improve the process of determining the depth map 101, the relative pose 102, or the semantic segmentation 103 based on the synthesized images 107. In particular, this is supported by a corresponding loss function 110 such as an L1 / view reconstruction function, a cross entropy function, or an L1 function.

[0098] In other words, the generator 106 and the discriminators 201 , 202 generate a synthesized image 107 by considering that the semantic edges in the synthesized image 107 should be the same as in the original input (i.e., in the real image 108 or the virtual image 109 ), wherein the synthesized image 107 has a higher texture variation.

[0099] Specifically, randomly providing the synthesized image 107 to the first discriminator 201 or the second discriminator 202 supports the generator 106 to generate the synthesized image 107 with higher texture variation.

[0100] To achieve this effect, during the training phase, the first discriminator 201 determines that the synthesized image 107 generated by the generator 106 based on the virtual image 109 is a false image, and determines that the virtual image 109 is the original image. For the same purpose, during the training phase, the second discriminator 202 determines that the synthesized image 107 generated by the generator 106 based on the real image 108 is a false image, and determines that the real image 108 is the original image.

[0101] like Figure 2 As further shown in FIG, during the training phase, device 100 optionally generates a learnable mask 203 based on synthetic image 107. Learnable mask 203 indicates pixels in synthetic image 107 that are important for further training neural network 104. Therefore, device 100 optionally further trains neural network 104 based on learnable mask 203 and synthetic image 107. Optionally, learnable mask 203 can be used to train neural network 104 to improve the process of determining at least one of the following: depth map 101, relative pose 102, and semantic segmentation 103.

[0102] Optionally, the learnable mask 203 is a semantic inner mask that indicates the region of interest in the synthesized image 107 through semantic segmentation information in the semantic inner mask.

[0103] like Figure 2 As further shown in FIG, during the training phase, device 100 optionally determines segmentation information 204 based on synthetic image 107. Segmentation information 204 enables associating portions of synthetic image 107 with labels, such as indicating that the predefined portion is at least one of the following: street, wall, tree, traffic light, sidewalk, sky, or house. Device 100 then generates a learnable mask 203 based on segmentation information 204 and synthetic image 107.

[0104] like Figure 2As further shown in FIG, during the training phase, the device 100 optionally determines pose information 205 based on the synthetic image 107. For example, the pose information 205 includes position coordinates and angles for vehicle orientation. The device 100 then generates a learnable mask 203 based on the pose information 205 and the synthetic image 107.

[0105] like Figure 2 As further shown in FIG, during the training phase, the device 100 optionally determines an inner mask 206 based on the synthesized image 107. Specifically, the estimated semantic segmentation information is incorporated into the inner mask 206 for training the self-supervised depth estimation model of the neural network 104. The device 100 then generates a learnable mask 203 based on the inner mask 206 and the synthesized image 107.

[0106] Specifically, training the neural network 104 based on the inner layer mask 206 and the segmentation information 204 can improve the process of the device 100 determining the depth map 101 based on the input image 105. The other two discriminators ( Figure 2 (not shown) can support moving the real image domain and the corresponding distribution of estimated depth and semantic segmentation to the virtual domain.

[0107] In other words, the device 100 is able to train the neural network 104 with the additional support of the two discriminators 201 , 202 to move the domain from real to virtual based on the estimated depth map and semantic segmentation results of the synthesized image 107 .

[0108] therefore, Figure 1 and Figure 2 The disclosed features can reduce the domain gap between virtual images 109 and real images 108 (or virtual datasets and real datasets) according to a self-supervised approach to improve the process of determining depth maps 101, semantic segmentation 103, and relative pose estimation 102.

[0109] Figure 3 A schematic diagram of the device 100 provided by an embodiment of the present invention is shown in more detail. Figure 3 The device 100 shown includes Figure 1 and Figure 2 All features and functions of network device 100, as well as the following optional features.

[0110] according to Figure 3, the generator 106 (which may also be referred to as a domain adaptation module or generator block) will now be described in more detail. The generator 106 may be used to train a model (e.g., a neural network 104) to create a new domain image (e.g., a synthesized image 107) having common features based on a virtual image domain and a real image domain. The adversarial loss function (i.e., loss function 110) may be used with a loss function based on image semantic edges and with two discriminator networks (i.e., a first discriminator 201 and a second discriminator 202), one for the virtual domain and the other for the real domain.

[0111] During the learning process, a first discriminator 201 and a second discriminator 202 are used. To reproduce the input image 105, a reconstruction loss is applied to the estimated public domain RGB image of the neural network 104. The input image 105 can be from the virtual domain or the real domain in a random pattern. The output of the generator 106 passes through a semantic gradient or edge-based loss function (i.e., loss function 110) and then through one of the first discriminator 201 and the second discriminator 202. The choice of discriminator is completely random. The reason for selecting only one discriminator is that the generator 106 calculates the result based on the judgment provided by the selected discriminator, which helps to blend the texture information of the real image 108 and the virtual image 109 by safely preserving the scene structure and edges. In addition, the randomness in the selection of the discriminator or the dataset (real or virtual images) helps the neural network 104 avoid entering a local minimum, instead causing the loss to fluctuate as it could reach a global minimum.

[0112] according to Figure 3 , the loss function 110 (which may also be referred to as an image gradient based loss function) is now described in more detail. In order to preserve image gradients, edges or scene structure, a robust loss function 110 based entirely on image semantic edges is provided. Semantic edge based learning supports the generator 106 generating images that maintain semantic gradient or edge information from the start of training while ensuring greater variation in texture on images (i.e., synthetic images 107) generated for training depth estimation (DE) models, semantic segmentation (SS) models and relative pose estimation (RPE) models. Therefore, computer vision application models are trained by using semantic edges as common important features for training models (DE, SS, RPE). The convolutions of the DE model, SS model and RPE model support learning computer vision applications independent of varying textures, shadows, lighting effects, weather conditions or color filtering.

[0113] according to Figure 3The first discriminator 201 and the second discriminator 202 (which may also be part of the domain adaptation module) will now be described in more detail. For the first discriminator 201 and / or the second discriminator 202, a Wasserstein discriminator can be used, which uses a ground-breaking distance to minimize the difference between the distribution of the virtual dataset (i.e., virtual images 109) and the distribution of the real image dataset (i.e., real images 108). Furthermore, a gradient penalty can be applied by at least one of the discriminators 201 and 202 to address the problem of vanishing or exploding gradients. The purpose of the first discriminator 201 can be to learn whether the input to the neural network 104 comes from the virtual dataset. During training based on the first discriminator 201, the virtual images 109 are considered real / authentic, while the output of the generator 106 is always considered fake / fake. The purpose of the second discriminator 202 is similar to that of the first discriminator 201, but operates in the exact opposite manner (i.e., in reverse). The second discriminator 202 always considers the real images 108 to be real / authentic and the output of the generator 106 to be fake / fake.

[0114] When training the neural network 104, the output of the generator 106 is always considered fake / false, and one of the discriminators 201, 202 is randomly selected to judge whether the generated RGB (i.e., the synthetic image 107) is virtual or real. By confusing the generator 106 with the first discriminator 201 and the second discriminator 202, the generator 106 will generate an image with mixed textures and features.

[0115] In a real-world scene, the depth map 101 is a representation of the 3D scene structure projected as a pattern based on camera sensor parameters and semantic information. For processing in the device 100, the semantic information may be purely patterns of the 3D scene, edges, and object shapes.

[0116] like Figure 3 As shown, the device 100 may optionally include a third discriminator 301 and a fourth discriminator 302. The third discriminator 301 may operate on the estimated depth map 101, while the fourth discriminator 302 may operate on the estimated semantic segmentation. The purpose of the third discriminator 301 and the fourth discriminator 302 is to further improve the process of determining the depth map 101 and the semantic segmentation 103 based on the input image 105.

[0117] Figure 4 4 shows a portion of an input image 401 and two portions of depth maps 402, 403 determined from the input image 401 by a device trained only on the virtual image 109. Figure 4As shown, when comparing portion 402 and portion 403, there are some holes and irregularities in the estimated depth maps 402, 403 of the real image (where the device has been trained solely on the virtual dataset). In this case, the third discriminator 301 and the fourth discriminator 302 help improve the estimated depth map or semantic segmentation based on the input texture (by training the device 100 on either the virtual or real dataset). As a result, the generator 106 is forced to generate better synthetic images 107 that have common features of the virtual and real domains.

[0118] Back to Figure 3 , we now describe how to train a neural network 104 based on the output of a generator for segmentation, a depth net, or a pose and mask net (i.e., based on segmentation information 204, depth information, pose information 205, and inner layer masks 206). As a supervised method for creating ground truth for improving the process of determining depth maps 101, relative pose 102, and semantic segmentation 103, self-supervised methods based on geometry and view reconstruction between images can be applied. For example, these images can come from stereo images or sequences, or from images of mapped data.

[0119] Using segmentation information 204 enables the creation of efficient learnable masks 203. Typically, when using a view reconstruction loss function to warp the network input to a stereo image, a sequence image, or a map relative image, outliers such as occlusions, non-overlaps, or edges are intractable areas for the neural network 104. By using semantic edges as the loss function 110 and by further training based on the segmentation information 204, the neural network 104 can be trained to further learn and improve the inner layer mask 206 and the learnable mask 203 to generate a better depth map 101 with the help of the view reconstruction loss.

[0120] according to Figure 5 、 Figure 6 and Figure 7 , various methods of training neural network 104 are now described.

[0121] Figure 5 The use of virtual images 109 (i.e., virtual datasets) to train depth maps 101 and semantic segmentation 103 is shown, for example, while running a self-supervised method ( Figure 5 (not shown in the figure). The main benefit of this approach is the continuous learning process. While the virtual dataset is always available for training, real videos are dynamically available to further improve the training. Therefore, the depth or segmentation model can be enriched with accurate virtual labels. Simultaneously, the determination results of device 100 are improved based on real images 108, which are used as the basis for the learning results of virtual images using third discriminator 301 and fourth discriminator 302.

[0122] Specifically, Figure 5 Active components of the apparatus 100 are shown for training from a virtual image 109. Since the virtual image 109 can be used to generate labels for the depth map 101 and semantic segmentation 103, etc., the virtual image 109 is used to train the neural network 104 via a loss function 110.

[0123] Figure 6 The network architecture of apparatus 100 for training using a sequence of real images 108 is shown. Specifically, the active components for training on real videos (sequence of real images 108) are shown. Since training is based on real videos, a self-supervised approach (based on a view reconstruction loss function) is used to compute depth maps 101 and relative poses 102 (odometry). Furthermore, estimated semantic segmentation 103 is used (forward propagation only) to refine inner layer masks 206 and / or learnable masks 203. Furthermore, only virtual images 109 (dataset) are used to train depth maps 101 and semantic segmentation 102, while the trained depth and segmentation models are forward propagated on real images 108 to adjust the estimated results on real images 108 to be similar to those estimated from virtual images 109. This is supported by using a third discriminator 301 and a fourth discriminator 302 to shift the domain from real to virtual, solely for the estimated depth and segmentation results.

[0124] Figure 7 The network architecture of the apparatus 100 for training a neural network 104 based on a single real image 108 is shown, without any loss function other than the discriminator used to train the model for depth map and semantic segmentation. More specifically, the network architecture is trained based on the real image 108 (dataset) without a depth-related loss function. In the example shown, the depth map on the real image is used to generate better results based on the third discriminator 301 and the fourth discriminator 302. The method shown supports the use of stereo virtual images 109 and stereo real images 108 without any ground truth labels to estimate the depth map 101 by using a self-supervised method. The method shown also supports the use of stereo virtual images 109 and a sequence of real images 108 without any ground truth labels to estimate the depth map 101 by using a self-supervised method.

[0125] Figure 8 The inference phase (also called the testing phase) of the device 100 is shown. In the inference phase shown in the figure, a depth map 101 and a semantic segmentation 103 of a given input image 105 of the real domain are determined. The device 100 also supports determining relative pose 102 ( Figure 8 not shown).

[0126] Figure 9Schematic diagram of method 900 provided by an embodiment of the present invention is shown. Method 900 is used to determine a depth map 101, a relative pose 102, or a semantic segmentation 103. Method 900 includes step 901: During an inference phase, a neural network 104 determines the depth map 101, the relative pose 102, or the semantic segmentation 103 based on an input image 105. Method 900 also includes step 902: During a training phase, a generator 106 generates a synthetic image 107 based on a real image 108 or a virtual image 109 and according to a loss function 110. Method 900 also includes step 903: the generator 106 trains the neural network 104 based on the synthetic image 107; wherein the loss function 110 includes a semantic edge function 111.

[0127] The invention has been described with reference to various exemplary embodiments and implementations. However, other variations will be apparent to and will be realized by those skilled in the art in practicing the claimed invention, based on a study of the drawings, the present disclosure, and the independent claims. In the claims and the specification, the word "comprising" does not exclude other elements or steps, and "a," "an" does not exclude a plurality. A single element or other unit may fulfil the functions of several entities or items recited in the claims. The recitation of certain measures in different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Claims

1. A training device (100), characterized in that The device (100) comprises: - a neural network (104) for determining a depth map (101), a relative pose (102) or a semantic segmentation (103) from an input image (105) during an inference phase; - Generator (106), used during the training phase to: generating a synthetic image (107) based on a real image (108) or a virtual image (109) and according to a loss function (110); Training the neural network (104) based on the synthesized image (107); wherein the loss function (110) includes a semantic edge function (111); The semantic edge function (111) is used to maintain semantic gradient information and / or edge information in the synthesized image (107); The invention also includes a first discriminator (201) and a second discriminator (202), and the generator (106) is further used to provide the synthesized image (107) to the first discriminator (201) or the second discriminator (202) in the training phase to train the neural network (104); The generator (106) is further configured to train the neural network (104) during the training phase based on the determination results of the discriminator (201, 202) providing the synthetic image (107); The generator (106) is further configured to randomly provide the synthesized image (107) to the first discriminator (201) or the second discriminator (202) during the training phase; The first discriminator (201) is further configured to determine, during the training phase, that a synthetic image (107) generated by the generator (106) based on a virtual image (109) is a false image, and to determine that the virtual image (109) is an original image; The second discriminator (202) is further configured to determine during the training phase whether the synthetic image (107) generated by the generator (106) based on the real image (108) is a false image, and to determine whether the real image (108) is an original image.

2. The device (100) according to claim 1, characterized in that It is also used to train the neural network (104) based on the synthetic image (107) to determine at least one of the following: a depth map (101), a relative pose (102), and a semantic segmentation (103).

3. The device (100) according to claim 1 or 2, characterized in that It is also used to generate a learnable mask (203) according to the synthetic image (107) during the training phase; and train the neural network (104) according to the learnable mask (203).

4. The device (100) according to claim 3, characterized in that The learnable mask (203) is a semantic inner mask.

5. The device (100) according to claim 3 or 4, characterized in that It is also used to train the neural network (104) based on the learnable mask (203) to determine at least one of the following: a depth map (101), a relative pose (102), and a semantic segmentation (103).

6. The device (100) according to any one of claims 3 to 5, characterized in that It is also used to determine segmentation information (204) according to the synthetic image (107) during the training phase, and generate the learnable mask (203) according to the segmentation information (204).

7. The device (100) according to any one of claims 3 to 6, characterized in that It is also used to determine posture information (205) according to the synthetic image (107) during the training phase, and generate the learnable mask (203) according to the posture information (205).

8. The device (100) according to any one of claims 6 to 7, characterized in that It is also used to determine an inner mask (206) according to the synthetic image (107) during the training phase, and generate the learnable mask (203) according to the inner mask (206).

9. A training method (900), characterized in that The method (900) comprises the following steps: - In the inference phase, the neural network (104) determines (901) a depth map (101) based on the input image (105), relative pose (102) or semantic segmentation (103); - During the training phase, The generator (106) generates (902) a synthetic image (107) based on the real image (108) or the virtual image (109) and according to the loss function (110); The generator (106) trains (903) the neural network (104) based on the synthetic image (107); During the training phase, the synthetic image (107) is provided to a first discriminator (201) or a second discriminator (202) to train the neural network (104); the first discriminator (201) is used to determine that the synthetic image (107) generated by the generator (106) based on the virtual image (109) is a false image, and that the virtual image (109) is an original image; the second discriminator (202) is used to determine that the synthetic image (107) generated by the generator (106) based on the real image (108) is a false image, and that the real image (108) is an original image; In the training phase, the generator (106) trains the neural network (104) based on the determination results of the first discriminator (201) and / or the second discriminator (202) providing the synthetic image (107); During the training phase, the generator (106) randomly provides the synthesized image (107) to the first discriminator (201) or the second discriminator (202); wherein the loss function (110) includes a semantic edge function (111); The semantic edge function (111) is used to maintain semantic gradient information and / or edge information in the composite image (107).

10. The method (900) according to claim 9, characterized in that Also includes: The neural network (104) is trained based on the composite image (107) to determine at least one of the following: a depth map (101), a relative pose (102), and a semantic segmentation (103).

11. The method (900) according to claim 9 or 10, characterized in that Also includes: In the training phase, a learnable mask (203) is generated based on the synthetic image (107); and the neural network (104) is trained based on the learnable mask (203).

12. The method (900) according to claim 11, characterized in that The learnable mask (203) is a semantic inner mask.

13. The method (900) according to claim 11 or 12, characterized in that Also includes: The neural network (104) is trained based on the learnable mask (203) to determine at least one of the following: a depth map (101), a relative pose (102), and a semantic segmentation (103).

14. The method (900) according to any one of claims 11 to 13, characterized in that Also includes: During the training phase, segmentation information (204) is determined based on the synthetic image (107), and the learnable mask (203) is generated based on the segmentation information (204).

15. The method (900) according to any one of claims 11 to 14, characterized in that Also includes: During the training phase, posture information (205) is determined based on the synthetic image (107), and the learnable mask (203) is generated based on the posture information (205).

16. The method (900) according to any one of claims 11 to 15, characterized in that Also includes: During the training phase, an inner mask (206) is determined based on the synthetic image (107), and the learnable mask (203) is generated based on the inner mask (206).

17. A storage medium, characterized in that: The method comprises instructions which, when called by a computer, cause the computer to execute the method (900) according to any one of claims 9 to 16.

Images