Reconstruction of depth information and other additional parameters from two-dimensional images

Abstract

Method (100) for reconstructing a location-dependent scalar additional quantity (5) of a scene (1) from a pixel-divided image (2) of this scene (1) with the steps: • Pixel values ​​(2a) of the image (2) are fed into a neural network (3) (110); • the pixel values ​​(2a) are processed by the neural network (3) to local difference information (4) (120), which each indicate how the location-dependent scalar additional size (5) of the scene (1) changes at the position specified by the respective pixel; • From the local difference information (4) this scalar additional information (5) is determined for further locations specified by pixels of the image (2) (130).

Description

[0001] The present invention relates to the enrichment of two-dimensional images with depth information and other additional variables that relate to the depicted scenery and are important, for example, for planning the behavior of vehicles or robots. State of the art

[0002] Autonomously moving agents, such as vehicles or robots, rely on continuous monitoring of their environment. A key source of information for this is camera images. Two-dimensional cameras are typically used because it is quite complex and expensive to cover the entire environment with stereoscopic or other three-dimensional cameras.

[0003] Therefore, two-dimensional images are enhanced by subsequently reconstructing the depth information using neural networks. (Y. Ming et al., "Deep learning for monocular depth estimation: A review," Neurocomputing 438, 14-33, doi: 10.1016 / j.neucom.2020.12.089 (2021)) provides an overview of common methods in this regard. Disclosure of the invention

[0004] The invention provides a method for reconstructing a location-dependent scalar additional variable of a scene from a pixel-divided image of that scene. Within this method, pixel values ​​of the image are fed to a neural network. This neural network processes the pixel values ​​into local difference information, each of which indicates how the location-dependent scalar additional variable of the scene changes at the position specified by the respective pixel. This scalar additional information is determined from the local difference information for other locations specified by pixels of the image.

[0005] In this case, the additional scalar quantity can, for example, contain depth information, which is a measure of the distance between a part of the scene and a camera used to capture the image. The depth information can therefore, for example, indicate the distance between matter-filled areas of the scene at the positions specified by the pixels, on the one hand, and the camera or a plane in which the camera is located, on the other. However, other additional scalar quantities are also conceivable, such as a density or other material property of matter present in the scene.

[0006] It was recognized that determining local difference information of the additional scalar quantity is significantly less error-prone than determining absolute values ​​of the additional scalar quantity. In particular, the missing individual values ​​are tolerated much better when determining local difference information than when determining absolute values. At the same time, for many applications, relative information of the additional scalar quantity, such as depth information, is completely sufficient. If, for example, a robot or other production machine is to grip components that are to be assembled during production, then the information about which areas of the scene shown in the recorded image are deeper than others is sufficient. The end result, namely the reliability with which the robot orWhen the production machine picks up the components, it is better to calculate only the information that is actually needed with good accuracy than to additionally calculate information that is not absolutely necessary and thereby sacrifice overall accuracy. Furthermore, especially with distant objects, such as in traffic situations, it is significantly more difficult to measure a distance absolutely than to measure relative changes in that distance.

[0007] In a particularly advantageous embodiment, the neural network determines local difference information based on a pixel, which contains the changes in the location-dependent scalar additional quantity between the position specified by the pixel, on the one hand, and the positions specified by various neighboring pixels, on the other. This then reveals in which direction in the image the scalar additional information changes and to what extent. For example, if a pixel identified by coordinates (i, j) has four neighbors, each of which is shifted by an amount of one pixel in one of the coordinates (i, j), there are four pieces of local difference information for the pixel (i, j). For example, the local difference information can be local depth differences LDD(i, j), which are related to the absolute depths D(i, j). For the four aforementioned neighborhood relationships, four local depth differences result for each pixel: LDD1,0(i,j)=D(i,j)−D(i+1,j), LDD0,1(i,j)=D(i,j)−D(i,j+1). LDD−1,0(i,j)=D(i,j)−D(i−1,j), LDD0,−1(i,j)=D(i,j)−D(i,j−1). This shows in particular in which direction the depth D changes most. For further refinement, those neighbors that are shifted by an amount of one pixel in both coordinates (i, j) can optionally be added.

[0008] Alternatively or in combination with this, differences between more distant pixels can also be included, such as LDD 0,2 (i, j).

[0009] In another particularly advantageous embodiment, at least one reference value of the location-dependent scalar additional information for at least one position in the scene is additionally included in the determination of the scalar additional information for additional locations. In this way, the determined relative scalar additional information can be converted to an absolute scale. If multiple reference values ​​are used, the accuracy of each of these individual reference values ​​is less critical.

[0010] In a further particularly advantageous embodiment, a cost function is used to evaluate the extent to which several differences of the scalar additional variable between two given locations in the scene, aggregated in different ways from several local difference information, are consistent with each other. Parameters that characterize the behavior of the neural network are optimized with the goal of improving the evaluation by the cost function. In this way, the neural network can be trained in a self-supervised manner, i.e., without the use of prior knowledge (“ground truth”). In particular, it can be required, for example, that the differences in all possible paths Φ=(p1,(dp1Φ,…,dpmΦ)) with Φ0 = p1 and Φ m = p2 between a first location p1 and a second location p2 aggregated differences D(p2) - D(p1) of the scalar additional quantity are equal to: D(p2)−D(p1)=∑i=1mLDDdpiΦ(Φi) for all paths Φ. The set of determined local difference information is then called "path-consistent." This is analogous, for example, to energy conservation in conservative force fields, in which the same work is always performed on every path between two given locations p1 and p2.

[0011] In particular, there may be applications where no training examples labeled with "ground truth" regarding the scalar additional variable are available. In these cases, only the self-supervised training mentioned above is available. However, this self-supervised training can also be combined with any configuration of "ground truth" supervised training. For this purpose, for example, a cost function from the supervised training and a cost function from the self-supervised training can be aggregated into an overall cost function, for example by adding them together.

[0012] In a further advantageous embodiment, the neural network is additionally designed to determine absolute values ​​of the location-dependent additional scalar variable for the positions specified by the pixels. In this way, the determination of the absolute values ​​on the one hand and the determination of local difference information (and from this, if applicable, relative values ​​of the additional scalar variable) on the other hand can support each other. In particular, for example, the determined absolute values ​​on the one hand and the determined local difference information on the other hand can be checked against each other. From the result of the plausibility check, a measure of the reliability of the absolute values ​​and / or the local difference information can then be determined. The information about this reliability is particularly important for the control of vehicles, robots or production machines. For example, the vehicle, the robot orThe more reliably depth information about a scene has been determined from captured images of that scene, the faster the production machine can move through it.

[0013] In another particularly advantageous embodiment, the neural network outputs a quotient of the local difference information and the value of the location-dependent additional variable at the position specified by the respective pixel. In the above example of depth information, this quotient, which is dependent on the location p, is given, for example, by NLDD(p)=LDD(p)D(p).

[0014] The advantage of normalizing to the additional scalar quantity, here D(p), is that the quotient is dimensionless, meaning it no longer contains a physical unit of measurement. It indicates the change in depth, for example, as a function of that depth. This is particularly important for many applications. For example, the rate of change of the distance to an object relative to the absolute distance is a measure of how imminent a collision with that object is.

[0015] In another particularly advantageous embodiment, the image augmented by the determined additional scalar variable is fed to a classifier network that assigns classification scores to the scenery and / or at least one object contained in the scenery with respect to one or more classes of a predefined classification. In this way, ambiguities that would remain when using only the image can be resolved. For example, it is possible to distinguish whether the scenery merely contains a photo or other image of an object or whether this object is actually physically present in the scenery.

[0016] In a further particularly advantageous embodiment, the image enriched by the determined additional scalar variable is included in the planning of the movement of a vehicle, a robot, and / or a production machine designed to pick up and assemble components.

[0017] The vehicle, robot, or production machine is controlled according to this planning. This increases the probability that the response of the vehicle, robot, or production machine to the control command is appropriate to the scene embodied by the image and the additional scalar information derived from it. This improves the overall performance of the vehicle, robot, or production machine.

[0018] The invention also provides a method for training a neural network for use in the method described above.

[0019] In this process, training images of at least one scene are provided. For better generalization, the training images can, for example, refer to many different scenes.

[0020] Pixel values ​​from the training images are fed into the neural network being trained. These pixel values ​​are then processed by the neural network to produce local difference information, which indicates how the location-dependent scalar additional variable of the scene changes at the position specified by the respective pixel.

[0021] Using a given cost function, the extent to which the determined local difference information is consistent with the specified target information regarding the location-dependent scalar additional variable in the scene is evaluated. This can be measured using any metric, depending on the type of target information available. Thus, not only is a direct comparison of the local difference information with the target information possible, but also, for example, an examination of the extent to which the determined local difference information contradicts the target information.

[0022] Parameters that characterize the behavior of the neural network are optimized with the goal of improving the evaluation by the cost function.

[0023] In a particularly advantageous embodiment, local target difference information is determined from specified target absolute values ​​for the location-dependent scalar additional variable. The local difference information determined by the neural network to be trained is compared with the local target difference information, resulting in a local deviation. The cost function depends on this local deviation. The target absolute values ​​can be determined particularly easily by measurement, for example, by scanning the scene using a lidar or other sensor technology.

[0024] In a further advantageous embodiment, absolute values ​​of the location-dependent scalar additional variable for the positions specified by the pixels are determined by the neural network and / or from the local difference information provided by the neural network. The absolute values ​​determined by the neural network to be trained and / or from the local difference information provided by the neural network are compared with specified target absolute values, resulting in an absolute deviation. The cost function also depends on this absolute deviation. In this way, the determination of local difference information by the neural network to be trained can be used as a regularization for the determination of the absolute values ​​by this same neural network.

[0025] For this purpose, new local target difference information can be determined, for example, from the same target absolute values ​​used for training with regard to determining absolute values, as described above. Training for path consistency can also be carried out, for example, in a self-supervised manner, as described above.

[0026] The methods can, in particular, be fully or partially computer-implemented. Therefore, the invention also relates to a computer program with machine-readable instructions that, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instances to execute one of the described methods. In this sense, control units for vehicles and embedded systems for technical devices that are also capable of executing machine-readable instructions are also to be regarded as computers. Compute instances can, for example, be virtual machines, containers, or serverless execution environments, which can be provided in a cloud, in particular.

[0027] The invention also relates to a machine-readable data carrier and / or a downloadable product containing the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e., downloaded by a user of the data network, and which can be offered for immediate download, for example, in an online shop.

[0028] Furthermore, one or more computers and / or compute instances may be equipped with the computer program, the machine-readable data carrier or the download product.

[0029] Further measures improving the invention are presented in more detail below together with the description of the preferred embodiments of the invention with reference to figures. Examples of implementation

[0030] It shows: Fig. 1 embodiment of the method 100 for reconstructing a location-dependent scalar additional quantity 5 from an image 2 of a scene 1; Fig. 2 Embodiment of the method 200 for training a neural network 3; Fig. 3 Two different ways of using given target absolute values ​​5a* as “ground truth” for training according to method 200.

[0031] Fig. 1 is a schematic flow diagram of an embodiment of the method 100 for reconstructing a location-dependent scalar additional quantity 5 from an image 2 of a scene 1.

[0032] In step 110, pixel values ​​2a of the image 2 are fed to a neural network 3.

[0033] In step 120, the pixel values ​​2a are processed by the neural network 3 to form local difference information 4. This local difference information 4 indicates how the location-dependent scalar additional variable 5 of the scene 1 changes at the position specified by the respective pixel.

[0034] According to block 121, the scalar additional quantity 5 may in particular contain, for example, depth information which is a measure of a distance between a part of the scenery 1 and a camera used to capture the image 2.

[0035] According to block 122, the neural network 3 can determine local difference information 4 based on an on-pixel, which contains the changes in the location-dependent scalar additional variable 5 between the position specified by the on-pixel on the one hand and the positions specified by various neighboring pixels on the other. This means that several pieces of local difference information 4 for different directions are then available for each on-pixel.

[0036] In step 130, the scalar additional information 5 for further locations specified by pixels of the image 2 is determined from the local difference information 4.

[0037] According to block 131, at least one reference information 5b of the location-dependent scalar additional information 5 for at least one position in the scenery 1 can be included in the determination of the scalar additional information 5 for further locations.

[0038] According to block 132, the neural network 3 can additionally be configured to determine absolute values ​​5a of the location-dependent scalar additional variable 5 for the positions specified by the pixels. In particular, for example, according to block 132a, the absolute values ​​5a determined by the neural network 3, on the one hand, and the determined local difference information 4, on the other hand, can then be checked against each other for plausibility. According to block 132b, a measure 4#, 5a# for the reliability of the absolute values ​​5a and / or the local difference information 4 can then be determined from the result of the plausibility check.

[0039] According to block 133, the neural network 3 can output a quotient of the local difference information 4 and the value of the location-dependent additional quantity 5 at the position specified by the respective pixel.

[0040] In the Fig. In the example shown in Figure 1, in step 140, a cost function 6 is used to evaluate the extent to which several differences of the scalar additional variable 5, aggregated from several pieces of local difference information 4 along different paths, between two given locations in the scene 1 are consistent with each other. As previously explained, the determined local difference information 4 is then called "path-consistent." This results in an evaluation 6a.

[0041] Subsequently, in step 150, parameters 3a, which characterize the behavior of neural network 3, are optimized to the goal of improving the evaluation 6a by the cost function 6. This optimization can be terminated upon reaching any termination criterion. The fully optimized state of parameters 3a is denoted by the reference symbol 3a* and also defines the fully trained state 3* of neural network 3.

[0042] In step 160, the image 2 enriched by the determined scalar additional variable 5 can be fed to a classifier network 7. This classifier network 7 assigns classification scores 7a to the scene 1 and / or at least one object contained in the scene 1 with respect to one or more classes of a predetermined classification.

[0043] In step 170, image 2 augmented by the determined scalar additional variable 5 can be incorporated into the planning 170a of the movement of a vehicle 50, a robot 60, and / or a production machine 70 configured to pick up and assemble components. Then, in step 180, the vehicle 50, the robot 60, or the production machine 70 can be controlled according to this planning 170a.

[0044] Fig. 2 is a schematic flow diagram of an embodiment of the method 200 for training a neural network 3.

[0045] In step 210, training images 2* of at least one scene 1 are provided.

[0046] In step 220, pixel values ​​2a of these training images 2* are fed to the neural network 3 to be trained.

[0047] In step 230, the pixel values ​​2a are processed by the neural network 3 to form local difference information 4. This local difference information 4 indicates how the location-dependent scalar additional variable 5 of the scene 1 changes at the position specified by the respective pixel.

[0048] In step 240, a given cost function 6 is used to evaluate the extent to which the determined local difference information 4 is consistent with the given target information 5* regarding the location-dependent scalar additional variable 5* in the scenery 1. This results in an evaluation 6a.

[0049] According to block 241, local target difference information 4* can be determined from specified target absolute values ​​5a* for the location-dependent scalar additional variable 5. According to block 241, the determined local difference information 4 can then be compared with the local target difference information 4*, resulting in a local deviation 4Δ. The cost function 6 can then depend on this local deviation 4Δ according to block 243.

[0050] According to block 244, absolute values ​​5a of the location-dependent scalar additional variable 5 for the positions specified by the pixels can be determined by the neural network 3 and / or from the local difference information 4 provided by the neural network 3. According to block 245, the absolute values ​​5a determined by the neural network 3 and / or from the local difference information 4 provided by the neural network 3 can then be compared 245 with predetermined target absolute values ​​5a*, resulting in an absolute deviation 5Δ. The cost function 6 can then also depend on this absolute deviation 5Δ according to block 246.

[0051] In step 250, parameters 3a, which characterize the behavior of neural network 3, are optimized to the goal of improving the evaluation 6a by cost function 6. This optimization can be terminated upon reaching any termination criterion. The fully optimized state of parameters 3a is denoted by reference symbol 3a* and also defines the fully trained state 3* of neural network 3.

[0052] Fig. 3 shows two different ways in which predetermined target absolute values ​​5a* can be used as “ground truth” for training according to method 200.

[0053] According to Fig. 3a, training images 2* are processed by the neural network 3, on the one hand, into local difference information 4 and, on the other hand, into absolute values ​​5a of the desired location-dependent additional scalar variable 5, here: depth information. Target absolute values ​​5a* of the additional scalar variable 5 are available as "ground truth" as target information 5*. These target absolute values ​​5a* can be used, on the one hand, for direct comparison with the absolute values ​​5a determined by the neural network, so that the resulting absolute deviation 5Δ can be included in the cost function 6 used in step 240 of the method 200. On the other hand, local target difference information 4* can be determined from the target absolute values ​​5a* in a deterministic manner, for example, using a two-dimensional convolutional layer.This local target difference information 4* can then be compared with the local difference information 4 determined by the neural network 3, so that the resulting local deviation 4Δ can also be included in the cost function 6. In addition, the path consistency of the local difference information 4 can be checked according to step 140 of the method 100.

[0054] In the Fig. 3b shown example determined in contrast to Fig. 3a, the neural network 3 does not directly calculate the absolute values ​​5a of the scalar additional quantity 5. Instead, these absolute values ​​5a are evaluated using at least one reference value 5b of the location-dependent scalar additional information 5 from the local difference information 4 provided by the neural network 3. Otherwise, the target absolute values ​​5a* are calculated in a completely analogous manner to Fig.3a is used to evaluate both directly the ultimately determined absolute values ​​5a and indirectly via local target difference information 4* the local difference information 4 supplied by the neural network 3. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Y. Ming et al., “Deep learning for monocular depth estimation: A review”, Neurocomputing 438, 14-33, doi: 10.1016 / j.neucom.2020.12.089 (2021

[0003]

Claims

[1] Method (100) for reconstructing a location-dependent scalar additional quantity (5) of a scene (1) from an image (2) of this scene (1) divided into pixels, comprising the steps: • Pixel values ​​(2a) of the image (2) are fed (110) to a neural network (3); • the pixel values ​​(2a) are processed (120) with the neural network (3) to produce local difference information (4) which each indicates how the location-dependent scalar additional variable (5) of the scenery (1) changes at the position specified by the respective pixel; • from the local difference information (4) this scalar additional information (5) is determined (130) for further locations specified by pixels of the image (2). [2] Method (100) according to claim 1, wherein the scalar additional quantity (5) includes depth information (121) which is a measure of a distance between a part of the scenery (1) and a camera used to capture the image (2). [3] Method (100) according to one of claims 1 to 2, wherein the neural network (3) determines (122) local difference information (4) starting from an on-pixel, which includes the changes in the location-dependent scalar additional variable (5) between the position specified by the on-pixel on the one hand and the positions specified by various neighboring pixels on the other hand. [4] Method (100) according to one of claims 1 to 3, wherein in the determination of the scalar additional information (5) for further locations, at least one reference indication (5b) of the location-dependent scalar additional information (5) for at least one position in the scenery (1) is additionally included (131). [5] Method (100) according to one of claims 1 to 4, wherein • a cost function (6) is used to evaluate (140) the extent to which several differences of the scalar additional quantity (5) between two given locations in the scenery (1), aggregated in different ways from several local difference information (4), are consistent with each other; and • Parameters (3a) that characterize the behavior of the neural network (3) are optimized (150) to the goal that the evaluation (6a) is improved by the cost function (6). [6] Method (100) according to one of claims 1 to 5, wherein the neural network (3) is additionally designed (132) to determine absolute values ​​(5a) of the location-dependent scalar additional quantity (5) for the positions specified by the pixels. [7] Method (100) according to claim 6, wherein • the determined absolute values ​​(5a) on the one hand and the determined local difference information (4) on the other hand are checked against each other for plausibility (132a) and • a measure (4#, 5a#) for the reliability of the absolute values ​​(5a) and / or the local difference information (4) is determined from the result of the plausibility check (132b). [8] Method (100) according to one of claims 1 to 7, wherein the neural network (3) outputs (133) a quotient of the local difference information (4) and the value of the location-dependent additional variable (5) at the position specified by the respective pixel. [9] Method (100) according to one of claims 1 to 8, wherein the image (2) enriched by the determined scalar additional variable (5) is fed (160) to a classifier network (7) which assigns classification scores (7a) to the scenery (1) and / or at least one object contained in the scenery (1) with respect to one or more classes of a predetermined classification. [10] Method (100) according to one of claims 1 to 9, wherein • the image (2) enriched by the determined scalar additional variable (5) is included (170) in the planning (170a) of the movement of a vehicle (50), a robot (60), and / or a production machine (70) designed to pick up and assemble components, and • the vehicle (50), the robot (60) or the production machine (70) is controlled (180) in accordance with this planning (170a). [11] Method (200) for training a neural network (3) for use in the method according to one of claims 1 to 10, comprising the steps: • training images (2*) of at least one scene (1) are provided (210); • Pixel values ​​(2a) of these training images (2*) are fed (220) to the neural network (3) to be trained; • the pixel values ​​(2a) are processed (230) by the neural network (3) to produce local difference information (4) which each indicates how the location-dependent scalar additional variable (5) of the scenery (1) changes at the position specified by the respective pixel; • using a given cost function (6), it is evaluated (240) to what extent the determined local difference information (4) is consistent with the given target information (5*) regarding the location-dependent scalar additional quantity (5*) in the scenery (1); and • Parameters (3a) that characterize the behavior of the neural network (3) are optimized (250) to the goal that the evaluation (6a) is improved by the cost function (6). [12] Method (200) according to claim 11, wherein • local target difference information (4*) is determined from specified target absolute values ​​(5a*) for the location-dependent scalar additional quantity (5) (241); • the determined local difference information (4) is compared (242) with the local target difference information (4*), so that a local deviation (4Δ) is generated; and • the cost function (6) depends on this local deviation (4Δ) (243). [13] Method (200) according to one of claims 11 to 12, wherein • Absolute values ​​(5a) of the location-dependent scalar additional quantity (5) for the positions specified by the pixels are determined (244) by the neural network (3) and / or from the local difference information (4) supplied by the neural network (3); • the absolute values ​​(5a) determined by the neural network (3) and / or from the local difference information (4) supplied by the neural network (3) are compared (245) with predetermined target absolute values ​​(5a*), so that an absolute deviation (5Δ) arises; and • the cost function (6) also depends on this absolute deviation (5Δ) (246). [14] A computer program comprising machine-readable instructions which, when executed on one or more computers and / or compute instances, cause the computer(s) and / or compute instances to perform a method (100, 200) according to any one of claims 1 to 13. [15] A machine-readable data carrier containing the computer program according to claim 14. [16] One or more computers and / or compute instances equipped with the computer program according to claim 14 and / or with the machine-readable data carrier according to claim 15.

Images