Interpolation method and system, and method and system for determining a map of a vehicle's surrounding environment

Abstract

The present disclosure relates to interpolation methods and systems and methods and systems for determining a map of a vehicle's surrounding environment, and more particularly to a computer-implemented method for interpolation, including the following steps performed by computer hardware components: determining a class of image input data structures that include a plurality of data points; determining a plurality of reference data points as a subset of the plurality of data points of the class of image input data structures; determining a class of image temporary data structures based on the class of image input data structures and a predetermined processing operation; replacing the data points of the class of image temporary data structures corresponding to the plurality of reference data points with the plurality of reference data points to obtain a class of image updated temporary data structures; and determining a class of image output data structures based on the class of image updated temporary data structures.

Description

Technical Field

[0001] The present disclosure relates to interpolation methods and systems and methods and systems for determining a map of a vehicle's surrounding environment. Background Art

[0002] Digital imaging devices (such as digital video cameras) are used in automotive applications for various purposes, such as detecting the surrounding environment of a vehicle (such as roads or other vehicles) or the interior of a vehicle (such as the driver or other passengers of the vehicle).

[0003] Images of different resolutions may be required for different tasks.

[0004] Therefore, there is a need to convert an image from one resolution to another. Summary of the Invention

[0005] The present disclosure provides a computer-implemented method, a computer system, and a non-transitory computer-readable medium. Embodiments are given in the description and the drawings.

[0006] In one aspect, the present invention relates to a computer-implemented method for interpolation, the method including the following steps performed (in other words: run) by computer hardware components: determining a class image input data structure that includes a plurality of data points; determining a plurality of reference data points as a subset of the plurality of data points of the class image input data structure; determining a class image temporary data structure based on the class image input data structure and a predetermined processing operation; replacing data points of the class image temporary data structure corresponding to the plurality of reference data points with the plurality of reference data points to obtain a class image updated temporary data structure; and determining a class image output data structure based on the class image updated temporary data structure.

[0007] Interpolation can be understood as determining pixels between reference pixels. Interpolation can be used to increase or decrease the (spatial) resolution of an image or to determine pixels between reference pixels. The (spatial) resolution of the class image output data structure can be higher than / lower than / equal to the (spatial) resolution of the class image input data structure. It should be understood that the (spatial) resolution can refer to the number of data points.

[0008] According to another aspect, the class image input data structure represents a matrix including a plurality of data points. Similarly, the class image temporary data structure and / or the class image output data structure can represent a matrix including a plurality of data points. The matrix can be a class grid data structure.

[0009] According to another aspect, the class image input data structure includes at least one of an image or a map. The image can be a visual image, and the data points can correspond to (or can be) pixels. The map can be a height map or a depth map, and the data points can correspond to (or can be) entries in the map.

[0010] According to another aspect, the class image input data structure can be sparse. Similarly, the class image temporary data structure and / or the class image output data structure can be sparse.

[0011] Sparse can mean that data is explicitly stored for data points having values different from 0 (which can also include data points having a value of 0), and all other data points (i.e., data points for which their values are not explicitly stored) are (implicitly) given as 0.

[0012] According to another aspect, reference data points are data points explicitly provided in the sparse representation.

[0013] Using the data points explicitly provided in the sparse representation (i.e., data points not implicitly given as 0) can ensure that the data points having the greatest impact on the reference data are used for interpolation.

[0014] According to another aspect, as a repeated process in multiple iterations, the computer hardware component performs the following steps: determining the class image temporary data structure for the current iteration based on the class image update temporary data structure and a predetermined processing operation of the previous iteration; and replacing the data points in the class image temporary data structure of the current iteration corresponding to the multiple reference data points with the multiple reference data points to obtain the class image update temporary data structure for the current iteration.

[0015] Replacing the data points in the class image temporary data structure of the current iteration corresponding to the multiple reference data points with the multiple reference data points can provide that the class image update temporary data structure for the current iteration is the same as the class image input data structure at least at the multiple reference data points.

[0016] According to another aspect, when a stop criterion is met, the repeated process is stopped, and the latest class image temporary data structure is determined as the class image output data structure.

[0017] According to another aspect, the stop criterion is based on at least one of the time available for determining the class image output data structure, the number of iterations in the repeated process, or the accuracy of the class image output data structure. The accuracy can be proportional to the number of loop iterations. The stop criterion as a time limit can be set to the maximum value for which the overall processing time (i.e., the time for evaluating the CNN plus the time for necessary projections plus the time for interpolation) can still be considered real-time. An alternative approach can be to measure the differences in subsequent class image output data structures and stop once the defined threshold is not met (in other words: stop once the class image output data structure does not change significantly from one iteration to the next).

[0018] According to another aspect, the predetermined processing operation includes at least one of average pooling (e.g., 2D (two-dimensional) average pooling), max pooling (e.g., 2D max pooling), min pooling (e.g., 2D min pooling), or convolution (e.g., 2D convolution).

[0019] It has been found that using one or more of these operations as the predetermined processing operation enhances the result.

[0020] In another aspect, the present disclosure relates to a computer-implemented method for determining a map of a vehicle's surrounding environment, the method comprising the following steps performed by computer hardware components: determining distance-related data; determining a class image input data structure based on the distance-related data; determining interpolation data based on a class image output data structure determined by an interpolation method as described herein; and determining a map of the vehicle's surrounding environment based on the interpolation data.

[0021] The distance-related data may be lidar data (e.g., lidar point cloud) or radar data, or any other type of information indicating distance data or from which distance information can be obtained.

[0022] It has been found that when used to determine the map, using the interpolation method described herein provides good results.

[0023] According to another aspect, the class image input data structure is determined based on a camera plane projection of the distance-related data. In this way, the distance-related data and the camera data can be made consistent.

[0024] According to another aspect, the computer-implemented method further comprises the following steps performed by computer hardware components: determining intermediate data based on an artificial neural network and the interpolation data; wherein, a map of the vehicle's surrounding environment is determined based on the intermediate data.

[0025] The intermediate data may include or may be a 2D camera probability map; and may further include a BEV (bird's-eye view) projection.

[0026] According to another aspect, the computer-implemented method may further comprise the following steps performed by computer hardware components: acquiring an image related to the vehicle's surrounding environment; wherein, the intermediate data is further determined based on the image. The image may be an RGB image.

[0027] In another aspect, the present disclosure relates to a computer system comprising a plurality of computer hardware components configured to perform several or all of the steps of the computer-implemented method described herein.

[0028] The computer system may include a plurality of computer hardware components (e.g., a processor (e.g., a processing unit or a processing network), at least one memory (e.g., a memory unit or a memory network), and at least one non-transitory data storage device). It should be understood that additional computer hardware components may be provided and used to perform the steps of the computer-implemented method in the computer system. The non-transitory data storage device and / or the memory unit may include a computer program for instructing a computer, e.g., using the processing unit and at least one memory unit, to perform several or all of the steps or aspects of the computer-implemented method described herein.

[0029] In another aspect, the present invention relates to a vehicle including the computer system.

[0030] In another aspect, the present disclosure relates to a non-transitory computer-readable medium including instructions for performing several or all of the steps or aspects of the computer-implemented method described herein. The computer-readable medium may be configured as: an optical medium, such as a compact disc (CD) or a digital versatile disc (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM), such as a flash memory; etc. Additionally, the computer-readable medium may be configured as a data storage device accessible via a data connection such as an Internet connection. The computer-readable medium may be, for example, an online data repository or cloud storage.

[0031] The present disclosure also relates to a computer program for instructing a computer to perform several or all of the steps or aspects of the computer-implemented method described herein.

[0032] By using the methods and systems according to various embodiments, an efficient system architecture including an efficient interpolation module can be provided, which can be transformed into an embedded real-time system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Exemplary embodiments and functions of the present disclosure are described herein in connection with the following schematically illustrated drawings:

[0034] Figure 1 is a neural network-based camera-lidar fusion system according to various embodiments.

[0035] Figure 2 is a flowchart illustrating an interpolation method according to various embodiments;

[0036] Figure 3 is a graph determination system according to various embodiments; and

[0037] Figure 4is a computer system having multiple computer hardware components configured to perform steps of a computer-implemented method for interpolation according to various embodiments, or configured to perform steps of a computer-implemented method for determining a map of a vehicle's surrounding environment according to various embodiments.

[0038] List of Reference Numerals

[0039] 100 Neural network-based camera-lidar fusion system according to various embodiments

[0040] 102 Camera data

[0041] 104 Convolutional layer

[0042] 106 Convolutional layer with optional dropout

[0043] 108 Lidar point cloud data

[0044] 110 Camera plane projection module

[0045] 112 Height map

[0046] 114 Interpolation module

[0047] 116 Depth map

[0048] 118 Interpolation module

[0049] 120 Convolutional layer

[0050] 122 Convolutional layer

[0051] 124 Convolutional layer with optional dropout

[0052] 126 Convolutional layer

[0053] 128 Probability map

[0054] 130 BEV projection module

[0055] 132 2D BEV probability map

[0056] 200 Flowchart illustrating an interpolation method according to various embodiments

[0057] 202 Step of determining a class of image input data structure including a plurality of data points

[0058] 204 Step of determining a plurality of reference data points as a subset of the plurality of data points of the class of image input data structure

[0059] 206 Step of determining a class of image temporary data structure based on the class of image input data structure and a predetermined processing operation

[0060] Step 208 of replacing data points corresponding to multiple reference data points in a class image temporary data structure with the multiple reference data points to obtain a class image updated temporary data structure; Step 210 of determining a class image output data structure based on the class image updated temporary data structure; 300 Diagram determination system

[0061] 302 Distance determination circuit

[0062] 304 Class image input data structure determination circuit

[0063] 306 Interpolation circuit

[0064] 310 Diagram determination circuit

[0065] 312 Electrical connection

[0066] 400 Computer system according to various embodiments

[0067] 402 Processor

[0068] 404 Memory

[0069] 406 Non - transitory data storage device

[0070] 408 Camera

[0071] 410 Distance sensor

[0072] 412 Connection Detailed description of the embodiments

[0073] The free space area (also referred to as the drivable area) can be a part of the road that is not occupied by any object and can be reached by the vehicle. To achieve a higher level of vehicle automation, detecting the free space area is an important activity that needs to be accurately and robustly addressed. Knowing which areas of the road surface are available for driving is actually a key prerequisite for performing safe trajectory planning and decision - making.

[0074] According to various embodiments, an efficient NN (neural network) - based low - level camera - lidar fusion system for drivable area determination is provided, which can include various components, and will be described in more detail herein.

[0075] Figure 1 A neural network - based camera - lidar fusion system 100 according to various embodiments is shown.

[0076] The system 100 may include (or may be) a fully convolutional neural network (CNN) having the following layers: an encoder (e.g., convolutional layers 104 and 122), a feature space (e.g., convolutional layers 106 and 124 with optional dropout), a decoder (e.g., convolutional layer 126, e.g., for late multi-point fusion), a block 110 for camera plane projection (e.g., using calibration data), and modules 114 and 118 for 2D interpolation.

[0077] LiDAR point cloud data 108 may be acquired and then projected onto the camera plane and restricted to the camera view boundaries (e.g., using the camera plane projection module 110). Based on the projected data, two separate images may be created: a height map 112 and a depth map 116. Both maps may be sparse and may thus be converted to a dense form by using 2D interpolation methods according to various embodiments, such as those described in Table 1. For example, a 2D interpolation 114 of the height map 112 may be obtained, and a 2D interpolation 118 of the depth map 118 may be obtained.

[0078] The 2D interpolation 114 of the height map 112 and the 2D interpolation 118 of the depth map 118 may then be fed into the CNN, where the first processing layer (which may be convolutional layer 120) may be responsible for their combination (or fusion) by way of concatenation and 2D convolution (e.g., providing early fusion).

[0079] The CNN may have an additional input through which camera data 102 (e.g., raw camera RGB (red-green-blue) image data) may be provided to the CNN.

[0080] Two branches of the CNN (the LiDAR branch and the camera branch) may have convolutional layers to encode the images (e.g., using 2D convolution and 2D max pooling) and output a large number of feature maps in the next 2D convolution (where the feature maps may be understood to be provided in the feature space). During CNN training, the dropout layer may be active in the feature space (in other words: the dropout layer associated with convolutional layers 106, 124 may be active). For example, the LiDAR branch of the neural network may include convolutional layer 122 which may form an encoder, and may also include convolutional layer 124 which may provide feature maps. For example, the camera branch of the neural network may include convolutional layer 104 which may form an encoder, and may also include convolutional layer 106 which may provide feature maps.

[0081] The CNN can end with a decoder part, where the encoder and the feature space are fused by concatenation and 2D convolution (e.g., late fusion is provided by convolutional layer 126). The network can produce a 2D probability map 128 that can be projected (e.g., using the BEV projection module 130) into a bird's-eye view, which can be the system output and can be a 2D BEV probability map 132.

[0082] The fully convolutional neural network according to various embodiments can be optimized to match real-time processing. The 2-D interpolation according to various embodiments has low complexity and it can be fully based on GPU-based operations. In addition, the interpolation method (or interpolation module) can be provided in a parallelized manner, such that the processing can be scaled to match the client requirements of real-time processing of new high-resolution cameras and lidars.

[0083] The lidar branches (122, 124) and camera branches (104, 106) of the CNN can be trained separately and can be regarded as independent systems. Then these branches can be combined and fine-tuned together. In this method, the good performance of the individual lidar and camera systems can guarantee the robustness of the entire fusion system.

[0084] In an alternative method, the lidar branches (122, 124) and camera branches (104, 106) of the CNN can be trained in a common training.

[0085] For example, the method can provide an efficient neural network-based low-level camera-lidar fusion system for drivable area determination.

[0086] A pseudocode representation of the interpolation method is provided in Table 1 below.

[0087] Input:

[0088] Image - a class matrix structure containing the original valid image pixels

[0089] ImageMask - a binary mask identifying the positions of valid image pixels

[0090] Output:

[0091] ResultImage - a class matrix structure containing the original valid pixels and interpolated image pixels

[0092] 1. Keep the original image values as OriginalImage

[0093] 2. Do not exceed the time limit:

[0094] a. Create a temporary image TmpImage by applying the following operations: 2D-AveragePooling, 2D-MaxPooling, 2D-MinPooling, 2D-Convolution (one or a combination of them)

[0095] b. Overwrite selected TmpImage pixels with valid pixel values from the original image - using ImageMask and OriginalImage

[0096] 3. Set ResultImage to TmpImage

[0097] Table 1: GPU (Graphics Processing Unit)-based 2D interpolation of sparse irregular images according to various embodiments.

[0098] Figure 2 Flowchart 200 illustrating an interpolation method according to various embodiments is shown. At 202, a class image input data structure can be determined, where the class image input data structure includes a plurality of data points. At 204, a plurality of reference data points can be determined as a subset of the plurality of data points of the class image input data structure. At 206, a class image temporary data structure can be determined based on the class image input data structure and a predetermined processing operation. At 208, the data points of the class image temporary data structure corresponding to the plurality of reference data points can be replaced with the plurality of reference data points to obtain a class image updated temporary data structure. At 210, a class image output data structure can be determined based on the class image updated temporary data structure.

[0099] According to various embodiments, the class image input data structure can represent a matrix including a plurality of data points. According to various embodiments, the class image input data structure includes at least one of an image or a graph. According to various embodiments, the class image input data structure can be sparse.

[0100] According to various embodiments, the reference data points can be data points explicitly provided in the sparse representation.

[0101] According to various embodiments, as a repeated process in multiple iterations, a computer hardware component performs the following steps: determining a class image temporary data structure of the current iteration based on the class image updated temporary data structure of the previous iteration and a predetermined processing operation; and replacing the data points of the class image temporary data structure of the current iteration corresponding to the plurality of reference data points with the plurality of reference data points to obtain a class image updated temporary data structure of the current iteration.

[0102] According to various embodiments, when a stop criterion is met, the repeated process can be stopped, and the latest class image temporary data structure can be determined as the class image output data structure.

[0103] According to various embodiments, the stop criterion may be based on at least one of the time available to determine the class image output data structure, the number of iterations in the repeated processing, or the accuracy of the class image output data structure.

[0104] According to various embodiments, the predetermined processing operation may include or may be at least one of average pooling, max pooling, min pooling, or convolution.

[0105] According to various embodiments, a method for determining a map of a vehicle's surrounding environment may be provided. The method may include: determining distance-related data; determining a class image input data structure based on the distance-related data; determining interpolation data based on the class image output data structure determined based on the above (interpolation) method; and determining a map of the vehicle's surrounding environment based on the interpolation data.

[0106] According to various embodiments, the class image input data structure may be determined based on the camera plane projection of the distance-related data.

[0107] According to various embodiments, intermediate data may be determined based on an artificial neural network and the interpolation data; wherein a map of the vehicle's surrounding environment is determined based on the intermediate data.

[0108] According to various embodiments, an image related to the vehicle's surrounding environment may be acquired, and the intermediate data may be further determined based on the image.

[0109] Each of steps 202, 204, 206, 208, 210 and the other steps described above may be performed by computer hardware components.

[0110] The interpolation as described above may be applied to process lidar point clouds in the camera plane.

[0111] However, it should be understood that the interpolation method may be used for other applications, such as any application that requires irregular grid / image interpolation. The interpolation method may reduce the computation time.

[0112] Figure 3 A map determination system 300 according to various embodiments is shown. The map determination system 300 may include a distance determination circuit 302, a class image input data structure determination circuit 304, an interpolation circuit 306, and a map determination circuit 310.

[0113] The distance determination circuit 302 may be configured to determine distance-related data.

[0114] The class image input data structure determination circuit 304 may be configured to determine a class image input data structure based on the distance-related data.

[0115] The interpolation circuit 306 can be configured to determine interpolation data based on the image-like output data structure determined based on the above interpolation method.

[0116] The map determination circuit 310 can be configured to determine a map around the vehicle based on the interpolation data.

[0117] The distance determination circuit 302, the image-like input data structure determination circuit 304, the interpolation circuit 306, and the map determination circuit 310 can be coupled to each other, for example, via an electrical connection 312 such as a cable or a computer bus or via any other suitable electrical connection to exchange electrical signals.

[0118] A "circuit" can be understood as any type of logical implementation entity, which can be a dedicated circuit or a processor that executes a program stored in a memory, firmware, or any combination thereof.

[0119] Figure 4 A computer system 400 is shown having a plurality of computer hardware components configured to perform steps of a computer-implemented method for interpolation according to various embodiments or configured to perform steps of a computer-implemented method for determining a map of the environment around a vehicle according to various embodiments. The computer system 400 can include a processor 402, a memory 404, and a non-transitory data storage device 406. A camera 408 and / or a distance sensor 410 (such as a radar sensor and / or a lidar sensor) can be provided as part of the computer system 400 (as Figure 4 shown), or can be provided external to the computer system 400.

[0120] The processor 402 can execute instructions provided in the memory 404. The non-transitory data storage device 406 can store a computer program including instructions that can be transferred to the memory 404 and then executed by the processor 402. The camera 408 and / or the distance sensor 410 can be used to determine distance-related data.

[0121] The processor 402, the memory 404, and the non-transitory data storage device 406 can be coupled to each other, for example, via an electrical connection 412 (such as a cable or a computer bus) or via any other suitable electrical connection to exchange electrical signals. The camera 408 and / or the distance sensor 410 can be coupled to the computer system 400, for example, via an external interface, or can be provided as part of the computer system (in other words: inside the computer system, for example, via the electrical connection 412).

[0122] The terms "coupled" or "connected" are intended to include directly "coupled" (e.g., via a physical link) or directly "connected" as well as indirectly "coupled" or indirectly "connected" (e.g., via a logical link), respectively.

[0123] It should be understood that what is described for one of the above methods can be similarly applied to the graph determination system 300 and / or the computer system 400.

Claims

1. A computer-implemented method for interpolation, the computer-implemented method comprising the following steps performed by computer hardware components: - determining (202) a class of image input data structures, the class of image input data structures including a plurality of data points; - determining (204) a plurality of reference data points as a subset of the plurality of data points of the class of image input data structures; - determining (206) a class of image temporary data structures based on the class of image input data structures and a predetermined processing operation, wherein, both the class of image input data structures and the class of image temporary data structures represent matrices including a plurality of data points, and wherein the predetermined processing operation includes at least one of average pooling, max pooling, min pooling, or convolution; - replacing (208) the data points of the class of image temporary data structures corresponding to the plurality of reference data points with the plurality of reference data points to obtain a class of image updated temporary data structures; and - determining (210) a class of image output data structures based on the class of image updated temporary data structures.

2. The computer-implemented method according to claim 1, Among them, the class of image input data structures including at least one of an image or a graph.

3. The computer-implemented method according to claim 1, Among them, the class of image input data structures being sparse; wherein the reference data points are data points explicitly provided in the sparse representation.

4. The computer-implemented method according to claim 1, Among them, as a repeated process in multiple iterations, the following steps are performed by the computer hardware components: - determining the class of image temporary data structures of the current iteration based on the class of image updated temporary data structures of the previous iteration and the predetermined processing operation; - replacing the data points of the class of image temporary data structures of the current iteration corresponding to the plurality of reference data points with the plurality of reference data points to obtain the class of image updated temporary data structures of the current iteration.

5. The computer-implemented method according to claim 4, Among them, stopping the repeated process when a stop criterion is met and determining the latest class of image temporary data structures as the class of image output data structures.

6. The computer-implemented method according to claim 5, Among them, the stop criterion being based on at least one of the time available for determining the class of image output data structures, the number of iterations in the repeated process, or the accuracy of the class of image output data structures.

7. A computer-implemented method for determining a graph of a vehicle's surrounding environment, the computer-implemented method comprising the following steps performed by computer hardware components: determining distance-related data; determining a class of image input data structures based on the distance-related data; determining interpolation data based on a class of image output data structures, the class of image output data structures being determined based on the method according to any one of claims 1 to 6; and determining a graph of the vehicle's surrounding environment based on the interpolation data.

8. The computer-implemented method according to claim 7, Among them, the class of image input data structures being determined based on a camera plane projection of the distance-related data.

9. The computer-implemented method according to claim 7 or 8, the computer-implemented method further comprising the following steps performed by the computer hardware components: Determining intermediate data based on an artificial neural network and based on the interpolation data; Among them, A map of the surrounding environment of the vehicle is determined based on the intermediate data.

10. The computer-implemented method according to claim 9, the computer-implemented method further comprising the following steps performed by the computer hardware components: Obtaining an image related to the surrounding environment of the vehicle; Among them, The intermediate data is further determined based on the image.

11. A computer system (400), the computer system comprising a plurality of computer hardware components, the plurality of computer hardware components being configured to perform the steps of the computer-implemented method according to any one of claims 1 to 10.

12. A vehicle comprising the computer system (400) according to claim 11.

13. A non-transitory computer-readable medium, the non-transitory computer-readable medium comprising instructions for performing the computer-implemented method according to any one of claims 1 to 10.

Images