METHODS AND SYSTEMS FOR DETERMINING THE DISTANCE OF AN OBJECT

Description

FIELD

[0001] The present disclosure relates to methods and systems for determining a distance of an object.BACKGROUND

[0002] Being aware of distances to the objects around the vehicle is an important aspect for (at least partially) autonomously driving vehicles.

[0003] Accordingly, there is a need to provide efficient and reliable methods and systems for determining distances to the objects around the vehicle.

[0004] Ding, M.; Zhang, Z.; Jiang, X.; Cao, Y., "Vision-Based Distance Measurement in Advanced Driving Assistance Systems". Appl. Sci. 2020, 10, 7276. https: / / doi.org / 10.3390 / app10207276, discloses a deep-learning-based application for the distance measurement between the subject vehicle and the target vehicle or pedestrian, which uses the forward-looking image captured by a vehiclemounted vision sensor to achieve depth map estimation and distance measurement.

[0005] Ramin Nabati et al., "Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles", 12th Workshop on Planning, Perception and Navigation for Intelligent Vehicles, IROS 2020, https: / / doi.org / 10.48550 / arXiv.2009.08428, discloses a radar-camera sensor fusion framework for object detection and distance estimation in autonomous driving scenarios. The proposed architecture uses a middle-fusion approach to fuse the radar point clouds and RGB images. The radar object proposal network uses radar point clouds to generate 3D proposals from a set of 3D prior boxes. These proposals are mapped to the image and fed into a Radar Proposal Refinement (RPR) network for objectness score prediction and box refinement. The RPR network utilizes both radar information and image feature maps to generate accurate object proposals and distance estimations. The radar-based proposals are combined with image-based proposals generated by a modified Region Proposal Network (RPN). The RPN has a distance regression layer for estimating distance for every generated proposal. The radar-based and image-based proposals are merged and used in the next stage for object classification.SUMMARY

[0006] The present disclosure provides a computer implemented method, a computer system and a non-transitory computer readable medium according to the independent claims. Embodiments are given in the subclaims, the description and the drawings.

[0007] In one aspect, the present disclosure is directed at a computer implemented method for determining a distance of an object, the method comprising the following steps performed (in other words: carried out) by computer hardware components: determining an image containing (in other words: showing) the object; determining a class of the object based on the image; determining a coarse estimation (in other words: coarse estimate) of the distance based on a distance sensor; and determining the distance of the object based on the coarse estimation and based on the class of the object.

[0008] The distance may be a scalar distance, or a vector with more than one components (for example a two-dimensional or three-dimensional vector, so that the distance may be used for determining a location in a two-dimensional or three-dimensional space relative to the ego vehicle on which the sensor is provided).

[0009] Determining the class may be understood as classifying the object based on the image into one of a plurality of classes.

[0010] For example, a technique for constructing a distance-oriented objects description using ML (machine learning) model training and re-projection of objects of interest imaged in 2D camera view onto the BEV (bird's eye view) coordinate system with distance compensation corrections may be provided.

[0011] A real time object detection system which may be applied in a self-driving car may be provided. This system may detect many object classes at once and may provide their localization with respect to distance between themselves and the ego vehicle. Description of an environment with positional knowledge of reprojected plane based detected objects may be provided.

[0012] According to another aspect, the class of the object is selected from a plurality of traffic participant types. The traffic participant types may be selected so that all objects of a specific type have a similar size.

[0013] According to another aspect, the plurality of traffic participant types comprise pedestrian, and / or bicycle, and / or car, and / or truck.

[0014] According to another aspect, the class of the object is selected from a plurality of orientations.

[0015] According to another aspect, the plurality of orientations comprises left, and / or upper-left, and / or upper, and / or upper right, and / or right, and / or bottom-right, and / or bottom, and / or bottom-left.

[0016] For example, a method for correction vectors construction from training classes assignment may be provided. The object classes may be semantically related, and / or size (for example width and height) related, and / or orientation specific (for example with granularity: bottom, bottom-left, left, upper-left, up, upper-right, right, bottom-right).

[0017] According to another aspect, the class may be determined using a machine learning method, for example an artificial neural network. The training method of the machine learning method (for example artificial neural network) may allow accurate reprojection of detected classes.

[0018] According to another aspect, the artificial neural network may be a SSD (single shot detector) type or a YOLO (you only look once) type. It will be understood that YOLO method is one of SSD type methods. The method for training of SSD type (or YOLO type) neural network may allow accurate reprojection of image detections to get BEV positional knowledge between objects. Machine learning models of SSD type (or YOLO type) may be trained on images only. It has been found that labelling of image 2D bounding boxes is fast, and that open training sets for image object detections are available.

[0019] According to another aspect, the computer implemented method may further comprise the following step carried out by the computer hardware components: determining a bounding box of the object in the image. The bounding box of the object may be determined based on the image.

[0020] According to another aspect, the coarse estimation may be determined further based on the bounding box.

[0021] According to another aspect, the coarse estimation is determined based on matching a measurement of the distance sensor with the bounding box.

[0022] The distance is determined based on a hash table for the class of the object.

[0023] The hash table comprises a correction for determining the distance based on the coarse estimation.

[0024] In another aspect, the present disclosure is directed at a computer system, said computer system comprising a plurality of computer hardware components configured to carry out several or all steps of the computer implemented method described herein. The computer system can be part of a vehicle.

[0025] The computer system may comprise a plurality of computer hardware components (for example a processor, for example processing unit or processing network, at least one memory, for example memory unit or memory network, and at least one non-transitory data storage). It will be understood that further computer hardware components may be provided and used for carrying out steps of the computer implemented method in the computer system. The non-transitory data storage and / or the memory unit may comprise a computer program for instructing the computer to perform several or all steps or aspects of the computer implemented method described herein, for example using the processing unit and the at least one memory unit.

[0026] In another aspect, the present disclosure is directed at a vehicle comprising the computer system as described herein, a camera configured to acquire the image, and the distance sensor.

[0027] In another aspect, the present disclosure is directed at a non-transitory computer readable medium comprising instructions for carrying out several or all steps or aspects of the computer implemented method described herein. The computer readable medium may be configured as: an optical medium, such as a compact disc (CD) or a digital versatile disk (DVD); a magnetic medium, such as a hard disk drive (HDD); a solid state drive (SSD); a read only memory (ROM), such as a flash memory; or the like. Furthermore, the computer readable medium may be configured as a data storage that is accessible via a data connection, such as an internet connection. The computer readable medium may, for example, be an online data repository or a cloud storage.

[0028] The present disclosure is also directed at a computer program for instructing a computer to perform several or all steps or aspects of the computer implemented method described herein.DRAWINGS

[0029] Exemplary embodiments and functions of the present disclosure are described herein in conjunction with the following drawings, showing schematically: Fig. 1an illustration of an image and corresponding bounding boxes; Fig. 2an illustration of distances determined by a distance sensor in the scenario of Fig. 1; Fig. 3an illustration of a coarse reprojection image, in which an object is illustrated; Fig. 4a flow diagram illustrating a method for determining a distance of an object according to various embodiments; Fig. 5a distance determination system according to various embodiments; and Fig. 6a computer system with a plurality of computer hardware components configured to carry out steps of a computer implemented method for determining a distance of an object according to various embodiments. DETAILED DESCRIPTION

[0030] It may be desirable for a self-driving (in other words: autonomous) car to be able to determine the distance to other objects and, after finding their placement, to determine a set of possible actions for the self-driving car. It will be understood that different objects have different meaning or significance, and while some object classes may be ignored, localization of others may be needed to determine the best and most secure way to drive for the self-driving car.

[0031] Machine learning method may allow to accurately find the localization of objects of interest in a form of 2D (two-dimensional) bounding boxes on a camera image plane. Determining 2D bounding boxes may include: 1) determining coordinates (in the image) of bounding box surrounding an object; 2) determining a type of the object (for example car, truck, pedestrian, construction vehicle, etc.); and 3) determining a certainty score (or a probability) of a detection.

[0032] Determining the 2D bounding boxes may be very efficient. 2D bounding box determination may work in real time on embedded devices, and may have very good accuracy distinguishing many classes. It will be understood that the number of classes that are distinguished may depend on the training data. For example, up to 80 classes may be modelled.

[0033] However, plane camera representation as such may give no information on how far from the ego vehicle (on which the camera which acquires the image is provided) actually the objects of interest are and what their 3D (three-dimensional) positions are with respect to actual distance among them. According to various embodiments, interchangeable and reliable systems returning such information in real time and in a very accurate way are provided.

[0034] According to various embodiments, quick and accurate construction of a representation allowing identification of places with relation to the ego vehicle of all objects of interests that were found by detection algorithm working on images may be provided. The architecture according to various embodiments may be applicable to various sensor setups.

[0035] According to various embodiments, distance-oriented objects description may be constructed by training of a machine learning method (for example an artificial neural network, for example of a SSD type), together with an accurate method for reprojection (in other words: determining 3D coordinates based on the 2D coordinates which are projected on the image sensor by the camera) using 'sensor with depth' (for example radar, lidar, 3D camera, time of flight camera) onto Bird's Eye View (BEV) coordinate system of all objects of interest found in 2D plane camera view. Furthermore, according to various embodiments, a further correction may be provided by compensation of distances for objects, which may be implied by choosing an appropriate training strategy which relies on specifying objects positions (heading direction) and sizes (width, length) encapsulating them into training classes types, making the classes more granular.

[0036] According to various embodiments, a coarse reprojection method may be provided, which approximates positioning of objects in the environment. Furthermore, a method to perform the training of ML (machine learning) method that is beneficial and allows to improve the overall accuracy of positions of reprojected objects may be provided.

[0037] To perform the reprojection itself, a sensor with depth or depth indicated from sensor implicitly (as mono / stereo camera depth) may be used. For example, the sensor may be a sensor lidar or radar sensor. The method for reprojecting using sensor with depth may make use of the inverted operations to those used when projecting, i.e., projecting 3D point cloud onto camera image. Moreover, a method to make an estimation of image point's depth may be provided since most likely looking at a specific point in image plane, it does not associate directly with any of projected points from sensor with depth (for example because that point is missed because of sparsity of the sensor points).

[0038] To have reasonably accurate view of the scene and to be able to identify the location in BEV, i.e., giving the distance from ego to all objects of interest, it may be assumed that there is at least one 'sensor with depth' point projected onto camera view in proximity of every object's 2D bounding box detection on the image sample, where proximity may be understood as at least one projected sensor's point in each bounding box. For instance, if it is required to detect further objects, a sensor may be provided which is able to cast points as far and as dense as needed so there are close projected points to our objects of interest. Thus, a reasonably reflected environment including all objects of interest may be provided.

[0039] According to various embodiments, reprojection may start with finding each 2D bounding box center. Then, having the central points, for each central point, the k (with an integer number of k) nearest projected 'sensor with depth' points may be determined with an efficient method using k-d trees method. Then, the average of their depths may be taken as an estimated depth for the bounding box center. Using this depth, the center may be reprojected onto 3D using inverse operations to these used while projecting 'sensor with depth' points onto camera plane. Then, the center points (and / or the bounding boxes) may be projected onto Bird's Eye View.

[0040] However, after these steps only, the obtained localization may be coarse. It may identify that in BEV the objects of interest occupy reprojected points but doesn't give an accurate representation of their positions.

[0041] From the information available with standard training of 2D bounding box detection neural networks, the type of an object may and its probability of existence may be determined (for example using the machine learning method of SSD type). According to various embodiments, enhancing this information may be used to make the positioning (in other words: the determination of the position) more robust. This may be provided by specific way of training of the detection neural network (for example the specific way of how the training of classes are provided).

[0042] According to various embodiments, a specific division of objects into training classes (such as pedestrian, sedan car, bicycle, ...) may be provided, where it may be able to indicate fixed size for all objects in specific class. By specifying a class, it may be assumed that its elements are of similar sizes, so that each class contains elements with at least substantially the same or similar sizes (for example width and / or length).

[0043] According to various embodiments, the neural network (for example of a SSD type) may be trained, subdividing the main classes onto the classes with similar width and height, as truck_type_a, truck_type_b, ....

[0044] According to various embodiments, the orientation of objects may be determined. For the reprojection setup, discrete representation of orientation (for example: left, upper-left, upper, upper-right, right, bottom-right, bottom, bottom-left) may be used. However, it will be understood that a more granular or a coarser representation may be used. The orientation may take a form of further subclass in the neural network. Then, for example, the class of an object may be "sedan_bottom_right" (i.e. a type of the object of class "sedan", and an orientation of the object of "bottom right").

[0045] Having the orientation and the type of an object, compensation may be provided to the reprojection of detection center from plane camera onto BEV. A default size value may be used for each class; for example, most sedan cars have similar sizes (looking from BEV we are interested only in their width and height). Then, we a default orientation may be applied to the BEV bounding box with indication of its front.

[0046] According to various embodiments, the compensation for the reprojected center point may specified in a hash-table where for each class (as truck_type_a_front_left), and a BEV correction vector may be provided. Using these vectors, while reprojecting each class, the compensation may be made. It will be understood that the orientation with known front and back of an object which is obtained by the learning strategy according to various embodiments may be used for other purposes.

[0047] According to various embodiments, each identified 2D bounding box may be reprojected, achieving accurate description of the environment in BEV with positional knowledge of detected objects. The architecture according to various embodiments may be used for different sensor setups. In place of various kinds of lidar sensors resulting in different densities of point clouds, for providing depth, depth predicted from mono or stereo camera, or depth coming from a radar reading may be provided.

[0048] Fig. 1 shows an illustration 100 of an image and corresponding bounding boxes (for example bounding box 102). Each bounding box may be related to a class of the object determined in the bounding box, and a confidence value (or probability value) indicating how reliable the determined information (bounding box including class is).

[0049] Fig. 2 shows an illustration 200 of distances determined by a distance sensor in the scenario of Fig. 1.

[0050] Fig. 3 shows an illustration 300 of a coarse reprojection image, in which an object 302 is illustrated. The localization (bounding box and its direction) of an object may further be corrected applying a correction vector and heading information implied by identified class type.

[0051] Fig. 4 shows a flow diagram 400 illustrating a method for determining a distance of an object according to various embodiments. At 402, an image containing the object may be determined. At 404, a class of the object may be determined based on the image. At 406, a coarse estimation of the distance may be determined based on a distance sensor. At 408, the distance of the object may be determined based on the coarse estimation and based on the class of the object.

[0052] According to various embodiments, the class of the object may be selected from a plurality of traffic participant types.

[0053] According to various embodiments, the plurality of traffic participant types may include pedestrian, and / or bicycle, and / or car, and / or truck.

[0054] According to various embodiments, the class of the object may be selected from a plurality of orientations.

[0055] According to various embodiments, the plurality of orientations may include left, and / or upper-left, and / or upper, and / or upper right, and / or right, and / or bottom-right, and / or bottom, and / or bottom-left.

[0056] According to various embodiments, the class may be determined using an artificial neural network.

[0057] According to various embodiments, the artificial neural network may be of a SSD type.

[0058] According to various embodiments, the method may further include determining a bounding box of the object in the image.

[0059] According to various embodiments, the coarse estimation may be determined further based on the bounding box.

[0060] According to various embodiments, the coarse estimation may be determined based on matching a measurement of the distance sensor with the bounding box.

[0061] The distance is determined based on a hash table for the class of the object.

[0062] The hash table comprises a correction for determining the distance based on the coarse estimation.

[0063] Each of the steps 402, 404, 406, 408 and the further steps described above may be performed by computer hardware components.

[0064] Fig. 5 shows a distance determination system 500 according to various embodiments. The distance determination system 500 may include an image determination circuit 502, a class determination circuit 504, a coarse estimation determination circuit 506, and a distance determination circuit 508.

[0065] The image determination circuit 502 may be configured to determine an image containing the object.

[0066] The class determination circuit 504 may be configured to determine a class of the object based on the image.

[0067] The coarse estimation determination circuit 506 may be configured to determine a coarse estimation of the distance based on a distance sensor.

[0068] The distance determination circuit 508 may be configured to determine the distance of the object based on the coarse estimation and based on the class of the object.

[0069] The image determination circuit 502, the class determination circuit 504, the coarse estimation determination circuit 506, and the distance determination circuit 508 may be coupled with each other, e.g. via an electrical connection 510, such as e.g. a cable or a computer bus or via any other suitable electrical connection to exchange electrical signals.

[0070] A "circuit" may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing a program stored in a memory, firmware, or any combination thereof.

[0071] Fig. 6 shows a computer system 600 with a plurality of computer hardware components configured to carry out steps of a computer implemented method for determining a distance of an object according to various embodiments. The computer system 600 may include a processor 602, a memory 604, and a non-transitory data storage 606. A camera 608 and a distance sensor 610 may be provided as part of the computer system 600 (like illustrated in Fig. 6), or may be provided external to the computer system 600.

[0072] The processor 602 may carry out instructions provided in the memory 604. The non-transitory data storage 606 may store a computer program, including the instructions that may be transferred to the memory 604 and then executed by the processor 602. The camera 608 may be used for determining the image containing the object. The distance sensor 610 may be used to determine the coarse estimation.

[0073] The processor 602, the memory 604, and the non-transitory data storage 606 may be coupled with each other, e.g. via an electrical connection 612, such as e.g. a cable or a computer bus or via any other suitable electrical connection to exchange electrical signals. The camera 608 and / or the distance sensor 610 may be coupled to the computer system 600, for example via an external interface, or may be provided as parts of the computer system (in other words: internal to the computer system, for example coupled via the electrical connection 612).

[0074] The terms "coupling" or "connection" are intended to include a direct "coupling" (for example via a physical link) or direct "connection" as well as an indirect "coupling" or indirect "connection" (for example via a logical link), respectively.

[0075] It will be understood that what has been described for one of the methods above may analogously hold true for the distance determination system 500 and / or for the computer system 600.Reference numeral list

[0076] 100illustration of an image and corresponding bounding boxes 102bounding box 200illustration of distances determined by a distance sensor in the scenario of Fig. 1 300illustration of a coarse reprojection image, in which an object is illustrated 302object 400flow diagram illustrating a method for determining a distance of an object according to various embodiments 402step of determining an image containing the object; 404step of determining a class of the object based on the image 406step of determining a coarse estimation of the distance based on a distance sensor 408step of determining the distance of the object based on the coarse estimation and based on the class of the object 500distance determination system 502image determination circuit 504class determination circuit 506coarse estimation determination circuit 508distance determination circuit 510connection 600computer system according to various embodiments 602processor 604memory 606non-transitory data storage 608camera 610distance sensor 612connection

Claims

1. Computer implemented method for determining a distance of an object, the method comprising the following steps carried out by computer hardware components: - determining (402) an image containing the object; - determining (404) a class of the object based on the image; - determining (406) a coarse estimation of the distance based on a distance sensor; and - determining (408) the distance of the object based on the coarse estimation and based on the class of the object; wherein the distance is determined based on a hash table for the class of the object; characterized in that the hash table comprises a correction for determining the distance based on the coarse estimation.

2. The computer implemented method of claim 1, wherein the class of the object is selected from a plurality of traffic participant types.

3. The computer implemented method of claim 2, wherein the plurality of traffic participant types comprise pedestrian, and / or bicycle, and / or car, and / or truck.

4. The computer implemented method of at least one of claims 1 to 3, wherein the class of the object is selected from a plurality of orientations.

5. The computer implemented method of claim 4, wherein the plurality of orientations comprises left, and / or upper-left, and / or upper, and / or upper right, and / or right, and / or bottom-right, and / or bottom, and / or bottom-left.

6. The computer implemented method of at least one of claims 1 to 5, wherein the class is determined using an artificial neural network.

7. The computer implemented method of claim 6, wherein the artificial neural network is of a single shot detector, SSD, type.

8. The computer implemented method of at least one of claims 1 to 7, further comprising the following step carried out by the computer hardware components: determining a bounding box of the object in the image.

9. The computer implemented method of claim 8, wherein the coarse estimation is determined further based on the bounding box.

10. The computer implemented method of claim 9, wherein the coarse estimation is determined based on matching a measurement of the distance sensor with the bounding box.

11. Computer system (600), the computer system (600) comprising a plurality of computer hardware components configured to carry out steps of the computer implemented method of at least one of claims 1 to 10.

12. Vehicle comprising: the computer system (600) of claim 11; a camera (608) configured to acquire the image; and the distance sensor (610).

13. Non-transitory computer readable medium comprising instructions for carrying out the computer implemented method of at least one of claims 1 to 10.