Camera-based distance measurement

Abstract

Systems, methods, and computer-readable media for distance measurement instructions can include obtaining an image of an environment surrounding a vehicle from a camera of the vehicle; searching for an anchor within the image, wherein the anchor is associated with at least one physical dimension of a known value; and when the anchor is found, determining a distance between the camera and the anchor based on (a) the at least one physical dimension of a known value, (b) an appearance of the at least one physical dimension of a known value in the image, and (c) a distance-appearance relationship that maps appearances to distances, wherein the distance-appearance relationship is generated by a calibration process that includes obtaining one or more calibration images of the anchor, and obtaining one or more distance measurements from the anchor.

Description

Technical Field

[0001] This invention relates to the field of distance measurement, and more specifically, to a method for distance measurement, a non-transitory computer-readable medium, and a computerized system. Background Technology

[0002] The three main types of sensors for autonomous vehicles (AVs) are cameras, radar, and lidar.

[0003] These three sensors work together to provide the AV with information about its surroundings and help the AV detect the speed and distance of nearby objects, as well as their three-dimensional shapes.

[0004] The lidar-based system provides real-time visualization of the environment surrounding the AV based on the distance of each lidar point. The computer of the lidar-based system converts the lidar points into a three-dimensional (3D) representation and can identify other vehicles, people, roads, buildings, etc., as a means to enable the AV to navigate safely in its surrounding environment.

[0005] The cost of the lidar-based system is very high—several orders of magnitude higher than that of a camera—making it unsuitable for installation in most vehicles.

[0006] There is a growing need to provide a system that is cost-effective and capable of generating deep information. Summary of the Invention

[0007] Systems, methods, and computer-readable media as shown in the specification may be provided. Attached Figure Description

[0008] The embodiments of this disclosure will be more fully understood and appreciated through the following detailed description, in conjunction with the accompanying drawings, wherein:

[0009] Figure 1 An example of a method is shown;

[0010] Figure 2 An example of a method is shown;

[0011] Figure 3 Examples of anchor images and dimensions are shown;

[0012] Figure 4 Examples of anchor images and dimensions are shown;

[0013] Figure 5 An example of calibrating a vehicle and its surrounding environment is shown; and

[0014] Figure 6 An example of a vehicle and its surrounding environment is shown. Detailed Implementation

[0015] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the invention.

[0016] The subject matter considered to be the present invention is specifically pointed out and explicitly claimed in the conclusion section of the specification. However, the organization and operation of the invention, as well as its objects, features, and advantages, can be best understood by referring to the following detailed description when read in conjunction with the accompanying drawings.

[0017] It should be understood that, for the sake of simplicity and clarity, the elements shown in the figures are not necessarily drawn to scale. For example, for clarity, the dimensions of some of these elements may be enlarged relative to others. Furthermore, where deemed appropriate, reference numerals may be repeated in the figures to identify corresponding or similar elements.

[0018] Since most of the embodiments shown in this invention can be implemented using electronic components and circuits known to those skilled in the art, the level of detail will not be as high as considered necessary as shown above in order to understand and comprehend the basic concepts of this invention and to avoid obscuring or deviating from the teachings of this invention.

[0019] Any reference to the methods in this specification should be interpreted in accordance with an apparatus or system capable of performing the methods and / or a non-transitory computer-readable medium storing instructions for performing the methods.

[0020] Any references to a system or device in this specification should be interpreted in accordance with methods that can be performed by said system, and / or in accordance with non-transitory computer-readable media that store instructions that can be performed by said system.

[0021] Any reference in the specification to a non-transitory computer-readable medium should be interpreted in accordance with an apparatus or system capable of executing instructions stored in said non-transitory computer-readable medium and / or a method suitable for executing said instructions.

[0022] Any combination of any modules or units listed in any drawing, any part of the specification and / or any claim may be provided.

[0023] The instructions and / or accompanying drawings may refer to images. Any reference to images should be interpreted in accordance with the video stream or one or more images.

[0024] The specification and / or drawings may refer to a processor. The processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), full-custom integrated circuits, or combinations of such integrated circuits.

[0025] Any combination of any steps of any method shown in the specification and / or figures may be provided.

[0026] Any combination of any subject matter that can be provided for any claim.

[0027] Any combination of systems, units, components, processors, and sensors shown in the specification and / or drawings may be provided.

[0028] Analysis of the content of a media unit can be performed by generating an identification marker for the media unit and comparing the identification marker with a reference identification marker. The reference identification marker can be arranged in one or more conceptual structures or in any other manner. The identification marker can be used for object detection or any other purpose.

[0029] The term "substantially" refers to insignificant deviations—such as differences not exceeding a few percent of the value, or differences in accuracy and / or resolution associated with the facial recognition process. The meaning of "substantially" can be defined in any way.

[0030] An active sensor is a sensor that emits radiation during the sensing process. The radiation can be optical radiation (LiDAR), radio frequency radiation (radar), sound waves (sonar), etc.

[0031] The terms first camera and second camera are used to indicate that the camera used during the calibration process may be different from the camera used after the calibration process, or it may be the same camera.

[0032] For the sake of simplicity, let's assume that the active sensor is a lidar.

[0033] A system, method, and non-transitory computer-readable medium may be provided that uses images acquired by a camera to obtain depth information. The camera is a two-dimensional camera and should not be a three-dimensional camera—e.g., a stereo camera.

[0034] The method obtains a mapping between anchors and their distance from the camera, representing their appearance in an image. An anchor is an object associated with at least one physical dimension of known value.

[0035] The association can mean that the at least one physical dimension is at least one dimension of the object.

[0036] The at least one dimension of the object can be any dimension—for example, the height, length, width, radius, diagonal, etc. of the object or any part thereof. For example, a known value of the dimension of a zebra crossing might be its width.

[0037] The association can mean that the at least one physical dimension is a spatial relationship (e.g., distance) related to a portion of the object. For example, the known dimensions of a zebra crossing might be the spacing between stripes of the same color.

[0038] The association can mean that the at least one physical dimension indicates the spatial relationship between the object and another object (e.g., distance, overlap, etc.). For example, the known value of the dimension of zebra stripes could be the spacing between stripes of the same color. Another example is the distance between adjacent road dividers.

[0039] The at least one physical dimension of the known value can be a known precise value (e.g., the width of the object can be 40 centimeters).

[0040] The known physical dimension can be within a certain range (e.g., the height of a Caucasian adult can be between 160 and 210 cm). A larger range may reduce the accuracy of distance determination, but may be better than no distance estimation at all.

[0041] Anchors can be determined in any way, such as by definition, based on metadata associated with the description of the anchor, or from a predefined list.

[0042] Non-limiting examples of anchors include traffic signs, different types of traffic lights (types refer to different models, different quantities, etc.), zebra crossings, road dividers, specific models of vehicles, different types of pedestrians (children, adults), etc.

[0043] A system, method, and non-transitory computer-readable medium may be provided for performing a calibration process or receiving the results of the calibration process.

[0044] The calibration process involves the use of cameras and active sensors, such as lidar.

[0045] Then, one or more images from the camera are acquired, and distance information is determined based on the results of the calibration process and the appearance of one or more anchors in the one or more images.

[0046] Figure 1 Calibration method 100 is shown.

[0047] Method 100 may begin with step 110, which involves receiving information about one or more anchors. The information may include an identifier for each object used as an anchor and at least one physical dimension of known value associated with each anchor.

[0048] Step 110 can be followed by step 120.

[0049] Step 120 may include acquiring multiple calibration images via a second camera, and acquiring distance information associated with the calibration images via an active sensor such as a lidar.

[0050] The distance information can be a lidar image or any other information about distance. The lidar image and the calibration image can be captured simultaneously or at least before the distance between the camera and its environment is changed.

[0051] The distance information is acquired when there is a known spatial relationship between the second camera and the lidar. The second camera and the lidar may be located in the same position, but this is not mandatory.

[0052] The second camera is a 2D camera, not a 3D camera.

[0053] The plurality of calibration images capture the one or more anchors. Each calibration image may capture at least one of the one or more anchors. Assuming there are multiple anchors, any one of the plurality of calibration images may include one, some, or all of the multiple anchors. For example, a calibration image may include a first anchor, a second calibration image may include a second anchor, a third calibration image may include the first anchor and the second anchor, and so on.

[0054] At least some of the plurality of calibration images can be acquired at different distances from the camera. Therefore, one or more calibration images can be acquired at the same distance.

[0055] Step 120 can be followed by step 130, which involves detecting the one or more anchors within the plurality of calibration images and obtaining distance measurements (via lidar) from the one or more anchors within the plurality of calibration images.

[0056] Step 130 may include performing one or more distance measurements using an active sensor, while utilizing the spatial relationship (aligned or at least aligned with a known mapping) between pixels of the image acquired by the active sensor and pixels of the one or more calibrated images to detect the anchor in the image acquired by the active sensor.

[0057] The lidar and the second camera can be aligned so that their fields of view are the same.

[0058] In this scenario, once an anchor is detected within the calibration image (acquired by the second camera), its position within the lidar image is also known, and a distance measurement associated with the anchor can be obtained from the lidar image.

[0059] This alignment eliminates the need to detect the anchor in the lidar image.

[0060] Step 130 may include performing one or more distance measurements by an active sensor, searching for the anchor in one or more images acquired by the active sensor, and searching for the anchor in the one or more calibrated images by applying an object detection algorithm.

[0061] The lidar and the second camera may not be aligned, but the mapping between the pixels of the calibration image and the pixels of the lidar image should be known, so that the position of the anchor in the lidar image can be deduced based on the position of the anchor in the calibration image and the mapping.

[0062] This mapping eliminates the need to detect the anchor in the lidar image.

[0063] According to another example, the second camera and the lidar can operate independently of each other, even without a known mapping between aligned pixels.

[0064] In this case, the lidar image should be processed to detect the anchor within the lidar image.

[0065] This is followed by associating the distance information provided from the lidar image with the anchor in the calibration image.

[0066] Step 130 can be followed by step 140, which generates a distance-appearance relationship that maps appearance to distance.

[0067] The appearance of a known value's size in an image can be the size of that size within the image, such as the number of pixels representing that size in the image. For example, a traffic light of known width is displayed as an H x W pixel box in an image, so the appearance is W pixels.

[0068] The result of step 120 is a plurality of tuples (the appearance of the anchor with known dimensions and the distance to the anchor), and the distance-appearance relationship can be generated during step 130 by any function that can generate a mapping based on instances of the plurality of tuples.

[0069] Figure 2 Method 200 is shown.

[0070] Method 200 may begin at step 210, which involves obtaining an image of the environment surrounding the vehicle from the vehicle's camera.

[0071] The vehicle's surroundings are anything within the camera's field of view.

[0072] Step 210 may be followed by step 220, which involves searching for anchors within the image, wherein the anchors are associated with at least one physical size of a known value.

[0073] Step 220 may be followed by step 230, which is to determine the distance between the camera and the anchor (when the anchor is found) based on (a) the at least one physical dimension of known value, (b) the appearance of the at least one physical dimension of known value in the image, and (c) a distance-appearance relationship that maps appearance to distance.

[0074] The distance-appearance relationship can be generated through a calibration process, which may include obtaining one or more calibration images of the anchor and obtaining one or more distance measurements from the anchor.

[0075] Method 100 is an example such as a calibration process.

[0076] Method 200 may include method 100, for example, as a preparatory step.

[0077] The one or more calibration images are multiple calibration images acquired at different distances from the second camera, and wherein the one or more distance measurements indicate different distances. The execution of the calibration process may include calculating the distance-appearance relationship based on the appearance of at least one physical dimension known in the multiple calibration images and the different distances.

[0078] The one or more distance measurements can be performed by an active sensor, wherein the execution of the calibration process includes searching for the anchor in one or more images acquired by the active sensor, and searching for the anchor in one or more calibration images by applying an object detection algorithm.

[0079] The one or more distance measurements can be performed by an active sensor, wherein the execution of the calibration process includes using the spatial relationship between pixels of the image acquired by the active sensor and pixels of the one or more calibration images to detect the anchor in the image acquired by the active sensor.

[0080] Step 220 may include approximating the anchor by a bounding box and measuring at least one dimension of the bounding box to provide a value that indicates the appearance of the at least one physical dimension known in the image.

[0081] An example of a defined bounding box is shown in U.S. Patent Application US 16 / 544,940, filed August 20, 2019, which is incorporated herein by reference.

[0082] Figure 3 and Figure 4 Various images are shown, including a variety of anchors and some dimensions with known values.

[0083] Figure 3 A first image 11 is shown, in which the width of the yield sign 21 is known (e.g., 26 cm), and it appears as an N1-pixel line in the first image. In the second image 12, the same yield sign 21 with the same known width appears as an N2-pixel line.

[0084] Assuming that the first image and the second image were obtained during the calibration process and at different distances from the giveaway sign (e.g., measured by an active sensor), these different distances and different appearances can be used to calculate the distance-appearance relationship.

[0085] Alternatively, assuming the first and second images are obtained after the calibration process, the appearance of the yield sign can provide an estimate of the distance of the yield sign from the vehicle at each moment.

[0086] It should be noted that if the image includes multiple anchors, the distances to multiple points within the image can be calculated based on the distances to these anchors and their positions in the image, for example, by extrapolation. It should also be noted that the latter can be based on the expected optical characteristics of the camera, such as distortion.

[0087] Figure 4 Examples of possible dimensions for known values ​​are shown, such as the diameter of the no-entry sign 22, the width of the give way sign 21, the height of an adult, the width of the zebra crossing, the width of the zebra stripes, and the spacing of zebra stripes of the same color.

[0088] Figure 5 The calibration process vehicle 90 is shown undergoing the calibration procedure. An active sensor 91 performs distance measurements 93 on a yield sign, which is also located within the field of view 94 of the camera 92.

[0089] Figure 6 The vehicle 90' ​​without active sensors is shown.

[0090] The vehicle 90' ​​may also include a camera 92, a processor 99, a vehicle controller 98, a storage unit 97, one or more vehicle computers (e.g., an autonomous driving controller or ADAS computer 96), and a communication unit 95 for communicating with other vehicles and / or remote computer systems (e.g., cloud computers).

[0091] The storage unit can store any data structure, such as a distance-appearance relation database.

[0092] The diagram also shows a yield sign with a certain width, which is displayed as N3 pixels, indicating that its distance from the camera is D3 according to the distance-appearance relation database 89.

[0093] While the foregoing written description of the invention enables those skilled in the art to make and use what is presently considered the best mode thereof, those skilled in the art will understand and appreciate that variations, combinations, and equivalents exist in the specific embodiments, methods, and examples described herein. Therefore, the invention should not be limited to the embodiments, methods, and examples described above, but rather to all embodiments and methods within the claimed scope and spirit of the invention.

[0094] In the foregoing specification, the invention has been described with reference to specific examples of embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0095] Furthermore, the terms “front,” “rear,” “top,” “bottom,” “above,” “below,” etc., used in the specification and claims are for describing, and not necessarily for describing, permanent relative positions. It should be understood that such terms are interchangeable where appropriate, enabling the embodiments of the invention described herein to operate, for example, in other orientations different from those shown or otherwise described herein.

[0096] Furthermore, when referring to setting a signal, status bit, or similar device to its logical true or logical false state, the terms "assert" or "set" and "negate" (or "deassert" or "clear") are used. If the logical true state is logic level 1, then the logical false state is logic level 0. And if the logical true state is logic level 0, then the logical false state is logic level 1.

[0097] Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative, and alternative embodiments may combine logic blocks or circuit elements or apply alternative decompositions to various logic blocks or circuit elements. Therefore, it should be understood that the architecture described herein is merely exemplary, and in fact, many other architectures can be implemented to achieve the same functionality.

[0098] Arbitrary arrangements of components that perform the same function are effectively “associated” to achieve the desired functionality. Therefore, regardless of the architecture or intermediate components, any two components combined in this paper to achieve a specific function can be considered “associated” with each other to achieve the desired functionality. Similarly, any two such associated components can also be considered “operably connected” or “operably coupled” to each other to achieve the desired functionality.

[0099] Furthermore, those skilled in the art will recognize that the boundaries between the above operations are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and the execution of operations may overlap at least partially in time. Additionally, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be varied in various other embodiments.

[0100] For example, in one embodiment, the examples shown can be implemented as circuitry located on a single integrated circuit or within the same device. Alternatively, these examples can be implemented as any number of separate integrated circuits or separate devices interconnected with each other in a suitable manner.

[0101] However, other modifications, variations, and substitutions are also possible. Therefore, the specification and drawings are considered illustrative rather than restrictive.

[0102] In the claims, any reference marks placed between parentheses should not be construed as limiting the claims. The word “comprising” does not exclude the presence of other elements or steps listed in the claims. Furthermore, as used herein, the terms “a” or “an” are defined as one or more. Moreover, the use of introductory phrases such as “at least one” and “one or more” in the claims should not be construed as implying that the introduction of another claim element by the indefinite article “a” (“a” or “an”) limits any particular claim containing such an introduced claim element to an invention containing only one such element, even if the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” (“a” or “an”). The same applies to the use of definite articles. Unless otherwise stated, terms such as “first” and “second” are used to arbitrarily distinguish the elements described by such terms. Therefore, these terms are not necessarily intended to indicate a temporal or other priority of such elements. The fact that certain measures are described only in mutually different claims does not indicate that combinations of these measures cannot be utilized.

[0103] While certain features of the invention have been shown and described herein, many modifications, substitutions, alterations, and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and alterations falling within the true spirit of the invention.

[0104] It should be understood that, for clarity, various features of embodiments of this disclosure described in the context of individual embodiments may also be provided in combination within a single embodiment. Conversely, for brevity, various features of embodiments of this disclosure described in the context of individual embodiments may also be provided individually or in any suitable sub-combination.

[0105] Those skilled in the art will understand that the embodiments of this disclosure are not limited to what has been specifically shown and described above. Rather, the scope of the embodiments of this disclosure is defined by the appended claims and their equivalents.

Claims

1. A method for measuring distance, the method comprising: A pre-established distance-appearance relationship mapping appearance to distance is obtained, wherein the distance-appearance relationship is generated by a calibration process, the calibration process including obtaining multiple calibration images at different distances from the anchor, obtaining multiple distance measurements from the anchor by an active sensor, wherein the multiple distance measurements indicate different distances, and obtaining distance information associated with the multiple calibration images based on calibration image-distance tuples at multiple different distances, wherein the calibration process is performed by a calibration process vehicle equipped with the active sensor; Images of the environment surrounding the measuring vehicle are acquired from the vehicle's camera. Search for the anchor within the image, wherein the anchor is associated with at least one physical dimension of a known value; When the anchor is located, the distance between the camera and the anchor is determined based on (a) the at least one physical dimension of known value, (b) the appearance of the at least one physical dimension of known value in the image, and (c) the pre-established distance-appearance relationship, without using the active sensor for distance measurement.

2. The method according to claim 1, wherein the camera is a first camera, and the plurality of calibration images are plurality of calibration images acquired at different distances from the second camera; wherein, The execution of the calibration process includes calculating the distance-appearance relationship based on the appearance of at least one physical dimension known in the plurality of calibration images and the different distances.

3. The method of claim 2, wherein the execution of the calibration process includes searching for the anchor in the images acquired by the active sensor and finding the anchor in the plurality of calibration images by applying an object detection algorithm.

4. The method of claim 2, wherein the execution of the calibration process includes using the spatial relationship between pixels of an image acquired by the active sensor and pixels of the plurality of calibration images to detect the anchor in the image acquired by the active sensor.

5. The method of claim 1, wherein the at least one physical dimension of the known value of the anchor is the size of the anchor.

6. The method of claim 1, wherein the at least one physical dimension of the known value of the anchor is the distance between the anchor and another anchor.

7. The method of claim 1, further comprising approximating the anchor by a bounding box and measuring at least one dimension of the bounding box to provide a value indicating the appearance of the at least one physical dimension of the known value in the image.

8. The method of claim 1, wherein the anchor includes at least one of the following: traffic signs, traffic lights, zebra crossings, road dividers, and vehicles.

9. A non-transitory computer-readable medium for storing the following instructions: A pre-established distance-appearance relationship mapping appearance to distance is obtained, wherein the distance-appearance relationship is generated by a calibration process, the calibration process including obtaining multiple calibration images at different distances from the anchor, obtaining multiple distance measurements from the anchor by an active sensor, wherein the multiple distance measurements indicate different distances, and obtaining distance information associated with the multiple calibration images based on calibration image-distance tuples at multiple different distances, wherein the calibration process is performed by a calibration process vehicle equipped with the active sensor; images of the environment surrounding the measurement vehicle are obtained from a camera of the measurement vehicle; the anchor is searched within the images, wherein the anchor is associated with at least one physical dimension of known value; when the anchor is found, the distance between the camera and the anchor is determined based on (a) the at least one physical dimension of known value, (b) the appearance of the at least one physical dimension of known value in the image, and (c) the pre-established distance-appearance relationship, without using the active sensor for distance measurement.

10. The non-transitory computer-readable medium of claim 9, wherein the camera is a first camera, and the plurality of calibration images are plurality of calibration images acquired at different distances from the second camera; wherein, The execution of the calibration process includes calculating the distance-appearance relationship based on the appearance of at least one physical dimension known in the plurality of calibration images and the different distances.

11. The non-transitory computer-readable medium of claim 10, wherein the execution of the calibration process includes searching for the anchor in images acquired by the active sensor and searching for the anchor in the plurality of calibration images by applying an object detection algorithm.

12. The non-transitory computer-readable medium of claim 10, wherein the execution of the calibration process includes using spatial relationships between pixels of an image acquired by the active sensor and pixels of the plurality of calibration images to detect the anchor in the image acquired by the active sensor.

13. The non-transitory computer-readable medium of claim 9, wherein the at least one physical dimension of the known value of the anchor is the size of the anchor.

14. The non-transitory computer-readable medium of claim 9, wherein the at least one physical dimension of the known value of the anchor is the distance between the anchor and another anchor.

15. The non-transitory computer-readable medium of claim 9, storing instructions for approximating the anchor by a bounding box and measuring at least one dimension of the bounding box to provide a value indicating the appearance of at least one physical dimension of the known value in the image.

16. A computerized system including a processor, the processor being configured to: acquire a pre-established distance-appearance relationship mapping appearance to distance, wherein the distance-appearance relationship is generated by a calibration process, the calibration process including acquiring multiple calibration images at different distances from an anchor, acquiring multiple distance measurements from the anchor via an active sensor, wherein the multiple distance measurements indicate different distances, and acquiring distance information associated with the multiple calibration images based on calibration image-distance tuples at multiple different distances, wherein the calibration process is performed by a calibration process vehicle equipped with the active sensor; receiving images of the environment surrounding the measurement vehicle from a camera of the measurement vehicle; searching for the anchor within the images, wherein the anchor is associated with at least one physical dimension of known value; and, when the anchor is found, determining the distance between the camera and the anchor based on (a) the at least one physical dimension of known value, (b) the appearance of the at least one physical dimension of known value in the image, and (c) the pre-established distance-appearance relationship, without using the active sensor for distance measurement.

17. The computerized system according to claim 16, wherein the camera is a first camera, and the plurality of calibration images are plurality of calibration images acquired at different distances from the second camera; wherein, The execution of the calibration process includes calculating the distance-appearance relationship based on the appearance of at least one physical dimension known in the plurality of calibration images and the different distances.

Images