Efficient algorithm for projecting world points into rolling shutter images

Abstract

An improved, efficient method for mapping world points from an environment (e.g., points generated by a LIDAR sensor of an autonomous vehicle) to locations (e.g., pixels) within a captured rolling shutter image of the environment is provided. This improved method allows accurate localization of world points in a rolling shutter image via an iterative process that converges in very few iterations. The method implements the localization process as an iterative process for determining the time at which a camera images a world point within a rolling shutter exposure period of the image. The method reduces the number of times a world point is projected into the normalized space of the camera image, typically converging in three or fewer iterations.

Description

[0001] Text files incorporated by reference

[0002] This application incorporates by reference in its entirety the contents of an ASCII-formatted text file named “camera_model.txt” (48,446 bytes in size, created on December 4, 2019) that was electronically filed on December 4, 2019, along with the instant application. Technical Field

[0003] The present application generally relates to efficient algorithms for projecting world points into rolling shutter images. Background Art

[0004] Active sensors, such as light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, and sound navigation and ranging (SONAR) sensors, can scan the environment by transmitting signals toward the environment and detecting reflections of the transmitted signals. Passive sensors, such as image sensors (e.g., cameras) and microphones, can detect signals originating from sources in the environment.

[0005] An example LIDAR sensor can determine the distance to environmental features while scanning through a scene to gather a "point cloud" indicating reflective surfaces. For example, individual points in the point cloud can be determined by sending a laser pulse and detecting a return pulse (if any) reflected from an object in the environment, and then determining the distance to the object based on the time delay between sending the pulse and receiving its reflection. Thus, a three-dimensional map of points indicating the locations of reflective features in the environment can be generated.

[0006] An example image sensor can capture an image of a scene visible to the image sensor. For example, the image sensor can include an array of complementary metal oxide semiconductor (CMOS) active pixel sensors or other types of light sensors. Each CMOS sensor can receive a portion of the light from the scene incident on the array. When the CMOS sensor is exposed to light from the scene, each CMOS sensor can then output a measurement of the amount of light incident on the CMOS sensor during the exposure time. With this arrangement, an image of the scene can be generated, where each pixel in the image indicates one or more values ​​(e.g., color, etc.) based on the output from the array of CMOS sensors. Summary of the invention

[0007] In one example, a method includes: (i) obtaining an indication of a point in an environment of an autonomous vehicle; (ii) obtaining information about the position and motion of the autonomous vehicle within the environment; (iii) obtaining an image of a portion of the environment of the autonomous vehicle, wherein the image includes a plurality of rows of pixels, and wherein the image is generated by a camera operating in a rolling shutter mode such that each row of pixels represents light sensed by the camera during a corresponding exposure time period; and (iv) mapping the point in the environment to a location within the image. Mapping the point in the environment to a location within the image includes: (a) determining an initial estimate time T0 of the camera sensing light from the point in the environment; (b) determining N updated estimate times T i , where N ≥ 1; and (c) based on the updated estimated time T N Determine the location within the image that corresponds to a point in the environment. The estimated time T for each update i The updating process includes: (1) determining the camera position at an estimated time T based on information about the position and motion of the autonomous vehicle. i-1 The pose of the camera is estimated at time T. i-1 The pose of the environment is projected to the projection position in the image, (3) the cost function is evaluated, which includes the estimated time T i-1 and a term based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image, and (4) determining an updated estimated time T based on the evaluated cost function i .

[0008] In another example, a non-transitory computer-readable medium having instructions stored therein, the instructions executable by a computing device to cause the computing device to perform operations. The operations include: (i) obtaining an indication of a point in an environment of an autonomous vehicle; (ii) obtaining information about the position and movement of the autonomous vehicle within the environment; (iii) obtaining an image of a portion of the environment of the autonomous vehicle, wherein the image includes multiple rows of pixels, and wherein the image is generated by a camera operating in a rolling shutter mode such that each row of pixels represents light sensed by the camera during a corresponding exposure time period; and (iv) mapping the point in the environment to a location within the image. Mapping the point in the environment to a location within the image includes: (a) determining an initial estimated time T0 that the camera senses light from the point in the environment; (b) determining N updated estimated times T i , where N ≥ 1; and (c) based on the updated estimated time T N Determine the location within the image that corresponds to a point in the environment. The estimated time T for each update i The updating process includes: (1) determining the camera position at an estimated time T based on information about the position and motion of the autonomous vehicle. i-1The pose of the camera is estimated at time T. i-1 The pose of the environment is projected to the projection position in the image, (3) the cost function is evaluated, which includes the estimated time T i-1 and a term based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image, and (4) determining an updated estimated time T based on the evaluated cost function i .

[0009] In yet another example, a system includes: (i) a light detection and ranging (LIDAR) sensor coupled to a vehicle; (ii) a camera coupled to the vehicle, wherein the camera is configured to obtain image data indicating an environment of the vehicle; and (iii) a controller, wherein the controller is operably coupled to the LIDAR sensor and the camera. The controller includes one or more processors configured to perform operations, the operations including: (a) operating the LIDAR sensor to generate a plurality of LIDAR data points indicating distances to one or more objects in the environment of the vehicle; (b) generating an indication of a point in the environment based on at least one LIDAR data point in the plurality of LIDAR data points; (c) operating the camera in a rolling shutter mode to generate an image of a portion of the environment of the vehicle, wherein the image includes a plurality of rows of pixels, and wherein each row of pixels represents light sensed by the camera during a corresponding exposure time period; (d) obtaining information about a position and motion of the autonomous vehicle within the environment; and (e) mapping the point in the environment to a position within the image. Mapping a point in the environment to a location within an image involves: (1) determining an initial estimate of the time T0 at which the camera senses light from the point in the environment; (2) determining N updated estimate times T i , where N ≥ 1; and (3) based on the updated estimated time T N Determine the location within the image that corresponds to a point in the environment. The estimated time T for each update i The updating process includes: (I) determining the camera position at an estimated time T based on information about the position and motion of the autonomous vehicle. i-1 The pose of the camera at the estimated time T i-1 The pose of the environment is projected to the projection position in the image, (III) the cost function is evaluated, which includes the estimated time T i-1 and a term based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image, and (IV) determining an updated estimated time T based on the evaluated cost function i .

[0010] These and other aspects, advantages and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with appropriate reference to the accompanying drawings. In addition, it should be understood that the descriptions provided in the Summary of the Invention section and elsewhere in this document are intended to illustrate the claimed subject matter by way of example and not limitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a simplified block diagram of a system according to an example embodiment.

[0012] Figure 2A A device including a rotating LIDAR sensor and a rolling shutter camera arrangement according to an example embodiment is shown.

[0013] Figure 2B yes Figure 2A Cross-sectional view of a rolling shutter camera arrangement.

[0014] Figure 2C yes Figure 2A Conceptual illustration of the operation of the device.

[0015] Figure 2D Shows Figure 2A A top view of the device.

[0016] Figure 2E Shows Figure 2A Another top view of the device.

[0017] Figure 3 A cross-sectional view of another rolling shutter camera arrangement is shown according to an example embodiment.

[0018] Figure 4A The motion of the camera relative to points in the environment is shown.

[0019] Figure 4B Shows Figure 4A The points shown are Figure 4A The projection onto the image generated by the camera is shown.

[0020] Figure 5 is a flowchart of a method according to an example embodiment. DETAILED DESCRIPTION

[0021] Exemplary embodiments are described herein. It should be understood that the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "exemplary" or "illustrative" is not necessarily to be construed as being preferred or advantageous over other embodiments or features. In the accompanying drawings, similar symbols generally identify similar components unless the context indicates otherwise. The example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure as generally described herein and shown in the accompanying drawings may be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0022] I. Example Sensor

[0023] Although the example sensors described herein include LIDAR sensors and cameras (or image sensors), other types of sensors are possible. A non-exhaustive list of example sensors that may be alternatively employed herein include RADAR sensors, SONAR sensors, sound sensors (e.g., microphones, etc.), motion sensors, temperature sensors, pressure sensors, etc. without departing from the scope of the present disclosure.

[0024] To this end, the example sensors herein may include active sensors that transmit signals (e.g., pulse trains or any other modulated signals) based on modulated power provided to the sensor and then detect reflections of the transmitted signals from objects in the surrounding environment. Alternatively or additionally, the example sensors herein may include passive sensors (e.g., cameras, microphones, antennas, pressure sensors, etc.) that detect external signals (e.g., background signals, etc.) originating from (multiple) external sources in the environment.

[0025] Now referring to the accompanying drawings, Figure 1 1 is a simplified block diagram of a system 100 including a sensor (e.g., a synchronization sensor) according to an example embodiment. As shown, the system 100 includes a power supply arrangement 102, a controller 104, one or more sensors 106, one or more sensors 108, a rotating platform 110, one or more actuators 112, a stationary platform 114, a rotary link 116, a housing 118, and a display 140.

[0026] In other embodiments, system 100 may include more, fewer, or different components. Furthermore, the components shown may be combined or divided in any number of ways. For example, sensor(s) 108 may be implemented as a single physical component (e.g., a camera ring). Alternatively, for example, sensor(s) 108 may be implemented as an arrangement of separate physical components. Other examples are also possible. Thus, Figure 1The functional blocks are shown only for convenience of description. Other example components, arrangements and / or configurations are also possible without departing from the scope of the present disclosure.

[0027] The power arrangement 102 may be configured to supply, receive, and / or distribute power to the various components of the system 100. To this end, the power arrangement 102 may include or otherwise take the form of a power source (e.g., a battery, etc.) disposed within the system 100 and connected to the various components of the system 100 in any feasible manner to provide power to the components. Additionally or alternatively, the power arrangement 102 may include or otherwise take the form of a power adapter configured to receive power from one or more external power sources (e.g., a power source disposed in a vehicle in which the system 100 is installed, etc.) and to send the received power to the various components of the system 100.

[0028] The controller 104 may include one or more electronic components and / or systems arranged to facilitate certain operations of the system 100. The controller 104 may be disposed within the system 100 in any feasible manner. In one embodiment, the controller 104 may be disposed at least partially within the central cavity region of the rotating link 116. In another embodiment, one or more functions of the controller 104 may alternatively be performed by one or more physically separate controllers each disposed within a corresponding component of the system 100 (e.g., sensor(s) 106, 108, etc.).

[0029] In some examples, the controller 104 may include or otherwise be coupled to wiring for communicating control signals to various components of the system 100 and / or for communicating data from various components of the system 100 to the controller 104. In general, the data received by the controller 104 may include sensor data based on detection of light by the LIDAR 106 and / or the camera(s) 108, etc. In addition, the control signals sent by the controller 104 may operate various components of the system 100, such as by controlling emission and / or detection of light or other signals by the sensor(s) 106 (e.g., LIDAR, etc.), controlling image pixel capture rate or time via a camera (e.g., included in the sensor(s) 108), and / or controlling the actuator(s) 112 to rotate the rotating platform 110, etc.

[0030] To this end, in some examples, the controller 104 may include one or more processors, data storage devices, and program instructions (stored in the data storage devices) executable by the one or more processors to cause the system 100 to perform various operations described herein. In some instances, the controller 104 may communicate with an external controller or the like (e.g., a computing system disposed in a vehicle, robot, or other mechanical device in which the system 100 is installed) to facilitate the transfer of control signals and / or data between the external controller and various components of the system 100.

[0031] Additionally or alternatively, in some examples, the controller 104 may include circuits that are wired to perform the various functions described herein. Additionally or alternatively, in some examples, the controller 104 may include one or more dedicated processors, servo systems, or other types of controllers. For example, the controller 104 may include a proportional integral derivative (PID) controller or other control loop feedback device that operates the actuator(s) 112 to adjust the rotation of the rotating platform 110 according to a specific frequency or phase. Other examples are also possible.

[0032] Sensors 106 and 108 may optionally include one or more sensors that scan the surrounding environment of system 100, such as LIDAR, cameras, gyroscopes, accelerometers, encoders, microphones, RADAR, SONAR, thermometers, etc.

[0033] The sensor(s) 106 may include any device configured to scan the surrounding environment by transmitting a signal and detecting the reflection of the transmitted signal. For example, the sensor(s) 106 may include any type of active sensor. To this end, as shown, the sensor 106 includes a transmitter 120 and a receiver 122. In some embodiments, the sensor 106 may also include one or more optical elements 124.

[0034] Transmitter 120 may be configured to transmit a signal toward the environment of system 100 .

[0035] In a first example, where the sensor 106 is configured as a LIDAR sensor, the transmitter 120 may include one or more light sources (not shown) that emit one or more light beams and / or pulses having wavelengths within a wavelength range. Depending on the configuration of the light source, the wavelength range may be, for example, in the ultraviolet portion, the visible portion, and / or the infrared portion of the electromagnetic spectrum. In some examples, the wavelength range may be a narrow wavelength range (such as provided by a laser and / or some light emitting diodes). In some examples, the light source(s) in the transmitter 120 may include a laser diode, a diode bar, a light emitting diode (LED), a vertical cavity surface emitting laser (VCSEL), an organic light emitting diode (OLED), a polymer light emitting diode (PLED), a light emitting polymer (LEP), a liquid crystal display (LCD), a micro-electromechanical system (MEMS), a fiber laser, and / or any other device configured to selectively transmit, reflect, and / or emit light to provide a plurality of emitted light beams and / or pulses.

[0036] In a second example, where the sensor 106 is configured as an active infrared (IR) camera, the transmitter 120 may be configured to emit IR radiation to illuminate the scene. To this end, the transmitter 120 may include any type of device (eg, a light source, etc.) configured to provide IR radiation.

[0037] In a third example, where the sensor 106 is configured as a RADAR sensor, the transmitter 120 may include one or more antennas configured to transmit a modulated radio frequency (RF) signal toward the environment of the system 100 .

[0038] In a fourth example, where the sensor 106 is configured as a SONAR sensor, the transmitter 120 may include one or more acoustic transducers, such as piezoelectric transducers, magnetostrictive transducers, electrostatic transducers, etc., configured to transmit modulated sound signals toward the environment of the system 100. In some implementations, the acoustic transducers may be configured to transmit sound signals within a specific wavelength range (e.g., infrasound, ultrasound, etc.). Other examples are also possible.

[0039] Receiver 122 may include one or more detectors configured to detect reflections of the signal transmitted by transmitter 120 .

[0040] In a first example, where the sensor 106 is configured as a RADAR sensor, the receiver 122 may include one or more antennas configured to detect reflections of the RF signal transmitted by the transmitter 120. To this end, in some embodiments, the one or more antennas of the transmitter 120 and the receiver 122 may be physically implemented as the same physical antenna structure.

[0041] In a second example, where the sensor 106 is configured as a SONAR sensor, the receiver 122 may include one or more sound sensors (e.g., microphones, etc.) configured to detect reflections of sound signals emitted by the transmitter 120. To this end, in some embodiments, one or more components of the transmitter 120 and the receiver 122 may be physically implemented as the same physical structure (e.g., the same piezoelectric transducer element).

[0042] In a third example, where the sensor 106 is configured as an active IR camera, the receiver 122 may include one or more light detectors (e.g., active pixel sensors, etc.) configured to detect a source wavelength of IR light transmitted by the transmitter 120 and reflected from the scene toward the receiver 122.

[0043] In a fourth example, where the sensor 106 is configured as a LIDAR sensor, the receiver 122 may include one or more light detectors (e.g., photodiodes, avalanche photodiodes, etc.) arranged to intercept and detect reflections of light pulses emitted by the transmitter 120 and reflected from one or more objects in the surrounding environment of the system 100. To this end, the receiver 122 may be configured to detect light having a wavelength in the same range as the wavelength of light emitted by the transmitter 120. In this way, for example, the sensor 106 (e.g., LIDAR) may distinguish reflected light pulses originating from the transmitter 120 from other light originating from external light sources in the environment.

[0044] In some examples, the receiver 122 may include a photodetector array, which may include one or more detectors each configured to convert detected light (e.g., within the wavelength range of the light emitted by the transmitter 120) into an electrical signal indicating the detected light. In practice, such a photodetector array may be arranged in one of various ways. For example, the detectors may be disposed on one or more substrates (e.g., a printed circuit board (PCB), a flexible PCB, etc.) and arranged to detect incident light. In addition, such a photodetector array may include any feasible number of detectors aligned in any feasible manner. In addition, the detectors in the array may take various forms. For example, the detector may take the form of a photodiode, an avalanche photodiode (e.g., a Geiger mode and / or a linear mode avalanche photodiode), a silicon photomultiplier (SiPM), a phototransistor, a camera, an active pixel sensor (APS), a charge coupled device (CCD), a cryogenic detector, and / or any other light sensor configured to receive focused light having a wavelength within the wavelength range of the emitted light.

[0045] In some embodiments, the sensor 106 (e.g., in a LIDAR configuration) can select or adjust the horizontal scanning resolution by changing the rotation rate of the LIDAR and / or adjusting the pulse rate of the light pulses emitted by the transmitter 120. As a specific example, the transmitter 120 can be configured to emit light pulses at a pulse rate of 15,650 light pulses per second. In this example, the LIDAR 106 can be configured to rotate at 10 Hz (i.e., 10 full 360° rotations per second). In this way, the receiver 122 can detect light at a horizontal angular resolution of 0.23°. In addition, the horizontal angular resolution of 0.23° can be adjusted by changing the rotation rate of the LIDAR 106 or by adjusting the pulse rate. For example, if the LIDAR 106 rotates at 20 Hz instead, the horizontal angular resolution can become 0.46°. Alternatively, if the transmitter 120 emits light pulses at a rate of 31,300 light pulses per second while maintaining a rotation rate of 10 Hz, the horizontal angular resolution can become 0.115°. Other examples are also possible. Additionally, in some examples, the LIDAR 106 may alternatively be configured to scan a particular field of view within less than a full 360° rotation of the LIDAR 106 .

[0046] The optical element(s) 124 may optionally be included in, or otherwise coupled to, the transmitter 120 and / or the receiver 122. In one example (e.g., where the sensor 106 includes a LIDAR sensor), the optical element(s) 124 may be arranged to direct light from a light source in the transmitter 120 toward the environment. In another example, the optical element(s) 124 may be arranged to focus and / or direct light from the environment toward the receiver 122. As such, the optical element(s) 124 may include any feasible combination of mirror(s), waveguide(s), filters, lenses, or any other optical component arranged to direct light propagating through a physical space and / or adjust certain light characteristics. For example, the optical element 124 may include a filter arranged to reduce or prevent light having a wavelength outside the wavelength range of light emitted by the transmitter 120 from propagating toward the receiver 122. For example, with this arrangement, the filter can reduce noise caused by background light propagating from the environment being scanned and originating from an external light source different from the light source of the transmitter 120.

[0047] The sensor(s) 108 may include any type of sensor configured to scan the surrounding environment. As shown, the sensor 108 includes an array of sensing elements 128. In addition, as shown, the sensor 108 may optionally include one or more optical elements 130.

[0048] In some examples, sensor(s) 108 may include active sensors that transmit signals and detect reflections thereof (e.g., LIDAR, RADAR, SONAR, etc.). Thus, although not shown, sensor(s) 108 may include a transmitter and a receiver similar to transmitter 120 and receiver 122, respectively. In other examples, sensor(s) 108 may include passive sensors that detect external signals originating from one or more external sources (e.g., microphones, cameras, image sensors, thermometers, etc.).

[0049] In a first example, where the sensor 108 is configured as a sound sensor, the sensing element 128 may include an array of microphones, each microphone detecting sound (eg, an external signal) incident on a corresponding microphone in the array.

[0050] In a second example, where the sensor(s) 108 are configured as one or more cameras, the camera(s) may include any camera (e.g., a still camera, a video camera, etc.) configured to capture an image of the environment in which the system 100 is located. For example, the camera of the sensor 108 may include any imaging device that detects and provides data indicative of an image. For example, the sensing element 128 may include one or more arrangements of light sensing elements, each of which provides a measurement of light incident thereon. To this end, the sensing element 128 may include a charge coupled device (CCD), an active pixel sensor, a complementary metal oxide semiconductor (CMOS) photodetector, an N-type metal oxide semiconductor (NMOS) photodetector, and the like.

[0051] Additionally, in some examples, data from sensing elements 128 may be combined to generate an image based on the arrangement of sensing elements 128. In one example, data from a two-dimensional (2D) array of sensing elements may correspond to a 2D array of image pixels in an image. In another example, a 3D arrangement of sensing elements (e.g., sensing elements arranged along a curved surface) may similarly be used to generate a 2D array of image pixels in an image. Other examples are also possible.

[0052] In some examples, the sensing element may optionally include multiple adjacent light detectors (or detectors of other types of signals), each of which is configured to detect light (or other signals) having a specific wavelength or range of wavelengths. For example, an image pixel may indicate color information (e.g., red, green, blue, or RGB) based on a combination of data from a first detector that detects red light intensity, a second detector that detects green light intensity, and a third detector that detects blue light intensity. Other examples are also possible.

[0053] In one embodiment, the sensor(s) 108 may be configured to detect visible light propagating from the scene. Additionally, in this embodiment, the receiver 122 (e.g., a LIDAR receiver) of the sensor 106 may be configured to detect non-visible light (e.g., infrared light, etc.) within the wavelength range of the light emitted by the transmitter 120. In this embodiment, the system 100 (or controller 104) may then combine the data from the sensor 106 (e.g., LIDAR) with the data from the sensor 108 (e.g., a camera) to generate a color three-dimensional (3D) representation (e.g., a point cloud) of the scanned environment.

[0054] In some examples, the sensor(s) 108 may include a plurality of cameras (e.g., a ring of cameras) disposed in a circular arrangement about an axis of rotation of the sensor 106 (e.g., a LIDAR). For example, a first camera may be arranged to image a first field of view (FOV) of the environment that at least partially overlaps with a range of pointing directions of the sensor 106 as the sensor 106 rotates about the axis (or as a signal transmitted by the transmitter 120 is otherwise manipulated to different pointing directions about the axis). Additionally, a second camera adjacent to and / or overlapping the first camera may image a second FOV adjacent to the first FOV of the first camera, and so on. In this manner, for example, the system 100 may be configured to capture a sequence of images of the respective FOVs simultaneously with (and / or synchronously or according to some other timing) a scan of the environment by the sensor 106 as the sensor 106 rotates about the axis.

[0055] In some examples, sensor(s) 108 may be configured to operate in a rolling shutter mode.

[0056] In a first example, where the sensor(s) 108 include a microphone array, each output from a microphone in the array may be associated with a respective exposure period of the corresponding sensing element (eg, microphone) to external sound incident on the sensor 108 .

[0057] In a second example, where the sensor(s) 108 include one or more cameras, each pixel or group of pixels output by the camera(s) may be associated with a corresponding exposure time period of the corresponding sensing element or group of sensing elements to external light. For example, the camera(s) 108 may together provide an array of adjacent rows of sensing elements 128. In addition, the camera(s) 108 may be configured to output a sequence of image pixels corresponding to the measurement of external light by the corresponding sensing elements in the array. For example, the camera(s) 108 may output a first row of image pixels based on data from a first row of sensing elements in the array, then output a second row of image pixels based on data from an adjacent second row of sensing elements in the array, and so on.

[0058] In this way, a first image pixel row can be associated with a first exposure time period during which the first row of sensing elements is exposed to light, a second image pixel row can be associated with a second exposure time period during which the adjacent second row of sensing elements is exposed to light, and so on. The first exposure time period can begin before the second exposure time period begins. For example, after a time delay from the start time of the first exposure time period (and optionally before the first exposure time period has passed), the (multiple) cameras 108 can begin exposing the adjacent second row of sensing elements. In addition, the first exposure time period can end before the second exposure time period ends. For example, the controller 104 can read an output from the first row of sensing elements after the first exposure time period ends and while the second row of sensing elements is still exposed to external light, and then read an output from the second row of sensing elements after the second exposure time period ends and while the third row of sensing elements is still exposed to external light, and so on.

[0059] In some examples, where sensor 106 includes an image sensor, system 100 may be configured to select an order in which to obtain a sequence of image pixels from sensing elements 128 in a rolling shutter mode based on the order in which transmitter 120 transmits light pulses (or other signals). For example, a given row of sensing elements in the array of sensing elements 128 may be aligned with (e.g., parallel to, etc.) an axis of rotation of the LIDAR (e.g., sensor 106). For example, if the axis of rotation of the LIDAR is a vertical axis, the given row may correspond to a vertical row of sensing elements (e.g., a vertical linear arrangement parallel to the axis of rotation of the LIDAR). In addition, transmitter 120 may be configured to repeatedly output multiple light beams in an arrangement of one or more vertical lines as the LIDAR (e.g., sensor 106) rotates about the axis. Thus, for example, sensor(s) 108 (e.g., camera(s)) may output a first row of image pixels using a first row of sensing elements that are arranged similarly (e.g., vertically, etc.) to the arrangement of the multiple light beams emitted by transmitter 120. Next, camera(s) 108 may then output a second row of image pixels using an adjacent second row of sensing elements in the rotational direction of the LIDAR (or other sensor 106). Thus, for example, the second row of image pixels may be aligned with a second vertical line of light beams emitted by transmitter 120 after sensor 106 is rotated toward the second row of sensing elements, and so on.

[0060] For example, by scanning vertical rows of sensing elements one by one, the sequence of image pixels obtained from camera(s) 108 may include a sufficient number of pixels captured at a time (and from a viewing direction) similar to the time and direction of the LIDAR light pulses (or other signals) emitted by transmitter 120 (e.g., when transmitter 120 is rotated about a vertical axis). However, for example, if camera(s) (e.g., sensor(s) 108) instead captures the sequence of image pixels using a horizontal first row of sensing elements, followed by a horizontal second row of sensing elements, etc., fewer image pixels may be captured at a time (and from a viewing direction) similar to the time and direction of the LIDAR light pulses.

[0061] The optical element(s) 130 may include any combination of optical components, such as lens(es), mirror(s), waveguide(s), filter(s), or any other type of optical component similar to the optical element(s) 124. Furthermore, the optical element(s) 130 may be arranged to focus, direct, and / or adjust light characteristics of incident light to propagate toward the sensing element 128. Furthermore, for example, where the sensor(s) 108 include a plurality of cameras, the optical element(s) 130 may include a plurality of respective camera lenses that focus external light onto respective image sensors of the cameras.

[0062] In some examples, optical element(s) 130 may include one or more filters that selectively send specific wavelengths of light toward one or more specific sensing elements of sensor 106 .

[0063] In a first example, the optical element(s) 130 may include one or more filters that attenuate optical wavelengths of light emitted by the transmitter 120. With this arrangement, for example, the system 100 may reduce noisy measurements (by the sensing element(s) 128) associated with high intensity light pulses (or other signals) emitted by the transmitter 120.

[0064] In a second example, the sensor(s) 108 may include a color image sensor (e.g., a Bayer filter sensor, a layered pixel sensor array, etc.) configured to indicate the color of incident light. In this example, the optical element(s) 130 may include a color filter array, where each filter of the array sends red, green, or blue light to a particular sensing element adjacent to the filter (and attenuates light of other wavelengths). The system 100 may then generate image pixels indicating color information (e.g., red, green, blue, etc.) (e.g., by combining outputs from multiple sensing elements sensing light of different colors).

[0065] In a third example, the optical element(s) 130 may include one or more filters that attenuate wavelengths of light (or other signals) emitted by the transmitter 120, and one or more other filters that allow transmission of these wavelengths. For example, the optical element(s) 130 may include a color filter array that includes a green filter, a red filter, and a blue filter. In this example, a relatively large number of color filters may be configured to attenuate wavelengths of the emitted light of the transmitter 120 to reduce the effects of the high intensity signal emitted by the transmitter 120. In addition, a relatively small number of color filters (e.g., one or more green filters, etc.) may be configured to (at least partially) allow transmission of wavelengths of light (or other signals) emitted by the transmitter 120. With this arrangement, the high intensity light of the transmitter 120 (reflected from objects in the environment of the system 100) can be used to illuminate one or more sensing elements in dark external light conditions (e.g., at night).

[0066] The rotating platform 110 can be configured to rotate around an axis. For example, the sensor 106 (and / or its transmitter 120 and receiver 122) can be supported by the rotating platform 110 (directly or indirectly) so that each of these components moves relative to the environment in response to the rotation of the rotating platform 110. Specifically, each of these components can rotate (simultaneously) relative to the axis so that the sensor 106 can obtain information from various directions. In some examples, the rotation axis of the rotating platform 110 is vertical, and the pointing direction of the sensor 106 can be adjusted horizontally by rotating the rotating platform 110 around its vertical rotation axis. The rotating platform 110 can be formed of any solid material suitable for supporting one or more components (e.g., sensor 106) mounted thereon.

[0067] In order to rotate the platform 110 in this manner, one or more actuators 112 may actuate the rotating platform 110. To this end, the actuators 112 may include motors, pneumatic actuators, hydraulic pistons, and / or piezoelectric actuators, among others.

[0068] With this arrangement, the controller 104 can operate the actuator 112 to rotate the rotating platform 110 in various ways to obtain information about the environment. In one example, the rotating platform 110 can rotate in either direction. In another example, the rotating platform 110 can perform a full rotation so that the sensor 106 (e.g., LIDAR) provides a 360° horizontal FOV of the environment. In addition, the rotating platform 110 can rotate at various frequencies so that the sensor 106 scans the environment at various refresh rates and / or scanning resolutions.

[0069] Alternatively or additionally, system 100 can be configured to adjust the pointing direction of the transmitted signal (transmitted by transmitter 120) in various ways. In some examples, the signal source (e.g., light source, antenna, acoustic transducer, etc.) of transmitter 120 can be operated according to a phased array configuration or other type of beam steering configuration.

[0070] In a first example, where the sensor 106 is configured as a LIDAR sensor, the light source in the transmitter 120 can be coupled to a phased array optical device (e.g., optical element 124) that controls the phase of the light wave emitted by the light source. For example, the controller 104 can be configured to adjust the phased array optical device (e.g., phased array beam steering) to change the effective pointing direction of the light signal emitted by the transmitter 120 (e.g., even if the rotating platform 110 is not rotating).

[0071] In a second example, where the sensor 106 is configured as a RADAR sensor, the transmitter 120 may include an antenna array, and the controller 104 may provide a corresponding phase shift control signal to each individual antenna in the array to modify the pointing direction of the combined RF signal from the array (e.g., phased array beam steering).

[0072] In a third example, where the sensor 106 is configured as a SONAR sensor, the transmitter 120 may include an acoustic transducer array, and the controller 104 may similarly operate the acoustic transducer array (e.g., via a phase shift control signal, etc.) to achieve a target pointing direction of the combined sound signal emitted by the array (e.g., even if the rotating platform 110 is not rotating, etc.).

[0073] In other examples, a deformable flexible structure (eg, MEMS, etc.) that can deform in response to a control signal from controller 104 can be used to control the pointing direction of sensor(s) 106 to adjust the steering direction of the signal transmitted by transmitter 120. Other examples are also possible.

[0074] The stationary platform 114 can take any shape or form and can be configured for coupling to various structures, such as coupling to the top of a vehicle, for example. In addition, the coupling of the stationary platform 114 can be performed via any feasible connector arrangement (e.g., bolts and / or screws). In this way, the system 100 can be coupled to a structure for various purposes, such as those described herein. In one example, the sensor(s) 108 can be coupled to the stationary platform 114. In this example, the sensor(s) 108 can remain stationary relative to the rotational movement of the sensor(s) 106 (or otherwise change the beam direction of the signal transmitted by the transmitter 120). In another example, the sensor(s) 108 can be mounted to another physical structure different from the stationary platform 114.

[0075] The rotating link 116 directly or indirectly couples the stationary platform 114 to the rotating platform 110. To this end, the rotating link 116 can take any shape, form, and material that provides for the rotating platform 110 to rotate about an axis relative to the stationary platform 114. In some examples, the rotating link 116 can take the form of a shaft or the like that rotates based on actuation from the actuator 112, thereby transmitting mechanical force from the actuator 112 to the rotating platform 110. In one embodiment, the rotating link 116 can have a central cavity in which one or more components of the system 100 can be disposed. In some examples, the rotating link 116 can also provide a communication link for transmitting data and / or instructions between the stationary platform 114 and the rotating platform 110 (and / or components thereon, such as (multiple) sensors 106, etc.).

[0076] The housing 118 can take any shape, form, and material, and can be configured to house one or more components of the system 100. In one example, the housing 118 can be a dome-shaped housing. In addition, in some examples, the housing 118 can be constructed of at least partially opaque material, which can allow at least some light to be blocked from entering the interior space of the housing 118, thereby helping to mitigate the thermal and noise effects of ambient light on one or more components of the system 100. Other configurations of the housing 118 are also possible. In some embodiments, the housing 118 can be coupled to the rotating platform 110 so that the housing 118 is configured to rotate around the above-mentioned axis based on the rotation of the rotating platform 110. In such an embodiment, the sensor(s) 106 can rotate with the housing 118. In other embodiments, the housing 118 can remain stationary while the sensor(s) 106 rotate within the housing 118. The system 100 can also include multiple housings similar to the housing 118 for housing certain subsystems or component combinations of the system 100. For example, system 100 may include a first housing for sensor(s) 106 and a separate housing for sensor(s) 108. Other examples are possible.

[0077] A display 140 may optionally be included in the system 100 to display information about one or more components of the system 100. For example, the controller 104 may operate the display 140 to display an image captured using a camera (e.g., sensor 108), a representation of the environment of the system 100 indicated by the LIDAR data from the sensor 106 (e.g., a 3D point cloud, etc.), and / or a representation of the environment based on a combination of data from the sensors 106 and 108 (e.g., a color point cloud, an image with a superimposed temperature indicator, etc.). To this end, the display 140 may include any type of display (e.g., a liquid crystal display, an LED display, a cathode ray tube display, a projector, etc.). In addition, in some examples, the display 140 may have a graphical user interface (GUI) for displaying and / or interacting with images captured by the sensor 108, LIDAR data captured using the sensor 106, and / or any other information about the various components of the system 100 (e.g., power remaining via the power supply arrangement 102). For example, a user may manipulate the GUI to adjust the scanning configuration of the sensors 106 and / or 108 (e.g., scanning refresh rate, scanning resolution, etc.).

[0078] Note that the various components of system 100 may be combined or separated into a variety of different arrangements. For example, although sensors 106 and 108 are shown as separate components, one or more components of sensors 106 and 108 may alternatively be physically implemented within a single device. Therefore, this arrangement of system 100 is described for exemplary purposes only and is not meant to be limiting.

[0079] Figure 2A A device 200 is shown according to an example embodiment including a rotating LIDAR sensor 206 and a camera ring 208. As shown, the device 200 includes the LIDAR 206, the camera ring 208 (e.g., an arrangement of rolling shutter cameras, etc.), a rotating platform 210, a stationary platform 214, a housing 218, a LIDAR lens 224, and camera lenses 230, 232, 234, which may be similar to, for example, the sensor(s) 106, the sensor(s) 108, the rotating platform 110, the stationary platform 114, the housing 118, the optical element 124, and the optical element 130, respectively.

[0080] As shown, the light beam 250 emitted by the LIDAR 206 propagates from the lens 224 toward the environment of the LIDAR 206 along the pointing direction of the LIDAR 206 and is reflected by one or more objects (not shown) in the environment as reflected light 260. Further, as shown, the LIDAR 206 may then receive (e.g., through the lens 224) the reflected light 260. Thus, for example, similar to the discussion above with respect to the sensor 106, the LIDAR 206 may provide data (e.g., a data point cloud, etc.) indicating the distance between one or more objects and the LIDAR 206 based on the detection(s) of the reflected light 260.

[0081] In addition, as shown, each camera of the camera ring 208 can receive and detect a corresponding portion of the external light 270 incident on the corresponding camera. To this end, the external light 270 can include light originating from one or more external light sources (such as the sun, street lights, etc.). For example, the external light 270 can include light propagating directly from the external light source toward the camera lenses 230, 232, and / or 234. Alternatively or additionally, the external light 270 can include light originating from the external light source and reflected by one or more objects (not shown) in the environment of the device 200 before propagating toward the lenses 230, 232, and / or 234. Thus, for example, the cameras of the camera ring 208 can generate one or more images of the environment based on the external light 270. In addition, each image generated by a particular camera can correspond to a particular FOV of that particular camera relative to the device 200.

[0082] To this end, in some examples, the camera ring 208 may include a plurality of cameras arranged in a ring form (e.g., a circular arrangement, an elliptical arrangement, etc.) relative to each other. Each of the plurality of cameras may be positioned (e.g., mounted to the device 200 and / or the camera ring 208) at a particular angle and / or orientation. Thus, for example, the FOV of a first camera may be adjacent to and / or partially overlap the FOVs of two other adjacent cameras. For example, with this arrangement, images from separate cameras may be combined into an image of a 360-degree FOV of the device 200. In addition, for example, during assembly or calibration of the device 200, the respective angles and / or orientations of each camera may be adjusted to reduce or prevent blind spots (e.g., areas of the surrounding environment that are not within the FOV of any camera in the camera ring 208). For example, the respective FOVs of two adjacent cameras may be aligned (e.g., by moving, rotating, and / or otherwise adjusting the relative mounting positions of the two cameras, etc.) such that an area of ​​the environment between the FOVs of the two cameras (e.g., a "blind spot") is less than a threshold distance from the device 200.

[0083] To facilitate this, in one embodiment, the camera ring 208 may optionally include a housing (e.g., annular, etc.) having one or more recesses that receive and / or support the cameras in specific corresponding mounting positions (e.g., angles, orientations, etc.). In another embodiment, an example system (e.g., system 100, a calibration system, etc.) may be configured to compare images captured by the cameras and, based on the comparison, determine alignment offsets that achieve corresponding target FOVs for corresponding cameras. The example system may also include and / or operate a robotic arm, an actuator, and / or any other alignment device to adjust the position of the cameras in the camera ring 208 according to the determined alignment offsets. Other examples are also possible.

[0084] In some examples, device 200 (or another computing device coupled thereto) may operate cameras of camera ring 208 and / or process images captured therefrom (e.g., combine portions of captured images, etc.) to form a cohesive circular vision of the environment of device 200. For example, a computing system (not shown) of device 200 or another device may match features in images captured by camera ring 208 to generate a combined, combined image that spans the FOV of the cameras.

[0085] In one embodiment, lens 230 may focus light from a first 90 degree FOV of device 200, lens 232 may focus light from an adjacent second 90 degree FOV, and so on. The first FOV may optionally partially overlap with the second FOV. In other embodiments, the FOV imaged by each camera may be greater or less than 90 degrees. In addition, consistent with the discussion above, the images captured by any camera in camera ring 208 may indicate various types of information, such as light intensity of different wavelengths (e.g., colors, etc.) in external light 270, and so on.

[0086] In some examples, the LIDAR 206 (and / or the housing 218) can be configured to have a substantially cylindrical shape and rotate about an axis 242 based on, for example, the rotation of the rotating platform 210 supporting the LIDAR 206. Further, in some examples, the axis of rotation 242 can be substantially vertical. Thus, for example, by rotating the LIDAR 206 about the axis 242, the device 200 (and / or a computing system operating the device 200) can determine a three-dimensional map of a 360-degree field of view of the environment of the device 200 based on data from the LIDAR 206. Additionally or alternatively, in some examples, the device 200 can be configured to tilt the axis of rotation of the rotating platform 210 (relative to the stationary platform 214) to adjust the FOV of the LIDAR 206. For example, the rotating platform 210 can include a tilted platform that tilts in one or more directions.

[0087] In some examples, as shown, the LIDAR lens 224 can have an optical power to collimate (and / or direct) the emitted light beam 250 toward the environment of the LIDAR 206, and to focus the reflected light 260 received from the environment onto the LIDAR receiver (not shown) of the LIDAR 206. In one example, the lens 224 has a focal length of approximately 120 mm. Other example focal lengths are also possible. By using the same lens 224 (rather than a transmitting lens for collimation and a receiving lens for focusing) to perform both functions, advantages in size, cost, and / or complexity can be provided. However, alternatively, the LIDAR 206 can include a separate transmitting lens and a receiving lens. Therefore, although not shown, the LIDAR 206 can alternatively include a transmitting lens that directs the emitted light 250 toward the environment, and a separate receiving lens that focuses the reflected light 260 for detection by the receiver of the LIDAR 206.

[0088] Note that the device 200 may include more, fewer, or different components than those shown, and one or more of the components shown may be combined or separated in different ways. In one example, instead of multiple camera lenses 230, 232, 234, the device 200 may alternatively include a single camera lens extending around the circumference of the camera ring 208. In another example, although the camera ring 208 is shown as coupled to the stationary platform 214, the camera ring 208 may alternatively be implemented as a separate physical structure. In yet another example, the camera ring 208 may be located above the LIDAR 206 without being rotated by the rotating platform 214. In yet another example, the camera ring 208 may include more or fewer cameras than shown. Other examples are also possible.

[0089] Figure 2B 2 shows a cross-sectional view of the camera ring 208 according to an example embodiment. Figure 2B , axis 242 (i.e., the axis of rotation of LIDAR 206) extends through the page. As shown, camera ring 208 includes four cameras 208a, 208b, 208c, 208d arranged around axis of rotation 242. Thus, in the example shown, each camera can be configured to image a corresponding 90 degree FOV of the environment of device 200. However, in other embodiments, camera ring 208 can include fewer or more cameras than shown. In a particular embodiment, camera ring 208 can alternatively include eight cameras, each of which is coupled to a corresponding lens that focuses light from (at least) a corresponding 45 degree FOV of the environment onto the image sensor of the camera. Other examples are possible. Thus, camera ring 208 can have a variety of different configurations, and thus the configuration shown includes four cameras only for ease of description.

[0090] Furthermore, as shown, camera 208a includes a lens 230 that focuses a first portion of external light (e.g., light 270) from the environment of device 200 onto image sensor 226 of camera 208a. Furthermore, as shown, camera 208b includes a lens 232 that focuses a second portion of the external light onto image sensor 246 of camera 232. Similarly, cameras 208c and 208d may be configured to focus respective portions of the external light onto respective image sensors of the cameras.

[0091] Furthermore, as shown, each image sensor may include, for example, an array of sensing elements similar to sensing element 128. For example, image sensor 226 of camera 208a may include an array of adjacent rows of sensing elements illustrated by sensing elements 228a-228f (e.g., which may be similar to sensing element 128). By way of example, a first row of sensing elements in image sensor 226 may include sensing element 228a and one or more other sensing elements (not shown) arranged vertically across the page (e.g., parallel to axis 242). Additionally, a second row of sensing elements adjacent to the first row may include sensing element 228b and one or more other sensing elements (not shown) arranged vertically across the page, and so on.

[0092] In this manner, for example, cameras 208a, 208b, 208c, 208d may together provide an array of adjacent rows of sensing elements arranged about axis 242 so as to be able to image respective corresponding portions of a 360 degree (horizontal) FOV around device 200. For example, a given row of sensing elements in image sensor 246 of camera 204b may include sensing element 248a (and one or more other sensors arranged across the page parallel to axis 242). Furthermore, in this example, a given row of sensing elements in camera 208b may also be adjacent to a row of sensing elements in camera 208a including sensing element 228f. Thus, in an example scenario, a sequence of image pixels obtained from camera ring 208 may include a row of image pixels obtained using data from a row of sensing elements including sensing element 228f, followed by a row of image pixels obtained using data from a row of sensing elements including sensing element 248a.

[0093] Note that the number of rows of sensing elements in each image sensor (e.g., sensors 226, 246, etc.) is shown only for ease of description. However, in some embodiments, image sensor 226 (and / or 246) may include more or fewer rows of sensing elements than shown. In one particular embodiment, image sensor 226 may alternatively include 3000 rows of sensing elements, and each row may include 1000 sensing elements (extending across the page). In this embodiment, camera 208a may therefore be configured to output an image of 3000×1000 pixels. Furthermore, in this embodiment, camera 208a may be configured to capture images at a rate of 60 Hz. Other camera configuration parameters are also possible.

[0094] Note that the sizes, shapes, and positions of the various components of device 200 are not necessarily drawn to scale, but are shown for ease of description only. Figure 2BThe sizes of the lenses 230, 232, 234, 236 and sensors 226, 246, etc. shown in the drawings may be different from the sizes shown. In another example, the distance between the lens 230 and the image sensor 226 may be different from the distance shown. In one embodiment, the distance from the lens 230 to the sensor 226 may correspond to approximately two times the diameter of the lens 230. However, in other embodiments, the image sensor 226 and the camera lens 230 may have other sizes, shapes, and / or positions relative to each other.

[0095] Figure 2C is a conceptual illustration of the operation of device 200 according to an example embodiment. Figure 2C In the illustration of FIG. 2 , the sensing element of the image sensor 226 of the camera 208 a is in the plane of the page. Note that for ease of description, Figure 2C Some components of device 200, such as camera lens 230 and LIDAR 206, are omitted from the illustration.

[0096] In some embodiments, device 200 can be configured to operate cameras 208a, 208b, 208c, and / or 208d in a rolling shutter configuration to obtain a sequence of image pixels. Figure 2C In the scenario of FIG. 2 , a first row of sensing elements including sensing elements 228 a and 228 g may be configured to measure the amount of external light incident thereon during a first exposure time period. Device 200 may also include an analog-to-digital converter (not shown) that reads and converts the measured values ​​of the first row of sensing elements (after the first exposure time period has elapsed) for transmission to a controller of device 200 (e.g., controller 104). After a time delay from the start time of the first exposure time period (and optionally before the end of the first exposure time period), device 200 may begin exposing a second row of sensing elements including sensing elements 228 b and 228 h for a second exposure time period. Thus, in some examples, the exposure time periods of multiple rows of sensing elements may partially overlap (e.g., the time delay between the start time of the first exposure time period and the start time of the second exposure time period may be less than the first exposure time period, etc.). In this way, a camera in a rolling shutter configuration may stagger the start time of the exposure time periods to increase the image refresh rate (e.g., by simultaneously exposing multiple rows of sensing elements during the overlapping portion of their respective exposure time periods).

[0097] Continuing with this scenario, after the second exposure time period has passed, the device 200 can then similarly measure the measurements of the second row of sensing elements and send them to the controller. This process can then be repeated until all rows of sensing elements (i.e., a complete image frame) are scanned. For example, after the start time of the second exposure time period (and optionally before the second exposure time period has passed), the device 200 can begin exposing the third row of sensing elements (adjacent to the second row) to external light 270, and so on.

[0098] Furthermore, as described above, the device 200 may be configured to acquire a sequence of image pixels in an order similar to the order in which the light pulses were emitted by the LIDAR 206. By doing so, for example, more image pixels captured by the cameras 208a-d may overlap with the LIDAR data (e.g., detected reflections of the emitted light pulses) (both in time and in the viewing direction) compared to implementations in which the sequence of image pixels is acquired in a different order.

[0099] continue Figure 2C For example, when the LIDAR 206 is in a first pointing direction or orientation about the axis 242, the light beams 250a, 250b, 250c may correspond to Figure 2A 250. In this scenario, device 200 can be configured to scan a first (vertical) row of sensing elements (e.g., including elements 228a and 228g) before scanning sensing elements in a second (vertical) row (e.g., including elements 228b and 228h). By doing so, image pixels captured using the first row of sensing elements are more likely to match detected reflections of light beams 250a-250c in both time and viewing direction. In this scenario, LIDAR 206 can then rotate (e.g., counterclockwise) about axis 242 and emit light beams 252a-252c. Device 200 can then obtain a second row of image pixels using a second row of sensing elements (e.g., including sensing elements 228b and 228h), which are more likely to be aligned with detected reflections of light beams 252a-252c (in both time and viewing direction), and so on.

[0100] In some embodiments, the device 200 may also be configured to obtain a row of image pixels in a sequence based on the order in which the light pulses / light beams are emitted by the LIDAR 206. As a variation of the above scenario, for example, if the LIDAR 206 emits light beams 250a, 250b, 250c in this order, the device 200 may be configured to obtain the row of image pixels associated with the first row of sensing elements in a similar order (e.g., starting with sensing element 228a and ending with sensing element 228g). However, for example, if the LIDAR 206 emits light beams 250c, 250b, 250a in this order, the device 200 may instead be configured to obtain the row of image pixels in the opposite order (e.g., starting with sensing element 228g and ending with sensing element 228a).

[0101] Additionally, in some embodiments, device 200 can be configured to adjust the time delay between capturing subsequent rows of image pixels in a sequence of image pixels based on the rotation rate of LIDAR 206. For example, if LIDAR 206 increases its rotation rate (e.g., via actuator(s) 112, etc.), device 200 can reduce the time delay between acquiring a first row of image pixels associated with a first row of sensing elements (e.g., including sensing elements 228a and 228g) and acquiring a second row of image pixels associated with an adjacent second row of sensing elements. As described above, for example, the exposure start time associated with each row of sensing elements can depend on the order and time in which the corresponding image pixels are acquired, and thus adjusting the time delay can improve the degree of matching the image pixel capture time (and viewing direction) with the corresponding LIDAR pulse emission time (and / or detection of the corresponding reflection).

[0102] Figure 2D A top view of the device 200 is shown. Figure 2D , LIDAR 206 can have a first pointing direction corresponding to a 0° angular position about axis 242 (e.g., toward the bottom of the page). For example, in this configuration, LIDAR 206 can scan and use camera 208c (including lens 234) (at Figure 2B The area of ​​the surrounding environment corresponding to the center of the captured image (best shown in FIG. 1 ).

[0103] Figure 2E Another top view of the device 200 is shown. Figure 2E , LIDAR 206 may have a second pointing direction (e.g., toward the top of the page) corresponding to an angular position of 180° about axis 242. Figure 2D After rotating half a full circle about axis 242 in the first pointing direction, LIDAR 206 may have Figure 2EIn addition, in this configuration, for example, LIDAR 206 can scan and use camera 208a (including lens 230) (in Figure 2B The area of ​​the environment corresponding to the center of the captured image (best shown in FIG. 1 ).

[0104] In some scenarios, when LIDAR 206 rotates about axis 242, the time period during which the FOV of LIDAR 206 overlaps the FOV of camera 208a may be less than the exposure time period (and readout time period) suitable for capturing images using camera 208a.

[0105] In one example scenario, where the camera 208a operates in a rolling shutter configuration (e.g., multiple rows of sensing elements in the camera 208a are exposed according to different exposure start times), the image sensor 226 of the camera 208a may measure and output an image frame (i.e., pixel data from all sensing elements of the image sensor 226) within a period of 60 ms. Furthermore, in this scenario, the LIDAR 206 may be configured to rotate at a rotation rate of 10 Hz (i.e., one full revolution around the axis 242 every 100 ms). Thus, the LIDAR 206 may scan a FOV range that overlaps with the FOV of the camera 208a within a period of 100 ms / 4=25 ms (e.g., from t=37.5 ms to t=62.5 ms). To account for the difference between the scanning durations of the camera and the LIDAR, in some embodiments, the device 200 may be configured to synchronize the LIDAR light pulses emitted by the LIDAR 206 with image pixels captured by some, but not all, of the image sensing elements in the camera.

[0106] For example, device 200 can be configured to trigger a particular camera to capture an image such that when LIDAR 206 is pointing in a particular pointing direction aligned with a particular area of ​​the image, that particular area of ​​the image (e.g., (multiple) vertical rows of image pixels at or near the center of the image, etc.) is exposed to external light.

[0107] Continuing with the above scenario, for example (in the case where the LIDAR 206 rotates at a frequency of 10 Hz), at time t=0 ms, the LIDAR 206 (e.g. Figure 2D ) may have a first pointing direction (e.g., angular position about axis 242 = 0°). In addition, at time t = 50 ms, LIDAR 206 (e.g., Figure 2E 242) can have a second pointing direction (eg, angular position about axis 242 = 180°).

[0108] In this scenario, the device 200 may be configured to synchronize the center of the exposure period of the image sensor 226 (inside the camera 208a) with the time (e.g., t=50ms) at which the FOV of the LIDAR 206 overlaps with the center of the FOV of the camera 208a. For example, where the exposure period of the image sensor 226 is 60ms, then at time t=30ms, the center vertical row of sensing elements in the image sensor 226 may be exposed to external light. In this example, the camera 208a may trigger image capture at time t=50-30=20ms to align the exposure of the vertical row(s) of sensing elements near the center of the image sensor 226 with the LIDAR light pulses (in both the temporal and spatial domains) emitted when the LIDAR 206 is scanning the FOV corresponding to the center of the image (e.g., at t=50ms).

[0109] With this arrangement, image pixels near the center of the image (e.g., captured using a vertical row including sensing element 228c or a row including sensing element 228d) may be relatively more aligned (with respect to timing and viewing direction) with the LIDAR light pulses emitted / detected when those image pixels were measured. On the other hand, image pixels captured using multiple rows of sensing elements away from the center of the image sensor may be relatively misaligned (in time or viewing direction) with the LIDAR light pulses emitted / detected when those image pixels were measured. For example, the FOV of the rotating LIDAR may overlap with the camera FOV of camera 208a between times t=37.5ms and t=62.5ms. However, in the above scenario, camera 208a may begin exposing the image including sensing element 228a (at time t=20ms) (i.e., outside the time range where the LIDAR's FOV overlaps with the camera's FOV). Figure 2C A row of sensing elements (best shown in FIG. 1 ).

[0110] To mitigate this misalignment, in some examples, cameras 208a, 208b, 208c, 208d can be configured to have partially overlapping FOVs. Figure 2B208d) can be configured to have a FOV that partially overlaps the FOV of the adjacent camera 208a. In addition, the exposure time period associated with the central area of ​​the image captured using camera 208d can be synchronized with the time (e.g., t=25ms) when the LIDAR 206 is pointed at the FOV associated with the center of the image captured using camera 208d. Thus, in these examples, the device 200 (or other computer) can use the aligned image pixel data from camera 208d (e.g., image pixels near the center of the captured image) instead of the misaligned image pixel data captured using camera 208a (e.g., image pixels farther from the center of the image) to map with the LIDAR data.

[0111] Figure 3 FIG. 4 shows a cross-sectional view of another rolling shutter camera arrangement 308 (eg, a camera ring) according to an example embodiment. The camera ring 308 may be similar to Figure 2B 3. Camera ring 208 is shown. As shown, for example, axis 342 can be a rotational axis of a LIDAR similar to axis 242. Additionally, for example, image sensor 326 can be similar to image sensor 226 (and / or 246), and can include an array of sensing elements exemplified by sensing elements 328a-328e, which can be similar to sensing elements 228a-228f. For example, image sensor 326 can include a first row of sensing elements including sensing elements 328a arranged linearly (e.g., perpendicular to the page) and one or more other sensing elements (not shown), and an adjacent second row of sensing elements including sensing elements 328b arranged linearly (e.g., perpendicular to the page) and one or more other sensing elements (not shown).

[0112] Although not shown, camera ring 308 may also include one or more camera lenses (e.g., similar to camera lenses 230, 232, 234, 236, etc.) that focus portions of external light incident on camera ring 308 toward corresponding sensing elements in image sensor 326. Additionally or alternatively, camera ring 308 may include one or more components shown in any of system 100 and / or device 200.

[0113] As shown, the camera ring 308 includes an image sensor 326 arranged along a curved surface (e.g., a circular surface) around an axis 342. In one example, the image sensor 326 can be implemented on a flexible substrate (e.g., a flexible PCB, etc.) on which an arrangement of sensing elements (including sensing elements 328a-328e, etc.) is mounted. Therefore, with this arrangement, each row of sensing elements in the image sensor 326 can be at the same given distance from the rotation axis 342 (e.g., a circular or cylindrical arrangement of sensing elements). In another example, the image sensor 326 can be implemented as a plurality of physically separated rows of sensing elements arranged adjacent to each other around the rotation axis 342. For example, each physically separated row of sensing elements can be located at the same given distance from the rotation axis as other rows. Other examples are possible. Regardless of the implementation, in the configuration of the camera ring 308, the curved surface on which each row of sensing elements in the image sensor 326 is mounted can improve the overlap (e.g., in terms of viewing direction) between the image pixels captured by the sensing elements and the light beams emitted by the LIDAR sensor rotating around the axis 342.

[0114] For example, the viewpoint of the LIDAR device (e.g., the position of the LIDAR lens) may move along a circular path as the LIDAR sensor rotates about axis 342. Thus, with this arrangement, the curved surface of image sensor 326 may resemble the circular path of emitted / detected LIDAR light pulses to increase the likelihood that image pixels collected by sensor 326 will match LIDAR light pulses detected along a similar curved path in the horizontal direction of LIDAR sensor rotation.

[0115] II. Example mapping from point to rolling shutter image

[0116] In various applications, it may be beneficial to map one or more points in an environment (e.g., points detected by a LIDAR sensor of an autonomous vehicle relative to the autonomous vehicle) to (multiple) locations (e.g., individual pixels) within an image of a portion of the environment. This may be done to generate information about the shape, composition, content, or other attributes of the environment to facilitate navigation in the environment, inventory of the contents of the environment, manipulation of the contents of the environment, or to facilitate other goals. For example, an image may be used to identify the contents of the environment (e.g., to identify objects of interest). In such an example, mapping detected points in the environment (e.g., points of a LIDAR point cloud) to locations within the image representing the identified content may allow the location of the identified content within the environment to be determined.

[0117] This method for locating a point in the environment to a position in an image of a portion of the environment can be part of a sensor fusion algorithm. A sensor fusion algorithm can be employed to merge data from multiple sensors (such as, for example, an image sensor and a LIDAR sensor) to generate a representation of the scanned environment. For example, a 3D representation of the scanned environment can indicate color information determined using an image sensor combined with other information determined using a LIDAR sensor (e.g., distance, depth, intensity, texture, reflected light pulse length, etc.).

[0118] Various methods may be employed to map points in the environment to locations within an image of the environment. These methods typically use information about the location and orientation or "pose" within the environment of the camera that generated the image. In examples where the camera is in motion relative to the environment (e.g., where the camera is part of a car, unmanned aerial vehicle, or some other autonomous or otherwise configured vehicle), it is desirable to determine the pose of the camera within the environment at a particular time period when the camera generated the image.

[0119] Figure 4A An example point 410 is shown in the environment (e.g., a point that is part of a point cloud generated by operation of a LIDAR sensor). A camera is also present in the environment and is operable to image a portion of the environment (e.g., a portion containing the example point 410). The camera is in motion (the direction of motion is in the range of Figure 4A Indicated by the arrow labeled "Movement" in Figure 4A , positions of the cameras at a first point in time, a second point in time, and a third point in time are shown by a first camera 420a, a second camera 420b, and a third camera 420c.

[0120] exist Figure 4A , the direction from the camera to the example point 410 at each time point is shown by a dashed arrow. Since the camera is in motion, the direction from the camera to the example point 410 relative to the orientation of the camera changes over time. Accordingly, the position of the example point 410 within the environment image captured by the camera will depend on the time when the camera generates the image and / or the time when the camera generates the image portion representing the light received from the environment content at the example point 410.

[0121] Figure 4B yes Figure 4AConceptual illustration of sensing elements 428a-h (e.g., CMOS photosensors, pixels of a CCD sensor) of an image sensor 426 of a camera in motion is shown. The lens, aperture, and / or other elements of the camera project light received from the environment from different trajectories to corresponding different locations on the image sensor 426. Accordingly, an example point 410 will be represented within an image generated by the camera at a location that depends on when the camera sensed light from the example point 410. Thus, for example, if camera 420a generates an image by sensing light from the example point 410 at a first point in time, the example point 410 will be mapped to a first location 415a (e.g., a first pixel) within the image. In another example, if camera 420a generates an image by sensing light from the example point 410 at a second point in time, the example point 410 will be mapped to a second location 415b (e.g., a second pixel) within the image. In yet another example, if the camera 420c generates an image by sensing light from the example point 410 at a third point in time, the example point 410 will be mapped to a third position 415c (eg, a third pixel) within the image.

[0122] In examples where the camera acts according to a "global shutter" mode or otherwise operates to generate an entire image substantially simultaneously (e.g., during a single exposure period), locating a point in the environment (e.g., 410) within the image frame may be relatively simple. For example, the position and orientation of the camera at (multiple) points in time when the image was generated may be determined and used to project the point into the image.

[0123] However, the process of projecting the dots into the image may be more complex when different portions of the image are generated at correspondingly different points in time, for example if the camera is operated in a "rolling shutter" mode. In this mode of operation, each row or column of pixels (or other light-sensitive elements of the camera) is operated during correspondingly different exposure times to sense light from the environment.

[0124] For example, a first row of sensing elements including sensing elements 428a and 428g can be configured to measure the amount of external light incident thereon during a first exposure time period. The camera can also include an analog-to-digital converter (not shown) that reads and converts the measured values ​​of the first row of sensing elements (after the first exposure time period has elapsed) for transmission to a controller (e.g., controller 104). After a time delay from the start time of the first exposure time period (and optionally before the end of the first exposure time period), the camera can begin exposing a second row of sensing elements including sensing elements 528b and 528h for a second exposure time period.

[0125] Thus, different portions (e.g., rows / columns of pixels) of an image generated in this manner represent light sensed by the camera during corresponding different time periods. Accordingly, during the acquisition of each portion (row / column) of the image, the movement of the camera during image acquisition will result in the camera having a different pose relative to the environment. Thus, in order to project a point (e.g., 410) from the environment to a location within the image, it may be necessary to determine which of a plurality of different poses to use.

[0126] Various methods can perform such projection and / or pose selection. In some examples, a camera pose can be determined for each exposure time (e.g., for each row and / or column) and used to project a point to a corresponding potential location within the image. Based on multiple projections, a particular one can be selected as an estimated projection of the point into the image. For example, for each potential location, the difference between the time the pose was used to generate the projection and the time the camera imaged the row / column at the projected location within the image can be determined. In such an example, the projection corresponding to the lowest amplitude difference can be selected as the estimated projection of the point into the image.

[0127] However, this approach involves determining a large number of camera poses and using these poses to project points into image frames. This may be computationally expensive (e.g., in terms of processor cycles, memory usage, data bus usage), energetically expensive (e.g., in terms of system energy used to calculate poses, projections, or other calculations), expensive in terms of time used to generate output, or otherwise disadvantageous. Accordingly, this brute force approach may be undesirable or infeasible in applications such as autonomous vehicle mapping, positioning, and / or navigation applications (which are constrained with respect to power, computing resources, latency / time budgets, or other factors) and / or where such mapping must be performed for a large number of points (e.g., for each point in a point cloud generated by a LIDAR sensor).

[0128] Methods described herein allow for low-cost, low-latency determination of the position of corresponding points within an environment (e.g., points from a LIDAR point cloud) within an image of a portion of the environment. These methods achieve these improvements by formulating the position estimation problem as a time estimation problem. Specifically, these methods estimate the time at which a camera senses light from the point in the environment (e.g., a particular exposure time for a row / column of a rolling shutter image) while generating an image of the environment into which the point will be projected.

[0129] The method can be applied iteratively by formulating the problem as a time estimation problem. A cost function can be determined for each time estimate and used to update the time estimate (e.g., based on the magnitude and / or sign of the cost function, the magnitude and / or sign of the residual of the cost function). Such a cost function can include the difference between a first term related to the estimated time (e.g., a first term equal to the estimated time defined relative to a reference time) and a second term based on a mapping from a projected position within an image of a point in the environment to the time at which the camera senses light represented at the projected position within the image. The second term can be determined, for example, by (i) projecting a point in the environment to a position within an image captured by the camera using a determined camera pose at the estimated time, and (ii) determining the time at which the camera senses light from the environment at the projected position within the image (e.g., the time at which a row / column of pixels at the projected position is exposed).

[0130] In practice, convergence of the estimate occurs in relatively few iterations (e.g., two to three iterations), resulting in relatively little performance of pose extrapolation / interpolation, point projection, or other computationally expensive tasks. Accordingly, these methods allow points in the environment to be projected to corresponding locations within an image of a portion of the environment using relatively less computational resources, power, and / or time when compared to alternative (e.g., brute force) methods.

[0131] Cost function terms related to the estimated time and the term related to the time at which the camera senses light represented at the projected position within the image may be defined relative to different reference times / periods. For example, a first term related to the estimated time may be based on the estimated time defined relative to the time of the anchor pose of the camera, and a second term related to the time at which the camera senses light represented at the projected position within the image may be based on a characteristic time of image capture (e.g., the timing of a first exposure time, an intermediate exposure time, or an end exposure time of a series of rolling shutter exposure times, the exposure timing of a principal point within the image). In such an example, the cost function may include an additional term related to an offset between the difference between the time references of the first term and the second term of the cost function.

[0132] An example of such a cost function is:

[0133] t h -IndexToTime(Z n (t h ))+t offset

[0134] t h is the time t relative to the anchor pose used to determine the camera pose at a given estimated time pose The estimated time at which light from a point in the environment (e.g., 410) is imaged to generate a corresponding portion of the environment image, as defined by Z.n (t h ) is used when estimating time t h The estimated camera pose is the position of the point in the environment projected onto the position within the image. The function IndexToTime() takes as input the position within the frame of the rolling shutter image and outputs the characteristic time t relative to the image. principal_point The time at which the light represented by the pixel row or column at the input position is sensed (eg, the exposure time of the pixel row / column). offset Indicates t h and the time difference between the different zero times / epochs of IndexToTime(). Therefore, in this example, t offset Can be equal to t pose -t principal_point The static value of .

[0135] IndexToTime() may provide a discontinuous output, for example, an output time that is one of an enumerated set of output times corresponding to a set of exposure times for a rolling shutter operation of the camera. Alternatively, IndexToTime() may provide a continuous output, for example, a linearized time that represents a linearized version of an enumerated set of output times corresponding to a set of exposure times for a rolling shutter operation of the camera.

[0136] In order to determine the estimated time, an initial estimated time is determined, and a cost function is iteratively applied to generate an output estimated time. This may include performing an iterative process set number of times, for example, between 2 and 4 times (including 2 and 4 times). Alternatively, the absolute or relative (for example, relative to the amplitude of the most recent estimated time) reduction of the cost function from one iteration to the next iteration may be evaluated (for example, compared to a threshold value), and used to determine whether the estimated time has converged. This threshold-based update process may be constrained to occur no more than a set threshold number of times, for example, the update process may terminate due to cost function convergence (for example, the amplitude of the cost function being reduced is less than a threshold amount) or due to the update process having been executed a threshold number of times. Once the iterative process has terminated, the output estimated time may be used to determine the projected position of a point in the environment within the image. This may be performed by repeating some of the processes employed in the iterative time estimation process (for example, using the output estimated time to determine the posture of the camera at the output time, and using the determined posture to project a point in the environment to a position within the image).

[0137] Updating the estimated time based on the cost function may include applying the output of the cost function (e.g., a residual determined by evaluating the cost function) to the estimated time so as to update the estimated time. The output of the cost function may be applied directly, or may be normalized. For example, before applying the output of the cost function to update the estimated time, the output of the cost function may be normalized using the Jacobian matrix of the cost function relative to the estimated time. Such a Jacobian matrix may be determined analytically using the derivative of one or more terms of the cost function and / or by a process of numerical estimation.

[0138] can be calculated in various ways when used in the estimated time t h The estimated camera pose is the position of the point in the environment projected into the image (Z in the example cost function above). n (t h )). For example, Z n (t) can be the normalized position within the image frame. An example of such a calculation is:

[0139]

[0140]

[0141] is the pose of the camera at time t, is the point in the environment to be projected into the image, proj(x,y) projects point x into the frame of the view of the pose is the point in the environment to be projected into the image, and proj(x,Y) projects point x into the frame of the view of the pose Y.

[0142] The pose of the camera at a particular time can be determined based on one or more anchor poses and corresponding one or more time points. The anchor pose can be determined based on global positioning data, magnetometer data, image data, wheel rotation data, LIDAR data, or some other information related to the position of the camera. Such pose data can be determined for the vehicle as a whole and then transformed (e.g., by applying offset translation and rotation) to derive the pose of the camera. In some examples, where the time of the known anchor pose is close enough to the time of the desired camera pose, the anchor pose can be applied without modification. However, for improved accuracy, the anchor pose can be extrapolated and / or interpolated to the time of interest. For example, the pose of the camera at a particular time t It can be achieved by taking the data from the camera at different times t pose An example of such a calculation is:

[0143]

[0144] is the anchoring posture time t pose The extrapolation may be performed in various ways. Examples of such extrapolation are:

[0145]

[0146]

[0147] is the camera at anchor pose time t pose The direction, The camera is at t pose location, The camera is at t pose The translation speed, The camera is at t pose In some examples, it may be sufficient to extrapolate the pose based only on the camera's translation velocity, for example, in examples where the camera's rotation velocity is of relatively low magnitude.

[0148] Additionally or alternatively, the camera pose at a particular time may be determined by interpolating a plurality of different known camera poses at corresponding different known time points. This may include linear interpolation between two known poses (e.g., two known poses corresponding to corresponding times before and after the time of the desired interpolated pose, respectively), nonlinear interpolation using more than two poses, or some other interpolation process.

[0149] In some examples, the position of a point in the environment that is determined and / or detected may move over time. For example, the point may be a detected position of a moving object in the environment, and translation speed, rotation speed, or other motion information about the moving object may also be detected. In such examples, the embodiments described herein may be modified to take into account the movement of the point within the environment over time. For example, It can represent the position of a point in the environment over time (as a function of time t). This time-dependent can replace the static Such a time-dependent position can be determined in various ways, for example, by extrapolating the position of a point from the position and velocity of the point detected at a single detection time (e.g., based on the position and velocity of the point at time t detect The location of the detected point and speed Either by interpolating the point's position between two different positions detected at two different times, or by using some other method to estimate the point's position in the environment as a function of time.

[0150] III. Example methods and computer-readable media

[0151] Figure 5 5 is a flow diagram of a method 500 according to an example embodiment. The method 500 presents an embodiment of a method that can be used with, for example, any of the system 100, the device 200, and / or the camera ring 308. The method 500 may include one or more operations, functions, or actions as shown in one or more of the blocks 502-514. Although the blocks are shown in a sequential order, in some instances, the blocks may be executed in parallel and / or in an order different from that described herein. In addition, based on the desired implementation, various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed.

[0152] In addition, for method 500 and other processes and methods disclosed herein, the flowchart illustrates the functions and operations of one possible implementation of the current embodiment. In this regard, each box may represent a module, a segment, a portion of a manufacturing or operating process, or a portion of a program code, which includes one or more instructions that can be executed by a processor to implement a specific logical function or step in the process. The program code can be stored on any type of computer-readable medium, for example, such as a storage device including a disk or hard drive. Computer-readable media can include non-temporary computer-readable media, for example, such as computer-readable media that stores data for a short period of time, like register memory, processor cache, and random access memory (RAM). Computer-readable media can also include non-temporary media, such as secondary or permanent long-term storage devices, such as read-only memory (ROM), optical disks or disks, compact disk read-only memory (CD-ROM). Computer-readable media can also be any other volatile or non-volatile storage system. For example, a computer-readable medium can be considered to be a computer-readable storage medium, or a tangible storage device. In addition, for method 500 and other processes and methods disclosed herein, Figure 5 Each block in the may represent circuits that are wired to perform the specific logical functions in the process.

[0153] At block 502, method 500 involves obtaining an indication of a point in an environment of an autonomous vehicle. For example, LIDAR sensor 206 may be operated to generate a plurality of LIDAR points associated with a location, shape, size, or other information about one or more objects in the environment of LIDAR sensor 206, and an indication of the point in the environment may be determined based on the one or more LIDAR points.

[0154] At box 504, method 500 involves obtaining information about the position and motion of the autonomous vehicle within the environment. For example, information from GPS, GLONASS or other navigation positioning receivers, LIDAR data, wheel rate / rotation data, inertial data from one or more accelerometers and / or gyroscopes, magnetic field data from magnetometers, image data about the environment from one or more cameras, or other information related to the position of the autonomous vehicle can be used to generate information about the position and motion of the autonomous vehicle within the environment. This can include performing sensor fusion, applying filters (e.g., Kalman filters) to estimates of the motion / position of the autonomous vehicle, or other processes that determine accurate estimates of the position and motion of the autonomous vehicle within the environment.

[0155] At block 506, method 500 involves obtaining an image of a portion of an environment of an autonomous vehicle. The image includes a plurality of rows of pixels. The image is generated by a camera operating in a rolling shutter mode such that each row of pixels represents light sensed by the camera during a corresponding exposure period.

[0156] At block 508, method 500 involves mapping a point in the environment to a location within the image. Mapping a point in the environment to a location within the image involves determining, at block 510, an initial estimated time T0 at which the camera sensed light from the point in the environment. This may include setting the initial estimated time based on a characteristic time of generation of the image (e.g., the time when a rolling shutter process begins or ends, the time when a "middle" set of pixels of the image is exposed). Mapping a point in the environment to a location within the image also involves determining, at block 512, N updated estimated times, where N ≥ 1. Mapping a point in the environment to a location within the image also involves determining, at block 514, an updated estimated time based on the updated estimated time T0. N Determine the locations within the image that correspond to points in the environment.

[0157] Determining the estimated time for N updates involves determining the estimated time T for each update through the update process i The updating process includes determining at block 513a based on information about the position and motion of the autonomous vehicle the position of the camera at an estimated time T i-1 The updating process further includes in block 513b based on the camera at the estimated time T i-1 The pose of the autonomous vehicle projects a point in the environment to a projected position within the image. This may include determining the camera at the estimated time T by interpolating between the obtained position and motion of the autonomous vehicle and one or more additional positions and motions of the autonomous vehicle, extrapolating the obtained position and motion of the autonomous vehicle to the estimated time, or by performing some additional or alternative process. i-1 posture.

[0158] The updating process also includes evaluating a cost function at block 513c, the cost function including (i) i-1 The cost function may be based on (i) a term based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image. These terms may be defined relative to the same or different “zero” times or periods. For example, terms based on estimated times may be defined relative to the time of information about the position and motion of the autonomous vehicle, and terms based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image may be defined relative to a characteristic time of the generation of the image (e.g., the time at which a rolling shutter process begins or ends, the time at which a “middle” set of pixels of the image is exposed). In these examples where these terms are defined relative to different “zero” times or periods, the cost function may additionally include an offset term to compensate for these differences.

[0159] The updating process also includes determining an updated estimated time T based on the evaluated cost function at block 513d. i The updating process may be performed a predetermined number of times, such as two, three, or four times. Alternatively, the updating process may include performing the updating process until the time estimates converge, for example, until the absolute change or relative change in the magnitude of the time estimates from one update to the next is less than a specified threshold level.

[0160] In some embodiments, method 500 involves determining a three-dimensional (3D) representation of an environment based on position data (e.g., data from a LIDAR sensor) and image information from an image (e.g., one or more pixels of an image). For example, an example system may combine LIDAR-based information (e.g., distance to one or more objects in the environment, etc.) with camera-based information (e.g., color, etc.) to generate a 3D representation. Other types of representations based on a combination of position and image data (e.g., 2D images, images with labels, enhanced images indicating shadow or texture information, etc.) are possible. Thus, in some embodiments, method 500 involves determining a representation of an environment based on color information indicated by an image and point position information (e.g., distance, depth, texture, reflectivity, absorptivity, reflected light pulse length, etc.) indicated by, for example, a LIDAR sensor.

[0161] In a first example, the system of method 500 can determine depth information of one or more image pixels in an image based on a point in the environment. For example, the system can assign depth values ​​to image pixels in an image. In addition, for example, the system can generate a 3D object data model (e.g., 3D rendering) of one or more objects in the environment (e.g., a color 3D model indicating 3D features in the environment, etc.) (e.g., via display 140). In another instance, the image processing system can identify and distinguish multiple objects in an image by comparing the depth information (indicated by the associated position of the point in the environment) of the corresponding object. Other applications are possible. Therefore, in some embodiments, method 500 involves mapping LIDAR data points collected using a LIDAR sensor to image pixels collected using one or more cameras. For example, LIDAR data can be mapped to a coordinate system of an image output by an image sensor or camera.

[0162] In a second example, the system of method 500 may assign colors to points in the environment (based on data from one or more cameras) or other known points in the environment (e.g., to individual points in a LIDAR point cloud). The example system may then generate (e.g., via display 140) a 3D rendering of the scanned environment that indicates distances to features in the environment and color (e.g., a colored point cloud, etc.) information indicated by the image sensor(s) of the one or more cameras. Thus, in some embodiments, method 500 involves mapping image pixels from an image to corresponding LIDAR or otherwise generated location data points. For example, image pixel data may be mapped to the coordinate system of LIDAR data output by a LIDAR sensor.

[0163] Method 500 may include additional or alternative elements. For example, 500 may include operating an autonomous vehicle (e.g., steering a vehicle, actuating a throttle of a vehicle, controlling torque and / or rotational output of one or more wheels, a motor, an engine, or other elements of a vehicle) based on a mapping of points in the vehicle's environment to locations (e.g., pixels) within an image. This may include using the mapping to identify and / or locate one or more objects or other contents of the environment. Additionally or alternatively, this may include using the mapping to determine a navigation plan and / or determine commands for operating the autonomous vehicle.

[0164] IV. in conclusion

[0165] The specific arrangements shown in the accompanying drawings should not be considered restrictive. It should be understood that other embodiments may include more or less of each element shown in a given figure. In addition, some of the elements shown may be combined or omitted. In addition, exemplary embodiments may include elements not shown in the figures. In addition, although various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes, not for limitation, and their true scope and spirit are indicated by the appended claims. Without departing from the spirit or scope of the subject matter presented herein, other embodiments may be utilized, and other changes may be made. It will be readily understood that, as generally described herein and shown in the accompanying drawings, aspects of the present disclosure may be arranged, replaced, combined, separated and designed in a variety of different configurations.

[0166] V. Example Implementations

[0167]

[0168]

[0169]

[0170]

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178]

[0179]

[0180]

[0181]

[0182]

[0183]

[0184]

[0185]

[0186]

[0187]

[0188]

[0189]

[0190]

[0191]

[0192]

[0193]

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200]

[0201]

[0202]

[0203]

[0204]

Claims

1. A method for projecting a point onto an image, comprising: obtaining an indication of a point in an environment of the autonomous vehicle; obtaining information about the position and movement of the autonomous vehicle within the environment; obtaining an image of a portion of an environment of the autonomous vehicle, wherein the image comprises a plurality of rows of pixels, and wherein the image is generated by a camera operating in a rolling shutter mode such that each row of pixels represents light sensed by the camera during a corresponding exposure time period; and Points in the environment are mapped to locations within the image by: determining an initial estimated time T0 at which the camera senses light from a point in the environment; Determine the estimated time of N updates, where N ≥ 1, and where the estimated time of each update is T i Determined by an update process, the update process comprising: The camera is determined to be at an estimated time T based on information about the position and motion of the autonomous vehicle. i-1 The posture, Based on the camera's estimated time T i-1 The posture projects the point in the environment to the projection position in the image, The cost function is evaluated, the cost function comprising (i) based on the estimated time T i-1 Item, and (ii) a term based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image, and Determine the updated estimated time T based on the evaluated cost function i ;as well as Based on the updated estimated time T N A location within the image corresponding to a point in the environment is determined.

2. The method according to claim 1, further comprising: The autonomous vehicle is operated to navigate within the environment based on the position determined within the image.

3. The method according to claim 1, wherein: Obtaining an indication of a point in an environment of the autonomous vehicle comprises: operating a light detection and ranging (LIDAR) sensor of the autonomous vehicle to generate a plurality of LIDAR data points indicative of distances to one or more objects in an environment of the vehicle; and An indication of a point in the environment is generated based on at least one LIDAR data point in the plurality of LIDAR data points.

4. The method according to claim 1, wherein: Based on the updated estimated time T N Determining a location within the image that corresponds to a point in the environment includes identifying a particular pixel within the image that corresponds to a point in the environment.

5. The method according to claim 4, further comprising: determining a depth value for a point in the environment based on the indication of the point in the environment; as well as The depth value is assigned to the particular pixel.

6. The method according to claim 1, further comprising: The presence of a feature in the environment is identified based on the image, wherein the feature is represented at least in part by particular pixels within the image.

7. The method according to claim 1, further comprising: The evaluated cost function is normalized by the Jacobian matrix of the evaluated cost function, where the updated estimated time T is determined based on the evaluated cost function. i The updated estimated time T is determined based on the normalized evaluated cost function. i .

8. The method according to claim 1, wherein: Obtaining information about the position and movement of the autonomous vehicle within the environment includes obtaining information about a first position of the autonomous vehicle at a first point in time and a second position of the autonomous vehicle at a second point in time, and wherein determining the pose of the camera at the estimated time includes interpolating information about the position of the autonomous vehicle at at least two different points in time to the estimated time.

9. The method according to claim 1, wherein: Obtaining information about the position and motion of the autonomous vehicle within the environment includes obtaining information about the position and translational velocity of the autonomous vehicle at a particular point in time, and wherein determining the position of the camera at an estimated time T i-1 The pose includes information about the position and translational velocity of the autonomous vehicle at that particular point in time extrapolated to an estimated time T i-1 .

10. The method according to claim 1, wherein: The updating process further comprises determining an estimated time T for the update i With the estimated time T i-1 The difference is less than a threshold amount, wherein the update is based on the estimated time T in response to at least one of the following: N Determine the location within the image corresponding to the point in the environment: (i) Determine the updated estimated time T N With the estimated time T N-1 The difference is less than a threshold amount, or (ii) determining that the update process has been performed a threshold number of times.

11. A non-transitory computer-readable medium having stored therein instructions executable by a computing device to cause the computing device to perform operations, wherein: The operations include: obtaining an indication of a point in an environment of the autonomous vehicle; obtaining information about the position and movement of the autonomous vehicle within the environment; obtaining an image of a portion of an environment of the autonomous vehicle, wherein the image comprises a plurality of rows of pixels, and wherein the image is generated by a camera operating in a rolling shutter mode such that each row of pixels represents light sensed by the camera during a corresponding exposure time period; and Points in the environment are mapped to locations within the image by: determining an initial estimated time T0 at which the camera senses light from a point in the environment; Determine the estimated time of N updates, where N ≥ 1, and where the estimated time of each update is T i Determined by an update process, the update process comprising: The camera is determined to be at an estimated time T based on information about the position and motion of the autonomous vehicle. i-1 The posture, Based on the camera's estimated time T i-1 The posture projects the point in the environment to the projection position in the image, The cost function is evaluated, the cost function comprising: (i) based on the estimated time T i-1 and (ii) a term based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image, and Determine the updated estimated time T based on the evaluated cost function i ;as well as Based on the updated estimated time T N A location within the image corresponding to a point in the environment is determined.

12. The non-transitory computer readable medium of claim 11, wherein: The operations also include: The autonomous vehicle is operated to navigate within the environment based on the position determined within the image.

13. The non-transitory computer readable medium of claim 11, wherein: Based on the updated estimated time T N Determining a location within the image that corresponds to a point in the environment includes identifying a particular pixel within the image that corresponds to a point in the environment.

14. The non-transitory computer readable medium of claim 13, wherein: The operations also include: determining a depth value for a point in the environment based on the indication of the point in the environment; and The depth value is assigned to the particular pixel.

15. The non-transitory computer readable medium of claim 11, wherein: Obtaining information about the position and motion of the autonomous vehicle within the environment includes obtaining information about the position and translational velocity of the autonomous vehicle at a particular point in time, and wherein determining the position of the camera at an estimated time T i-1 The pose includes information about the position and translational velocity of the autonomous vehicle at that particular point in time extrapolated to an estimated time T i-1 .

16. A system for projecting a point onto an image, comprising: a light detection and ranging (LIDAR) sensor coupled to the vehicle; a camera coupled to the vehicle, wherein the camera is configured to obtain image data indicative of an environment of the vehicle; and A controller, wherein the controller is operably coupled to the LIDAR sensor and the camera, and wherein the controller includes one or more processors configured to perform operations comprising: operating the LIDAR sensor to generate a plurality of LIDAR data points indicative of distances to one or more objects in an environment of the vehicle; generating an indication of a point in the environment based on at least one LIDAR data point in the plurality of LIDAR data points; operating the camera in a rolling shutter mode to generate an image of a portion of an environment of the vehicle, wherein the image comprises a plurality of rows of pixels, and wherein each row of pixels represents light sensed by the camera during a corresponding exposure time period; obtaining information about the position and movement of the autonomous vehicle within the environment; and Points in the environment are mapped to locations within the image by: determining an initial estimated time T0 at which the camera senses light from a point in the environment; Determine the estimated time of N updates, where N ≥ 1, and where the estimated time of each update is T i Determined by an update process, the update process comprising: The camera is determined to be at an estimated time T based on information about the position and motion of the autonomous vehicle. i-1 The posture, Based on the camera's estimated time T i-1 The posture projects the point in the environment to the projection position in the image, The cost function is evaluated, the cost function comprising (i) based on the estimated time T i-1 and (ii) a term based on a mapping from the projection position to the time at which the camera senses light represented at the projection position within the image, and Determine the updated estimated time T based on the evaluated cost function i ;as well as Based on the updated estimated time T N A location within the image corresponding to a point in the environment is determined.

17. The system of claim 16, wherein: The operations also include: The autonomous vehicle is operated to navigate within the environment based on the position determined within the image.

18. The system of claim 16, wherein: Based on the updated estimated time T N Determining a location within the image that corresponds to a point in the environment includes identifying a particular pixel within the image that corresponds to a point in the environment.

19. The system of claim 18, wherein: The operations also include: determining a depth value for a point in the environment based on the indication of the point in the environment; and The depth value is assigned to the particular pixel.

20. The system of claim 16, wherein: Obtaining information about the position and motion of the autonomous vehicle within the environment includes obtaining information about the position and translational velocity of the autonomous vehicle at a particular point in time, and wherein determining the position of the camera at an estimated time T i-1 The pose includes information about the position and translational velocity of the autonomous vehicle at that particular point in time extrapolated to an estimated time T i-1 .

Images