Image scan line timestamp

Abstract

Techniques for adding temporal data to scan lines of an image frame are disclosed. In some examples, an image sensor may perform rolling shutter image capture to generate scan lines. In some examples, data captured by another sensor may be associated with at least a portion of the scan line based at least in part on temporal data added to the scan line. In addition, techniques for synchronizing data capture by multiple sensors are disclosed. For example, rolling shutter image capture performed by an image sensor may be synchronized with data capture performed by another sensor.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application number 201980063099.0, application date September 25, 2019, and invention name “Image Scan Line Timestamp”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This PCT international application claims priority to U.S. patent application No. 16 / 142,865, filed on September 26, 2018, entitled “IMAGE SCAN LINETIMESTAMPING,” and U.S. patent application No. 16 / 142,966, filed on September 26, 2018, entitled “MULTI-SENSOR DATA CAPTURE SYNCHRONIZATION,” the entire contents of both of which are incorporated by reference into this application. Background Art

[0004] Various methods, apparatus, and systems utilize image data from image sensors. For example, a robotic platform (e.g., an autonomous vehicle) may use image data and / or other sensor data to navigate through an environment that includes obstacles (e.g., other vehicles, buildings, pedestrians, etc.). In some examples, temporal information (e.g., a timestamp) may be associated with the entire image frame, which may present various challenges, such as inaccurate association of the capture of a particular portion of an image frame with a particular portion of data from another sensor due to a temporal difference between the capture of the particular portion of the image frame and the particular portion of data from another sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical components or features.

[0006] Figure 1 An example environment is shown that includes a vehicle having one or more time stamp components for adding time data to scan lines of an image frame (eg, rolling shutter image capture) according to examples of the present disclosure.

[0007] Figure 2 is a diagram showing an example of temporal data being added to scan lines of an image frame (eg, via a timestamp component) according to examples of the present disclosure.

[0008] Figure 3is a block diagram illustrating example components for implementing an example process flow including adding temporal data to a scan line and delaying and / or shifting the end of line data associated with the scan line according to examples of the present disclosure.

[0009] Figure 4 is a block diagram illustrating example components for implementing an example process flow including obtaining time data for associating with a scan line according to examples of the present disclosure.

[0010] Figure 5A is a diagram illustrating an example of synchronizing rolling shutter image capture of an image sensor field of view with LIDAR data capture of a LIDAR sensor field of view according to an example of the present disclosure.

[0011] Figure 5B is a diagram illustrating an example of synchronizing rolling shutter image capture of multiple image sensor fields of view with LIDAR data capture of a LIDAR sensor field of view according to an example of the present disclosure.

[0012] Figure 6 is a schematic block diagram illustrating an example of image processing components for correcting a distorted image using scan line time data according to an example of the present disclosure.

[0013] Figure 7 is a flow chart illustrating an example process for adding temporal data to one or more scan lines of an image frame according to examples of the present disclosure.

[0014] Figure 8 is a flow chart illustrating an example process for determining associations between data from multiple sensors based at least in part on temporal data added to scan lines of an image frame according to an example of the present disclosure.

[0015] Fig. 9 is a flow chart illustrating an example process for modifying an image to correct or compensate for distortion effects based at least in part on temporal data added to scan lines of an image frame according to an example of the present disclosure.

[0016] Fig.10 is a flow chart illustrating an example process for synchronizing rolling shutter image capture of a first field of view (associated with a first sensor) with data capture of a second field of view (associated with a second sensor), according to an example.

[0017] Fig.11 is a block diagram of an example system for implementing the techniques described herein according to an example of the present disclosure. DETAILED DESCRIPTION

[0018] In some systems, time information (e.g., a timestamp) may be associated with an entire image frame. Such time information may limit the system's ability to accurately associate the capture of a particular portion of an image frame with a particular portion of data from another sensor, for example, due to a time difference between the capture of a particular portion of an image frame and a particular portion of data from another sensor. In addition, in some examples, because a single timestamp is associated with the entire image, performing additional processing is inaccurate or otherwise not feasible (e.g., camera calibration, 3D reconstruction, tracking, feature extraction, etc.). The present disclosure is directed to techniques for adding time data to portions of an image at the time of capture (e.g., scan lines of an image frame during rolling shutter image capture). In addition, the present disclosure is directed to techniques for synchronizing data capture from multiple sensors. Among other things, the techniques discussed herein may enable specific portions of an image frame to be accurately aligned with specific portions of data from another sensor (e.g., another sensor modality, such as LIDAR and / or other image-based sensor systems).

[0019] In some examples, an image sensor of a vehicle (e.g., an autonomous vehicle) can capture an image of a scene within a first field of view of the image sensor. In some examples, image capture can be performed by scanning across the scene (e.g., vertically or horizontally) to capture a portion of the scene (e.g., a first scan line) before capturing another portion of the scene (e.g., a second scan line), such as in rolling shutter image capture. Rolling shutter image capture can produce scan lines of an image frame. The scan lines can include scan line data (e.g., pixel data) and a line data end. The line data end can indicate the end of the scan line.

[0020] In various examples, a computing system of a vehicle can receive scan lines and add corresponding time data (e.g., a timestamp) to one or more scan lines. For example, the computing system can add the time data to each scan line. In some examples, the time data can be added to the scan lines such that the time data is between the scan line data and the end of the line data. According to some examples, for example, the end of the line data associated with the scan line can be shifted from a first position to a second position to accommodate the addition of the time data.

[0021] In some examples, a LIDAR sensor of a vehicle may capture LIDAR data (e.g., LIDAR points) within a second field of view of the LIDAR sensor, which may at least partially overlap with a first field of view of the image sensor. According to some embodiments, a computing system may associate at least a portion of the LIDAR data with at least a portion of the scan line based at least in part on time data added to the scan line (and / or based at least in part on time data associated with a portion of the LIDAR data) and a timestamp associated with the acquisition of the LIDAR data. In addition, the computing system may associate (e.g., fuse) the image data with the LIDAR data based at least in part on the association of a portion of the LIDAR data with a portion of the scan line, thereby generating associated data. In some examples, the associated data may be used as input to control the movement of the vehicle.

[0022] In some embodiments, the computing system can modify the distorted image using the temporal data added to one or more scan lines to produce a corrected image. For example, the distorted image can include one or more distortion effects caused by motion associated with rolling shutter image capture. As non-limiting examples, the distortion effects can include wobble, skew, spatial aliasing, and / or temporal aliasing, etc. Modifying the distorted image can include correcting or compensating for the distortion effects.

[0023] According to some examples, the second field of view associated with the LIDAR sensor can be movable relative to the first field of view associated with the image sensor. In some examples, the second field of view can be rotatable (e.g., around one or more axes).

[0024] In some examples, the computing system may synchronize image data capture (performed by the image sensor) with LIDAR data capture (performed by the LIDAR sensor). For example, during a time period when the LIDAR sensor is capturing LIDAR data of at least a portion of a scene corresponding to the field of view of the image sensor, the computing system may trigger the image sensor to perform rolling shutter image capture of the scene. In various examples, the computing system may implement synchronization based on one or more synchronization conditions. As non-limiting examples, the synchronization conditions may include an amount of overlap between a first field of view (associated with the image sensor) and a second field of view (associated with the LIDAR sensor), and / or an overlap between a first specific portion of the first field of view and a second specific portion of the second field of view, etc. As a non-limiting example, the capture of image data may be triggered so that the center of the image is captured at substantially the same time that the rotating LIDAR sensor is aligned with the field of view of the image sensor.

[0025] In some examples, the computing system may determine a first orientation of a first field of view and / or a first posture (position and orientation) associated with an image sensor. In addition, the computing system may determine a second orientation of a second field of view and / or a second posture associated with a LIDAR sensor. In some examples, the corresponding orientations and / or corresponding postures associated with the image sensor and / or the LIDAR sensor may be tracked. According to some examples, the orientations and / or postures may be tracked relative to each other. The computing system may use the field of view orientation information and / or posture information as input for causing the image sensor to initiate rolling shutter image capture of the first field of view, for example, such that at least a first portion of the first field of view (associated with the image sensor) overlaps at least a second portion of the second field of view (associated with the LIDAR sensor) according to a synchronization condition. In at least some examples, such initiation may be timed so as to optimize (e.g., maximize) the overlap of the corresponding fields of view. In various examples, during rolling shutter image capture, the second field of view associated with the LIDAR sensor may move relative to the first field of view associated with the image sensor. As a non-limiting example, a rotating LIDAR sensor may be associated with a field of view that moves relative to a field of view associated with an image sensor.

[0026] The techniques discussed herein can improve the functionality of a computing device in a variety of ways. For example, the techniques discussed herein can include adding temporal data to individual scan lines of an image frame, which can allow a computing device to accurately align (or otherwise associate) pixels of an image frame with LIDAR points, for example, to enable accurate multi-modal sensor fusion, sensor calibration, 3D reconstruction, multi-modal calibration, etc. For example, by making such an association using temporal data associated with individual scan lines, a vehicle's computing system can generate a more accurate representation of the vehicle's environment than by using temporal data associated with the entire image frame. As a result, the vehicle's computing system may be able to improve its detection of objects (e.g., obstacles) and its trajectory and / or route planning, for example, to control the vehicle's movement to avoid collisions with obstacles.

[0027] As another example, by synchronizing sensor data capture using the techniques described herein, a computing system of a vehicle may be able to accurately associate image data with LIDAR data (e.g., temporally and / or spatially). In some implementations, synchronization may allow for a determination that data captured by an image sensor and a LIDAR sensor are associated with each other when the data is received, which may reduce the amount of processing required to determine an association between the data after the data is received by the computing system of the vehicle. These and other improvements to the functionality of computing systems are discussed herein.

[0028] The techniques described herein can be implemented in a variety of ways. Example implementations are provided below with reference to the following figures. Although discussed in the context of self-driving cars, the methods, devices, and systems described herein can be applied to a variety of systems (e.g., robotic platforms, manually driven vehicles, etc.) and are not limited to self-driving vehicles. In another example, these techniques can be utilized in an aviation or navigation context, or in any system that uses machine vision. In addition, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0029] Figure 1 An example environment 100 is shown that includes a vehicle 102 having one or more time stamp components for adding time data to scan lines of an image frame (e.g., a rolling shutter image capture) according to an example of the present disclosure. In some examples, the environment 100 may include the vehicle 102 having one or more time stamp components for adding time data to scan lines of an image frame (e.g., a rolling shutter image capture). Figure 2-11 One or more features, components, and / or functionality of the described examples.

[0030] In some examples, the vehicle 102 may be configured to travel across a surface 104 (e.g., a road surface) to, for example, transport people and / or cargo from a first location to a second location. For purposes of illustration, the vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration in 2013, which describes a vehicle capable of performing all safety-critical functions throughout a trip without the driver (or occupant) being expected to control the vehicle at any time. In that case, since the vehicle may be configured to control all functions from start to stop, including all parking functions, it may not be occupied. This is merely an example, and the systems and methods described herein may be incorporated into any vehicle and / or robot, including vehicles and / or robots ranging from vehicles that require manual control by a driver at all times to vehicles that are partially autonomously controlled, such that they can be autonomously controlled without driver attention or assistance under certain driving conditions (e.g., when operating on limited-access highways), but they require driver attention during other driving conditions (e.g., while operating on urban city streets in an urban area, or at least during certain parking functions).

[0031] In some examples, the vehicle 102 may be a car having four wheels and each wheel having a corresponding tire. Other types and configurations of vehicles may be considered, such as vans, sport utility vehicles, crossovers, trucks, buses, agricultural vehicles, and construction vehicles. The vehicle 102 may be powered by one or more internal combustion engines, one or more electric motors, hydrogen power, or any combination thereof. In addition, although the exemplary vehicle 102 may have four wheels, the systems and methods described herein may be incorporated into vehicles having fewer or more wheels, tires, and / or tracks. The exemplary vehicle 102 may have four-wheel steering and may operate with substantially the same performance characteristics in all directions, for example, such that when traveling in a first direction, the first end of the vehicle 102 may be at the front end of the vehicle 102, and when traveling in an opposite second direction, the first end may become the rear end of the vehicle 102. Similarly, when traveling in a second direction, the second end of the vehicle 102 may be the front end of the vehicle 102, and when traveling in an opposite first direction, the second end may become the rear end of the vehicle 102. These example features can facilitate greater maneuverability, for example, in tight spaces or crowded environments such as parking lots and urban areas.

[0032] According to various examples, the vehicle 102 may include a sensor 106. Further, in various examples, the vehicle 102 may include a vehicle computing system 108. The vehicle computing system 108 may include one or more components that enable the vehicle computing system 108 and / or the vehicle 102 to perform various operations. In various examples, the vehicle computing system 108 may include a timestamp component 110 and / or time data 112. As will be discussed in further detail herein, the vehicle computing system 108 may use the timestamp component 110 to add the time data 112 to one or more scan lines of an image frame.

[0033] According to some implementations, a sensor 106 of a vehicle 102 (e.g., an image sensor such as a rolling shutter image capture device) may capture an image of a scene within a field of view associated with the sensor 106. In some examples, the image capture may be performed by scanning across the scene (e.g., vertically or horizontally) to capture a portion of the scene (e.g., a first scan line) before capturing another portion of the scene (e.g., a second scan line), such as in rolling shutter image capture. Figure 1An example representation of a rolling shutter image capture 114 of a scene within a field of view of a sensor 106 of a vehicle 102 is included. As shown in this non-limiting example, the rolling shutter image capture 114 can be performed by scanning horizontally across the scene. The rolling shutter image capture can produce scan lines (e.g., scan line 1, scan line 2, ..., scan line n) of an image frame 116. The scan lines can include scan line data (e.g., pixel data 118). Additionally, in some examples, the scan lines can be associated with and / or can include line data ends (e.g., as shown in FIG. 1 ). Figure 2 ). In some cases, the end of line data associated with a particular scan line may indicate the end of the scan line. In various examples, the vehicle computing system 108 may use the timestamp component 110 to add time data 112 to one or more scan lines, for example, as described herein with reference to Figure 2-4 In some non-limiting examples, the time stamp component 110 may function to add time data 112 to each scan line of the image frame 116 .

[0034] In some examples, sensors 106 may include light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, ultrasonic transducers, sound navigation and ranging (SONAR) sensors, position sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement unit, accelerometer, magnetometer, gyroscope, etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.), wheel encoders, microphones, environmental sensors (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), etc. Sensors 106 may generate sensor data that may be utilized by a vehicle computing system 108 associated with vehicle 102.

[0035] Figure 2 is a diagram showing an example 200 of adding temporal data (eg, via timestamp component 110) to a scan line of an image frame according to an example of the present disclosure. In some examples, example 200 may include the example of the present disclosure. Figure 1 and Figure 3-11 One or more features, components, and / or functionality of the described examples.

[0036] In some examples, an image frame (e.g., as described above with reference to Figure 1 Scan lines (e.g., scan line 1, scan line 2, ..., scan line n) of the image frame 116 described above may be generated by rolling shutter image capture 114. For example, rolling shutter image capture 114 may be generated by an image sensor (e.g., as described above with reference to Figure 1 The described image sensor 106 of the vehicle 102 is performed.

[0037] like Figure 2 As shown, a scan line may include and / or be associated with scan line data 202 (e.g., including pixel data 118) and / or an end of line data 204 (e.g., a signal indicating the end of a scan line). For example, scan line 1 may include first scan line data and a first end of the line data, scan line 2 may include second scan line data and a second end of the line data, and so on. Figure 2 Scan line data 202 and the end of line data 204 are shown as different components of the scan line, but in various examples, the end of line data 204 may be considered a part of scan line data 202 .

[0038] According to some examples, the timestamp component 110 can receive one or more scan lines as input. Figure 2 As shown, the time stamp component 110 can add time data 112 (e.g., a time stamp) to the scan line and output a modified scan line including the time data 112. For example, the time stamp component 110 can add the first time data to scan line 1, the second time data to scan line 2, and so on. The first time data can be different from the second time data. For example, the first time data can be a first time stamp or otherwise indicate (e.g., by the time stamp component 110) the first time (and / or first time period) at which scan line 1 is generated and / or received. The second time data can be a second time stamp or otherwise indicate the second time (and / or second time period) at which scan line 2 is generated and / or received. In some cases, the time stamp component 110 can add the time data 112 to each scan line of the image frame. According to various examples, the time stamp component 110 can add the time data 112 to the scan line so that the time data 112 is between the scan line data 202 and the end of the line data 204 associated with the scan line. In some embodiments, the end of the line data 204 can be shifted, for example, to accommodate the time data 112. For example, the end of the line data 204 may be at a first position in (or relative to) the scan line before adding the time data 112. The time stamp component 110 may shift the end of the line data 204 from the first position to a second position. In the second position, the time data 112 may be between the scan line data 202 and the end of the line data 204. Furthermore, in the second position, the end of the line data 204 may be disposed at an end portion of the scan line and / or otherwise indicate the end of the scan line.

[0039] In some examples, although described as additional data appended to the end of each scan line, such end of line data 204 may not be stored at the end of each line. As a non-limiting example of this, a "width" may be associated with the amount of data used for each scan line. Such a width may be available, for example, in a header of an image frame. In such examples, although described as appending the end of line data 204 to the end of each scan line, it should be understood that no data may be written, but rather the associated width may be increased to account for the additional data in the associated temporal data.

[0040] In some examples, each scan line of an image frame can have the same exposure time. In other examples, one or more scan lines of an image frame can have a different exposure time than one or more other scan lines of the image frame. For example, scan line 1 can have a first exposure time, and scan line 2 can have a second exposure time different from the first exposure time.

[0041] Figure 3 is a block diagram illustrating an exemplary component 300 for implementing an exemplary process flow including adding time data to a scan line and shifting the end of line data associated with the scan line according to an example of the present disclosure. In some examples, the component 300 may include the Figure 1 , 2 and one or more features, components and / or functions of the examples described in and 4-11.

[0042] According to some examples, component 300 may include timestamp component 110, sensor 302 (e.g., image sensor), and / or memory 304. In some examples, timestamp component 110 may receive scan lines from sensor 302. In a non-limiting example, timestamp component 110 may receive a stream or sequence of scan lines from sensor 302 via one or more interfaces (e.g., mobile industry processor interface (MIPI)) and / or one or more other components (which are configured to transfer data (e.g., image data, scan lines, etc.) between sensor 302 and timestamp component 110).

[0043] The time stamp component 110 can add time data 112 to individual scan lines, for example, as described herein with reference to Figure 2 In some examples, adding the time data 112 to the scan line may include appending the time data 112 to the scan line data 202 for the scan line. Additionally, the timestamp component 110 may shift 306 the end of the line data 204 associated with the scan line from a first position to a second position, e.g., as described herein with reference to Figure 2In some examples, shifting the end of the line data 204 can include appending the end of the line data 204 to the time data 112. According to various examples, the timestamp component 110 can output a modified scan line including the time data 112, such as Figure 3 In some non-limiting examples, the modified scan lines may be transferred to the memory 304, for example, via direct memory access (DMA).

[0044] In various examples, there may be a delay between each scan line received from the sensor 302 (e.g., by the timestamp component 110). For example, the delay between receiving a first scan line data and a second scan line data may be associated with the exposure time of the second scan line data. The delay may be large enough so that the addition of time data (e.g., by the timestamp component 110) may not introduce memory overhead.

[0045] Figure 4 is a block diagram illustrating an example component 400 for implementing an example process flow including obtaining time data associated with a scan line according to an example of the present disclosure. In some examples, the component 400 may include the Figure 1-3 and Figure 5A-11 One or more features, components, and / or functionality of the described examples.

[0046] In some examples, assembly 400 may include timestamp assembly 110, image sensor 402, LIDAR sensor 404, system processor clock 406, counter 408, and / or precision time protocol (PTP) clock 410. Timestamp assembly 110 may receive image data (e.g., scan line 412). Additionally, in some embodiments, timestamp assembly 110 may receive time data (e.g., as described herein). Figure 1-3 The time data 112 described above is added to each scan line 412, and image data including the scan line with the time data 114 is output.

[0047] In various examples, the timestamp component 110 can determine the time data associated with the scan line. According to some non-limiting examples, the timestamp component 110 can use data from the counter 408 to determine the time data associated with the scan line. For example, the counter 408 can be subordinate to the system processor clock 406. In some examples, the system processor clock 406 can be a clock hosted by a system processor (not shown). In some examples, the image sensor 402 can receive data from the system processor clock 406 and / or can be controlled by the system processor. In some examples, the system processor clock can receive data from the PTP clock 410. In addition, the LIDAR sensor 404 can receive data from the PTP clock 410 in some examples.

[0048] According to some examples, the LIDAR sensor 404 can output LIDAR data (e.g., LIDAR points 416). In some examples, at least a portion of the time data added to the scan line 412 to produce the scan line with the time data 414 can be used to determine the association between at least a portion of the image data and at least a portion of the LIDAR data. For example, in some examples, the component 400 can include a sensor association component that receives the image data (output from the time stamp component 110) and the LIDAR data (output from the LIDAR sensor 404). In some embodiments, the sensor association component 418 can use the time data associated with the scan line to determine the association between at least a portion of the scan line and at least a portion of the LIDAR data. In addition, in some examples, the LIDAR data can be associated with a time stamp (e.g., for a rotating LIDAR, the columns can be associated with the same time stamp), and the sensor association component 418 can use the time data associated with the scan line and the time stamp associated with the LIDAR data to determine the association between a portion of the scan line and a portion of the LIDAR data.

[0049] As a non-limiting example, sensor association component 418 can associate at least a portion of the pixel data of the scan line with one or more LIDAR points 416 of the LIDAR data based at least in part on the temporal data of the scan line. In some cases, the association can indicate that a portion of the pixel data and the LIDAR point were captured substantially simultaneously (e.g., within a time period that satisfied a temporal threshold). In some examples, sensor association component 418 can associate (e.g., fuse) a portion of the image data with a portion of the LIDAR data to produce associated data 420. For example, the association can be based at least in part on an association between the portion of the pixel data and the LIDAR point. In some examples, associated data 420 can be used to control a vehicle (e.g., with reference to a vehicle). Figure 1 Described is the movement of vehicle 102).

[0050] Figure 5A is a diagram illustrating an example 500a of synchronizing rolling shutter image capture of an image sensor field of view with LIDAR data capture of a LIDAR sensor field of view according to an example of the present disclosure. In some examples, example 500a may include reference to Figure 1-4 and one or more features, components and / or functions of the examples described in 5B-11.

[0051] In some examples, example 500a may include image sensor 502, LIDAR sensor 504, and / or computing system 506 (e.g., see Figure 1108). Image sensor 502 may have a first field of view 508. LIDAR sensor 504 may have a second field of view 510. In various examples, the field of view may be associated with a portion of the environment sensed by the sensor at a given time. In some examples, computing system 506 may include sensor synchronization component 512, synchronization condition 514, timestamp component 110, and / or time data 112.

[0052] According to some examples, the second field of view 510 of the LIDAR sensor 504 can be movable relative to the first field of view 508 of the image sensor 502. In some examples, the second field of view 510 can be rotatable (e.g., about one or more axes). In some non-limiting examples, the second field of view 510 can be rotatable 360 ​​degrees. In some examples, the amount of rotation can be less than 360 degrees. Figure 5A As shown, the rotation 516 of the second field of view 510 is in a clockwise direction. However, it should be understood that in some examples, the rotation 516 can be in a counter-clockwise direction and / or can change direction.

[0053] In various examples, the sensor synchronization component 512 can facilitate synchronization of image data capture (e.g., performed by the image sensor 502) with LIDAR data capture (e.g., performed by the LIDAR sensor 504). For example, the sensor synchronization component 512 can trigger the image sensor 502 to perform rolling shutter image capture 518 of the scene during a time period when the LIDAR sensor 504 is capturing LIDAR data of at least a portion of the scene. By synchronizing sensor data capture in this manner, the computing system 506 may be able to accurately associate image data with LIDAR data (e.g., temporally and / or spatially). In various examples, the sensor synchronization component 512 can implement synchronization according to a synchronization condition 514. As non-limiting examples, the synchronization condition 514 can include an amount of overlap between the first field of view 508 and the second field of view 510, and / or an overlap between a first particular portion of the first field of view 508 and a second particular portion of the second field of view 510, and so on. In at least some examples, the synchronization can be such that a central scan line in a captured image corresponds to a LIDAR data capture that is substantially directed toward the field of view of the image sensor. In such examples, the amount of LIDAR data associated with the same area of ​​the environment as associated with the image data is optimized (eg, maximized).

[0054] In some examples, the sensor synchronization component 512 can determine a first orientation of a first field of view 508 of the image sensor 502. In addition, the sensor synchronization component 512 can determine a second orientation of a second field of view 510 of the LIDAR sensor 504. In some examples, the respective orientations of the image sensor 502 and / or the LIDAR sensor 504 can be tracked. According to some examples, the first orientation and the second orientation can be tracked relative to each other. The sensor synchronization component 512 can use the field of view orientation information (e.g., the first orientation, the second orientation, etc.) as input for causing the image sensor 502 to initiate a rolling shutter image capture 518 of the first field of view 508, for example, such that at least a first portion of the first field of view 508 overlaps at least a second portion of the second field of view 510 according to the synchronization condition 514. In various examples, the second field of view 510 can move relative to the first field of view 508 during the rolling shutter image capture 518.

[0055] According to some implementations, the sensor synchronization component 512 can time the rolling shutter image capture 518 such that at least a portion of the scan lines of an image frame associated with a scene (e.g., scan line 1, scan line 2, ..., scan line n) are captured while the LIDAR sensor 504 is capturing LIDAR data associated with at least a portion of the scene (e.g., at least a portion of the scene is within the second field of view 510 of the LIDAR sensor 504). In some examples, the sensor synchronization component 512 can time the rolling shutter image capture 518 such that a majority of the scan lines of an image frame are captured during a time period associated with the LIDAR sensor 504, such as Figure 5A As shown, the LIDAR sensor 504 captures a second field of view 510 of at least a portion of the scene. In some examples, the sensor synchronization component 512 can time the rolling shutter image capture 518 so that a particular scan line (or a group of scans) is captured in association with a particular portion of the second field of view 510. As a non-limiting example, the sensor synchronization component 512 can time the rolling shutter image capture 518 so that a central scan line (e.g., a center-most scan line, a center group of scan lines, etc.) can be captured in association with LIDAR data captured in a center portion of the second field of view 510.

[0056] In some examples, the sensor synchronization component 512 can determine a first pose of the image sensor 502. The first pose can include a first position and / or a first orientation (e.g., x, y, z position, roll, pitch, and / or yaw) associated with the image sensor 502. In addition, the sensor synchronization component 512 can determine a second pose of the LIDAR sensor 504. The second pose can include a second position and / or a second orientation (e.g., x, y, z position, roll, pitch, and / or yaw) associated with the LIDAR sensor 504. In some examples, the respective poses of the image sensor 502 and / or the LIDAR sensor 504 can be tracked. According to some examples, the first pose and the second pose can be tracked relative to each other. The sensor synchronization component 512 can use the pose information (e.g., the first pose, the second pose, etc.) as input for causing the image sensor 502 to initiate a rolling shutter image capture 518 of the first field of view 508, for example, such that at least a first portion of the first field of view 508 overlaps at least a second portion of the second field of view 510 according to the synchronization condition 514.

[0057] In some examples, the sensor synchronization component 512 can estimate the exposure time of the image frame of the rolling shutter image capture 518. For example, the exposure time can be estimated before performing the rolling shutter image capture 518. In some examples, the sensor synchronization component 512 can receive exposure time data associated with a previous rolling shutter image capture. The exposure time of the image frame of the rolling shutter image capture 518 can be estimated based at least in part on the exposure time data associated with the previous rolling shutter image capture (e.g., by calculating the time difference between the timestamps associated with the first scan line and the last scan line). According to various examples, the sensor synchronization component 512 can use the estimated exposure time as an input for causing the image sensor 502 to initiate the rolling shutter image capture 518 of the first field of view 508, for example, such that at least a first portion of the first field of view 508 overlaps with at least a second portion of the second field of view 510 according to the synchronization condition 514.

[0058] According to some examples, the sensor synchronization component 512 can use the time data associated with the one or more scan lines as an input to cause the image sensor 502 to initiate a rolling shutter image capture 518 of the first field of view 508, e.g., such that at least a first portion of the first field of view 508 overlaps with at least a second portion of the second field of view 510 according to the synchronization condition 514. In some examples, the sensor synchronization component 512 can cause the image sensor 502 to initiate the rolling shutter image capture 518 based at least in part on the time data associated with the one or more scan lines of a previous rolling shutter image capture. Additionally or alternatively, the sensor synchronization component 512 can cause the image sensor 502 to adjust an exposure time of at least one scan line of the rolling shutter image capture 518, e.g., based at least in part on the time data associated with the one or more scan lines of the rolling shutter image capture 518 and / or the time data associated with the one or more scan lines of a previous rolling shutter image capture.

[0059] Figure 5B is a schematic diagram illustrating an example 500b of synchronizing rolling shutter image capture of multiple image sensor fields of view with LIDAR data capture of a LIDAR sensor field of view according to an example of the present disclosure. In some examples, example 500b may include reference to Figure 1-5A and one or more features, components and / or functions of the examples described in 6-11.

[0060] In some examples, example 500b may include reference to Figure 5A The image sensor 502 (also referred to herein as the “first image sensor 502”), the LIDAR sensor 504, and / or the computing system 506 (in Figure 5B Additionally, example 500b may include a second image sensor 520 having a third field of view 522.

[0061] According to some examples, the second field of view 510 of the LIDAR sensor 504 can be movable relative to the first field of view 508 (of the first image sensor 502) and / or the third field of view 522 (of the second image sensor 520). In some examples, the second field of view 510 can be rotatable (e.g., around one or more axes). In some non-limiting examples, the second field of view 510 can be rotatable 360 ​​degrees. In some examples, the amount of rotation can be less than 360 degrees.

[0062] like Figure 5BAs shown, during the first time period, the second field of view 510 of the LIDAR sensor 504 may at least partially overlap with the first field of view 508 of the first image sensor 502. Additionally, during the second time period 526, the second field of view 510 of the LIDAR sensor 504 may at least partially overlap with the third field of view of the second image sensor 520. For example, the second field of view 510 of the LIDAR sensor 504 may have moved (e.g., via rotation 516) by a certain amount (e.g., based at least in part on the rotation speed of the second field of view 510) between the first time period 524 and the second time period 526.

[0063] In some examples, the sensor synchronization component 512 (eg, Figure 5A ) can facilitate synchronization of image data capture (e.g., by first image sensor 502 and / or second image sensor 520, etc.) with LIDAR data capture (e.g., by LIDAR sensor 504). Figure 5A As discussed, the sensor synchronization component 512 can trigger the first image sensor 502 to perform rolling shutter image capture 518 (also referred to herein as “first rolling shutter image capture 518”) of the first scene during a time period when the LIDAR sensor 504 is capturing LIDAR data of at least a portion of the first scene (e.g., during a first time period 524). Additionally or alternatively, the sensor synchronization component 512 can trigger the second image sensor 520 to perform second rolling shutter image capture 528 of the second scene during a time period when the LIDAR sensor 504 is capturing LIDAR data of at least a portion of the second scene (e.g., during a second time period 526). In various examples, the sensor synchronization component 512 can trigger the second image sensor 520 to perform second rolling shutter image capture 528 of the second scene based on the synchronization condition 514 (e.g., during a second time period 526). Figure 5A As non-limiting examples, the synchronization condition 514 may include a first central portion of the image frame (generated by the second rolling shutter image capture 528) being substantially aligned with a second central portion of the second field of view 510 (of the LIDAR sensor 504) and / or a majority of scan lines (e.g., of the image frame generated by the second rolling shutter image capture 528) captured during a second time period 526 associated with the second field of view 510 (of the LIDAR sensor 504), etc.

[0064] In some examples, the sensor synchronization component 512 can determine the orientation of the third field of view 522 of the second image sensor 520. In addition, the sensor synchronization component 512 can determine the orientation of the second field of view of the LIDAR sensor 504. In some examples, the respective orientations of the second image sensor 520 and / or the LIDAR sensor 504 can be tracked. According to some examples, the orientations can be tracked relative to each other. For example, the sensor synchronization component 512 can determine the orientation of the first field of view relative to the second field of view. The sensor synchronization component 512 can use the field of view orientation information as an input to cause the second image sensor 520 to initiate a second rolling shutter image capture 528 of the third field of view 522, for example, such that according to the synchronization condition 514, at least a portion of the third field of view 522 (of the second image sensor 520) overlaps with at least a portion of the second field of view 510 (of the LIDAR sensor 504). In various examples, during the second rolling shutter image capture 528 , the second field of view 510 (of the LIDAR sensor 504 ) may move relative to the third field of view 522 (of the second image sensor 520 ).

[0065] According to some implementations, the sensor synchronization component 512 can time the second rolling shutter image capture 528 so that at least a portion of a scan of an image frame associated with the second scene is captured while the LIDAR sensor 504 is capturing LIDAR data associated with at least a portion of the second scene (e.g., at least a portion of the second scene is within the second field of view 510 of the LIDAR sensor 504). In some examples, the sensor synchronization component 512 can time the second rolling shutter image capture 528 so that a majority of the scan lines of the image frame are captured within a time period associated with the second field of view 510 of the LIDAR sensor 504 capturing at least a portion of the second scene (e.g., within the second time period 526). In some examples, the sensor synchronization component 512 can time the second rolling shutter image capture 528 so that a particular scan line (or a group of scan lines) is captured in association with a particular portion of the second field of view 510 of the LIDAR sensor 504. As a non-limiting example, the sensor synchronization component 512 can time the second rolling shutter image capture 528 so that a center portion of a scan line (e.g., a center scan line, a center group of scan lines, etc.) can be captured in association with LIDAR data captured in a middle portion of the second field of view 510 of the LIDAR sensor 504.

[0066] In some examples, the sensor synchronization component 512 can determine a pose of the second image sensor 520. The pose of the second image sensor 520 can include a position and / or orientation (e.g., x, y, z position, roll, pitch, and / or yaw) associated with the second image sensor 520. In addition, the sensor synchronization component 512 can determine a pose of the LIDAR sensor 504. The pose of the LIDAR sensor 504 can include a position and / or orientation (e.g., x, y, z position, roll, pitch, and / or yaw) associated with the LIDAR sensor 504. In some examples, the respective poses of the second image sensor 520 and / or the LIDAR sensor 504 can be tracked. The sensor synchronization component 512 can use the pose information as an input for causing the second image sensor 520 to initiate a second rolling shutter image capture 528 of the third field of view 522, e.g., such that, according to the synchronization condition 514, at least a portion of the third field of view 522 (of the second image sensor 520) overlaps at least a portion of the second field of view 510 (of the LIDAR sensor 504).

[0067] In some examples, the sensor synchronization component 512 can estimate an exposure time of an image frame of the second rolling shutter image capture 528. For example, the exposure time can be estimated before performing the second rolling shutter image capture 528. In some examples, the sensor synchronization component 512 can receive exposure time data associated with a previous rolling shutter image capture of the second image sensor 520. The exposure time of the image frame of the second rolling shutter image capture 528 can be estimated based at least in part on the exposure time data associated with the previous rolling shutter image capture. According to various examples, the sensor synchronization component 512 can use the estimated exposure time as an input for causing the second image sensor 520 to initiate a second rolling shutter image capture 528 of the third field of view 522, e.g., such that, according to the synchronization condition 514, at least a portion of the third field of view 522 (of the second image sensor 520) overlaps with at least a portion of the second field of view 510 (of the LIDAR sensor 504).

[0068] According to some examples, the sensor synchronization component 512 can use the time data associated with the one or more scan lines as input to cause the second image sensor 520 to initiate a second rolling shutter image capture 528 of the third field of view 522, e.g., such that at least a portion of the third field of view 522 (of the second image sensor 520) overlaps with at least a portion of the second field of view 510 (of the LIDAR sensor 504) according to the synchronization condition 514. In some examples, the sensor synchronization component 512 can cause the second image sensor 520 to initiate the second rolling shutter image capture 528 based at least in part on the time data associated with the one or more scan lines of a previous rolling shutter image capture of the second image sensor 520. Additionally or alternatively, the sensor synchronization component 512 can cause the second image sensor 520 to adjust an exposure time of at least one scan line of the second rolling shutter image capture 528, e.g., based at least in part on the time data associated with the one or more scan lines of the second rolling shutter image capture 528 and / or the time data associated with the one or more scan lines of a previous rolling shutter image capture of the second image sensor 520.

[0069] In some examples, first time data (e.g., a first timestamp) associated with one or more scan lines (e.g., of an image frame generated by a rolling shutter image capture (e.g., first rolling shutter image capture 518 or second rolling shutter image capture 528, etc.)) can be determined based at least in part on time data from a system processor clock synchronized with a precision time protocol (PTP) clock, for example, as described herein with reference to Figure 4 Additionally or alternatively, second time data (e.g., a second timestamp) associated with the measurement captured by LIDAR sensor 504 can be determined based at least in part on the time data from the PTP clock. In some examples, first image sensor 502 and / or second image sensor 520 can initiate rolling shutter image capture based at least in part on the first timestamp and / or the second timestamp.

[0070] Figure 6 is a schematic block diagram illustrating an example 600 of an image processing component that uses scan line time data to correct a distorted image according to an example of the present disclosure. In some examples, example 600 may include the example of the present disclosure. Figure 1-5B and one or more features, components and / or functions of the examples described in 7-11.

[0071] According to some examples, the example 600 may include one or more image processing components 602. In some examples, the image processing component 602 may reside in the vehicle computing system 108 (hereinafter referred to as Figure 1 described), computing system 506 (referenced herein Figure 5A), vehicle computing device 1104 (referenced herein Fig.11 ) and / or computing device 1140 (described herein with reference to Fig.11 described).

[0072] In some examples, the image processing component 602 can be configured to perform image processing on the image. In example 600, the image processing component 602 receives a distorted image 604 (e.g., image data associated with the distorted image) as input. In some examples, the distorted image 604 can be associated with an image frame having time data, such as individual scan lines added by the timestamp component 110, which can also be used to perform image correction, sensor calibration, cross-modal calibration, 3D reconstruction, tracking, feature extraction, multimodal sensor fusion, etc. In various examples, the distorted image 604 may include one or more distortion effects, such as distortion effects caused by motion associated with rolling shutter image capture. As non-limiting examples, the distortion effects may include wobbling, skewing, spatial aliasing, and / or temporal aliasing. Because the image data may be obtained via rolling shutter image capture of an object, and the rolling shutter image capture may be performed by a motion image sensor, the distorted image 604 may include distortion effects. That is, each scan line may be associated with a corresponding portion of the object being imaged at one or more corresponding distances (eg, in view of relative motion between the image sensor and the object).

[0073] According to some examples, the image processing component 602 may receive as input distance data 606 associated with one or more scan lines of a rolling shutter image capture. As a non-limiting example, the distance data 606 may be associated with LIDAR data captured by a LIDAR sensor. The LIDAR sensor may capture LIDAR data associated with an object, and the LIDAR data may include distance data 606 representing a distance between the LIDAR sensor and the object at a given time. The distance data 606 may be used to determine the distance between the image sensor and the object at a given time. The image processing component 602 may modify the distorted image 604 based at least in part on the distance data 606 and / or the time data added to one or more scan lines of the image frame associated with the distorted image 604 to correct and / or compensate for the distortion effects of the distorted image 604. The image processing component 602 may output a corrected image 608 (e.g., image data associated with the corrected image). In some examples, because image processing component 602 has corrected and / or compensated for the distortion effects of distorted image 604 , the distortion of corrected image 608 may be less than that of distorted image 604 .

[0074] Figure 7is a flow chart illustrating an example process 700 for adding temporal data to one or more scan lines of an image frame according to an example of the present disclosure. In some examples, process 700 may include Figure 1-6 and 8-11 describe one or more features, components and / or functions of the examples.

[0075] At 702, process 700 can include capturing and / or receiving scan line data (e.g., first scan line data) associated with a scan line (e.g., a first scan line) of an image frame. For example, the first scan line data associated with the first scan line can be captured by an image sensor. In some examples, the image sensor can perform rolling shutter image capture that produces scan lines including the first scan line and one or more additional scan lines.

[0076] At 704, process 700 can include appending the time data to the scan line data. In some examples, for example, Figure 2 and 3 As shown, the first time data may be appended to the first scan line data. At 706, process 700 may include delaying a first end of line data associated with the first scan line data. In some examples, the first end of the line data may be delayed based at least in part on appending the first time data, e.g., Figure 2 and 3 The first end of the line data may indicate an end of the first scan line.

[0077] At 708, process 700 may include determining whether the image frame includes an additional scan line. If it is determined at 708 that the image frame includes an additional scan line, process 700 may include appending time data to the scan line data of the additional scan line (at 704) and delaying the end of the line data associated with the scan line data of the additional scan line (at 706). For example, it may be determined that the image frame includes a second scan line associated with second scan line data. The second time data may be appended to the second scan line data. In addition, the second end of the line data may be associated with the second scan line data and may be delayed based at least in part on appending the second time data to the second scan line data. The second end of the line data may indicate the end of the second line. In any example described herein, the end of the line data may not be appended. As such a non-limiting example, a scan line width may be defined such that an amount of data is associated with each scan line. In such an example, the width associated with the data and made available to the image interpreter may be increased to account for the additional data indicating the additional time data at the end of each scan line.

[0078] In some examples, the first scan line data can be associated with a first exposure time, and the second scan line data can be associated with a second exposure time different from the first exposure time. In other examples, the first exposure time and the second exposure time can be the same.

[0079] If it is determined at 708 that the image frame does not include additional scan lines, process 700 may include appending image frame time data to the image frame at 710. At 712, process 700 may include appending end of frame data to the image frame. The end of frame data may indicate the end of the image frame.

[0080] Figure 8 is a flow chart illustrating an example process 800 for determining associations between data from multiple sensors based at least in part on temporal data added to scan lines of an image frame according to an example of the present disclosure. In some examples, process 800 may include the following: Figure 1-7 and one or more features, components and / or functions of the examples described in 9-11.

[0081] At 802, process 800 may include receiving sensor data. For example, at 804, process 800 may include receiving first data from a first sensor (e.g., an image sensor). At 806, process 800 may include receiving second data from a second sensor (e.g., a LIDAR sensor). In some examples, the first sensor may have a first sensor modality, and the second sensor may have a second sensor modality different from the first sensor modality. In some non-limiting examples, the second sensor may include another image sensor, a LIDAR sensor, a radar sensor, a time-of-flight sensor, etc. In some examples, sensor data may be captured and / or received periodically, continuously (e.g., in near real time), and / or when a triggering event occurs (e.g., when motion is detected (at a certain speed), etc.).

[0082] At 808, process 800 may include determining first time data associated with the scan line. In some examples, the first time data may be determined based at least in part on data obtained from a counter and / or a clock. For example, the first time data may be obtained from a counter (e.g., as described herein with reference to Figure 4 The counter can be obtained from a clock belonging to a system processor (e.g., as described herein with reference to Figure 4 In some examples, the system processor clock can be derived from a PTP clock (e.g., a system processor clock 406 described herein). Figure 4 The PTP clock 410 described above receives data.

[0083] At 810, process 800 may include adding first temporal data to the scan line. For example, the first temporal data may be added to the scan line such that the first temporal data is between the scan line data and an end of the line data associated with the scan line, e.g., as Figure 2 and 3 In some examples, the line data end can be shifted from a first position to a second position, for example to accommodate the first time data between the end of the line data and the scan line data. The scan line data may include pixel data associated with pixels exposed to light during rolling shutter image capture of the scan line.

[0084] At 812, process 800 may include determining second time data associated with the measurement result. For example, the measurement result may include data captured by the second sensor. In some examples, the measurement result may be a LIDAR point captured by the LIDAR sensor. In a non-limiting example, the second time data may include the time (and / or time period) at which the measurement result is captured and / or received.

[0085] At 814, process 800 may include determining an association between a measurement result and at least a portion of a scan line. For example, the association may be determined based at least in part on the first time data and / or the second time data. In some examples, the measurement result and a portion of the scan line may be associated based at least in part on determining that a time difference between the first time data and the second time data satisfies a threshold time difference. For example, the threshold time difference may be set to a low value so that if the measurement result and a portion of the scan line are captured substantially simultaneously, they are associated with each other. In some examples, the association between the measurement result and a portion of the scan line may be mapped into a database. In some examples, the measurement result and a portion of the scan line may be associated with each other regardless of the time difference between the first time data and the second time data.

[0086] At 816, process 800 can include associating (e.g., fusing) at least a portion of the first data with at least a portion of the second data to produce associated data. For example, associating the data can be based at least in part on an association between the measurements and the scan lines. In some examples, at least a portion of the first data can be associated with at least a portion of the second data based at least in part on determining that a time difference between the first temporal data and the second temporal data satisfies a threshold time difference.

[0087] At 818, process 800 may include controlling the movement of a vehicle (e.g., an autonomous vehicle). For example, the movement of the vehicle may be controlled based at least in part on the associated data. As a non-limiting example, the associated data may indicate the location of an obstacle in the vehicle environment. A trajectory and / or route to avoid a collision between the vehicle and the obstacle may be determined based at least in part on the associated data. The vehicle may be controlled to move along the trajectory and / or route.

[0088] Fig. 9 is a flow chart illustrating an example process 900 for modifying an image to correct or compensate for distortion effects based at least in part on temporal data added to scan lines of an image frame according to an example of the present disclosure. In some examples, process 900 may include Figure 1-8 , 10 and 11 describe one or more features, components and / or functions of the examples.

[0089] At 902, process 900 can include obtaining scan line data from a rolling shutter image capture. For example, an image sensor can perform rolling shutter image capture. The rolling shutter image capture can generate scan lines of an image frame, for example, as described herein with reference to Figure 1 and Figure 2 as described.

[0090] At 904, process 900 can include adding corresponding temporal data to the scan lines. For example, first temporal data can be appended to first scan line data of a first scan line, second temporal data can be appended to second scan line data of a second scan line, and so on.

[0091] At 906, process 900 may include determining whether the image is distorted. If it is determined at 906 that the image is distorted, process 900 may include modifying at least a portion of the image to correct or compensate for the distortion effects (at 908), for example, as described herein with reference to Figure 6 In various examples, the distortion effects may be caused by motion associated with rolling shutter image capture (e.g., relative motion between the image sensor and objects in the environment). As non-limiting examples, the distortion effects may include wobbling, skewing, spatial aliasing, and / or temporal aliasing. In some examples, the image may be modified based at least in part on temporal data added to one or more scan lines.

[0092] At 910, process 900 may include controlling the movement of a vehicle (e.g., an autonomous vehicle). For example, the movement of the vehicle may be controlled based at least in part on the modified image. As a non-limiting example, the modified image may indicate the location of an obstacle in the vehicle environment. A trajectory and / or route to avoid a collision between the vehicle and the obstacle may be determined based at least in part on the modified image. The vehicle may be controlled to move along the trajectory and / or route.

[0093] If it is determined at 906 that the image is not distorted, process 900 may include controlling movement of the vehicle based at least in part on the image (at 910 ).

[0094] Fig.10 is a flow chart illustrating an example process 1000 for synchronizing rolling shutter image capture of a first field of view (associated with a first sensor) with data capture of a second field of view (associated with a second sensor) according to examples disclosed herein. In some examples, process 1000 may include the following: Figure 1-9 and 11. One or more features, components and / or functions of the examples described.

[0095] At 1002, process 1000 may include determining a sensor orientation, which may include a relative pose (position and / or orientation) of the sensor. For example, at 1004, process 1000 may include determining a first orientation of a first field of view associated with a first sensor. At 1006, process 1000 may include determining a second orientation of a second field of view associated with a second sensor. In some examples, the field of view may include a cone of view (e.g., a 3D field of view for capturing data by the sensor). In some examples, the first sensor may include an image sensor. Further, as non-limiting examples, the second sensor may include another image sensor, a LIDAR sensor, a radar sensor, and / or a time-of-flight sensor, etc.

[0096] According to some examples, the second field of view associated with the second sensor can be movable relative to the first field of view associated with the first sensor. In some examples, the second field of view can be rotatable, for example, as described herein with reference to Figure 5A and 5B as described.

[0097] In some examples, determining the sensor orientation (at 1002) can include determining a first posture of a first sensor and / or determining a second posture of a second sensor. The first posture can include a first position and / or a first orientation (e.g., x, y, z position, roll, pitch, and / or yaw) associated with the first sensor. The second posture can include a second position and / or a second orientation (e.g., x, y, z position, roll, pitch, and / or yaw) associated with the second sensor. In some examples, the corresponding postures of the first sensor and / or the second sensor can be tracked. According to some examples, the first posture and the second posture can be tracked relative to each other.

[0098] At 1008, process 1000 may include determining whether to initiate rolling shutter image capture according to one or more synchronization conditions. If it is determined at 1008 that rolling shutter image capture is initiated according to the synchronization conditions, process 1000 may include causing the first sensor to initiate rolling shutter image capture of the first field of view such that, according to the synchronization conditions, at least a first portion of the first field of view overlaps with at least a second portion of the second field of view (at 1010). As non-limiting examples, the synchronization conditions may include an amount of overlap between the first field of view and the second field of view, and / or an overlap between a first particular portion of the first field of view and a second particular portion of the second field of view, and so on.

[0099] According to some embodiments, rolling shutter image capture can be timed such that at least a portion of scan lines of an image frame associated with a scene is captured by a first sensor at the same time as a second sensor is capturing other data associated with at least a portion of the scene (e.g., at least a portion of the scene is within a second field of view associated with the second sensor). In some examples, rolling shutter image capture can be timed such that a majority of scan lines of an image frame are captured within a time period associated with a second sensor, which captures a second field of view of at least a portion of the scene. In some examples, rolling shutter image capture can be timed such that a particular scan line (or group of scan lines) is captured in association with a particular portion of the second field of view. As a non-limiting example, rolling shutter image capture can be timed such that a central portion of scan lines (e.g., a central scan line, a central group of scan lines, etc.) can be captured in association with data captured by the second sensor in a middle portion of the second field of view.

[0100] In some examples, orientation information (e.g., a first orientation, a second orientation, etc.) can be used as input for causing the first sensor to initiate rolling shutter image capture of the first field of view, e.g., such that at least a first portion of the first field of view overlaps at least a second portion of the second field of view according to a synchronization condition. In various examples, during the rolling shutter image capture, the second field of view can move relative to the first field of view.

[0101] In some examples, gesture information (e.g., a first gesture, a second gesture, etc.) can be used as input for causing a first sensor to initiate rolling shutter image capture of a first field of view, such as such that, according to a synchronization condition, at least a first portion of the first field of view overlaps with at least a second portion of the second field of view.

[0102] In some examples, an exposure time of an image frame of a rolling shutter image capture can be estimated. For example, the exposure time can be estimated before performing the rolling shutter image capture. In some examples, the exposure time of the image frame of the rolling shutter image capture can be estimated based at least in part on exposure time data associated with a previous rolling shutter image capture. According to various examples, the estimated exposure time can be used as an input for causing a first sensor to initiate rolling shutter image capture of a first field of view, for example, such that at least a first portion of the first field of view overlaps with at least a second portion of the second field of view according to a synchronization condition.

[0103] According to some examples, time data associated with one or more scan lines can be used as input for causing the first sensor to initiate a rolling shutter image capture of the first field of view, e.g., such that at least a first portion of the first field of view overlaps with at least a second portion of the second field of view according to a synchronization condition. In some examples, the first sensor can be instructed to initiate a rolling shutter image capture based at least in part on the time data associated with one or more scan lines of a previous rolling shutter image capture. Additionally or alternatively, the first sensor can be instructed to adjust an exposure time of at least one scan line of a rolling shutter image capture, e.g., based at least in part on the time data associated with one or more scan lines of the rolling shutter image capture and / or time data related to one or more scan lines of a previous rolling shutter image capture.

[0104] If it is determined at 1008 that rolling shutter image capture according to a synchronization condition is not initiated, process 1000 can continue to determine the sensor orientation (at 1002) and / or determine whether to initiate rolling shutter image capture according to a synchronization condition (at 1008). In various examples, the sensor orientation can be determined periodically and / or continuously (e.g., in near real time).

[0105] Fig.11 is a block diagram of an example system 1100 for implementing the techniques described herein according to an example of the present disclosure. In some examples, the system 1100 may include the Figure 1-10 One or more features, components, and / or functionality of the described examples.

[0106] In some examples, system 1100 may include a vehicle 1102 (e.g., as described herein with reference to Figure 1 The vehicle 1102 may include a vehicle computing device 1104, one or more sensor systems 1106, one or more transmitters 1108, one or more communication connections 1110, at least one direct connection 1112, and one or more drive modules 1114.

[0107] The vehicle computing device 1104 may include one or more processors 1116 and a memory 1118 communicatively coupled to the one or more processors 1116. In the example shown, the vehicle 1102 is an autonomous vehicle; however, the vehicle 1102 may be any other type of vehicle, or any other system having at least an image capture device (e.g., a smartphone with camera functionality). In the example shown, the memory 1118 of the vehicle computing device 1104 stores a positioning component 1120, a perception component 1122, a planning component 1124, one or more system controllers 1126, a timestamp component 110, a sensor association component (418), a sensor synchronization component 512, and / or an image processing component 602. Although in the example shown, the vehicle 1102 is an autonomous vehicle; however, the vehicle 1102 may be any other type of vehicle, or any other system having at least an image capture device (e.g., a smartphone with camera functionality). Fig.11 1102 for illustrative purposes as residing in memory 1118, it is contemplated that positioning component 1120, perception component 1122, planning component 1124, one or more system controllers 1126, timestamp component 110, sensor association component 418, sensor synchronization component 512, and / or image processing component 602 may additionally or alternatively be accessible to vehicle 1102 (e.g., stored on or otherwise accessible to memory remote from vehicle 1102). In some cases, vehicle computing device 1104 may correspond to Figure 1 The vehicle computing system 108 .

[0108] In at least one example, the positioning component 1120 may include receiving data from the sensor system 1106 to determine the position and / or orientation (e.g., x, y, z position, roll, pitch, or yaw) of the vehicle 1102. For example, the positioning component 1120 may include and / or request / receive a map of the environment, and may continuously determine the position and / or orientation of the autonomous vehicle within the map. In some cases, the positioning component 1120 may utilize SLAM (simultaneous localization and mapping), CLAMS (simultaneous calibration, localization, and mapping), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. to receive image data, LIDAR data, radar data, IMU data, GPS data, wheel encoder data, etc. to accurately determine the position of the autonomous vehicle. In some cases, the positioning component 1120 may provide data to various components of the vehicle 1102 to determine an initial position of the autonomous vehicle for generating a trajectory and / or determining to retrieve map data including an occlusion grid from a memory, as discussed herein.

[0109] In some cases, the perception component 1122 may include functionality to perform object detection, segmentation, and / or classification. In some examples, the perception component 1122 may provide processed sensor data indicating the presence of an entity adjacent to the vehicle 1102 and / or the classification of the entity as an entity type (e.g., car, pedestrian, bicycle, animal, building, tree, pavement, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perception component 1122 may provide processed sensor data indicating one or more characteristics associated with a detected entity (e.g., a tracked object) and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), entity velocity, entity acceleration, entity range (size), etc. Characteristics associated with the environment may include, but are not limited to, the presence of another entity in the environment, the status of another entity in the environment, the time of day, the day of the week, the season, weather conditions, darkness / light indications, and the like.

[0110] Typically, the planning component 1124 can determine a path for the vehicle 1102 to follow through the environment. For example, the planning component 1124 can determine various routes and trajectories and various levels of detail. For example, the planning component 1124 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For ease of discussion, a route can be a sequence of waypoints for traveling between two locations. As a non-limiting example, waypoints include streets, intersections, global positioning system (GPS) coordinates, etc. In addition, the planning component 1124 can generate an instruction for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 1124 can determine how to guide the autonomous vehicle from a first waypoint in a sequence of waypoints to a second waypoint in a sequence of waypoints. In some examples, the instruction can be a trajectory or a portion of a trajectory. In some examples, multiple trajectories can be generated substantially simultaneously (e.g., within a technical tolerance) according to a back-off level technology, wherein one of the multiple trajectories is selected for navigation by the vehicle 1102.

[0111] In some examples, planning component 1124 can use temporal logic, such as linear temporal logic and / or signal temporal logic, to evaluate one or more trajectories of vehicle 1102. Details of utilizing temporal logic in planning component 1124 are discussed in U.S. Application Serial No. 15 / 632,147, the entire contents of which are incorporated herein by reference.

[0112] In at least one example, the vehicle computing device 1104 may include one or more system controllers 1126 that may be configured to control steering, propulsion, braking, safety, transmitter, communication, and other systems of the vehicle 1102. These system controllers 1126 may communicate with and / or control corresponding systems of the drive module 1114 and / or other components of the vehicle 1102.

[0113] The memory 1118 may further include one or more maps (not shown) that can be used by the vehicle 1102 to navigate within the environment. For purposes of discussion, a map may be any number of data structures modeled in two, three, or N dimensions that can provide information about the environment, such as, but not limited to, topology (e.g., intersections), streets, mountains, roads, terrain, and general environment. In some cases, a map may include, but is not limited to: texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.); spatial information (e.g., image data projected onto a grid, individual "surfels" (e.g., polygons associated with individual colors and / or intensities), reflectivity information (e.g., specular reflectivity information, retroreflectivity information, BRDF information, BSSRDF information, etc.). In one example, a map may include a three-dimensional grid of the environment. Grid. In some cases, the map can be stored in a tile format so that individual tiles of the map represent discrete portions of the environment and can be loaded into working memory as needed. In at least one example, the one or more maps may include at least one map (e.g., an image and / or a grid). In some examples, the vehicle 1102 can be controlled based at least in part on the map. That is, the map can be used in conjunction with the positioning component 1120, the perception component 1122, and / or the planning component 1124 to determine the location of the vehicle 1102, identify objects in the environment, and / or generate routes and / or trajectories for navigating within the environment.

[0114] In some examples, one or more maps may be stored on a remote computing device (e.g., computing device 1140) accessible via network 1142. In some examples, multiple maps may be stored based on characteristics (e.g., type of entity, time of day, day of week, season of year, etc.), for example. Storing multiple maps may have similar memory requirements, but increases the speed of accessing data in the maps.

[0115] In some cases, aspects of some or all of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, in some cases, the components in memory 1118 (and memory 1146 discussed below) may be implemented as a neural network.

[0116] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to produce an output. Each layer in a neural network may also contain another neural network, or may contain any number of layers (whether convolutional or not). As can be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a large class of algorithms in which an output is generated based on learned parameters.

[0117] Although discussed in the context of neural networks, any type of machine learning may be used in accordance with the present disclosure. For example, machine learning algorithms may include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimated scatter plot smoothing (LOESS), example-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomizer 3 (ID3), chi-squared automatic interaction detection (CHAID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), association rules, learning algorithms (e.g., perceptron, back-propagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Samon map, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), hybrid discriminant analysis (MDA), quadratic discriminant analysis (QDA), elastic discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (hybrid), gradient boosting machine (GBM), gradient boosting regression tree (GBRT), random forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0118] Additional examples of architectures include neural networks, e.g., ResNet70, ResNet101, VGG, DenseNet, PointNet, etc.

[0119] In at least one example, the sensor system 1106 may include a LIDAR sensor, a RADAR sensor, an ultrasonic transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., an inertial measurement unit (IMU)), an accelerometer, a magnetometer, a gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, etc.), a camera (e.g., RGB, IR, intensity, depth, time of flight, etc.), a microphone, a wheel encoder, an environmental sensor (e.g., a temperature sensor, a humidity sensor, a light sensor, a pressure sensor, etc.), etc. The sensor system 1106 may include multiple examples of each of these or other types of sensors. For example, the LIDAR sensor may include individual LIDAR sensors located at the corners, front, back, sides, and / or top of the vehicle 1102. As another example, the camera sensor may include multiple cameras disposed at various locations around the exterior and / or interior of the vehicle 1102. The sensor system 1106 may provide input to the vehicle computing device 1104. Additionally or alternatively, the sensor system 1106 may transmit sensor data to one or more computing devices via the one or more networks 1146 at a specific frequency, after a predetermined period of time, in near real time, etc. In some cases, the sensor system 1106 may correspond to Figure 1 Sensor 106.

[0120] The vehicle 1102 may also include one or more emitters 1108 for emitting light and / or sound, as described above. In this example, the emitters 1108 include internal audio and visual emitters to communicate with passengers of the vehicle 1102. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, tactile emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.), etc. In this example, the emitters 1108 also include external emitters. By way of example and not limitation, in this example, the external emitters include lights for indicating a direction of travel signal or other indicators of vehicle action (e.g., indicator lights, signs, light arrays, etc.), as well as one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) to audibly communicate with pedestrians or other nearby vehicles, one or more of the audio emitters including acoustic beam steering technology.

[0121] The vehicle 1102 may also include one or more communication connections 1110 that enable communication between the vehicle 1102 and one or more other local or remote computing devices. For example, the communication connection 1110 may facilitate communication with other local computing devices on the vehicle 1102 and / or the driver module 1114. Moreover, the communication connection 1110 may allow the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). The communication connection 1110 also enables the vehicle 1102 to communicate with a remote teleoperation computing device or other remote service.

[0122] The communication connection 1110 may include a physical and / or logical interface for connecting the vehicle computing device 1104 to another computing device or a network (e.g., the network 1142). For example, the communication connection 1110 may implement Wi-Fi-based communication, such as via frequencies defined by the IEEE 802.11 standard, such as short-range wireless frequencies such as cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that enables the corresponding computing device to interface with other computing devices.

[0123] In at least one example, the vehicle 1102 may include one or more drive modules 1114. In some examples, the vehicle 1102 may have a single drive module 1114. In at least one example, if the vehicle 1102 has multiple drive modules 1114, the individual drive modules 1114 may be positioned on opposite ends (e.g., front and rear, etc.) of the vehicle 1102. In at least one example, the drive module 1114 may include one or more sensor systems to detect conditions of the drive module 1114 and / or the surrounding environment of the vehicle 1102. As an example and not a limitation, the sensor system may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors, for acoustically detecting objects around the drive module, LIDAR sensors, RADAR sensors, etc. Some sensors (e.g., wheel encoders) may be unique to the drive module 1114. In some cases, the sensor system on driver module 1114 may overlap or supplement a corresponding system of vehicle 1102 (eg, sensor system 1106 ).

[0124] The drive module 1114 may include many vehicle systems, including a high-voltage battery, an electric motor for driving the vehicle, an inverter for converting direct current from the battery into alternating current for use by other vehicle systems, a steering system (which includes a steering motor and a steering rack (which may be electric)), a braking system including a hydraulic or electric actuator, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate traction loss and maintain control, an HVAC system, lighting (e.g., lighting for illuminating the vehicle's external environment, such as front / tail lights), and one or more other systems (e.g., cooling system, safety system, on-board charging system, other electrical components, such as DC / DC converters, high-voltage junctions, high-voltage cables, charging systems, charging ports, etc.). Additionally, the drive module 1114 may include a drive module controller that may receive and pre-process data from a sensor system and control the operation of various vehicle systems. In some examples, the drive module controller may include one or more processors and a memory coupled to the one or more processors in a communicative manner. The memory may store one or more modules to perform various functions of the drive module 1114. Additionally, the driver modules 1114 also include one or more communication connections that enable communication by the respective driver module with one or more other local or remote computing devices.

[0125] In at least one example, direct connection 1112 can provide a physical interface to couple drive module 1114 to the body of vehicle 1102. For example, direct connection 1112 can allow for the transfer of energy, fluid, air, data, etc. between drive module 1114 and the vehicle. In some cases, direct connection 1112 can further releasably secure drive module 1114 to the body of vehicle 1102.

[0126] In at least one example, the components discussed herein can process the sensor data as described above, and can send their respective outputs to the one or more computing devices 1140 via the one or more networks 1142. In at least one example, the components discussed herein can send their respective outputs to the one or more computing devices 1140 at a particular frequency, after a predetermined period of time, in near real time, and the like.

[0127] In some examples, vehicle 1102 may send sensor data to one or more computing devices 114 via network 1142. In some examples, vehicle 1102 may send raw sensor data to computing device 1140. In other examples, vehicle 1102 may send processed sensor data and / or representations of sensor data to computing device 1140. In some examples, vehicle 1102 may send sensor data to computing device 1140 at a particular frequency, after a predetermined period of time has passed, in near real time, etc. In some cases, vehicle 1102 may send sensor data (raw or processed) to computing device 1140 as one or more log files.

[0128] The computing device 1140 can include a processor 1144 and a memory 1146 that stores a map component 1148 , a timestamp component 110 , a sensor association component 418 , a sensor synchronization component 512 , and / or an image processing component 602 .

[0129] The processor 1116 of the vehicle 1102 and the processor 1144 of the computing device 1140 may be any suitable processor capable of executing instructions to process data and perform the operations described herein. By way of example and not limitation, the processors 1116 and 1144 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data to convert the electronic data into other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may be considered processors as long as they are configured to implement coded instructions.

[0130] Memories 1118 and 1146 are examples of non-transitory computer-readable media. Memories 1118 and 1146 can store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods described herein and the functions of various systems. In various embodiments, the memory can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory type memory, or any other type of memory capable of storing information. The architecture, system, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the figures are merely examples relevant to the discussion herein.

[0131] It is to be understood that for illustrative purposes, the components discussed herein are described as separate. However, the operations performed by the various components may be combined or performed in any other components.

[0132] It should be noted that although Fig.11 1140 and / or components of the computing device 1140 may be associated with the vehicle 1102. That is, the vehicle 1102 may perform one or more functions associated with the computing device 1140 and vice versa. Additionally, aspects of the timestamp component 110, the sensor association component 418, the sensor synchronization component 512, and / or the image processing component 602 may be executed on any device discussed herein.

[0133] Figure 7-10 Example processes according to examples of the present disclosure are shown. These processes are shown as logical flow charts, each operation of which represents a series of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media, which perform the recorded operations when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be understood as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement processing. In some embodiments, one or more operations can be omitted.

[0134] Sample Clauses

[0135] A. A system includes: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations including the following steps: receiving first scan line data of an image frame associated with rolling shutter image capture from an image sensor; appending first time data to the first scan line data; adjusting a width associated with the first scan line data based at least in part on a size of the first time data; receiving second scan line data of an image frame associated with rolling shutter image capture from the image sensor; and appending second time data to the second scan line data.

[0136] B. The system of paragraph A, wherein the operation further comprises: receiving LIDAR data including a LIDAR point from the LIDAR sensor, the LIDAR data being associated with third time data; and associating the LIDAR point with at least a portion of the first scan line data based at least in part on the first time data and the third time data.

[0137] C. A system as described in paragraph A or B, wherein: the first scan line data is associated with a first exposure time; and the second scan line data is associated with a second exposure time different from the first exposure time.

[0138] D. A system as described in any of paragraphs A to C, wherein the operation also includes: modifying at least a portion of the image to correct or compensate for distortion effects caused by motion associated with rolling shutter image capture to produce a modified image, based at least in part on the first time data, wherein the distortion effects include one or more of the following: wobble; skew; spatial aliasing; or temporal aliasing; and controlling the movement of the autonomous vehicle based at least in part on the modified image.

[0139] E. The system as described in any of paragraphs A to D further includes: a precision time protocol (PTP) clock; and a system processor clock synchronized with the PTP clock; wherein the operation further includes: obtaining data from a counter subordinate to the system processor clock; and determining the first time data based at least in part on the data.

[0140] F. A method comprising: receiving first scan line data associated with a first scan line of an image frame; adding first temporal data to the first scan line; receiving second scan line data associated with a second scan line of the image frame; and adding the second data to the second scan line.

[0141] G. A method as described in paragraph F, wherein: receiving first scan line data includes: receiving the first scan line data from a first sensor having a first sensor modality; and the method also includes: receiving data from a second sensor having a second sensor modality, the data including measurement results; and determining an association between the measurement results and at least a portion of the first scan line data based at least in part on the first time data.

[0142] H. The method of paragraph G, wherein: the first sensor comprises an image sensor; and the second sensor comprises at least one of: an additional image sensor; a LIDAR sensor; a radar sensor; or a time-of-flight sensor.

[0143] I. The method described in paragraph G or H also includes: associating at least a portion of the first scan line data with at least a portion of the data from the second sensor to produce associated data based at least in part on the association between the measurement results and at least a portion of the first scan line; and controlling the movement of the autonomous driving vehicle based at least in part on the associated data.

[0144] J. The method described in any of paragraphs F to I also includes: obtaining data from a counter of a system processor clock belonging to the autonomous vehicle, wherein the system processor clock is synchronized with a precision time protocol (PTP) clock; and determining first time data based at least in part on the data.

[0145] K. A method as described in paragraph J, wherein: receiving first scan line data includes: receiving the first scan line data from a first sensor having a first sensor modality; the method also includes: receiving other data from a second sensor having a second sensor modality, the other data including a measurement result, wherein the measurement result is associated with a timestamp based on time data from a PTP clock; and determining an association between the measurement result and at least a portion of the first scan line data based at least in part on the first time data and the timestamp associated with the measurement result.

[0146] L. A method as described in any of paragraphs F to K, wherein: the first scan line data is associated with a first exposure time; and the second scan line data is associated with a second exposure time different from the first exposure time.

[0147] M. The method of any of paragraphs F to M, further comprising: modifying at least a portion of the image based at least in part on the first time data to correct or compensate for distortion effects caused by motion associated with rolling shutter image capture.

[0148] N. A method as described in paragraph M, wherein the distortion effect includes at least one of the following: wobble; skew; spatial aliasing; or temporal aliasing.

[0149] O. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: receiving scan line data associated with scan lines of an image frame captured by a rolling shutter image; and adding time data to the scan lines.

[0150] P. One or more non-transitory computer-readable media as described in paragraph O, wherein: the scan line data associated with the scan line is first scan line data associated with the first scan line; the time data is first time data; and the operation further includes: receiving second scan line data associated with a second scan line of an image frame captured by a rolling shutter image; and adding the second time data to the second scan line.

[0151] Q. One or more non-transitory computer-readable media as described in paragraphs O or P, wherein: receiving scan line data includes: receiving the scan line data from an image sensor; and the operation also includes: receiving measurement result data from at least one of the following: an additional image sensor; a LIDAR sensor; a radar sensor; or a time-of-flight sensor; wherein the measurement result data includes measurement results associated with at least a portion of the environment near the vehicle; determining an association between the measurement results and one or more pixels of the scan line data based at least in part on the time data.

[0152] R. One or more non-transitory computer-readable media as described in paragraph Q, wherein: the vehicle includes an autonomous vehicle, the association includes distance data, and the operation also includes: controlling the movement of the autonomous vehicle based at least in part on the associated data.

[0153] S. One or more non-transitory computer-readable media as described in paragraphs Q or R, wherein: the measurement result data includes distance data associated with the object; and the operation also includes: modifying at least a portion of an image associated with the object based at least in part on the time data and the distance data to correct or compensate for distortion effects caused by motion associated with rolling shutter image capture.

[0154] T. One or more non-transitory computer-readable media as described in any of paragraphs O to S: The operation also includes: obtaining data from a counter of a system processor clock belonging to the autonomous vehicle, wherein the system processor clock is synchronized with a precision time protocol (PTP) clock; and determining time data based at least in part on the data.

[0155] U. A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising the following steps: determining a first pose of an image sensor, wherein the image sensor is associated with a first field of view; determining a second pose of a LIDAR sensor, wherein the LIDAR sensor is associated with a second field of view; and causing the image sensor to initiate rolling shutter image capture of the first field of view so that, based on one or more synchronization conditions, at least a first portion of the first field of view overlaps with at least a second portion of the second field of view, wherein the rolling shutter image capture produces an image frame comprising: a first scan line associated with a first exposure time; and a second exposure time associated with a second exposure time different from the first exposure time.

[0156] V. A system as described in paragraph U, wherein: the image sensor is a first image sensor; the rolling shutter image capture is a first rolling shutter image capture; the one or more synchronization conditions include a first set of one or more synchronization conditions; and the operation also includes: determining a pose of a second image sensor, wherein the second image sensor is associated with a field of view; and causing the second image sensor to initiate a second rolling shutter image capture of the field of view such that, according to a second set of one or more synchronization conditions, at least a portion of the field of view associated with the second image sensor overlaps with at least a portion of a second field of view associated with the LIDAR sensor.

[0157] W. A system as described in paragraphs U or V, wherein the one or more synchronization conditions include at least one of the following: a first central portion of the image frame is substantially aligned with a second central portion of a second field of view associated with the LIDAR sensor; or a majority of the scan lines of the image frame are captured during a time period associated with the second field of view.

[0158] X. In a system as described in any of paragraphs U to W, the operation further includes: receiving scan line data associated with a scan line of a previous rolling shutter image capture from the image sensor and during a time period prior to the rolling shutter image capture; and appending time data to the scan line data; wherein the image sensor is caused to initiate the rolling shutter image capture based at least in part on the time data.

[0159] Y. A method comprising: determining an orientation of a first field of view associated with a first sensor relative to a second field of view associated with a second sensor; and causing the first sensor to initiate rolling shutter image capture of the first field of view so that, based on one or more synchronization conditions, at least a portion of the first field of view overlaps with at least a second portion of the second field of view, wherein the rolling shutter image capture produces an image frame comprising: a first scan line associated with a first exposure time; and a second scan line associated with a second exposure time different from the first exposure time.

[0160] Z. A method as described in paragraph Y, wherein: the first sensor includes an image sensor of the autonomous driving vehicle; and the second sensor includes a LIDAR sensor of the autonomous driving vehicle.

[0161] AA. The method described in paragraph Y or Z further includes: determining a first posture of the first sensor, wherein the first posture includes: a first position; and a first orientation; and determining a second posture of the second sensor, wherein the second posture includes: a second position; and a second orientation; wherein, the first sensor initiates rolling shutter image capture based at least in part on the first posture and the second posture.

[0162] BB. A method as described in any of paragraphs Y to AA, wherein: the first sensor includes a first image sensor; the orientation is a first orientation; the rolling shutter image capture is a first rolling shutter image capture; the second sensor includes a LIDAR sensor; the one or more synchronization conditions include a first set of one or more synchronization conditions; the method also includes: determining a second orientation of the second image sensor associated with the field of view relative to the LIDAR sensor; and causing the second image sensor to initiate a second rolling shutter image capture of the field of view based at least in part on the second orientation so that according to the second set of one or more synchronization conditions, at least a portion of the field of view overlaps with at least a portion of a second field of view associated with the LIDAR sensor.

[0163] CC. A method as described in any of segments Y to BB, wherein: one or more synchronization conditions include at least one of the following: a first central portion of the image frame is substantially aligned with a second central portion of a second field of view associated with a second sensor; or a majority of scan lines of the image frame are captured during a time period associated with the second field of view.

[0164] DD. The method of any of paragraphs Y to CC, further comprising: estimating an exposure time of the image frame; wherein causing the first sensor to initiate rolling shutter image capture based at least in part on the exposure time.

[0165] EE. The method as described in paragraph DD also includes: receiving exposure time data associated with a previous rolling shutter image capture from the first sensor and during a time period before the rolling shutter image capture; wherein the exposure time of the estimated image frame is at least partially based on the exposure time data associated with the previous rolling shutter image capture.

[0166] FF. The method as described in any of paragraphs Y to EE also includes: determining a first timestamp associated with a scan line of an image frame based at least in part on first time data from a system processor clock synchronized with a precision time protocol (PTP) clock; wherein, the first sensor initiates rolling shutter image capture based at least in part on the first timestamp.

[0167] GG. The method as described in paragraph FF also includes: determining a second timestamp associated with the measurement results captured by the second sensor based at least in part on second time data from the PTP clock; wherein causing the first sensor to initiate rolling shutter image capture is based at least in part on the second timestamp.

[0168] HH. A method as described in any of paragraphs Y to GG, wherein: the first sensor includes an image sensor; the image frame includes scan lines within a first central portion of the image frame; the second sensor includes a LIDAR sensor; and causing the first sensor to initiate rolling shutter image capture includes: causing the image sensor to initiate rolling shutter image capture such that the scan lines are captured during a time period associated with a second central portion of a second field of view associated with the LIDAR sensor.

[0169] II. The method of any of paragraphs Y to HH, wherein during rolling shutter image capture, the second field of view moves relative to the first field of view.

[0170] JJ. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: determining an orientation of a first field of view associated with a first sensor relative to a second field of view associated with a second sensor; and causing the image sensor to initiate rolling shutter image capture of the first field of view so that, based on one or more synchronization conditions, at least a first portion of the first field of view overlaps with at least a second portion of the second field of view, wherein the rolling shutter image capture produces an image frame comprising: a first scan line associated with a first exposure time; and a second scan line associated with a second exposure time different from the first exposure time.

[0171] KK. One or more non-transitory computer-readable media as described in paragraph JJ, wherein one or more synchronization conditions include at least one of the following: a first central portion of the image frame is substantially aligned with a second central portion of a second field of view associated with the second sensor; or a majority of the scan lines of the image frame are captured within a time period associated with the second field of view.

[0172] LL. One or more non-transitory computer-readable media as described in paragraphs JJ or KK, wherein the operation also includes: receiving first exposure time data associated with a first scan line of a previous image frame associated with a previous rolling shutter image capture from a first sensor and during a time period before a rolling shutter image capture; receiving second exposure time data associated with a last scan line of a previous image frame from the first sensor and during a time period before a rolling shutter image capture; determining an exposure time of an image frame of the rolling shutter image capture based at least in part on the first exposure time data and the second exposure time data; wherein causing the image sensor to initiate a rolling shutter image capture is based at least in part on the exposure time.

[0173] MM. One or more non-transitory computer-readable media as described in any of paragraphs JJ to LL, wherein the operation also includes: determining a first timestamp associated with a scan line of an image frame based at least in part on first time data from a system processor clock synchronized with a precision time protocol (PTP) clock; wherein causing the first sensor to initiate rolling shutter image capture is based at least in part on the first timestamp.

[0174] NN. One or more non-transitory computer-readable media as described in paragraph MM, the operation also includes: determining a second timestamp associated with a measurement result captured by a second sensor based at least in part on second time data from a PTP clock; wherein, the first sensor is caused to initiate rolling shutter image capture based at least in part on the second timestamp.

[0175] Although the above exemplary clauses are described with respect to a specific implementation, it should be understood that in the context of this document, the content of the exemplary clauses may also be implemented via a method, an apparatus, a system, a computer-readable medium, and / or another implementation.

[0176] in conclusion

[0177]

[0066] While one or more examples of the techniques described herein have been described, various alterations, additions, permutations, and equivalents thereof are also included within the scope of the techniques described herein.

[0178] In the description of the examples, reference is made to the accompanying drawings forming a part of this article, which show by way of illustration a specific example of the subject matter claimed for protection. It should be understood that other examples may be used, and changes or substitutions such as structural changes may be made. Such examples, changes or modifications do not necessarily deviate from the scope relative to the subject matter intended for protection. Although the steps of this article may be presented in a certain order, in some cases, the order may be changed so that certain inputs may be provided at different times or in different orders without changing the functions of the described systems and methods. The disclosed processes may also be performed in different orders. In addition, it is not necessary to perform various calculations in the disclosed order, and other examples of alternative orders using calculations may be easily implemented. In addition to being reordered, these calculations may also be decomposed into sub-calculations with the same results.

Claims

1. A system comprising: one or more processors; as well as One or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: determining a first pose of an image sensor, wherein the image sensor is associated with a first field of view; determining a second pose of the LIDAR sensor, wherein the LIDAR sensor is associated with a second field of view; and causing the image sensor to initiate rolling shutter image capture of the first field of view such that, based on one or more synchronization conditions, at least a first portion of the first field of view overlaps at least a second portion of the second field of view, wherein the rolling shutter image capture produces an image frame comprising: a first scan line associated with a first exposure time; and A second scan line is associated with a second exposure time different from the first exposure time.

2. The system of claim 1, wherein: The image sensor is a first image sensor; The rolling shutter image capture is a first rolling shutter image capture; The one or more synchronization conditions include a first set of one or more synchronization conditions; and The operations also include: determining a pose of a second image sensor, wherein the second image sensor is associated with the field of view; and causing the second image sensor to initiate a second rolling shutter image capture of the field of view such that at least a portion of the field of view associated with the second image sensor overlaps with at least a portion of the second field of view associated with the LIDAR sensor according to a second set of one or more synchronization conditions.

3. The system according to claim 1, wherein: The one or more synchronization conditions include at least one of the following: a first central portion of the image frame substantially aligned with a second central portion of the second field of view associated with the LIDAR sensor; or A majority of the scan lines of the image frame are captured during a time period associated with the second field of view.

4. The system of claim 1, wherein the operations further comprise: receiving scan line data associated with a scan line of a previous rolling shutter image capture from the image sensor and during a time period prior to the rolling shutter image capture; as well as Append time data to scanline data; Wherein causing the image sensor to initiate the rolling shutter image capture is based at least in part on the temporal data.

5. A method comprising: determining an orientation of a first field of view associated with the first sensor relative to a second field of view associated with the second sensor; as well as causing the first sensor to initiate rolling shutter image capture of the first field of view such that, based on one or more synchronization conditions, at least a portion of the first field of view overlaps at least a second portion of the second field of view, wherein the rolling shutter image capture produces an image frame comprising: a first scan line associated with a first exposure time; and A second scan line is associated with a second exposure time different from the first exposure time.

6. The method according to claim 5, wherein: The first sensor comprises an image sensor of the autonomous driving vehicle; and The second sensor includes a LIDAR sensor of the autonomous vehicle.

7. The method according to claim 5, further comprising: Determining a first posture of the first sensor, wherein the first posture comprises: First position; and First orientation; and Determining a second posture of the second sensor, wherein the second posture comprises: Second position; and Second orientation; Wherein causing the first sensor to initiate the rolling shutter image capture is based at least in part on the first gesture and the second gesture.

8. The method according to claim 5, wherein: The first sensor comprises a first image sensor; The orientation is a first orientation; The rolling shutter image capture is a first rolling shutter image capture; The second sensor comprises a LIDAR sensor; The one or more synchronization conditions include a first set of one or more synchronization conditions; The method further comprises: determining a second orientation of a second image sensor associated with the field of view relative to the LIDAR sensor; as well as Based at least in part on the second orientation, causing the second image sensor to initiate a second rolling shutter image capture of the field of view such that at least a portion of the field of view overlaps with at least a portion of the second field of view associated with the LIDAR sensor according to a second set of one or more synchronization conditions.

9. The method according to claim 5, wherein: The one or more synchronization conditions include at least one of the following: a first central portion of the image frame is substantially aligned with a second central portion of a second field of view associated with the second sensor; or A majority of the scan lines of the image frame are captured during a time period associated with the second field of view.

10. The method according to claim 5, further comprising: estimating an exposure time of the image frame; Wherein causing the first sensor to initiate the rolling shutter image capture is based at least in part on the exposure time.

11. The method according to claim 10, further comprising: receiving, from the first sensor and during a time period prior to the rolling shutter image capture, exposure time data associated with a previous rolling shutter image capture; Wherein estimating the exposure time of the image frame is based at least in part on the exposure time data associated with the previous rolling shutter image capture.

12. The method according to claim 5, further comprising: determining a first time stamp associated with a scan line of the image frame based at least in part on first time data from a system processor clock synchronized with a precision time protocol (PTP) clock; Wherein causing the first sensor to initiate the rolling shutter image capture is based at least in part on the first timestamp.

13. The method according to claim 12, further comprising: determining a second timestamp associated with a measurement captured by the second sensor based at least in part on second time data from the PTP clock; Wherein causing the first sensor to initiate the rolling shutter image capture is based at least in part on the second timestamp.

14. The method according to claim 5, wherein: The first sensor comprises an image sensor; The image frame includes scan lines within a first central portion of the image frame; The second sensor comprises a LIDAR sensor; as well as Causing the first sensor to initiate rolling shutter image capture includes: The image sensor is caused to initiate the rolling shutter image capture such that the scan line is captured during a time period associated with a second central portion of the second field of view associated with the LIDAR sensor.

15. The method according to claim 5, wherein: During the rolling shutter image capture, the second field of view moves relative to the first field of view.

16. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein: When executed, the instructions cause the one or more processors to perform operations including: determining an orientation of a first field of view associated with the first sensor relative to a second field of view associated with the second sensor; as well as causing the first sensor to initiate rolling shutter image capture of the first field of view such that, based on one or more synchronization conditions, at least a first portion of the first field of view overlaps at least a second portion of the second field of view, wherein the rolling shutter image capture produces an image frame comprising: a first scan line associated with a first exposure time; and A second scan line is associated with a second exposure time different from the first exposure time.

17. The one or more non-transitory computer-readable media of claim 16, wherein: The one or more synchronization conditions include at least one of the following: a first central portion of the image frame substantially aligned with a second central portion of the second field of view associated with the second sensor; or A majority of the scan lines of the image frame are captured during a time period associated with the second field of view.

18. The one or more non-transitory computer-readable media of claim 16, the operations further comprising: receiving, from the first sensor and during a time period prior to the rolling shutter image capture, first exposure time data associated with a first scan line of a previous image frame associated with a previous rolling shutter image capture; receiving, from the first sensor and during the time period prior to the rolling shutter image capture, second exposure time data associated with a last scan line of the previous image frame; determining an exposure time for the image frame of the rolling shutter image capture based at least in part on the first exposure time data and the second exposure time data; Wherein causing the first sensor to initiate the rolling shutter image capture is based at least in part on the exposure time.

19. The one or more non-transitory computer-readable media of claim 16, the operations further comprising: determining a first time stamp associated with a scan line of the image frame based at least in part on first time data from a system processor clock synchronized with a precision time protocol (PTP) clock; Wherein causing the first sensor to initiate the rolling shutter image capture is based at least in part on the first timestamp.

20. The one or more non-transitory computer-readable media of claim 19, the operations further comprising: determining a second timestamp associated with a measurement captured by the second sensor based at least in part on second time data from the PTP clock; Wherein causing the first sensor to initiate the rolling shutter image capture is based at least in part on the second timestamp.

Images