Flexible synchronization of lidar with camera for unmanned vehicles

Abstract

In one aspect, a computing device of an autonomous driving vehicle (ADV) is configured to determine a first control signal for a lidar (Lidar) sensor of the ADV and a second control signal for a camera of the ADV (602), provide the first control signal to the lidar sensor and provide the second control signal to the camera (604), and processing the lidar output of the lidar sensor and the camera output of the camera to detect one or more features of the lidar output or the camera output (606). In response to detecting the one or more features, the computing device adjusts the first control signal or the second control signal (608).

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to operating an autonomous vehicle. More specifically, embodiments of the present disclosure relate to synchronizing a lidar sensor with a camera of an autonomous vehicle. Background Art

[0002] A vehicle operating in an automated mode (e.g., driverless mode) can relieve occupants, particularly the driver, of some driving-related tasks. When a vehicle is operating in an automated mode, onboard sensors can be used to navigate the vehicle to different locations, allowing the vehicle to be driven with minimal human intervention, or in some cases without any occupants.

[0003] An autonomous driving vehicle (ADV) may include one or more image sensors (e.g., cameras, lidar sensors, etc.) to capture the surroundings of the ADV. The surroundings may include the physical environment around the ADV, such as roads, other vehicles, buildings, people, objects, etc. Each image sensor may generate an image stream. The number of image sensors may vary from vehicle to vehicle. The individual image sensors may be placed in different positions to capture the environment from their perspective, for example, capturing the environment from a given position at a given angle relative to the ADV.

[0004] Lidar and cameras are the two main types of sensors in ADVs. Lidar sensors determine range (variable distance) by using lasers to aim at an object or surface and measuring the time it takes for the reflected light to return to a receiver. Lidar sensors can also make a digital 3D representation of the ADV's surroundings, which can include, for example, walls, buildings, pedestrians, vehicles, trees, and other objects. Lidar uses ultraviolet, visible, or near-infrared light to image an object. Cameras can sense light in the surrounding environment and generate images based on the sensed light. ADVs can include one or more cameras to capture the environment from different angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The various aspects are shown in the accompanying drawings by way of example and not limitation, wherein similar reference numerals represent similar elements. It should be noted that the "one" or "certain" aspect mentioned in the present disclosure does not necessarily refer to the same aspect, which represents at least one. In addition, for simplicity and to reduce the total number of drawings, the drawings provided may be used to illustrate the features of multiple aspects, and for a given aspect, not all elements in the drawings are necessary. It should be understood that some of the embodiments shown may be combined with other embodiments even if not shown in each figure.

[0006] Figure 1 A block diagram of a networking system according to some embodiments is shown.

[0007] Figure 2 A block diagram of an example of an autonomous vehicle is shown, according to some embodiments.

[0008] Figure 3A A block diagram of an example of an autonomous driving system for use with an autonomous vehicle according to some embodiments is shown.

[0009] Figure 3B A block diagram of an example of an autonomous driving system for use with an autonomous vehicle according to some embodiments is shown.

[0010] Figure 4 A block diagram of a system architecture for autonomous driving according to some embodiments is shown.

[0011] Figure 5 An example of a computing device that may be configured as a sensor synchronization module in accordance with some embodiments is shown.

[0012] Figure 6 An example method for synchronizing a lidar sensor and a camera of an ADV is shown in accordance with some embodiments.

[0013] Figure 7 An example workflow is shown for managing control of a lidar sensor and camera to improve synchronization and sensor fusion of an ADV in accordance with some embodiments. DETAILED DESCRIPTION

[0014] Various embodiments and aspects of the present disclosure will be described with reference to the following details, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative and should not be construed as limiting the present disclosure. In order to thoroughly understand the various embodiments of the present disclosure, many specific details are described. However, in some cases, in order to briefly discuss the embodiments of the present disclosure, well-known or conventional details are not described.

[0015] The "one embodiment" or "an embodiment" mentioned in the specification means that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present disclosure. The "in one embodiment" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0016] Sensor fusion is the processing of information from multiple sensors to understand the environment around an autonomous driving vehicle (ADV), which requires highly accurate and synchronized data between multiple sensors. ADVs on the road may operate in harsh conditions, such as being subjected to dynamic forces (e.g., vibration, shock) and high and low temperatures. Over time, sensor data quality may be affected due to timing changes or offsets. Environmental conditions may cause timing changes in various sensor data, which over time can affect the ADV's ability to correctly perceive its surroundings.

[0017] In traditional systems, software timestamp information can be included in the perception data to synchronize measurements of sensor data. However, due to differences in software processes, the control signals for various sensors are not accurate. Some control systems can provide hardware-level signal control for data acquisition. Hardware-level signal control can include camera triggers or other waveforms that sensors can use as input and perform actions based on this. Due to the harsh conditions of ADV environments and the importance of sensor data fusion from various sensors, simple control signals alone may not provide consistent and reliable results. Traditional systems lack management solutions to properly ensure the synchronization of lidar sensors with cameras.

[0018] According to some embodiments, the sensor synchronization module of the ADV may include hardware design to provide flexible and redundant sensor signal control with fast response. The module or system may include multi-level state monitoring and control to improve system stability and accuracy of control signals. The system can detect when the lidar sensor and camera exhibit behavior that jeopardizes data synchronization and respond accordingly.

[0019] In one aspect, a method performed by a computing device of an autonomous vehicle (ADV) includes: determining a first control signal for a lidar (Light Detection and Ranging) sensor of the ADV and a second control signal for a camera of the ADV; providing the first control signal to the lidar sensor and the second control signal to the camera; processing a lidar output of the lidar sensor and a camera output of the camera to detect one or more features of the lidar output or the camera output; and adjusting the first control signal (of the lidar sensor) or the second control signal (of the camera) in response to detecting the one or more features. In this way, the method can combine feedback (e.g., output of the sensor) to determine whether the data from the sensor presents a risk and respond accordingly.

[0020] In some examples, determining the first control signal and determining the second control signal include selecting the first control signal based at least on a lidar type of the lidar sensor and selecting the second control signal based at least on a camera type of the camera.

[0021] In some examples, adjusting the first control signal or the second control signal includes adjusting the first control signal or the second control signal to synchronize the lidar sensor with the camera. This may include calculating a control signal so that the lidar and the camera capture images at synchronized intervals (even if at different rates), or aligning at least one image of the lidar with the camera within a given control cycle. In some examples, this may include aligning the lidar sensor with the camera based on a common reference (e.g., a common clock).

[0022] In some examples, one or more features include drift in the lidar output or camera output. Drift can be understood as a deviation from the expected timing of events (e.g., image capture) of the lidar sensor or camera or both, and the actual timing. Over time, these deviations may cause the timing offset of each event to increase. Thus, the system can detect drift and respond by adjusting the control signal of the lidar sensor or camera or both.

[0023] Additionally or alternatively, one or more features include a temporal misalignment between the lidar output and the camera output. The temporal misalignment can be determined based on a difference between a timestamp frame of the lidar sensor and a timestamp frame of the camera. The system can detect such a misalignment and respond by adjusting a control signal of the lidar sensor or the camera or both.

[0024] In some examples, in response to one or more features satisfying a fault threshold, the system records the one or more features (e.g., in a fault log). The log can be stored in a computer-readable memory (e.g., a non-volatile memory). Each occurrence of a feature can be stored as a record and provide a history of sensor behavior for fault detection or troubleshooting. Additionally or optionally, the system can trigger a fault response, such as providing a notification alert to a user. In some examples, satisfying the fault threshold can include: the timing of the lidar output or camera output differs from the expected timing (e.g., manufacturer specifications) by a threshold amount. In some examples, satisfying the fault threshold can include: there is a missing data in the lidar output or camera output.

[0025] In some examples, adjusting the first control signal or the second control signal includes: switching from a non-frame-by-frame control signal to a frame-by-frame control signal. For example, a frame-by-frame control signal may include pulses, each of which represents a command for the sensor to perform an action (e.g., capture an image, open a shutter, etc.). Such a control signal enables the system to have more refined control over the lidar sensor or camera. The non-frame-by-frame control signal may include a pulse-per-second control signal that pulses once per second or other interval pulses on a non-frame-by-frame basis. The lidar sensor or camera may receive such a signal and synchronize its own events within that second based on the pulse-per-second. In some cases, when the internal timing mechanism of the lidar sensor or camera does not perform as specified (e.g., the timing is irregular or does not meet the specification), such a non-frame-by-frame control signal cannot provide sufficient synchronization. In other examples, the non-frame-by-frame control signal may include a command to the lidar or camera to operate indefinitely at an "X" capture rate, in which case more reliance is placed on the internal timing mechanism of the lidar sensor or camera. Thus, the system can switch from non-frame-by-frame control signals to frame-by-frame control signals (with greater granularity and control) to reduce reliance on the internal mechanisms of a given sensor (e.g., a lidar sensor or camera). This adjustment is beneficial when these internal mechanisms are out of specification or malfunctioning.

[0026] In some examples, adjusting the first control signal or the second control signal includes switching from a non-frame-by-frame control signal to a frame-by-frame control signal. For example, if the lidar sensor is misaligned with the camera under frame-by-frame control, or one of the sensors is falsely triggered or behaves abnormally under frame-by-frame control, the system can switch its control signal to a non-frame-by-frame control signal to try to change the behavior of the sensor using a different type of control signal.

[0027] In some examples, adjusting the first control signal or the second control signal includes increasing or decreasing the frequency of the first control signal or the second control signal. For example, the system can decrease the frequency or rate of the signal and analyze the response of the sensor. If the sensor timing is improved or the synchronization is improved, the system can maintain the adjusted control signal or continue to decrease the frequency or rate of the signal. If the timing is deteriorated, the frequency or rate can be increased. Furthermore, the timing can be monitored until the system determines whether increasing or decreasing the control signal rate improves the timing or synchronization of the sensor.

[0028] Figure 1 FIG. 1 is a block diagram showing an autonomous driving network configuration according to an embodiment of the present disclosure. Figure 1, the network configuration 100 includes an autonomous driving vehicle (ADV) 101, which can be communicatively connected to one or more servers 103-104 via a network 102. One ADV is shown in the figure, but multiple ADVs can be connected to each other and / or to servers 103-104 via network 102. Network 102 can be any type of wired or wireless network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a satellite network, or a combination thereof (wired or wireless). Servers 103-104 can be any type of server or server cluster, such as a Web or cloud server, an application server, a back-end server, or a combination thereof. Servers 103-104 can be data analysis servers, content servers, traffic information servers, map and point of interest (MPOI) servers, or location servers, etc.

[0029] An ADV refers to a vehicle that can be configured for an automated mode, in which the vehicle requires little or no input from the driver to navigate an environment. Such an ADV may include a sensor system having one or more sensors configured to detect information related to the environment in which the vehicle operates. The vehicle and its associated controller use the detected information to navigate the environment. The ADV 101 can operate in a manual mode, a fully automated mode, or a partially automated mode.

[0030] In one embodiment, ADV 101 includes, but is not limited to, an autonomous driving system (ADS) 110, a vehicle control system 111, a wireless communication system 112, a user interface system 113, and a sensor system 115. ADV 101 may also include certain common components included in an ordinary vehicle, such as an engine, wheels, a steering wheel, a transmission, etc., which may be controlled by the vehicle control system 111 and / or ADS 110 using various communication signals and / or commands, such as acceleration signals or commands, deceleration signals or commands, steering signals or commands, braking signals or commands, etc.

[0031] Components 110-115 may be communicatively connected to each other via an interconnect, a bus, a network, or a combination thereof. For example, components 110-115 may be communicatively connected to each other via a controller area network (CAN) bus. The CAN bus is a vehicle bus standard designed to allow microcontrollers and devices to communicate with each other in applications without a host computer. The CAN bus is a message-based protocol originally designed for multi-way electrical wiring in automobiles, but is also used in many other situations.

[0032] See also Figure 2 In one embodiment, the sensor system 115 includes, but is not limited to, one or more cameras 211, a global positioning system (GPS) unit 212, an inertial measurement unit (IMU) 213, a radar unit 214, and a laser radar (LIDAR) unit 215. The GPS unit 212 may include a transceiver for providing information related to the position of the ADV. The IMU unit 213 may sense the position and orientation changes of the ADV based on inertial acceleration. The radar unit 214 may represent a system that uses radio signals to sense objects within the local environment of the ADV. In some embodiments, in addition to sensing objects, the radar unit 214 may also sense the speed and / or heading of the object. The laser radar unit 215 may use lasers to sense objects in the environment where the ADV is located. The laser radar unit 215 may include one or more laser sources, a laser scanner, and one or more detectors and other system components. The camera 211 may include one or more devices for capturing images of the environment around the ADV. The camera 211 may be a camera and / or a video camera. The camera may be mechanically moved by, for example, mounting the camera on a rotating and / or tilting platform.

[0033] The sensor system 115 may also include other sensors, such as a sonar sensor, an infrared sensor, a steering sensor, a throttle sensor, a brake sensor, and an audio sensor (e.g., a microphone). The audio sensor may be used to capture sounds from the surrounding environment of the ADV. The steering sensor may be used to sense the steering angle of a steering wheel, a wheel, or a combination thereof. The throttle sensor and the brake sensor may sense the throttle position and the brake position of the vehicle, respectively. In some cases, the throttle sensor and the brake sensor may be integrated into an integrated throttle / brake sensor.

[0034] In one embodiment, the vehicle control system 111 includes but is not limited to a steering unit 201, a throttle unit 202 (also referred to as an acceleration unit), and a brake unit 203. The steering unit 201 is used to adjust the direction or heading of the vehicle. The throttle unit 202 is used to control the speed of the motor or engine, thereby controlling the speed and acceleration of the vehicle. The brake unit 203 slows down the speed of the wheels or tires of the vehicle by providing friction, thereby decelerating the vehicle. It should be noted that Figure 2 The components shown in may be implemented as hardware, software, or a combination thereof.

[0035] See again Figure 1, the wireless communication system 112 allows the ADV 101 to communicate with external systems, such as devices, sensors, and other vehicles. For example, the wireless communication system 112 can communicate wirelessly with one or more devices directly, or with one or more devices through a communication network such as servers 103-104 on the network 102. The wireless communication system 112 can use any cellular communication network or wireless local area network (WLAN), such as WiFi, to communicate with another component or system. The wireless communication system 112 can communicate directly with devices (e.g., mobile devices, display devices, speakers of passengers in the vehicle 101) using, for example, infrared links, Bluetooth, etc. The user interface system 113 can be part of a peripheral device implemented in the vehicle 101, including, for example, a keyboard, a touch screen display device, a microphone, and a speaker.

[0036] Some or all of the functions of ADV 101 may be controlled or managed by ADS 110, especially when operating in an automated driving mode. ADS 110 includes necessary hardware (e.g., processor, memory, storage) and software (e.g., operating system, planning and routing programs) to receive information from sensor system 115, control system 111, wireless communication system 112, and / or user interface system 113, process the received information, plan a route or path from a starting point to a destination, and then drive vehicle 101 based on the planning and control information. Alternatively, ADS 110 may be integrated with vehicle control system 111.

[0037] For example, a user as a passenger may specify a starting location and a destination of a trip, for example, through a user interface. ADS 110 obtains trip-related data. For example, ADS 110 may obtain location and route data from an MPOI server, which may be part of servers 103-104. The location server provides location services, and the MPOI server provides map services and points of interest at some locations. Alternatively, these locations and MPOI information may be cached locally in a persistent storage device of ADS 110.

[0038] When ADV 101 moves along the route, ADS 110 can also obtain real-time traffic information from a traffic information system or server (TIS). It should be noted that the servers 103-104 can be operated by a third-party entity. Alternatively, the functions of servers 103-104 can be integrated into ADS 110. Based on real-time traffic information, MPOI information, location information, and real-time local environmental data (such as obstacles, objects, nearby vehicles) detected or sensed by sensor system 115, ADS 110 can plan the best route, and drive vehicle 101 according to the planned route, for example, through control system 111, to reach the designated destination safely and efficiently. Server 103 can be a data analysis system that performs data analysis services for various clients. In one embodiment, data analysis system 103 includes data collector 121 and machine learning engine 122. Data collector 121 collects driving statistics 123 from various vehicles such as ADS or ordinary vehicles driven by drivers. The driving statistics data 123 include the following information: driving commands issued at different times (e.g., throttle commands, brake commands, steering commands) and vehicle responses captured by vehicle sensors (e.g., speed, acceleration, deceleration, direction). The driving statistics data 123 may also include information describing the driving environment at different times, such as routes (including starting locations and target locations), MPOI, road conditions, weather conditions, etc.

[0039] The machine learning engine 122 generates or trains a set of rules, algorithms, and / or predictive models 124 for various purposes based on driving statistics 123. The algorithms 124 can then be uploaded to the ADV for real-time use during autonomous driving.

[0040] Figure 3A and Figure 3B FIG. 3 is a block diagram showing an example of an automated driving system for use with an ADV according to one embodiment. The system 300 may be implemented as Figure 1 A portion of ADV 101 includes, but is not limited to, ADS 110, control system 111, and sensor system 115. Figure 3A and Figure 3B , ADS 110 includes but is not limited to a positioning module 301 , a perception module 302 , a prediction module 303 , a decision module 304 , a planning module 305 , a control module 306 , and a route selection module 307 [fill in other modules here].

[0041] Some or all of the modules 301 to 307 may be implemented in software, hardware, or a combination thereof. For example, these modules may be installed in the persistent storage device 352, loaded into the memory 351, and executed by one or more processors (not shown). It should be noted that some or all of these modules may be implemented in conjunction with Figure 2 Some or all modules of the vehicle control system 111 are communicatively connected or integrated together. Some modules in the modules 301-307 can be integrated together to form an integrated module.

[0042] The positioning module 301 determines the current location of the ADV 101 (e.g., using the GPS unit 212) and manages any data related to the user's itinerary or route. The positioning module 301 (also known as the map and route module) manages any data related to the user's itinerary or route. The user can log in, for example, through a user interface, and specify the starting location and destination of the itinerary. The positioning module 301 communicates with other components of the ADV 101, such as the map and route data 311, to obtain itinerary related data. For example, the positioning module 301 can obtain location and route data from a location server and an MPOI server. The location server provides location services, and the MPOI server provides map services and POIs of some locations, which can be cached as part of the map and route data 311. When the ADV 101 moves along the route, the positioning module 301 can also obtain real-time traffic information from a traffic information system or server.

[0043] Based on the sensor data provided by the sensor system 115 and the positioning information obtained by the positioning module 301, the perception module 302 determines the perception information of the surrounding environment. The perception information may represent the information that an average driver perceives about the surrounding environment of the vehicle he is driving. The perception information may include, for example, lane configuration in the form of objects, traffic light signals, relative positions of other vehicles, pedestrians, buildings, crosswalks or other traffic-related signs (e.g., stop signs, yield signs), etc. The lane configuration includes information describing one or more lanes, such as lane shape (e.g., straight or curved), lane width, number of lanes on the road, one-way or two-way lanes, merging or splitting lanes, exit lanes, etc.

[0044] The perception module 302 may include a computer vision system or functionality of a computer vision system to process and analyze images captured by one or more cameras to identify objects and / or features in the ADV environment. Objects may include traffic signals, road boundaries, other vehicles, pedestrians, and / or obstacles, etc. The computer vision system may use object recognition algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system may map the environment, track objects, and estimate the speed of objects, etc. The perception module 302 may also detect objects based on other sensor data provided by other sensors such as radar and / or lidar.

[0045] For each object, the prediction module 303 predicts the behavior of the object in a specific situation. The prediction is performed based on the perception data, which combines a set of map / route information 311 and traffic rules 312 to perceive the driving environment at a certain moment. For example, if the object is a vehicle traveling in the opposite direction and the current driving environment includes an intersection, the prediction module 303 will predict whether the vehicle is likely to go straight or turn. If the perception data indicates that there is no traffic light at the intersection, the prediction module 303 can predict that the vehicle may have to come to a complete stop before entering the intersection. If the perception data indicates that the vehicle is currently in a left-turn lane or a right-turn lane, the prediction module 303 will predict that the vehicle will be more likely to turn left or right, respectively.

[0046] For each object, the decision module 304 makes a decision on how to respond to the object. For example, for a particular object (e.g., another vehicle in an intersecting path) and its metadata describing the object (e.g., speed, direction, turn angle), the decision module 304 decides how to respond to the object (e.g., overtake, yield, stop, pass). The decision module 304 can make such decisions based on a set of rules such as traffic rules or driving rules 312, which can be stored in the persistent storage device 352.

[0047] The route selection module 307 is configured to provide one or more routes or paths from the starting point to the destination. For a given trip from the starting location to the destination location, for example, received from a user, the route selection module 307 obtains the route and map information 311 and determines all possible routes or paths from the starting location to the destination location. The route selection module 307 can generate a reference line in the form of a topographic map for each route it determines from the starting location to the destination location. The reference line is an ideal route or path that is not interfered by other vehicles, obstacles, or traffic conditions. That is, if there are no other vehicles, pedestrians, or obstacles on the road, the ADV should follow the reference line completely or approximately. Thereafter, the topographic map is provided to the decision module 304 and / or the planning module 305. The decision module 304 and / or the planning module 305 examine all possible routes to select and modify one of the optimal routes based on other data provided by other modules, such as traffic conditions from the positioning module 301, the driving environment perceived by the perception module 302, and the traffic conditions predicted by the prediction module 303. The actual path or route of controlling the ADV may be close to or different from the reference line provided by the route selection module 307, depending on the specific driving environment at the time.

[0048] Based on the decision for each perceived object, the planning module 305 uses the reference line provided by the route selection module 307 as a basis to plan the path or route of the ADV and driving parameters (such as distance, speed and / or turning angle). That is, for a given object, the decision module 304 decides what to do with the object, and the planning module 305 determines how to do it. For example, for a given object, the decision module 304 can decide to pass through the object, and the planning module 305 can determine whether to pass from the left or right side of the object. The planning module 305 generates planning and control data, including information describing how the vehicle 101 moves in the next movement cycle (such as the next route / path segment). For example, the planning and control data can indicate that the vehicle 101 moves 10 meters at a speed of 30 miles per hour (mph), and then changes to the right lane at a speed of 25 miles per hour.

[0049] Based on the planning and control data, the control module 306 controls and drives the ADV according to the route or path defined by the planning and control data by sending appropriate commands or signals to the vehicle control system 111. The planning and control data includes sufficient information to drive the vehicle from a first point on the route or path to a second point along the route or path using appropriate vehicle settings or driving parameters (e.g., throttle, brake, steering commands) at different times.

[0050] In one embodiment, the planning stage is performed with multiple planning cycles, which are also called driving cycles, for example, every 100 milliseconds. For each planning cycle or driving cycle, one or more control commands will be issued based on the planning and control data. That is, every 100 milliseconds, the planning module 305 plans the next route segment or path segment, for example, including the target position and the time required for the ADV to reach the target position. Alternatively, the planning module 305 can also specify a specific speed, direction and / or steering angle, etc. In one embodiment, the planning module 305 plans the route segment or path segment for the next predetermined time period of, for example, 5 seconds. For each planning cycle, the planning module 305 plans the target position of the current cycle (for example, the next 5 seconds) based on the target position planned in the previous cycle. Afterwards, the control module 306 generates one or more control commands (for example, throttle, brake, steering control commands) based on the planning and control data of the current cycle.

[0051] It should be noted that the decision module 304 and the planning module 305 can be integrated into an integrated module. The decision module 304 / planning module 305 can include a navigation system or the functionality of a navigation system to determine the driving path of the ADV. For example, the navigation system can determine a series of speeds and directional headings so that the ADV generally proceeds along a lane-based path to the final destination while its driving path substantially avoids perceived obstacles. The destination can be set based on user input via the user interface system 113. The navigation system can dynamically update the driving path while the ADV is running. The navigation system can combine data from a GPS system and one or more maps to determine the driving path of the ADV.

[0052] In some aspects, the autonomous driving system 110 includes a sensor synchronization module 360. The sensor synchronization module 360 ​​can be configured to determine a first control signal for a lidar sensor of the ADV and a second control signal for a camera of the ADV. The lidar sensor and the camera sensor can be part of the sensor system 115. For example, the lidar sensor can correspond to the lidar unit 215, and the camera sensor can correspond to any camera 211. The sensor synchronization module 360 ​​can determine the control signals of the lidar sensor and the camera respectively according to the type of the lidar sensor and the type of the camera.

[0053] The sensor synchronization module 360 ​​may provide a first control signal to the lidar sensor and a second control signal to the camera. The sensor synchronization module 360 ​​may process the lidar output of the lidar sensor and the camera output of the camera to detect one or more features of the lidar output or the camera output. The one or more features may be related to the timing of the output. For example, the corresponding output may include one or more frames with timestamp information. The one or more features may indicate lidar sensor drift, lidar sensor alignment with the camera, or other features.

[0054] In response to detecting one or more features, the sensor synchronization module 360 ​​can adjust the first control signal or the second control signal. For example, the frequency of the control signal can be increased or decreased to reduce the misalignment between the lidar sensor and the camera. For another example, if the lidar sensor exhibits drift or other timing issues, the control signal of the lidar sensor can be changed from one type of control signal (e.g., pulse per second (PPS) or command signal) to a different type of signal. This can be repeated until the one or more features (e.g., timing-related issues) are no longer detected in the output of the lidar sensor or camera.

[0055] The sensor synchronization module 360 ​​may record one or more features and trigger a fault response in response to one or more features satisfying a fault threshold. The satisfying of the fault threshold may include: the timing of the lidar output or the camera output differs from the expected timing by a threshold amount. For example, if the output of the camera or lidar sensor has early or late frames, or drifts beyond a threshold amount, or the alignment deviation of the sensors (relative to each other) exceeds a threshold range, or a combination of the above, the sensor synchronization module 360 ​​may detect any of these as features and store a record of such events in the sensor error log 362.

[0056] Additionally or alternatively, the failure threshold being met may include: data from the lidar output or the camera output is missing. This may be considered a serious feature. The error log may include an error code, text indicating error details (e.g., "lidar frame X missing", "camera frames exceed expected amount", etc.). The error log 362 may be stored in a computer readable memory (e.g., non-volatile computer readable memory). Other aspects of the sensor synchronization module will be further described in other sections.

[0057] Figure 4 400 is a block diagram showing a system architecture of an autonomous driving system according to an embodiment. Figure 3A and Figure 3B The system architecture of the autonomous driving system is shown.

[0058] See also Figure 4 , the system architecture 400 includes, but is not limited to, an application layer 401, a planning and control (PNC) layer 402, a perception layer 403, a driver layer 404, a firmware layer 405, and a hardware layer 406. The application layer 401 may include a user interface or configuration application that interacts with a user or passenger of the autonomous vehicle, such as functionality associated with the user interface system 113. The PNC layer 402 may include functionality of at least the planning module 305 and the control module 306. The perception layer 403 may include functionality of at least the perception module 302.

[0059] In one embodiment, there are additional layers including the functions of the prediction module 303 and / or the decision module 304. Alternatively, these functions may be included in the PNC layer 402 and / or the perception layer 403.

[0060] The system architecture 400 also includes a driver layer 404, a firmware layer 405, and a hardware layer 406. The firmware layer 405 may at least represent the functions of the sensor system 115, which may be implemented in the form of a field programmable gate array (FPGA). The hardware layer 406 may represent the hardware of the autonomous driving vehicle, such as the control system 111. The layers 401-403 may communicate with the firmware layer 405 and the hardware layer 406 through the device driver layer 404.

[0061] Various aspects of the sensor synchronization module may be implemented through a combination of layers such as the hardware layer 406 , the firmware layer 405 , the driver layer 404 , and the application layer 401 .

[0062] Figure 5 An example of a computing device 504 that may be configured as a sensor synchronization module 516 in accordance with some embodiments is shown. The sensor synchronization module 516 may correspond to the examples shown elsewhere.

[0063] The computing device 504 may include processing logic, such as the processing device 502, which may include memory, such as a volatile memory device (such as a random access memory (RAM)), a non-volatile memory device (such as a flash memory), and / or other types of memory devices. The processing device 504 may include electronic circuits, programmable logic, a processor (e.g., a central processing unit (CPU), a microprocessor, a digital signal processor, etc.), or a combination thereof. The processing device 504 may be configured to perform the operations described in the present disclosure.

[0064] The processing device 504 can be configured as a sensor synchronization module 516. In some examples, the sensor synchronization module 516 can include one or more computer applications running on the processing device 502 to perform the operations described herein. In some examples, the sensor synchronization module 516 can be configured as hardware (e.g., programmable logic or electronic circuitry), or a combination of hardware and software.

[0065] Computing device 504 may be integrated with ADV 522. Similarly, lidar sensor 506 and camera 508 may correspond to sensor system 215 and be integrated with ADV 522 to sense the environment of ADV 522. These sensors may be electrically integrated with other described electronic components via printed circuit boards, wires, wireless technology, etc. to help the ADV sense its environment.

[0066] The computing device 504 may be configured to determine a first control signal 512 for the lidar sensor 506 of the ADV and a second control signal 514 for the camera 508 of the ADV. The computing device may provide the first control signal 512 to the lidar sensor 506 and the second control signal 514 to the camera 508. For example, each control signal may be provided via a conductor (e.g., one or more wires), or wirelessly, or both. In some examples, the control signal may be a digital signal, while in other examples, the control signal may be an analog signal. In some examples, the computing device 504 may include hardware, software, or a combination thereof, and may generate a waveform or a digital command as an output control signal.

[0067] The first control signal 512 may be a pulse per second (PPS) signal that the LiDAR sensor uses to align rotation and image capture. For example, the first control signal 512 may include pulses or square waves with a frequency of 1 Hz. The LiDAR sensor may include one or more actuators and position sensors, and operate the actuators to align their rotation with the first control signal 512 (e.g., by using pulses per second as a basis for performing rotation and image capture). The LiDAR sensor may be configured to rotate 'X' turns per second, aligned with the first control signal 512. It should be understood that

[0068] The second control signal 514 may include a trigger signal, such as a waveform, pulse, square wave, or other trigger signal. The camera may include hardware and software to receive the trigger signal and respond to each trigger (e.g., a rising edge or a falling edge of the waveform), and the response may be to open the shutter, receive light through the aperture, close the shutter, and process the sensed light to generate an image.

[0069] The computing device 504 can process the lidar output 520 of the lidar sensor 506 and the camera output 518 of the camera 508 to detect one or more features 510 of the lidar output or the camera output. The lidar output 520 can include a sequence of frames, where each frame can include information such as an image, a corresponding position of the image (e.g., an orientation or rotation angle), and a timestamp corresponding to the image. The camera output 518 can include a sequence of frames, where each frame can include a captured image and a timestamp corresponding to the captured image. Each output can include additional information, such as an ID of the corresponding sensor (e.g., a camera ID or a lidar ID), a brand or model of the sensor, or other information identifying which sensor generated the output, or a sensor type. Although not shown, the output of each sensor can be provided to the ADS 110, for example, for perception and other autonomous driving tasks.

[0070] In some examples, determining the first control signal and determining the second control signal include selecting the first control signal based at least on a lidar type of the lidar sensor and selecting the second control signal based at least on a camera type of the camera. In some examples, the lidar type and the camera type can be determined based on outputs of the respective sensors.

[0071] Additionally or alternatively, the corresponding type of each sensor can be determined based on system configuration data. For example, the computing device 504 can access a system configuration file (e.g., stored in a computer-readable memory) that indicates the brand, model, or type of lidar and camera. The lidar type can include the type of control signal that the onboard lidar sensor 506 can use as input. Similarly, the camera type can include the type of control signal that the onboard camera 508 can use as input. Based on this information, the sensor synchronization module 516 can select a default control signal for each sensor, such as control signals 512 and 514. The default control signal can be selected as one of the compatible control signal types for the sensor type.

[0072] In response to detecting one or more features, computing device 504 can adjust first control signal 512 or second control signal 514 or both. The adjustment can synchronize the lidar sensor with the camera. For example, the camera's control signal 514 can be adjusted to slow down or speed up, thereby reducing the time misalignment between camera output 518 and lidar output 520.

[0073] In some examples, one or more features 510 may include drift in the lidar output or the camera output. For example, the sensor synchronization module 516 may analyze multiple frames of camera output 518 to determine whether the timing of image capture has drifted. If the drift exceeds a threshold amount, the sensor synchronization module 516 may switch the control signal 514 from a first type of control signal (e.g., a non-triggered signal) to a second type of control signal (e.g., a triggered control signal). Similarly, the sensor synchronization module 516 may analyze multiple frames of lidar output 520 to determine whether the timing of the lidar sensor 506 has drifted. If the drift exceeds a threshold amount, the sensor synchronization module 516 may switch the control signal 512 from a first type of control signal (e.g., a pulse per second control signal) to a second type of control signal (e.g., a triggered control signal).

[0074] In some examples, adjusting the first control signal or the second control signal includes switching from a non-frame-by-frame control signal to a frame-by-frame control signal. Alternatively, adjusting the first control signal or the second control signal includes switching from a non-frame-by-frame control signal to a frame-by-frame control signal. A triggered control signal can cause the lidar sensor or camera to perform capture each time it is triggered (e.g., a rising edge or a falling edge). Therefore, the triggered control signal can be understood as a frame-by-frame control signal. The non-frame-by-frame control signal can be a pulse-per-second control signal, or a command signal such as ("run at 1000Hz"), in which case the receiving sensor will rely on its own internal timing mechanism for image capture.

[0075] In some examples, adjusting the first control signal or the second control signal includes increasing or decreasing the frequency of the first control signal or the second control signal. The sensor synchronization module 516 can recheck the corresponding output of the sensor after the adjustment to determine whether the one or more characteristics are still present. The sensor synchronization module 516 can continue to increase or decrease the frequency of the first control signal or the second control signal until the one or more characteristics are no longer present.

[0076] In some examples, one or more features 510 may include a time misalignment between the lidar output and the camera output. The sensor synchronization module 516 may compare the timestamps of the lidar output 520 and the camera output 518 frame by frame to determine whether the outputs are aligned in time. For example, the capture rate of the camera 508 may correspond to the capture rate of the lidar sensor 506 at a known or expected ratio. The sensor synchronization module 516 may compare the timestamp of every N camera output frames with the timestamp of every S lidar sensor output frames to determine whether they are aligned in time.

[0077] Figure 6An example method 600 of synchronizing a lidar sensor and a camera of an ADV according to some embodiments is shown. The method may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, processors, processing devices, central processing units (CPUs), system-on-chips (SoCs), etc.), software (e.g., instructions running / executing on a processing device), firmware (e.g., microcode), or a combination thereof.

[0078] See also Figure 6 , method 600 illustrates example functions used by various embodiments. Although the method discloses specific functional blocks ("blocks"), these blocks are only examples. That is, embodiments are well suited to executing variations of the blocks described in various other blocks or methods. It should be understood that the blocks in method 500 may be executed in an order different from the order shown, and not all blocks in the method are executed.

[0079] At box 602, processing logic determines a first control signal for a lidar sensor of the ADV and a second control signal for a camera of the ADV. In some examples, determining the first control signal and determining the second control signal include selecting the first control signal based at least on the lidar type of the lidar sensor, and selecting the second control signal based at least on the camera type of the camera. For example, the processing logic may determine the lidar type and the camera type with reference to a system configuration or setting stored in an electronic memory. The lidar type may indicate a control signal that is compatible with the lidar type. If multiple control signals are supported, the processing logic may use a default control signal for the lidar (e.g., a pulse per second control signal) and a default trigger signal for the camera (e.g., at a predefined frequency).

[0080] At box 604, the processing logic provides a first control signal to the lidar sensor and a second control signal to the camera. For example, the processing logic can generate electronic control signals (e.g., waveforms or digital messages) for the lidar sensor and the camera, respectively, and transmit each electronic control signal to an input port of each sensor. Each sensor can be configured to decode the signal and perform a capture event using the time of the control signal as a time reference for its own action. For the lidar sensor, it can include rotating a specified amount and generating an image from the rotated position. For the camera, it can include opening a shutter, generating an image based on information sensed by light through the camera aperture, and closing the shutter.

[0081] At box 606, processing logic can process the lidar output of the lidar sensor and the camera output of the camera to detect one or more features of the lidar output or the camera output. This can include automatically scanning each output in real time (e.g., without manual input) to determine whether the sensor is drifting. The output can be scanned frame by frame. Drift can be perceived when the time between output frames (indicated by timestamps) increases or decreases over time. Similarly, processing logic can compare the timestamps of the output frames of two sensors to determine whether the outputs of the two sensors are misaligned in time.

[0082] At block 608, in response to detecting the one or more features, the processing logic may adjust the first control signal or the second control signal. For example, if the one or more features are detected in the lidar output, the first control signal (of the lidar sensor) may be adjusted. If the one or more features are detected in the camera output, the second control signal (of the camera) may be adjusted. In some examples, if the one or more features indicate misalignment, the processing logic may adjust the frequency of the trigger signal of the camera to reduce the misalignment.

[0083] The processing logic may obtain the sensor outputs to detect whether the one or more characteristics are present (e.g., at block 606), and if so, the processing logic may adjust the control signal to another compatible control signal (e.g., a frame-by-frame control signal). In some cases, the processing logic may try only those compatible control signals, and try different control signals until the outputs of both sensors no longer exhibit the detected one or more characteristics.

[0084] In some examples, at block 608, adjusting the first control signal or the second control signal includes adjusting the first control signal or the second control signal to synchronize the lidar sensor with the camera. Adjusting the first control signal or the second control signal may include switching from a non-frame-by-frame control signal to a frame-by-frame control signal, or switching from a non-frame-by-frame control signal to a frame-by-frame control signal. In some examples, adjusting the first control signal or the second control signal includes increasing or decreasing a frequency of the first control signal or the second control signal.

[0085] In some examples, one or more characteristics include drift of the lidar output or the camera output. The control signal of the drift sensor can be adjusted. For example, if the lidar output drifts, the processing logic can switch from a pulse per second control signal to a frame by frame control signal.

[0086] The processing logic may perform additional fault monitoring operations. For example, in response to one or more features satisfying a fault threshold, the processing logic may record the one or more features and trigger a fault response. The satisfying of the fault threshold may include: the timing of the lidar output or the camera output differs from the expected timing by a threshold amount. For example, if the drift of the lidar output is greater than a manufacturer-specified tolerance "X", the processing logic may trigger a fault response, such as generating a notification or alert to a user, or storing a digital record of the fault. Additionally or alternatively, the satisfying of the fault threshold includes the presence of missing data in the lidar output or the camera output.

[0087] Figure 7 A system 700 is shown for managing control of a LiDAR sensor and a camera to improve synchronization and sensor fusion for ADVs, in accordance with some embodiments.

[0088] At control block 712, system 700 may determine a first control signal 720 for a lidar sensor 702 of an ADV and a second control signal 722 for a camera 722 of the ADV. The system may provide the first control signal 720 to the lidar sensor 702 and the second control signal 722 to the camera 704. The control signal may be provided as an electrical signal, such as an analog signal, a digital signal, a waveform, or other electrical signal. The electrical signal may be a time-varying electrical signal whose frequency indicates a capture rate of the sensor. Additionally or alternatively, the control signal may be provided as a message (e.g., a command message) that may be transmitted via one or more known communication protocols (e.g., TCP / IP).

[0089] The system configuration box 714 can select the laser radar type and the camera type and match them to each other or to different types of control signals. The laser radar sensor 702 can use one or more different types of control signals. Similarly, the camera 704 can use one or more different types of control signals. The system configuration 714 may include a computer-readable memory for storing the type of the laser radar sensor 702 and the type of the camera 704. The type may include a brand, a model, or an enumerated type. The type may include different control signals compatible with the laser radar sensor 702 or the camera 704. For example, the data of the system configuration 714 may indicate that the laser radar sensor 702 can only use a pulse per second signal, while the camera 704 can use a trigger signal or a non-trigger signal input (e.g., a digital command). In some examples, the system can analyze output 716 and output 718 to determine the laser radar sensor type or camera type. The type can be determined based on an output signature (e.g., format) or metadata in the output.

[0090] At first level block 706, system 700 may process lidar output 716 of lidar sensor 702 and camera output 718 of camera 704 to detect one or more features of the lidar output or camera output. As described above, these features may involve time misalignment indicated by comparison of timestamps of the outputs, or drift of one or both sensors, or a combination thereof. In some cases, the outputs may have missing frames, or data may be missing from a frame. The first level block may support time synchronization, such as adjusting a PPS signal or other control signal to synchronize (e.g., align in time) the camera with image capture by the lidar sensor.

[0091] At monitoring block 710 , system 700 may record detected features, such as individual sensor performance and data drift between different sensors. In response to detecting one or more features, the system may adjust first control signal 720 , second control signal 722 , or both at control block 712 .

[0092] In some examples, system 700 may include a second level block 708 for detecting and responding to more severe features, such as when LiDAR time synchronization is missing or fails. In the second level block 708, the system may process output 716 and output 718 to detect whether there are any severe features, such as whether LiDAR time synchronization is missing or fails, or whether frames are missing, or whether data is missing in frames. If such features are detected in any output, the system will raise a flag and execute monitoring block 710 to store event records. Each record may include a timestamp corresponding to the detected feature (e.g., a failure) and a description of the detected feature.

[0093] At control block 712, system 700 may make per-second signal adjustments (e.g., per-second pulse signal adjustments), or per-frame control signal adjustments (e.g., trigger signals), or both to control signal 720. At control block 712, the system may switch between adjustments based on first-stage output 724 and second-stage output 726. The adjusted control signal included in first-stage output 724 may adjust control signal 722 or control signal 720 based on detected features such as drift or misalignment. The adjusted control signal included in second-stage output 726 may adjust control signal 722 or control signal 720 based on detected severe features such as whether a loss or failure in lidar time synchronization occurs, whether a frame is missing, whether data is missing in a frame, whether a drift or misalignment exceeds a threshold amount, or a combination thereof.

[0094] The system 700 can select the first level output 724 or the second level output 726 to adjust the control based on the system configuration 714, the logic executed by the monitoring block 710, or a combination thereof. For example, if the second level output 726 does not exist, the system configuration 714 can indicate that the first level output 724 is used. If the second level output 726 exists (e.g., a serious feature is detected), the control block 712 can use the second level output 726. At the control block 712, the system can implement circuits or other hardware to generate control signals 722 and 720. These control signals can be provided to the corresponding sensors through wired or wireless communication components (e.g., PCB, connector, transceiver, etc.).

[0095] System 700 can perform the operations shown repeatedly and in real time. Although there is minimal delay due to buffering, transmission or processing, real time can be understood as performing operations and immediately determining the adjusted control signal when each sensor generates an output. Each of blocks 714, 706, 708, 710 and 712 can be understood as a module, which can be implemented by one or more computing devices and can include hardware, software or a combination thereof.

[0096] In some examples, certain operations and blocks in the system may be configured using a programmable logic device such as a field programmable gate array (FPGA) or a system on chip (SoC), which may be an integral part of the processing logic.

[0097] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally is conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulation of physical quantities.

[0098] It should be remembered, however, that all of these terms and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless explicitly noted in the above discussion, it should be understood that throughout the specification, discussions using terms such as those set forth in the appended claims refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system registers and memories and converts that data into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage devices, transmission devices, or display devices.

[0099] Embodiments of the present disclosure also relate to devices for performing the operations herein. Such a computer program is stored in a non-transitory computer-readable medium. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium (e.g., a read-only memory (ROM), a random access memory (RAM), a disk storage medium, an optical storage medium, a flash memory device).

[0100] The processes or methods depicted in the foregoing figures may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. The processes or methods are described above in terms of some sequential operations, but it should be understood that some of the operations described may be performed in a different order. In addition, some operations may be performed in parallel rather than sequentially.

[0101] The embodiments of the present disclosure are not described with reference to any particular programming language. It should be appreciated that a variety of programming languages ​​can be used to implement the teachings of the embodiments of the present disclosure described herein.

[0102] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. Obviously, various modifications may be made to the present disclosure without departing from the broader spirit and scope of the present disclosure as described in the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0103] In some aspects, the present disclosure may include terms such as "at least one of [element A] and [element B]". The term may represent one or more elements. For example, "at least one of A and B" may represent "A", "B", or "A and B". Specifically, "at least one of A and B" may represent "at least one of A and at least one of B" or "at least one of A or B". In some aspects, the present disclosure may include terms such as "[element A], [element B] and / or [element C]". The term may represent any element or any combination thereof. For example, "A, B and / or C" may represent "A", "B", "C", "A and B", "A and C", "B and C", or "A, B and C".

Claims

1. A method performed by a computing device of an autonomous vehicle ADV, comprising: Determining a first control signal for a Lidar sensor of the autonomous vehicle and a second control signal for a camera of the autonomous vehicle; providing the first control signal to the lidar sensor and providing the second control signal to the camera; processing a lidar output of the lidar sensor and a camera output of the camera to detect one or more features of the lidar output or the camera output; and In response to detecting the one or more characteristics, the first control signal or the second control signal is adjusted.

2. The method according to claim 1, wherein: Determining the first control signal and determining the second control signal include: selecting the first control signal based on at least a lidar type of the lidar sensor, and selecting the second control signal based on at least a camera type of the camera.

3. The method according to claim 1, wherein: Adjusting the first control signal or the second control signal includes: adjusting the first control signal or the second control signal so that the lidar sensor is synchronized with the camera.

4. The method according to claim 1, wherein: The one or more characteristics include drift of the lidar output or the camera output.

5. The method according to claim 1, wherein: The one or more characteristics include a time misalignment between the lidar output and the camera output.

6. The method according to claim 1, further comprising: In response to the one or more characteristics satisfying a fault threshold, the one or more characteristics are recorded and a fault response is triggered.

7. The method according to claim 6, wherein: Meeting the fault threshold includes: the timing of the laser radar output or the camera output differs from the expected timing by a threshold amount.

8. The method according to claim 6, wherein: Meeting the fault threshold includes: data output by the laser radar or data output by the camera is missing.

9. The method according to claim 1, wherein: Adjusting the first control signal or the second control signal includes switching from a non-frame-by-frame control signal to a frame-by-frame control signal.

10. The method according to claim 1, wherein: Adjusting the first control signal or the second control signal includes switching from a non-frame-by-frame control signal to a frame-by-frame control signal.

11. The method according to claim 1, wherein: Adjusting the first control signal or the second control signal includes: increasing or decreasing the frequency of the first control signal or the second control signal.

12. A computing device of an autonomous driving vehicle (ADV), configured to: Determining a first control signal for a Lidar sensor of the autonomous vehicle and a second control signal for a camera of the autonomous vehicle; providing the first control signal to the lidar sensor and providing the second control signal to the camera; processing a lidar output of the lidar sensor and a camera output of the camera to detect one or more features of the lidar output or the camera output; and In response to detecting the one or more characteristics, the first control signal or the second control signal is adjusted.

13. The computing device of claim 12, wherein: Determining the first control signal and determining the second control signal include: selecting the first control signal based on at least a lidar type of the lidar sensor, and selecting the second control signal based on at least a camera type of the camera.

14. The computing device of claim 12, wherein: Adjusting the first control signal or the second control signal includes: adjusting the first control signal or the second control signal so that the lidar sensor is synchronized with the camera.

15. The computing device of claim 12, wherein: The one or more characteristics include drift of the lidar output or the camera output.

16. The computing device of claim 12, wherein: The one or more characteristics include a time misalignment between the lidar output and the camera output.

17. An autonomous driving vehicle ADV, comprising: Lidar sensor; camera; A processing device, the processing device being configured to: determining a first control signal for a lidar sensor of the autonomous vehicle and a second control signal for a camera of the autonomous vehicle; providing the first control signal to the lidar sensor and providing the second control signal to the camera; processing a lidar output of the lidar sensor and a camera output of the camera to detect one or more features of the lidar output or the camera output; and In response to detecting the one or more characteristics, the first control signal or the second control signal is adjusted.

18. The method according to claim 17, wherein: Adjusting the first control signal or the second control signal includes switching from a non-frame-by-frame control signal to a frame-by-frame control signal.

19. The method according to claim 17, wherein: Adjusting the first control signal or the second control signal includes switching from a non-frame-by-frame control signal to a frame-by-frame control signal.

20. The method according to claim 17, wherein: Adjusting the first control signal or the second control signal includes: increasing or decreasing the frequency of the first control signal or the second control signal.

Images