Time-of-flight camera using a passive image sensor and an existing light source

Abstract

A vehicle can use a passive sensor (e.g., a passive image sensor) with existing on-board light sources to perform time-of-flight (ToF) calculations or determinations to generate depth data and depth images of a scene surrounding the vehicle. For example, a vehicle can use a car camera (e.g., a passive sensor that does not emit light) and can pair the camera with one or more light sources on the vehicle to construct a ToF camera. To enable this construction of a ToF camera using an existing car camera and existing light sources, a controller of the vehicle synchronizes the passive car camera with the on-board light sources. In some examples, the vehicle can enable high dynamic range (HDR) of the ToF camera by changing properties of light emitted from the light sources.

Description

Technical Field

[0001] This disclosure generally pertains to vehicle systems, and specifically to the determination of Time of Flight (ToF) within vehicle systems. Background Technology

[0002] In recent years, transportation methods have changed significantly. This change is partly due to considerations of the limited availability of natural resources, the surge in personal technology, and the societal shift towards adopting more environmentally friendly transportation solutions. These considerations have encouraged the development of many new types of flexible fuel vehicles, hybrid electric vehicles, and electric vehicles.

[0003] Although these vehicles appear new, they are typically implemented as conventional subsystems simply connected to alternative power sources. In fact, the design and construction of these vehicles are constrained by standard chassis size, shape, materials, and transport concepts. Among other things, these limitations prevent the utilization of the advantages of new technologies, power sources, and supporting architectures. Specifically, these vehicles may not be able to utilize or employ different technologies, sensors, and equipment to perform time-of-flight (ToF) determinations or calculations for different purposes (e.g., to assist in autonomous driving operations, to identify passengers within the vehicle, etc.). Attached Figure Description

[0004] Figure 1 A vehicle according to an embodiment of this disclosure is shown;

[0005] Figure 2 A plan view of a vehicle according to at least some embodiments of this disclosure is shown;

[0006] Figure 3A This is a block diagram of an embodiment of the communication environment of a vehicle according to an embodiment of the present disclosure;

[0007] Figure 3B This is a block diagram of an embodiment of an internal sensor in a vehicle according to an embodiment of this disclosure;

[0008] Figure 3C This is a block diagram of an embodiment of a vehicle navigation system according to an embodiment of the present disclosure;

[0009] Figure 4 An embodiment of a vehicle dashboard according to one embodiment of this disclosure is shown;

[0010] Figure 5 This is a block diagram of an embodiment of the vehicle's communication subsystem;

[0011] Figure 6 This is a block diagram of the computing environment related to the implementation scheme proposed in this paper;

[0012] Figure 7This is a block diagram of a computing device associated with one or more of the components described herein;

[0013] Figure 8 This is an example block diagram illustrating time-of-flight (ToF) determination using a passive image sensor and existing light source, according to various aspects of this disclosure;

[0014] Figure 9 This is an example light source operation used to perform ToF determination based on aspects of this disclosure;

[0015] Figure 10 This is an example sensor operation used to perform ToF determination based on aspects of this disclosure; and

[0016] Figures 11 to 12 This is a flowchart illustrating aspects of this disclosure. Detailed Implementation

[0017] Time-of-flight (ToF) sensors have been developed for use in determining the range from a camera (e.g., or a similar sensor or apparatus) to an object. For example, a ToF camera is an active sensor capable of both emitting and receiving light. A ToF camera emits a light pulse with known characteristics (e.g., pulse signal, pulse frequency, spectral composition, intensity, etc.) and measures the time elapsed between the emission of the light and the reception of the reflected light by the sensor. Based on the time elapsed between emitting the light pulse and receiving the reflected light pulse, the distance between the ToF camera and the reflecting target (i.e., any object near the ToF camera) can be calculated based on the speed of light.

[0018] In a typical implementation, the sensor system includes a light source that outputs light of a selected wavelength or wavelength range. The sensor system may also include an optical bandpass filter or an optical longpass filter. The time required for light to travel from the light source output, reflect off an object (e.g., a reflective target) within the field of view of the ToF camera, and return to the sensor can be used to calculate the range or distance from the ToF camera to the object. In some implementations, as described herein, vehicles can use ToF sensors and cameras (e.g., optical imaging, detection and ranging (LiDAR) sensors, or other ranging and imaging systems) to detect objects outside or inside the vehicle for various purposes (e.g., to assist in autonomous driving operations, to identify passengers within the vehicle, etc.). However, the operation of ToF sensors and cameras is relatively power-intensive and has limited applications (i.e., depth detection).

[0019] Some vehicles include multiple cameras and sensors to achieve different intended purposes. For example, a vehicle may include cameras and sensors to enable automated operation of the vehicle (e.g., autonomous driving), assist the driver or operator of the vehicle (e.g., cameras for reversing and parking, navigation, etc.), or perform additional operations on the vehicle. Unlike ToF cameras, these cameras and sensors (e.g., traditional automotive cameras) are passive sensors. That is, these passive cameras and sensors do not actively emit light and do not need to utilize light emission to operate. In any case, these sensors are passive, and therefore, without resource-intensive computer vision post-processing of the scene, these sensors do not have the ability to determine the distance to a target.

[0020] In addition to the multiple cameras and sensors on the vehicle, the vehicle may also include one or more light sources designed to serve additional purposes or assist the operation of the cameras and sensors. For example, the vehicle may include various onboard light sources, such as, but not limited to, headlights, taillights, turn signals, additional hazard lights, reversing lights, interior lights, ground lighting, etc. However, similar to existing relatively power-dense cameras designated for depth or distance detection, these light sources on the vehicle may also be passive, because these light sources can emit light but cannot sense light or generate information based on sensed light, and therefore cannot determine the distance to a target (e.g., perform Time-of-Flight (ToF) determination).

[0021] As described herein, vehicles can use passive sensors (e.g., passive image sensors) with existing onboard light sources (e.g., passive light sources) to perform Time-of-Flight (ToF) calculations or determinations. For example, a vehicle can use a conventional automotive camera (which is a non-illuminating passive sensor) and pair it with various light sources on the vehicle to construct a ToF camera capable of providing depth information. To achieve this construction of a ToF camera using an automotive camera and light sources, the vehicle (e.g., or a controller within the vehicle) synchronizes the passive automotive camera with the onboard light sources.

[0022] As an example, a vehicle can use a reversing camera and taillights to form or construct a ToF camera. In this example, at the start of each light pulse emitted from the taillight, an electronic signal is sent from the taillight (e.g., via a physical cable) to the reversing camera (e.g., a light-emitting diode (LED) taillight is pulsed during operation, and a halogen taillight can be made to apply the pulse). The reversing camera receives this electronic signal and starts an internal timer. The internal timer stops when the reversing camera receives a light pulse emitted from the taillight that has now been reflected from the surrounding environment behind the vehicle. The calculation of the distance between the ToF camera (e.g., the reversing camera connected to the taillight in this example) and any reflecting object is given by the following Equations 1 and 2. Equation 2 can be considered a general representation of Equation 1, where d represents "distance", t represents "internal timer reading", and c represents the speed of light.

[0023] Distance = Internal timer reading * Speed ​​of light * 0.5 (1)

[0024]

[0025] While a reversing camera and taillights have been used as examples in this discussion, the same concept of calculating and determining distance to objects can be applied to other combinations of car cameras with onboard light sources around the vehicle. For example, a vehicle could use a combination of turn signals on the fenders and cameras on the side mirrors, a combination of headlights and a front parking camera, a combination of headlights and one or more front advanced driver assistance system (ADAS) cameras, or additional suitable light sources and sensors from the vehicle to form a Time-of-Flight (ToF) camera.

[0026] Additionally, vehicles can utilize automotive cameras and various light sources to achieve high dynamic range (HDR) for ToF cameras. In some examples, one or more light sources include large-sized light sources (e.g., large headlights), light sources with discrete components within them (e.g., active elements separated from passive sensors within the light source), or both. Based on these characteristics of the light sources, the dynamic range of the system described herein can be improved. For example, the intensity and / or frequency of the pulses emitted by the light source can be increased or decreased based on the level of “noise” generated in the sensor output. This adjustment of pulse intensity and / or frequency can be used in situations where multiple objects with different reflectivities exist in the same scene. That is, the light source can be pulsed with different intensities and / or frequencies, allowing the camera to capture images from both low-reflectivity and high-reflectivity objects. The low-reflectivity and high-reflectivity images can be superimposed using appropriate HDR techniques to form an HDR ToF image (e.g., a depth image of a scene with both bright and dark areas).

[0027] In some examples, the vehicle can vary the number of pulsed LEDs in the light source, which is part of the ToF camera described herein. For instance, the vehicle can start by applying pulses to a small string of LEDs in the ToF camera and can add more LEDs until the range for performing ToF calculations is met. In some examples, the vehicle can use the same camera designed to achieve HDR but with different pulsed light. Additionally, the light sources can be made to operate independently of each other and can therefore be used to independently illuminate different areas around the vehicle for use with multiple passive sensors placed at different locations around the vehicle.

[0028] In some examples, a vehicle can use multiple automotive cameras or sensors and multiple light sources to construct multiple Time-of-Flight (ToF) cameras around the vehicle. For example, a vehicle can use passive sensors (e.g., cameras) placed around the vehicle and synchronized with different light sources to support multiple ToF cameras. In some examples, multiple cameras (e.g., passive sensors) can be synchronized with the same light source to expand the effective field of view of the system. That is, the vehicle can enable multiple cameras to sense from a single light source, thereby expanding the field of view that can be captured from a single light source.

[0029] Additionally, when operating and configuring a ToF camera, the vehicle can employ a variable field of view for the camera. For example, depending on the vehicle's speed, the vehicle (e.g., or a controller within the vehicle) can actuate a light source (e.g., a headlight) such that the cone angle of the beam emitted from the light source widens at lower speeds and narrows at higher speeds. In some examples, the vehicle can adjust the cone angle of the beam emitted from the light source by using small electric motors built into the housing of the light source (e.g., or built into individual lamps within the light source), which respond to information provided by sensors on or within the vehicle to determine how fast the vehicle is traveling, the degree to which the driver is turning the steering wheel, whether the vehicle is going uphill or downhill, etc., and adjust the light source accordingly. Alternatively or alternatively, if a light source comprises multiple individual lamps (e.g., a headlight may include multiple LEDs), the vehicle can selectively turn on or off a subset of the individual lamps to adjust the cone angle or additional characteristics of the beam emitted from the light source. This actuation or adjustment of the light source allows the system (e.g., the vehicle / controller operating the constructed ToF camera) to collect depth and angular position information at higher resolution based on the vehicle's speed. Alternatively, the vehicle can use both small and large field-of-view cameras to detect different parts of the scene surrounding the vehicle.

[0030] In some examples, the vehicle may employ one or more additional features to enhance the operation of the ToF sensor system constructed from a combination of existing cameras and sensors on the vehicle and a light source used by the vehicle. For example, the vehicle (e.g., via a controller) may amplify the wavelength of the light source (e.g., headlights) so that the camera / sensor can better identify light emitted from the light source and reflected back to the camera / sensor from the surrounding area. Alternatively, the vehicle may use a complete replacement of the camera / sensor, wherein the camera / sensor is completely modified to monitor reflected light from the light source (e.g., all pixels of the camera / sensor are controlled to monitor reflected light), or the vehicle may have a partially modified camera / sensor to monitor reflected light (e.g., a subset of the pixels of the camera / sensor is controlled to monitor reflected light). In some examples, the system and ToF camera described herein may be used for both direct ToF calculations and / or indirect ToF calculations, wherein direct ToF calculations use Equations 1 and 2 above, while indirect ToF calculations involve using different phases of light emitted from the light source to calculate the distance to nearby objects.

[0031] This disclosure provides cameras, systems, or devices for Time-of-Flight (ToF) calculations that improve efficiency in power consumption, data transmission, and data processing. For example, instead of using relatively power-intensive sensors (e.g., lidar sensors or other ranging and imaging systems) specifically designed for ToF determination, this disclosure can utilize existing sensors (e.g., cameras) and light sources on the vehicle to perform ToF calculations. Embodiments of this disclosure will be described in conjunction with vehicles, and in some embodiments with electric vehicles, rechargeable electric vehicles, and / or hybrid electric vehicles, and associated systems. Hereinafter, implementations of this disclosure will be described in detail based on the accompanying drawings. Further, in the following embodiments, the same reference numerals will be assigned to the same or equivalent parts or elements, and redundant descriptions will be omitted.

[0032] Figure 1 A perspective view of a vehicle 100 according to an embodiment of this disclosure is shown. The vehicle 100 includes a front portion 110, a rear portion or tail portion, a roof 130, at least one side portion 160, a chassis 140, and an interior portion 150 (e.g., interior space). In any case, the vehicle 100 may include a frame 104 and one or more body panels 108 mounted or attached thereto. The vehicle 100 may include one or more interior components (e.g., components within the vehicle interior 150 or user space of the vehicle 100), exterior components (e.g., components outside the vehicle interior 150 or user space of the vehicle 100), a drive system, a control system, structural components, etc.

[0033] Although shown as a passenger car, it should be understood that the vehicle 100 described herein can include any means of transport or any type of transport vehicle designed to move one or more tangible objects, such as people, animals, goods, etc. The term "vehicle" does not require the means of transport to be moving or capable of moving. Typical vehicles may include, but are not limited to, passenger cars, trucks, motorcycles, buses, automobiles, trains, rail transport vehicles, ships, vessels, maritime transport vehicles, submarine transport vehicles, airplanes, space shuttles, aircraft, human-powered transport vehicles, etc.

[0034] In some embodiments, vehicle 100 may include a plurality of sensors, devices, and / or systems capable of assisting driving operations, such as automatic or semi-automatic control. Examples of various sensors and systems may include, but are not limited to, one or more of the following: cameras (e.g., standalone images, stereo images, combined images, etc.), infrared (IR) sensors, radio frequency (RF) sensors, ultrasonic sensors (e.g., transducers, transceivers, etc.), radar sensors (e.g., object detection sensors and / or systems), lidar systems, ranging sensors and / or devices (e.g., encoders, etc.), orientation sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), navigation sensors and systems (e.g., Global Positioning System (GPS), etc.), and other ranging, imaging, and / or object detection sensors. Sensors may be disposed within the vehicle interior 150 of vehicle 100 and / or on the exterior of vehicle 100. In some embodiments, sensors and systems may be disposed in one or more parts of vehicle 100 (e.g., frame 104, body panels, compartments, etc.).

[0035] Vehicle sensors and systems can be selected and / or configured to suit the level of operation associated with vehicle 100. Among other things, the number of sensors used in the system can be varied to increase or decrease the information available to the vehicle control system (e.g., affecting the controllability of vehicle 100). Alternatively or additionally, sensors and systems may be part of one or more ADAS systems associated with vehicle 100. In any case, sensors and systems can be used to provide driver assistance at any level of operation (e.g., from fully manual operation to fully automatic operation, etc.), as described herein.

[0036] Various control and / or operation levels of a vehicle can be described as corresponding levels of autonomy for implementing driving operations associated with vehicle 100. For example, in Level 0 or fully manual driving operation, the driver (e.g., a human driver) can be responsible for all driving control operations associated with the vehicle (e.g., steering, acceleration, braking, etc.). Level 0 can be referred to as the "non-automation" level. In Level 1, the vehicle can be responsible for a limited number of driving operations associated with the vehicle, while the driver remains responsible for most driving control operations. Examples of Level 1 vehicles can include vehicles where throttle control and / or braking operations (e.g., cruise control operations, etc.) can be controlled by the vehicle. Level 1 can be referred to as the "driver assistance" level. In Level 2, the vehicle can collect (e.g., via one or more driver assistance systems, sensors, etc.) information about the vehicle's environment (e.g., surrounding area, road, traffic, environmental conditions, etc.) and use the collected information to control driving operations associated with the vehicle (e.g., steering, acceleration, braking, etc.). In Level 2 automated vehicles, the driver can be required to perform other aspects of driving operations not controlled by the vehicle. Level 2 can be referred to as the "partial automation" level. It should be understood that levels 0 through 2 all involve the driver monitoring the vehicle's driving operations.

[0037] At Level 3, the driver can disengage from all driving operations of the vehicle except when the vehicle requests operator action or intervention to control one or more driving operations. In other words, the driver can disengage from control of the vehicle unless requested to take over. Level 3 may be referred to as the "conditional automation" level. At Level 4, the driver can disengage from all driving operations of the vehicle, and the vehicle can maintain control of driving operations even if the user fails to respond to an intervention request. Level 4 may be referred to as the "high automation" level. At Level 5, the vehicle can control all driving operations associated with the vehicle in all driving modes. A Level 5 vehicle can continuously monitor traffic, vehicles, road, and / or environmental conditions while driving. At Level 5, no human driver interaction is required in any driving mode. Accordingly, Level 5 may be referred to as the "full automation" level. It should be understood that at Levels 3 to 5, the vehicle and / or one or more automated driving systems associated with the vehicle monitor the vehicle's driving operations and the driving environment. Here, it should be understood that the example embodiments can be used in conjunction with other automation levels not specifically identified herein.

[0038] like Figure 1As shown, vehicle 100 may include, for example, at least one of the following: a ranging and imaging system 112 (e.g., lidar, etc.), imaging sensors 116A, 116F (e.g., cameras for color image generators, IR sensors, etc.), a radio object detection and ranging system sensor 116B (e.g., radar, RF, etc.), an ultrasonic sensor 116C, and / or other object detection sensors 116D, 116E. In some embodiments, lidar system 112 and / or sensors may be mounted on the roof 130 of vehicle 100. In one embodiment, radar sensor 116B may be located at least at the front 110, rear 120, or side 160 of vehicle 100. Among other things, radar sensors may be used to monitor and / or detect the positions of other vehicles, pedestrians, and / or other objects in or near vehicle 100. Although shown as being associated with one or more areas of vehicle 100, it should be understood that... Figure 1 and Figure 2 Any of the sensors and systems 116A to 116K, 112 shown may be installed in, on, and / or around the vehicle 100 at any location, area, and / or zone of the vehicle 100.

[0039] For reference Figures 8 to 12 In more detail, Figure 1 and Figure 2 One or more sensors and systems 116A to 116K, 112 shown may include passive cameras that can be synchronized with at least one light source 118 on vehicle 100 to temporarily form a Time-of-Flight (ToF) sensor system. For example, vehicle 100 may form a ToF camera system using light source 118 (e.g., headlights, taillights, turn signals, etc.) and passive image sensors 116 (e.g., cameras, such as cameras on side mirrors, front parking cameras, ADAS cameras, etc.). In this case, light source 118 and sensor 116 are synchronized and controlled by vehicle 100 (e.g., via a controller in vehicle 100) to perform ToF determination and calculation. That is, vehicle 100 may control light source 118 to emit light with certain characteristics such that sensor 116 can sense light reflected from any nearby objects in the surrounding area around the vehicle to perform ToF calculations to determine the distance from vehicle 100 to these nearby objects. In some examples, these ToF calculations are then used to support autonomous driving operations, semi-autonomous driving operations, manual driving operations, or combinations thereof.

[0040] Vehicle 100 can switch or transform between a ToF system using passive components as described herein (e.g., passive cameras and passive light sources that have been synchronized to form a ToF system) and a ToF system specifically designed for performing distance measurements (e.g., lidar sensors or other ranging and imaging systems). For example, vehicle 100 can switch between a ToF system using passive components and a ToF system designed for distance measurements based on whether vehicle 100 is operating in a low-power mode (e.g., a low-power mode corresponds to using a ToF system utilizing passive components that consume less power than a power-intensive lidar system), input from the driver of vehicle 100 (e.g., switching between any two ToF systems), information from sensors and systems 116A to 116K, 112 (e.g., the corresponding ToF system can be switched based on different external conditions for vehicle 100 (such as whether it is raining, night or day), or based on different triggers not explicitly indicated herein. In some examples, the low-power mode can be selected or input by the driver of vehicle 100, or it can be activated by the sensors or controllers of vehicle 100 (e.g., if the battery power of vehicle 100 drops below a threshold).

[0041] Now for reference Figure 2 A plan view of a vehicle 100 according to an embodiment of this disclosure will be described. Specifically, Figure 2 A vehicle sensing environment 200 is illustrated, at least partially defined by sensors and systems 116A to 116K, 112 disposed within, above, and / or around a vehicle 100. Each sensor 116A to 116K may include an operational detection range R and an operational detection angle. The operational detection range R may define an effective detection limit or distance for the sensors 116A to 116K. In some cases, this effective detection limit may be defined as the distance from a portion of the sensors 116A to 116K (e.g., a lens, sensing surface, etc.) to a point in space offset from the sensors 116A to 116K. The effective detection limit may define a distance beyond which the sensing capability of the sensors 116A to 116K deteriorates, fails to function, or becomes unreliable. In some embodiments, the effective detection limit may define a distance within which the sensing capability of the sensors 116A to 116K can provide accurate and / or reliable detection information. The operational detection angle may define at least one angle of the span of the sensors 116A to 116K, or at least one angle between a horizontal limit and / or a vertical limit. As can be understood, the operational detection limits and operational detection angles of sensors 116A to 116K together define the effective detection areas 216A to 216D of sensors 116A to 116K (e.g., effective detection area and / or volume, etc.).

[0042] In some embodiments, vehicle 100 may include a ranging and imaging system 112, such as lidar. The ranging and imaging system 112 may be configured to detect visual information in the environment surrounding vehicle 100. The visual information detected in the environment surrounding the ranging and imaging system 112 (e.g., via one or more sensors and / or a system processor, etc.) may be processed to generate a complete 360-degree view of the environment 200 surrounding the vehicle. The ranging and imaging system 112 may be configured to generate a changing 360-degree view of the environment 200 in real time, for example, while vehicle 100 is being driven. In some cases, the ranging and imaging system 112 may have an effective detection limit 204, which is a distance in 360 degrees outward from the center of vehicle 100. The effective detection limit 204 of the ranging and imaging system 112 defines an observation area 208 (e.g., region and / or volume, etc.) around vehicle 100. Any object falling outside the observation area 208 is in an undetected area 212 and will not be detected by the ranging and imaging system 112 of vehicle 100.

[0043] Sensor data and information can be collected by one or more sensors or systems 116A to 116K, 112 of vehicle 100 monitoring the vehicle's sensing environment 200. This information can be processed (e.g., via a processor, computer vision system, etc.) to determine targets (e.g., objects, signs, people, markings, roads, road conditions, etc.) within one or more detection zones 208, 216A to 216D associated with the vehicle's sensing environment 200. In some cases, information from multiple sensors 116A to 116K can be processed to form composite sensor detection information. For example, a first camera 116A and a second camera 116F targeting the forward direction of travel of vehicle 100 can serve as a first sensor 116A and a second sensor 116F. In this example, images collected by cameras 116A, 116F can be combined to form stereo image information. This composite information can enhance the capabilities of individual sensors 116A to 116K by, for example, increasing the ability to determine the depth associated with targets in one or more detection zones 208, 216A to 216D. The same or similar image data can be collected by a rear-view camera (e.g., sensor 116G, 116H) targeting the vehicle 100 in the direction of rearward travel.

[0044] In some embodiments, multiple sensors 116A to 116K can be effectively combined to increase the sensing area and provide increased sensing coverage. For example, multiple radar sensors 116B disposed on the front 110 of a vehicle can be combined to provide a coverage area 216B spanning the entire front 110 of the vehicle. In some cases, multiple radar sensors 116B can cover a detection area 216B that includes one or more other sensor detection areas 216A. These overlapping detection areas can provide redundant sensing, enhanced sensing, and / or provide more sensing detail within a specific portion (e.g., area 216A) of a larger area (e.g., area 216B). Alternatively or additionally, the sensors 116A to 116K of the vehicle 100 can be arranged to produce full coverage via one or more sensing areas 208, 216A to 216D around the vehicle 100. In some areas, the sensing areas 216C of two or more sensors 116D, 116E can intersect at an overlapping area 220. In some areas, the angles and / or detection limits of two or more sensing areas 216C, 216D (e.g., of two or more sensors 116E, 116J, 116K) may meet at a virtual intersection 224.

[0045] Vehicle 100 may include a plurality of sensors 116E, 116G, 116H, 116J, 116K disposed near the rear 120 of vehicle 100. These sensors may include, but are not limited to, imaging sensors, cameras, IR, radio object detection and ranging sensors, radar, RF, ultrasonic sensors and / or other object detection sensors. Among other things, these sensors 116E, 116G, 116H, 116J, 116K can detect targets approaching or near the rear of vehicle 100. For example, another vehicle approaching the rear 120 of vehicle 100 may be detected by one or more of the ranging and imaging system (e.g., lidar) 112, the rear-view cameras 116G, 116H and / or the rear-facing radar sensors 116J, 116K. As described above, images from the rear-view cameras 116G, 116H may be processed to generate a stereo view of targets visible to both cameras 116G, 116H (e.g., providing depth, etc., associated with the object or environment). As another example, vehicle 100 may be in motion, and one or more of the ranging and imaging system 112, forward-facing cameras 116A, 116F, forward-facing radar sensor 116B, and / or ultrasonic sensor 116C can detect targets in front of vehicle 100. This approach can provide critical sensor information to the vehicle control system at at least one of the aforementioned levels of autonomous driving. For example, when vehicle 100 is driving autonomously (e.g., Level 3, Level 4, or Level 5) and detects other vehicles stopped in its path, sensor detection information can be sent to vehicle control system 100 to control driving actions associated with vehicle 100 (e.g., braking, deceleration, etc.) (in this example, slowing vehicle 100 down to avoid a collision with another stopped vehicle). As yet another example, vehicle 100 may be maneuverable, and one or more of the ranging and imaging system 112 and / or side-facing sensors 116D, 116E (e.g., radar, ultrasonic, camera, combinations thereof, and / or other types of sensors) can detect targets located to the side of vehicle 100. It should be understood that sensors 116A to 116K can detect targets that are both on the side 160 and at the front 110 of vehicle 100 (e.g., diagonally positioned with respect to the centerline of vehicle 100 as it travels from the front 110 to the rear 120). Alternatively, sensors 116A to 116K can detect targets that are both on the side 160 and at the rear 120 of vehicle 100, or simultaneously at both locations (e.g., diagonally positioned with respect to the centerline of vehicle 100).

[0046] Figures 3A to 3CThis is a block diagram of an embodiment of the communication environment 300 of a vehicle 100 according to an embodiment of this disclosure. The communication system 300 may include one or more vehicle driving sensors and systems 304, a sensor processor 340, a sensor data storage 344, a vehicle control system 348, a communication subsystem 350, control data 364, a computing device 368, a display device 372, and other components 374 that may be associated with the vehicle 100. These associated components may be electrically and / or communicatively coupled to each other via at least one bus 360. In some embodiments, one or more associated components may transmit and / or receive signals to at least one of a navigation source 356A, a control source 356B, or some other entity 356N via a communication network 352.

[0047] According to at least some embodiments of this disclosure, communication network 352 may include any type of known communication medium or set of communication media and may use any type of protocol, such as SIP, TCP / IP, SNA, IPX, AppleTalk, etc., to transmit messages between endpoints. Communication network 352 may include wired and / or wireless communication technologies. The Internet is an example of communication network 352, which constitutes an Internet Protocol (IP) network comprising numerous computers, computing networks, and other communication devices located around the world, connected via numerous telephone systems and other means. Other examples of communication network 352 include, but are not limited to, standard Common Old-Style Telephone Systems (POTS), Integrated Services Digital Network (ISDN), Public Switched Telephone Network (PSTN), Local Area Networks (LANs) such as Ethernet, Token Ring networks, and / or similar networks, Wide Area Networks (WANs), virtual networks including but not limited to Virtual Private Networks (“VPNs”); the Internet, intranets, extranets, cellular networks, infrared networks; wireless networks (e.g., in the IEEE 802.9 protocol suite, known in the art). This refers to networks operating under any of the protocols and / or any other wireless protocols, as well as any other types of packet-switched or circuit-switched networks and / or any combination of these and / or other networks known in the art. Furthermore, it is understood that the communication network 352 is not necessarily limited to any one network type, but may include many different networks and / or network types. The communication network 352 may include multiple different communication media, such as coaxial cable, copper cable / wire, fiber optic cable, antennas for transmitting / receiving wireless messages, and combinations thereof.

[0048] The driving vehicle sensors and systems 304 may include one or more navigation sensors 308 (e.g., GPS, etc.), one or more orientation sensors 312, one or more ranging sensors 316, one or more lidar sensors 320, one or more radar sensors 324, one or more ultrasonic sensors 328, one or more cameras 332 (e.g., for color image generation), one or more IR sensors 336, and / or other sensors or systems 338. These driving vehicle sensors and systems 304 may be combined with... Figure 1 and Figure 2 The described sensors and systems are similar to (if not identical to) 116A to 116K, 112.

[0049] Navigation sensor 308 may include one or more sensors having a receiver and an antenna, the sensors being configured to utilize a satellite-based navigation system comprising a navigation satellite network capable of providing geolocation and time information to at least one component of vehicle 100. Examples of navigation sensor 308 described herein may include, but are not limited to, at least one of the following: GLO TM Series of GPS and GLONASS combined sensors GPS 15x TM Series of sensors GPS 16x TM A series of sensors with high-sensitivity receivers and antennas. GPS 18xOEM series high-sensitivity GPS sensors, Dewetron DEWE-VGPS series GPS sensors, GlobalSat 1-Hz series GPS sensors, other industrial equivalent navigation sensors and / or systems, and can perform navigation and / or geolocation functions using any known or future-developed standards and / or architectures.

[0050] Orientation sensor 312 may include one or more sensors configured to determine the orientation of vehicle 100 relative to at least one reference point. In some embodiments, orientation sensor 312 may include at least one pressure transducer, stress / strain gauge, accelerometer, gyroscope, and / or geomagnetic sensor. Examples of the navigation sensor 308 described herein may include, but are not limited to, at least one of the following: Bosch Sensortec BMX 160 series low-power absolute orientation sensor, Bosch Sensortec BMX055 9-axis sensor, Bosch Sensortec BMI055 6-axis inertial sensor, Bosch Sensortec BMI160 6-axis inertial sensor, Bosch Sensortec BMF055 9-axis inertial sensor (accelerometer, gyroscope, and magnetometer) with integrated Cortex M0+ microcontroller, Bosch Sensortec BMP280 absolute barometric pressure sensor, Infineon TLV493D-A1B6 3D magnetic sensor, Infineon TLI493D-W1B6 3D magnetic sensor, Infineon TL series 3D magnetic sensors, Murata Electronics SCC2000 series combined gyroscope sensor and accelerometer, Murata Electronics The SCC1300 series combines gyroscope sensors with accelerometers, other industrial equivalent orientation sensors, and / or systems that can perform orientation detection and / or determination functions using any known or future-developed standards and / or architectures.

[0051] The ranging sensor and / or system 316 may include one or more components configured to determine the change of position of vehicle 100 over time. In some embodiments, the ranging system 316 may utilize data from one or more other sensors and / or systems 304 to determine the position (e.g., distance, location, etc.) of vehicle 100 relative to a previously measured position of vehicle 100. Alternatively or additionally, the ranging sensor 316 may include one or more encoders, Hall speed sensors, and / or other measuring sensors / devices configured to measure wheel speed, rotation, and / or revolutions over time. Examples of the ranging sensor / system 316 described herein may include, but are not limited to, at least one of the following: Infineon TLE4924 / 26 / 27 / 28C high-performance speed sensor, Infineon TL4941plusC(B) single-chip differential Hall wheel speed sensor, Infineon TL5041plusC giant magnetoresistive (GMR) effect sensor, Infineon TL series magnetic sensors, EPC 25SP Accu-CoderPro. TMIncremental shaft encoders; EPC 30M compact incremental encoders employing advanced magnetic sensing and signal processing technology; EPC 925 absolute shaft encoders; EPC 958 absolute shaft encoders; EPC MA36S / MA63S / SA36S absolute shaft encoders; Dynapar... TM F18 commutation optical encoder, Dynapar TM The HS35R series phase array encoder sensors, other industrial equivalent ranging sensors and / or systems, can be used with any known or future-developed standard and / or architecture to perform changes in position change detection and / or determination functions.

[0052] The lidar sensor / system 320 may include one or more components configured to use laser illumination to measure distance to a target. In some embodiments, the lidar sensor / system 320 may provide 3D imaging data of the environment surrounding vehicle 100. The imaging data may be processed to generate a full 360-degree view of the environment surrounding vehicle 100. The lidar sensor / system 320 may include a laser generator configured to generate a plurality of target-illuminating laser beams (e.g., laser channels). In some embodiments, the plurality of laser beams may be aimed or directed at a rotating reflective surface (e.g., a mirror) and guided outward from the lidar sensor / system 320 into the measurement environment. The rotating reflective surface may be configured to rotate continuously 360 degrees about an axis such that the plurality of laser beams are guided within a full 360-degree range around vehicle 100. A photodiode receiver of the lidar sensor / system 320 may detect when light emitted from the plurality of laser beams into the measurement environment returns (e.g., reflected echo) to the lidar sensor / system 320. The lidar sensor / system 320 can calculate the distance from vehicle 100 to the illuminated target based on the time associated with the emission of light to the return of the detected light. In some embodiments, the lidar sensor / system 320 can generate more than 2 million points per second and has an effective operating range of at least 100 meters. Examples of the lidar sensor / system 320 described herein may include, but are not limited to, at least one of the following: LiDAR TM HDL-64E 64-channel LiDAR sensor LiDAR TM HDL-32E 32-channel LiDAR sensor LiDAR TM PUCK TM VLP-16 16-channel LiDAR sensor, Leica Geosystems Pegasus: Two mobile sensor platform LIDAR-Lite v3 measurement sensor, Quanergy M8 lidar sensor, Quanergy S3 solid-state lidar sensor The LeddarVU is a compact solid-state fixed-beam lidar sensor, other industrial equivalent lidar sensors and / or systems, and can be used with any known or future-developed standard and / or architecture to perform the detection of illuminated targets and / or obstacles in the environment surrounding vehicle 100.

[0053] Radar sensor 324 may include one or more radio components configured to detect objects / targets in the environment of vehicle 100. In some embodiments, radar sensor 324 may determine, over time, the distance, position, and / or motion vector (e.g., angle, velocity, etc.) associated with a target. Radar sensor 324 may include a transmitter configured to generate and transmit electromagnetic waves (e.g., radio waves, microwaves, etc.) and a receiver configured to detect the returned electromagnetic waves. In some embodiments, radar sensor 324 may include at least one processor configured to interpret the returned electromagnetic waves and determine the positional characteristics of the target. Examples of radar sensor 324 as described herein may include, but are not limited to, at least one of the following: Infineon RASIC. TM The RTN7735PL transmitter and RTN7745PL / 46PL receiver sensors, Autoliv ASP vehicle radar sensors, Delphi L2C0051TR 77GHz ESR electronically scanned radar sensors, Fujitsu Ten Ltd. automotive compact 77GHz 3D electronically scanned millimeter-wave radar sensors, other industrial equivalent radar sensors and / or systems, can be used to perform radio target / obstacle detection in the environment surrounding a vehicle 100 using any known or future-developed standard and / or architecture.

[0054] The ultrasonic sensor 328 may include one or more components configured to detect objects / targets in the environment of the vehicle 100. In some embodiments, the ultrasonic sensor 328 may determine, over time, the distance, position, and / or motion vector (e.g., angle, velocity, etc.) associated with the target. The ultrasonic sensor 328 may include an ultrasonic transmitter and receiver, or transceiver, configured to generate and emit ultrasonic waves and interpret the return echoes of those waves. In some embodiments, the ultrasonic sensor 328 may include at least one processor configured to interpret the returned ultrasonic waves and determine the positional characteristics of the target. Examples of the ultrasonic sensor 328 described herein may include, but are not limited to, at least one of the following: Texas Instruments TIDA-00151 automotive ultrasonic sensor interface IC sensor, MB8450 ultrasonic proximity sensor ParkSonar TM -EZ ultrasonic proximity sensor, Murata Electronics MA40H1S-R open-structure ultrasonic sensor, Murata Electronics MA40S4R / S open-structure ultrasonic sensor, Murata Electronics MA58MF14-7N waterproof ultrasonic sensor, other industrial equivalent ultrasonic sensors and / or systems, and can use any known or future-developed standard and / or architecture to perform ultrasonic detection of targets and / or obstacles in the environment surrounding vehicle 100.

[0055] Camera sensor 332 may include one or more components configured to detect image information associated with the environment of vehicle 100. In some embodiments, camera sensor 332 may include a lens, a filter, an image sensor, and / or a digital image processor. One aspect of this disclosure is that multiple camera sensors 332 can be used together to generate stereo images, thereby providing depth measurements. Examples of camera sensors 332 as described herein may include, but are not limited to, at least one of the following: ON The system includes an MT9V024 global shutter VGAGS CMOS image sensor, a Teledyne DALSAFalcon2 camera sensor, a CMOSIS CMV50000 high-speed CMOS image sensor, other industrial equivalent camera sensors, and / or systems. The camera sensor 332 can perform visual target and / or obstacle detection in the environment surrounding the vehicle 100 using any known or future-developed standard and / or architecture. In at least one example embodiment, the camera sensor 332 can be synchronized with one or more existing light sources (e.g., headlights) to form a ToF sensor system.

[0056] IR sensor 336 may include one or more components configured to detect image information associated with the environment of vehicle 100. IR sensor 336 may be configured to detect targets in low-light, dark, or poorly lit environments. IR sensor 336 may include an IR emitting element (e.g., an IR light-emitting diode (LED) and an IR photodiode. In some embodiments, the IR photodiode may be configured to detect returned IR light with the same or approximately the same wavelength as emitted by the IR emitting element. In some embodiments, IR sensor 336 may include at least one processor configured to interpret the returned IR light and determine the positional characteristics of the target. IR sensor 336 may be configured to detect and / or measure the temperature associated with a target (e.g., an object, pedestrian, other vehicle, etc.). Examples of IR sensor 336 as described herein may include, but are not limited to, at least one of the following: a photodiode lead salt IR array sensor, a photodiode OD-850 near-infrared LED sensor, a photodiode SA / SHA727 steady-state IR emitter and IR detector. LS microbolometer sensor, TacFLIR 380-HD InSbMWIR FPA and HD MWIR thermal sensors VOx 640x 480 pixel detector sensor, Delphi IR sensor, other industrial equivalent IR sensors and / or systems, and perform IR visual target and / or obstacle detection in the environment surrounding vehicle 100 using any known or future-developed standard and / or architecture.

[0057] The vehicle 100 may also include one or more internal sensors 337. The internal sensors 337 can measure characteristics of the internal environment of the vehicle 100. These internal sensors 337 can be combined as follows: Figure 3B Described.

[0058] Navigation system 302 may include any hardware and / or software for manually or automatically navigating the vehicle. Navigation system 302 may be, for example, combined with... Figure 3C Described.

[0059] In some embodiments, the vehicle sensor and system 304 may include other sensors 338 and / or combinations of the aforementioned sensors 306 to 337. Alternatively or additionally, one or more of the aforementioned sensors 306 to 337 may include one or more processors configured to process and / or interpret signals detected by the one or more sensors 306 to 337. In some embodiments, the processing of at least some sensor information provided by the vehicle sensor and system 304 may be performed by at least one sensor processor 340. Raw and / or processed sensor data may be stored in a sensor data memory 344 storage medium. In some embodiments, the sensor data memory 344 may store instructions used by the sensor processor 340 to process the sensor information provided by the sensor and system 304. In any case, the sensor data memory 344 may be a disk drive, an optical storage device, a solid-state storage device such as random access memory (“RAM”) and / or read-only memory (“ROM”), which may be programmable, flash-updatable, etc.

[0060] The vehicle control system 348 can receive processed sensor information from the sensor processor 340 and determine an aspect of the vehicle 100 to be controlled. Controlling an aspect of the vehicle 100 may include presenting information via one or more display devices 372 associated with the vehicle, sending commands to one or more computing devices 368 associated with the vehicle, and / or controlling the driving operations of the vehicle. In some embodiments, the vehicle control system 348 may correspond to one or more computing systems that control the driving operations of the vehicle 100 according to the aforementioned level of driving autonomy. In one embodiment, the vehicle control system 348 can operate the speed of the vehicle 100 by controlling the output signals to the accelerometer and / or braking system of the vehicle. In this example, the vehicle control system 348 may receive sensor data describing the environment surrounding the vehicle 100 and determine, based on the received sensor data, to adjust the acceleration, power output, and / or braking of the vehicle 100. The vehicle control system 348 may also control the steering and / or other driving functions of the vehicle 100.

[0061] The vehicle control system 348 can communicate in real time with driving sensors and system 304, thereby forming a feedback loop. Specifically, upon receiving sensor information describing the target conditions in the environment surrounding vehicle 100, the vehicle control system 348 can automatically change the driving operation of vehicle 100. Then, the vehicle control system 348 can receive subsequent sensor information describing any changes to the target conditions detected in the environment due to the changed driving operation. This continuous cycle of observation (e.g., via sensors, etc.) and action (e.g., selected control or non-control of vehicle operation, etc.) allows vehicle 100 to operate automatically in the environment.

[0062] In some embodiments, one or more components of the vehicle 100 (e.g., driving vehicle sensor 304, vehicle control system 348, display device 372, etc.) can communicate with one or more entities 356A to 356N via communication network 352 through communication subsystem 350 of the vehicle 100. Figure 5 An embodiment of the communication subsystem 350 is described in more detail. For example, the navigation sensor 308 may receive global positioning, location, and / or navigation information from the navigation source 356A. In some embodiments, and to name only a few examples, the navigation source 356A may be a Global Navigation Satellite System (GNSS) similar to (if not identical to) NAVSTAR GPS, GLONASS, EU Galileo, and / or BeiDou Navigation Satellite System (BDS).

[0063] In some embodiments, the vehicle control system 348 may receive control information from one or more control sources 356B. The control source 356 may provide vehicle control information, including automatic driving control commands, vehicle operation override control commands, etc. The control source 356 may correspond to an automated vehicle control system, a traffic control system, an administrative control entity, and / or some other control server. One aspect of this disclosure is that the vehicle control system 348 and / or other components of the vehicle 100 may exchange messages with the control source 356 via a communication network 352 and through a communication subsystem 350.

[0064] Information associated with controlling the driving operations of vehicle 100 can be stored in the control data memory 364 storage medium. The control data memory 364 can store instructions used by the vehicle control system 348 to control the driving operations of vehicle 100, historical control information, autonomous driving control rules, etc. In some embodiments, the control data memory 364 can be a disk drive, an optical storage device, a solid-state storage device such as random access memory (“RAM”) and / or read-only memory (“ROM”), which can be programmable, flash-updatable, etc.

[0065] In addition to the mechanical components described herein, vehicle 100 may also include multiple user interface devices. User interface devices receive human input and convert it into mechanical motion or electrical signals or stimuli. Human input may be one or more of the following: motion (e.g., body movement in two or three-dimensional space, body part movement, etc.), voice, touch, and / or physical interaction with components of vehicle 100. In some embodiments, human input may be configured to control one or more functions of vehicle 100 and / or systems of vehicle 100 described herein. The user interface may include, but is not limited to, at least one graphical user interface of the following: display device, steering wheel or steering mechanism, gear lever or buttons (e.g., including parking position, neutral position, reverse position and / or drive position, etc.), accelerator control pedal or mechanism, brake control pedal or mechanism, power control switching, communication device, etc.

[0066] Figure 3B A block diagram illustrating an embodiment of internal sensors 337 of vehicle 100 is shown. These internal sensors 337 may be arranged into one or more groups, at least in part based on their functionality. For example, the interior space of vehicle 100 may include environmental sensors, one or more user interface sensors, and / or safety sensors. Alternatively or additionally, sensors associated with different devices within the vehicle (e.g., smartphones, tablets, laptops, wearable devices, etc.) may also be present.

[0067] Environmental sensors may include sensors configured to collect data relating to the interior environment of vehicle 100. Examples of environmental sensors may include, but are not limited to, one or more of the following: oxygen / air sensor 301, temperature sensor 303, humidity sensor 305, light / photoelectric sensor 307, and more. Oxygen / air sensor 301 may be configured to detect the quality or characteristics of the air in the interior space 108 of vehicle 100 (e.g., including the ratio and / or type of gases in the air within vehicle 100, hazardous gas levels, safe gas levels, etc.). Temperature sensor 303 may be configured to detect temperature readings of one or more objects, users 216, and / or areas within vehicle 100. Humidity sensor 305 may detect the amount of water vapor present in the air within vehicle 100. Light / photoelectric sensor 307 may detect the amount of light present in vehicle 100. Furthermore, light / photoelectric sensor 307 may be configured to detect different levels of light intensity associated with light within vehicle 100.

[0068] User interface sensors may include sensors configured to collect data relating to one or more users (e.g., drivers and / or one or more passengers) in vehicle 100. As will be understood, user interface sensors may include sensors configured to collect data from user 216 in one or more areas of vehicle 100. Examples of user interface sensors may include, but are not limited to, one or more of the following: IR sensor 309, motion sensor 311, weight sensor 313, wireless network sensor 315, biometric sensor 317, camera (or image) sensor 319, audio sensor 321, and more.

[0069] The IR sensor 309 can be used to measure IR light emitted from at least one surface in the vehicle 100, a user, or other object. Among other things, the IR sensor 309 can be used to measure temperature, form images (especially under low light conditions), identify the user 216, and even detect motion in the vehicle 100.

[0070] Motion sensor 311 can detect the movement and / or motion of objects within vehicle 100. Optionally, motion sensor 311 can be used alone or in combination to detect movement. For example, when a passenger in the rear of vehicle 100 unbuckles their seatbelt and begins to move around in vehicle 100, the user may be operating vehicle 100 (e.g., while driving). In this example, the passenger's movement can be detected by motion sensor 311. In response to the detection of such movement and / or the direction associated with such movement, the passenger can be prevented from touching and / or engaging at least some vehicle control features. As understood, such movement / motion can be alerted to the user, allowing the user to take action to prevent the passenger from interfering with vehicle control. Optionally, the number of motion sensors in the vehicle can be increased to improve the accuracy of associating motion detected in vehicle 100.

[0071] Weight sensor 313 can be used to collect data related to objects and / or users in different areas of vehicle 100. In some cases, weight sensor 313 may be included in the seat and / or floor of vehicle 100. Optionally, vehicle 100 may include wireless network sensor 315. Sensor 315 may be configured to detect one or more wireless networks within vehicle 100. Examples of wireless networks may include, but are not limited to, those utilizing… Wi-Fi TM Wireless communication using ZigBee, IEEE 802.11, and other wireless technology standards. For example, a mobile hotspot can be detected within vehicle 100 via wireless network sensor 315. In this case, vehicle 100 can determine to share the detected mobile hotspot using and / or via one or more other devices associated with vehicle 100.

[0072] Biometric sensor 317 can be used to identify and / or record characteristics associated with a user. It is contemplated that biometric sensor 317 may include at least one of the following: an image sensor, an IR sensor, a fingerprint reader, a weight sensor, a pressure measuring element, a force transducer, a heart rate monitor, a blood pressure monitor, and the like described herein.

[0073] Camera sensor 319 can record still images, video, and / or combinations thereof. Camera sensor 319 can be used alone or in combination to identify objects, users, and / or other features within vehicle 100. Among other things, two or more camera sensors 319 can be used in combination to form stereoscopic and / or three-dimensional (3D) images. Stereoscopic images can be recorded and / or used to determine depth associated with objects and / or users within vehicle 100. Furthermore, the combined use of camera sensors 319 can determine complex geometry associated with features for user identification. For example, camera sensor 319 can be used to determine dimensions between various features of a user's face (e.g., depth / distance from the user's nose to the user's cheek, linear distance between the centers of the user's eyes, and more). These dimensions can be used to verify, record, and even modify the features used to identify the user. Camera sensor 319 can also be used to determine movement associated with objects and / or users within vehicle 100. It should be understood that the number of image sensors used in vehicle 100 can be increased to provide greater dimensional accuracy and / or views of images detected within vehicle 100. In at least one example embodiment, the camera sensor 319 may be synchronized with one or more existing light sources (e.g., internal lights) to form a ToF sensor system.

[0074] Audio sensor 321 can be configured to receive audio input from a user of vehicle 100. Audio input from the user can correspond to voice commands, conversations detected in vehicle 100, telephone calls made in vehicle 100, and / or other auditory expressions made in vehicle 100. Audio sensor 321 can include, but is not limited to, microphones and other types of acoustic-electric transducers or sensors. Optionally, internal audio sensor 321 can be configured to receive sound waves and convert them into equivalent analog or digital signals. Internal audio sensor 321 can be used to determine one or more locations associated with various sounds in vehicle 100. The location of a sound can be determined based on comparisons of volume levels, intensities, etc., between sounds detected by two or more internal audio sensors 321. For example, a first audio sensor 321 can be located in a first area of ​​vehicle 100, and a second audio sensor 321 can be located in a second area of ​​vehicle 100. If the first audio sensor 321 detects a sound at a first volume level, and the second audio sensor 321 detects a sound at a second, higher volume level in the second area of ​​vehicle 100, it can be determined that the sound is closer to the second area of ​​vehicle 100. As can be understood, the number of sound receivers used in vehicle 100 can be increased (e.g., more than two, etc.) to increase the accuracy of measurements around sound detection and the location or source of sound (e.g., via triangulation, etc.).

[0075] Safety sensors may include sensors configured to collect data relating to the safety of the user and / or one or more components of the vehicle 100. Examples of safety sensors may include, but are not limited to, one or more of the following: force sensor 325, mechanical motion sensor 327, orientation sensor 329, restraint sensor 331, and more.

[0076] Force sensor 325 may include one or more sensors configured within vehicle 100 to detect forces observed in vehicle 100. An example of force sensor 325 may include a force transducer that converts the measured force (e.g., force, weight, pressure, etc.) into an output signal. Mechanical motion sensor 327 may correspond to an encoder, accelerometer, damped mass, and the like. Optionally, mechanical motion sensor 327 may be adapted to measure gravity (i.e., G-force) observed within vehicle 100. Measuring the G-force observed within vehicle 100 can provide valuable information relating to acceleration, deceleration, impact, and / or forces that one or more users in vehicle 100 may have experienced. Orientation sensor 329 may include an accelerometer, gyroscope, magnetic sensor, etc., configured to detect orientation associated with vehicle 100.

[0077] Restraint sensor 331 may correspond to a sensor associated with one or more restraint devices and / or systems in vehicle 100. Seat belts and airbags are examples of restraint devices and / or systems. As can be understood, restraint devices and / or systems may be associated with one or more sensors configured to detect the state of the device / system. This state may include extension, engagement, retraction, disengagement, deployment, and / or other electrical or mechanical conditions associated with the device / system.

[0078] The associated device sensor 323 may include any sensor associated with a device in vehicle 100. As previously described, typical devices may include smartphones, tablets, laptops, etc. It is anticipated that vehicle control system 348 may employ the various sensors associated with these devices. For example, a typical smartphone may include image sensors, IR sensors, audio sensors, gyroscopes, accelerometers, wireless network sensors, fingerprint readers, and more. One aspect of this disclosure is that one or more of these associated device sensors 323 may be used by one or more subsystems of vehicle 100.

[0079] Figure 3C One or more GPS / navigation subsystems 302 are shown. The navigation subsystems 302 can be any existing or future-built navigation system that can use location data, such as from a Global Positioning System (GPS), to provide navigation information or control vehicle 100. The navigation subsystems 302 may include components such as, but not limited to, one or more of the following: GPS antenna / receiver 331, location module 333, map database 335, etc. Typically, the components or modules 331 to 335 can be hardware, software, firmware, computer-readable media, or a combination thereof.

[0080] The GPS antenna / receiver 331 can be any antenna, GPS disk, and / or receiver capable of receiving signals from GPS satellites or other navigation systems. These signals can be demodulated, converted, interpreted, etc., by the GPS antenna / receiver 331 and provided to the location module 333. Therefore, the GPS antenna / receiver 331 can convert time signals from the GPS system and provide location (e.g., coordinates on a map) to the location module 333. Alternatively, the location module 333 can interpret the time signals as coordinates or other location information.

[0081] The location module 333 may be a controller for a satellite navigation system designed for use in vehicle 100. The location module 333 may acquire orientation data from GPS antenna / receiver 331 to locate a user or vehicle 100 on a road within the unit's map database 335. The location module 333 may use the road database 335 to provide orientation for other locations along roads also in the database 335. When GPS signals are unavailable, the location module 333 may apply dead reckoning to estimate distance data from sensors 304, including but not limited to one or more of the following: speed sensors, gyroscopes, accelerometers, etc., attached to the drivetrain of vehicle 100. Alternatively or concurrently, the location module 333 may use known locations of Wi-Fi hotspots, cellular tower data, etc., to determine the position of vehicle 100 using techniques such as Time Difference of Arrival (TDOA) and / or Frequency Difference of Arrival (FDOA).

[0082] Map database 335 may include any hardware and / or software for storing information about maps, Geographic Information System (GIS) information, location information, etc. Map database 335 may include any data definitions or other structures for storing this information. Typically, map database 335 may include a road database, which may include one or more vector maps of areas of interest. Street names, street numbers, house numbers, and other information may be encoded as geographic coordinates, allowing users to find a desired destination using street addresses. Points of interest (waypoints) may also be stored along with their geographic coordinates. For example, points of interest may include information such as high-speed cameras, gas stations, public parking lots, and "stopped here" (or "you parked here"). Map database 335 may also include road or street features such as speed limits, the location of stop lights / stop signs, lane markings, school locations, etc. The content of the map database may be generated or updated by a server connected via a wireless system communicating with the Internet, even as vehicle 100 travels along existing streets, thus producing an up-to-date map.

[0083] When operating at L4 or L5 and based on sensor information from external and internal vehicle sensors, the vehicle control system 348 can control the vehicle's driving behavior in response to the current vehicle position, sensed object information, sensed vehicle occupant information, vehicle-related information, external environment information, and navigation information from the map database 335.

[0084] Sensed object information refers to information sensed about objects outside the vehicle. Examples include: living objects, such as animals and their attributes (e.g., animal type, current spatial location, current activity, etc.) and pedestrians and their attributes (e.g., identity, age, gender, current spatial location, current activity, etc.); and inanimate objects and their attributes, such as other vehicles (e.g., current vehicle status or activity (parked or in motion or currently used level of automation), occupant or operator identity, vehicle type (truck, car, etc.), vehicle spatial location, etc.), curbs (terrain and spatial location), potholes (size and spatial location), lane markings (type or color and spatial location), signs (type or color and spatial location, such as speed limit signs, yield signs, stop signs, and other restrictive or warning signs), traffic lights (e.g., red, yellow, blue, green, etc.), buildings (spatial location), walls (height and spatial location), roadblocks (height and spatial location), etc.

[0085] Sensed occupant information refers to information sensed about occupants inside the vehicle. Examples include the number and identity of occupants and their attributes (e.g., seating position, age, gender, gaze direction, biometrics, authentication information, preferences, historical behavioral patterns (such as current or historical user driving behavior, historical user routes, destination and waypoint preferences), nationality, ethnicity and race, language preferences (e.g., Spanish, English, Chinese, etc.), current occupant role (e.g., operator or passenger), occupant priority ranking (e.g., the vehicle owner is ranked higher than child occupants), and electronic calendar information (e.g., Outlook). TM (and medical information and history, etc.)

[0086] Vehicle-related information refers to the sensed information about the selected vehicle. Examples include vehicle manufacturer, type, model, year of manufacture, current geographic location, current vehicle status or activity (parked or in motion or currently used level of automation), vehicle specifications and capabilities, currently sensed vehicle operating parameters, and other information.

[0087] External environment information refers to sensed information about the external environment of the selected vehicle. Examples include road type (paved, gravel, brick, etc.), road conditions (e.g., wet, dry, icy, snowy, etc.), weather conditions (e.g., outdoor temperature, pressure, humidity, wind speed and wind direction, etc.), ambient light conditions (e.g., time of day), and the development level of the vehicle's surroundings (e.g., urban or rural).

[0088] In a typical implementation, the automated vehicle control system 348 constructs a three-dimensional map of the space near the vehicle based on feedback from certain sensors, particularly lidar and radar sensors positioned around the vehicle's perimeter. This enables the automated vehicle control system 348 to identify and spatially locate living and inanimate objects. Other sensors, such as inertial measurement units, gyroscopes, wheel encoders, sonar sensors, motion sensors, and outward-facing cameras (e.g., for performing computer vision processing), can provide further contextual information to generate a more accurate three-dimensional map. Navigation information is combined with the three-dimensional map to provide short-, medium-, and long-range route tracking and route selection. The vehicle control system 348 processes real-world information, along with GPS data and vehicle speed, to accurately determine the precise location of each vehicle (down to a few centimeters), while making corrections for nearby living and inanimate objects.

[0089] The vehicle control system 348 can process, essentially in real-time, aggregate mapping information of the occupants of the current vehicle and other nearby living or inanimate objects, model (or predict) their behavior, and issue appropriate commands regarding vehicle operation based on the aggregate mapping information and the modeled behavior. While some commands (such as stopping at red lights and stop signs) are hard-coded into the vehicle, other responses are learned and recorded through configuration file updates based on previous driving experience. Examples of learned behaviors include: a slowly moving or stopped vehicle or an emergency vehicle in the right lane indicates a higher probability that a car following it will attempt to overtake it; potholes, rocks, or other foreign objects in the road are equivalent to a higher probability that the driver will swerve to avoid them; and traffic congestion in one lane means that other drivers moving in the same direction are more likely to overtake in adjacent lanes or drive on the shoulder.

[0090] Figure 4An embodiment of an instrument panel 400 of vehicle 100 is shown. The instrument panel 400 of vehicle 100 includes: a steering wheel 410, a vehicle operation display 420 (e.g., configured to present and / or display driving data, such as speed, measured air resistance, vehicle information, entertainment information, etc.), one or more auxiliary displays 424 (e.g., configured to present and / or display information separate from the operation display 420, entertainment applications, movies, music, etc.), a head-up display 434 (e.g., configured to display any information previously described, including but not limited to, guidance information for a route to a destination, obstacle warning information for warning of potential collisions, or some or all of the main vehicle operation data such as speed, drag, etc.), a power management display 428 (e.g., configured to display data corresponding to the vehicle 100's power level, standby power, charging status, etc.), and an input device 432 (e.g., a controller, touchscreen, or other interface device configured to interface with one or more displays in the instrument panel or components of vehicle 100. The input device 432 may be configured as a joystick, mouse, touchpad, tablet computer, 3D gesture capture device, etc.). In some embodiments, the input device 432 may be used to manually manipulate a portion of the vehicle 100 into a charging position (e.g., move the charging pad to a desired separation distance, etc.).

[0091] While one or more displays of the dashboard 400 may be touchscreen displays, it should be understood that vehicle operation displays may be displays that cannot receive touch input. For example, an operation display 420 spanning the interior space centerline 404 and crossing first zone 408A and second zone 408B may be isolated from receiving input from touch, particularly input from passengers. In some cases, displays providing vehicle operation or critical system information and interfaces may be restricted from receiving touch input and / or configured as non-touchscreen displays. This type of configuration can prevent dangerous errors when providing touch input, where such input could lead to accidents or undesirable control.

[0092] In some embodiments, one or more displays of the dashboard 400 may be mobile devices and / or applications residing on mobile devices such as smartphones. Alternatively or additionally, any information described herein may be presented to one or more portions 420A to 420N of the operating display 420 or other displays 424, 428, 434. In one embodiment, one or more displays of the dashboard 400 may be physically detachable from or removable from the dashboard 400. In some cases, detachable displays may remain tethered to the dashboard.

[0093] Portions 420A to 420N of the operating display 420 can be dynamically reconfigured and / or resized to accommodate any information display described. Alternatively, the number of portions 420A to 420N used to visually present information via the operating display 420 can be dynamically increased or decreased as needed, and is not limited to the configuration shown.

[0094] Figure 5 Hardware diagrams are shown of communication components that may optionally be associated with vehicle 100 according to embodiments of this disclosure.

[0095] The communication component may include one or more wired or wireless devices, such as transceivers and / or modems, which not only allow communication between the various systems disclosed herein but also allow communication with other devices such as devices on a network and / or devices on a distributed network such as the Internet and / or in the cloud and / or with one or more other vehicles.

[0096] The communication subsystem 350 may also include vehicle-to-vehicle and vehicle-to-vehicle communication capabilities, such as hotspot and / or access point connections for any one or more of vehicle occupants and / or vehicle-to-vehicle communication.

[0097] Additionally, although not specifically shown, the communication subsystem 350 may include one or more communication links (which may be wired or wireless) and / or communication buses (managed by the bus manager 574), including one or more of the following: CAN bus, OBD-II, ARCINC 429, Byteflight, CAN (Controller Area Network), D2B (Domestic Digital Bus), FlexRay, DC-BUS, IDB-1394, IEBus, I2C, ISO 9141-1 / -2, J1708, J1587, J1850, J1939, ISO 11783, Keyword Protocol 2000, LIN (Local Internetwork), MOST (Media-Oriented System Transmission), Multifunction Vehicle Bus, SMARTwireX, SPI, VAN (Vehicle Area Network), etc., or generally any communication protocol and / or standard (one or more).

[0098] Various protocols and communications can communicate wirelessly and / or via one or more transmission media, such as single wire, twisted pair, fiber optic, IEEE 1394, MIL-STD-1553, MIL-STD-1773, power line communication, etc. (All of the above standards and protocols are incorporated herein by reference in their entirety).

[0099] As discussed, the communication subsystem 350 enables communication between any vehicle-to-vehicle systems and subsystems and with non-co-located resources (such as those accessible via networks like the Internet).

[0100] In addition to well-known components (omitted for clarity), the communication subsystem 350 includes interconnecting elements, including one or more of the following: one or more antennas 504, interleaver / deinterleaver 508, analog front-end (AFE) 512, memory / storage device / cache 516, controller / microprocessor 520, MAC circuitry system 522, modulator / demodulator 524, encoder / decoder 528, multiple connectivity managers 534, 558, 562, 566, GPU 540, accelerometer 544, multiplexer / demultiplexer 552, transmitter 570, receiver 572, and additional radio components (such as Wi-Fi PHY / Module 580, Wi-Fi / BT MAC module 584, (one or more) additional transmitters 588 and (one or more) additional receivers 592). The various components in device 350 are connected via one or more links / buses 5 (also not shown for clarity).

[0101] Device 350 may have one or more antennas 504 for wireless communication, such as multiple-input multiple-output (MIMO) communication and multi-user multiple-input multiple-output (MU-MIMO) communication. LTE, 4G, 5G, Near Field Communication (NFC), etc., and are generally used for any type of wireless communication. One or more antennas 504 may include, but are not limited to, one or more of the following: directional antennas, omnidirectional antennas, monopole antennas, patch antennas, loop antennas, microstrip antennas, dipole antennas, and any other antennas (one or more) suitable for communication transmission / reception. In an exemplary embodiment, MIMO transmission / reception may require specific antenna spacing. In another exemplary embodiment, MIMO transmission / reception can achieve spatial diversity, thereby allowing different channel characteristics at each antenna. In yet another embodiment, MIMO transmission / reception can be used to allocate resources to multiple users, for example, within vehicle 100 and / or in another vehicle.

[0102] One or more antennas 504 typically interact with an analog front-end (AFE) 512, which is necessary for the proper processing of received modulated signals and the conditioning of transmitted signals. The AFE 512 may functionally be located between the antenna and the digital baseband system to convert analog signals to digital signals for processing and vice versa.

[0103] Subsystem 350 may also include a controller / microprocessor 520 and a memory / storage device / cache 516. Subsystem 350 may interact with the memory / storage device / cache 516, which may store information and operations necessary for configuring and transmitting or receiving the information described herein. The memory / storage device / cache 516 may also be used in conjunction with the controller / microprocessor 520 to execute application programming or instructions, and for temporary or long-term storage of program instructions and / or data. As an example, memory / storage device / cache 520 may include computer-readable devices, RAM, ROM, DRAM, SDRAM, and / or other storage devices (one or more) and media.

[0104] The controller / microprocessor 520 may include a general-purpose programmable processor or controller for executing application programming or instructions associated with subsystem 350. Furthermore, the controller / microprocessor 520 may perform operations for configuring and transmitting / receiving information, as described herein. The controller / microprocessor 520 may include multiple processor cores and / or implement multiple virtual processors. Optionally, the controller / microprocessor 520 may include multiple physical processors. As an example, the controller / microprocessor 520 may include a specially configured application-specific integrated circuit (ASIC) or other integrated circuit, a digital signal processor (one or more), a controller, hardwired electronic or logic circuitry, a programmable logic device or gate array, a dedicated computer, etc.

[0105] Subsystem 350 may further include one or more transmitters 570, 588 and one or more receivers 572, 592, which can transmit and receive signals to and from other devices, subsystems and / or other destinations using the one or more antennas 504 and / or links / buses. Subsystem 350 circuitry includes a media access control or MAC circuitry 522. MAC circuitry 522 provides control over access to the wireless medium. In an exemplary embodiment, MAC circuitry 522 may be arranged to contend for the wireless medium and configure frames or packets transmitted over the wired / wireless medium.

[0106] Subsystem 350 may optionally include a security module (not shown). This security module may contain information about, but not limited to, security parameters required to connect the device to one or more other devices or other available networks (one or more), and may include WEP or WPA / WPA-2 (optionally +AES and / or TKIP) secure access keys, network keys, etc. The WEP secure access key is a secure cipher used by the Wi-Fi network. Knowing this code allows the wireless device to exchange information with the access point and / or another device. Information exchange can be performed via encoded messages, where the WEP access code is typically selected by the network administrator. WPA is an additional security standard also used in conjunction with network connectivity, where encryption is stronger than WEP.

[0107] In some embodiments, the communication subsystem 350 further includes a GPU 540, an accelerometer 544, and Wi-Fi / BT / BLE ( The system includes a low-power PHY module 580 and a Wi-Fi / BT / BLE MAC module 584, as well as an optional wireless transmitter 588 and an optional wireless receiver 592. In some embodiments, the GPU 540 may be a graphics processing unit or visual processing unit including at least one circuit and / or chip that manipulates and modifies memory to accelerate the creation of images in a frame buffer for output to at least one display device. The GPU 540 may include one or more of the following: a display device connection port, a printed circuit board (PCB), a GPU chip, a metal-oxide-semiconductor field-effect transistor (MOSFET), memory (e.g., single data rate random access memory (SDRAM), double data rate random access memory (DDR) RAM, etc., and / or combinations thereof), auxiliary processing chips (e.g., processing video output capabilities, processing and / or other functions besides the GPU chip, etc.), capacitors, heat sinks, temperature control or cooling fans, motherboard connections, shielding, etc.

[0108] Various connectivity managers 534, 558, 562, and 566 manage and / or coordinate communication between subsystem 350 and one or more systems disclosed herein, as well as one or more other devices / systems. Connectivity managers 534, 558, 562, and 566 include a charging connectivity manager 534, a vehicle database connectivity manager 558, a remote operating system connectivity manager 562, and a sensor connectivity manager 566.

[0109] The charging connectivity manager 534 can not only coordinate the physical connectivity between vehicle 100 and the charging equipment / vehicle, but also communicate with one or more of the following: an electricity management controller, one or more third parties, and optionally a billing system(s). As an example, vehicle 100 can establish communication with the charging equipment / vehicle to: coordinate the interconnectivity between the two (e.g., by aligning the charging socket on the vehicle with the charger space on the charging vehicle), and optionally share navigation information. Once charging is complete, the amount of electricity supplied can be tracked and optionally forwarded to, for example, a third party for billing. In addition to managing the connectivity used for exchanging electricity, the charging connectivity manager 534 can also transmit information such as billing information to the charging vehicle and / or the third party. This billing information can be, for example, the vehicle owner, the vehicle's driver / occupant(s), company information, or any information that can generally be used to charge the appropriate entity for the electricity received.

[0110] The Vehicle Database Connectivity Manager 558 allows subsystems to receive and / or share information stored in the vehicle database. This information can be shared with other vehicle components / subsystems and / or other entities such as third parties and / or charging systems. This information can also be shared with one or more vehicle occupant devices, such as an app on a driver's mobile device used to track information about the vehicle and / or dealers or service / maintenance providers. Typically, any information stored in the vehicle database can optionally be shared with any one or more other devices optionally subject to any privacy or confidentiality constraints.

[0111] The remote operating system connectivity manager 562 facilitates communication between vehicle 100 and any one or more automated vehicle systems. This communication may include one or more of the following: navigation information, vehicle information, other vehicle information, weather information, occupant information, or any information generally related to the remote operation of vehicle 100.

[0112] The Sensor Connectivity Manager 566 facilitates communication between any one or more vehicle sensors (e.g., driving vehicle sensors and system 304, etc.) and any one or more other vehicle systems. The Sensor Connectivity Manager 566 can also facilitate communication between any one or more sensors and / or vehicle systems and any other destination (such as a service company, an application, or any destination that typically requires sensor data).

[0113] According to one exemplary embodiment, any communication discussed herein can be transmitted via one or more conductors used for charging. An exemplary protocol that can be used for these communications is power line communication (PLC). A PLC is a communication protocol that uses power lines to simultaneously carry data and alternating current (AC) power transmission or distribution. It is also known as power line carrier, power line digital subscriber line (PDSL), power communication, power line networking (PLN). In a DC environment within a vehicle, a PLC can be used in conjunction with a CAN bus, a LIN bus over power lines (DC-LIN), and a DC-BUS.

[0114] The communications subsystem may also optionally manage one or more identifiers, such as IP (Internet Protocol) addresses (one or more) associated with the vehicle, and one or more of these or other systems, subsystems, components, and / or devices. These identifiers may be used in conjunction with any one or more connectivity managers discussed herein.

[0115] Figure 6 A block diagram is shown of a computing environment 600 that can be used as a server, user computer, or other system provided and described herein. The computing environment 600 includes one or more user computers or computing devices, such as a vehicle computing device 604, a communication device 608, and / or more devices 612. These computing devices 604, 608, 612 may include general-purpose personal computers (by way of example only, including those running various versions of Microsoft Corp.'s...) and / or Apple Corp.'s Personal computers and / or laptops running operating systems; and / or running a wide variety of commercial operating systems. Workstation computers operating systems such as UNIX or similar operating systems. These computing devices 604, 608, and 612 can also have any of a variety of applications, including, for example, database client and / or server applications and web browser applications. Alternatively, these computing devices 604, 608, and 612 can be any other electronic device capable of communicating via network 352 and / or displaying and navigating web pages or other types of electronic documents or information, such as thin client computers, internet-enabled mobile phones, and / or personal digital assistants. Although an exemplary computing environment 600 with two computing devices is shown, any number of user computers or computing devices can be supported.

[0116] The computing environment 600 may also include one or more servers 614, 616. In this example, server 614 is shown as a web server, and server 616 is shown as an application server. Web server 614 can be used to handle requests for web pages or other electronic documents from computing devices 604, 608, 612. Web server 614 can run an operating system, including any of those discussed above and any commercially available server operating system. Web server 614 can also run a wide variety of server applications, including SIP (Session Initiation Protocol) servers, HTTP(s) servers, FTP servers, CGI servers, database servers, etc. Servers, etc. In some cases, network server 614 can publish available operations as one or more network services.

[0117] The computing environment 600 may also include one or more file and / or application servers 616, which, in addition to an operating system, may include one or more applications accessible to clients running on one or more computing devices 604, 608, 612. Servers 616 and / or 614 may be one or more general-purpose computers capable of executing programs or scripts in response to computing devices 604, 608, 612. As an example, servers 616, 614 may execute one or more network applications. Network applications may be implemented as one or more scripts or programs written in any programming language, such as... C Application servers 616 may also include C++ and / or any scripting language, such as Perl, Python, or TCL, and any combination of programming / scripting languages. One or more application servers may also include a database server, including but not limited to those that can access databases from... These are the commercially purchased database servers, which can handle requests from database clients running on computing devices 604, 608, and 612.

[0118] Web pages created by servers 614 and / or 616 can be forwarded to computing devices 604, 608, and 612 via network (file) servers 614 and 616. Similarly, network server 614 can receive web page requests, network service calls, and / or input data from computing devices 604, 608, and 612 (e.g., user computers, etc.) and can forward web page requests and / or input data to network (application) server 616. In another embodiment, server 616 can be used as a file server. Although for ease of description, Figure 6A separate network server 614 and file / application server 616 are shown; however, those skilled in the art will recognize that the functions described with respect to servers 614 and 616 can be performed by a single server and / or multiple dedicated servers, depending on the specific requirements and parameters of the implementation. Computer systems 604, 608, 612, network (file) server 614, and / or network (application) server 616 can be used as... Figures 1 to 6 The system, device or component described in the document.

[0119] The computing environment 600 may also include a database 618. Database 618 may reside in various locations. As an example, database 618 may reside on a local (and / or on) storage medium of one or more computers 604, 608, 612, 614, 616. Alternatively, the database may be located remotely from any or all of computers 604, 608, 612, 614, 616 and communicate with one or more of these computers (e.g., via network 352). Database 618 may reside in a storage area network (“SAN”) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to computers 604, 608, 612, 614, 616 may be stored locally on the respective computers and / or remotely, as appropriate. Database 618 may be a relational database, such as Oracle, suitable for storing, updating, and retrieving data in response to commands in SQL format.

[0120] Figure 7 An embodiment of a computer system 700 is illustrated, on which the aforementioned server, user computer, computing device, or other system or component may be deployed or executed. The computer system 700 is shown as including hardware elements electrically coupled via a bus 704. The hardware elements may include one or more central processing units (CPUs) 708; one or more input devices 712 (e.g., mouse, keyboard, etc.); and one or more output devices 716 (e.g., display devices, printers, etc.). The computer system 700 may also include one or more storage devices 720. As an example, the storage devices 720 may be disk drives, optical storage devices, solid-state storage devices such as random access memory (“RAM”) and / or read-only memory (“ROM”), which may be programmable, flash-updatable, and / or similar.

[0121] Computer system 700 may additionally include a computer-readable storage medium reader 724; a communication system 728 (e.g., a modem, network interface card (wireless or wired), infrared communication device, etc.); and working memory 736, which may include RAM and ROM devices as described above. Computer system 700 may also include a processing acceleration unit 732, which may include a DSP, a dedicated processor, and / or the like.

[0122] The computer-readable storage medium reader 724 can also be connected to computer-readable storage media, which together (and optionally, in conjunction with one or more storage devices 720) comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently containing computer-readable information. The communication system 728 can allow the exchange of data with the network and / or any other computer described above with respect to the computer environment described herein. Furthermore, as disclosed herein, the term "storage medium" can refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, magnetic core memory, disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information.

[0123] Computer system 700 may also include software elements shown as currently residing within working memory 736, including operating system 740 and / or other code 744. It should be understood that alternative embodiments of computer system 700 may have many variations different from those described above. For example, custom hardware and / or specific elements that may be implemented in hardware, software (including portable software such as applets), or both may also be used. Furthermore, connections to other computing devices, such as network input / output devices, may be employed.

[0124] The examples of processors 340 and 708 described herein may include, but are not limited to, at least one of the following: 800 and 801, featuring 4G LTE integration and 64-bit computing. 620 and 615, with 64-bit architecture A7 processor, M7 motion coprocessor, series, Core TM Series processors, Series processors, Atom TM Series processors, Intel Series processors, i5-4670K and i7-4770K 22nm Haswell, i5-3570K 22nm Ivy Bridge, FX TM Series processors, FX-4300, FX-6300 and FX-8350 32nm Vishera Kaveri processor, Texas Jacinto C6000 TM Automotive infotainment processor, Texas OMAP TM Automotive-grade mobile processors Cortex TM -M processor, Cortex-A and ARM926EJ-S TM Processors, other industrial equivalent processors; and can perform computing functions using any known or future-developed standards, instruction sets, libraries, and / or architectures.

[0125] Figure 8 Block diagram 800, according to aspects of this disclosure, is shown as an embodiment for performing Time-of-Flight (ToF) determination using passive sensors and existing light sources on vehicle 100. Block diagram 800 can be implemented as referenced. Figures 1 to 7 The aspects of the vehicle 100 described, or may be referenced as follows. Figures 1 to 7 The vehicle described is implemented in various ways. For example, block diagram 800 may include controller 804 (e.g., reference 804). Figure 3A and Figure 3C The vehicle control system 348 described includes one or more light sources 808 and one or more cameras 812 (e.g., reference 348). Figure 1 and Figure 2 The sensor 116 and reference described Figure 3B Various sensors are described, such as light / photoelectric sensor 307, IR sensor 309, camera sensor 319, etc. As shown, one or more light sources 808 include a first light source 808A, a second light source 808B, etc., up to the Nth light source 808N; and one or more cameras 812 include a first camera 812A, a second camera 812B, etc., up to the Nth camera 812N.

[0126] As described herein, controller 804 can control one or more light sources 808 to emit output light 816 having known characteristics, and can control one or more cameras 812 to monitor and receive any reflected light 824 corresponding to the output light 816 that has been reflected from objects 820 in the surrounding environment around vehicle 100. By controlling one or more light sources 808 and one or more cameras 812 as described herein, controller 804 can generate depth data of the scene using light emitted from one or more light sources 808 and reflected from the scene around vehicle 100 back to one or more cameras 812 (e.g., output light 816). The depth data can correspond to depth values ​​of one or more pixels of camera 812, wherein these depth values ​​are assigned to each pixel based on the time it takes for the emitted light to reflect from an object back to the pixel of camera 812. The depth values ​​can then be used to generate a depth image of the scene, for example, by assigning grayscale values ​​to each depth value and generating an image with the assigned grayscale values.

[0127] In operation, controller 804 may control one or more cameras 812 to start timers, or may start an internal timer of controller 804 itself in response to one or more light sources 808 emitting output light 816. This timer / internal timer stops when one or more cameras 812 detects reflected light 824. The amount of time taken for the output light 816 emitted from one or more light sources 808 to be received at one or more cameras 812 via reflected light 824 can then be used for Time-of-Flight (ToF) calculations (e.g., using Equations 1 and 2 as described above) to determine the position of object 820 relative to vehicle 100. Based on the determined position of object 820, the vehicle may enhance autonomous driving operations (e.g., to avoid collisions with object 820) and / or semi-autonomous driving operations or manual driving operations (e.g., by alerting the driver or operator of vehicle 100 to the proximity of object 820 to vehicle 100). Alternatively or additionally, controller 804 may control one or more internal cameras and one or more internal sensors within vehicle 100 in the same or similar manner to perform ToF determination for identifying objects within vehicle 100.

[0128] In some examples, controller 804 may be configured to operate in two modes: a first mode (e.g., normal or manufacturer-expected operation for one or more light sources 808 and one or more cameras 812) and a second mode (e.g., for performing Time-of-Flight (ToF) determination using one or more light sources 808 and one or more cameras 812). For example, the first mode includes one or more light sources 808 providing a first function to vehicle 100 and one or more cameras 812 providing a second function to vehicle 100. The first function may include controller 804 controlling one or more light sources 808 to illuminate the scene around vehicle 100 using light of visible wavelengths emitted from one or more light sources 808, and the second function may include controller 804 controlling one or more cameras 812 to capture image data for the scene, wherein the image data is used to generate an image of the scene (e.g., a color image, a black-and-white image, etc.). The first and second functions can be performed independently of each other because controller 804 can provide the ToF function without synchronizing one or more light sources 808 with one or more cameras 812. In this way, the operation of controller 804 on light sources 808 and cameras 812 is performed independently of each other in the first mode.

[0129] Alternatively, controller 804 may control one or more light sources 808 and one or more cameras 812 to operate together in a second mode to provide a third function. The third function may include generating depth data based on ToF calculations and determinations as described herein. In some examples, controller 804 may operate simultaneously in both the first and second modes. For example, controller 804 may operate continuously in the first mode to control one or more light sources 808 and one or more cameras 812 to provide the first and second functions, respectively. Accordingly, controller 804 may also operate continuously in the second mode to control one or more light sources 808 and one or more cameras 812 to provide the third function, or it may operate in the second mode as desired or required (e.g., the second mode and the third function are activated in response to triggers such as: human input to the vehicle interface, detection of certain ambient light conditions, detection of certain vehicle surroundings (e.g., detection of high-traffic areas or low-traffic areas), etc.).

[0130] In some examples, controller 804 may switch or transition between a first mode and a second mode for one or more light sources 808 and one or more cameras 812 to provide corresponding first and second functions or to provide a third function. For example, vehicle 100 may switch between the first and second modes based on triggers such as whether vehicle 100 is operating in a low-power mode for collecting depth data, input from the driver of vehicle 100 instructing one or more light sources 808 and(one or more) cameras 812 to collect depth data, or information from the sensors and systems of vehicle 100, or based on different triggers not explicitly indicated herein. In some examples, the low-power mode may be selected or input by the driver of vehicle 100, or may be activated by the sensors or controllers of vehicle 100 (e.g., if the battery power of vehicle 100 drops below a threshold). Alternatively or additionally, controller 804 may activate or operate according to the second mode based on different factors of the environment surrounding vehicle 100 (e.g., whether it is raining, whether it is day or night, traffic volume around vehicle 100, etc.). The controller 804 can also activate or operate the light source 808 and camera 812 at the rear of the vehicle according to the second mode based on different operations performed on the vehicle 100 (for example, if the vehicle 100 is reversing, the light source 808 and camera 812 at the rear of the vehicle can be directed to perform a third function; if the autonomous driving of the vehicle 100 is enabled or in use, the additional light source 808 and camera 812 can be directed to perform a third function, etc.).

[0131] In some examples, each of one or more light sources 808 may include multiple light sources (e.g., multiple LEDs within a single light source 808), wherein the vehicle 100 may vary the number of LEDs pulsed at a light source 808. For example, the vehicle 100 may be activated by pulsed light to a small string of LEDs within one or more light sources 808, and more LEDs may be added until one or more conditions for performing a Time-of-Flight (ToF) calculation are met. Such conditions for performing a ToF calculation may include the range of object detection, the type of object detected, the reflectivity of the detected object, etc. Alternatively or additionally, the conditions under which the controller 804 and the vehicle 100 adjust the number of pulsed LEDs may include information about the environment surrounding the vehicle 100 received via additional sensors of the vehicle 100. For example, the number of pulsed LEDs may be adjusted by the controller 804 and the vehicle 100 based on certain weather conditions (e.g., more LEDs may be pulsed in inclement weather such as rain, fog, or snow), ambient lighting (e.g., more LEDs may be pulsed when ambient lighting is low), or other factors. In some examples, controller 804 can control light source 808 to emit light with different properties (e.g., by controlling multiple light sources within light source 808 to emit light with different light intensities) to generate depth data based on reflected light 824 with different properties.

[0132] The intensity of the pulses emitted by light source 808 can be increased or decreased based on the level of "noise" generated in the sensor output. This adjustment of pulse intensity and / or frequency can be used when multiple objects with different reflectivities exist in the same scene. That is, light source 808 can be pulsed at different intensities (e.g., stronger / weaker light) and / or frequencies, enabling camera 812 to capture images from both low-reflectivity and high-reflectivity objects. The low-reflectivity and high-reflectivity images can be superimposed using appropriate image processing techniques to form an HDR depth image (e.g., a high dynamic range depth image of the scene). In some examples, vehicle 100 can use a single camera 812 to generate HDR images while using different light sources 808 to provide light pulsed in different ways to generate depth data.

[0133] Additionally, the controller 804 can construct a ToF sensor system for generating depth data by associating a single light source 808 with multiple cameras 812. For example, the controller 804 can control the single light source 808 and the multiple cameras 812 to expand the field of view of an image captured according to the depth data (e.g., a depth image of the scene around vehicle 100). Accordingly, the multiple cameras 812 (e.g., passive sensors) can be synchronized with the same light source 808 to expand the effective field of view of the system so that vehicle 100 can enable the multiple cameras 812 to sense reflected light originating from a single light source 808, thereby expanding the field of view that can be captured from the single light source 808.

[0134] In some examples, vehicle 100 can employ a variable camera field of view for one or more cameras 812. That is, the controller 804 of vehicle 100 can control the radiation pattern of one or more light sources 808 to vary based on the vehicle's speed. For example, depending on the speed of vehicle 100, controller 804 can actuate one or more light sources 808 (e.g., headlights) such that the cone angle of the output light 816 emitted from the one or more light sources 808 widens at lower speeds or as the speed of vehicle 100 decreases, and narrows at higher speeds or as the speed of vehicle 100 increases. This actuation or adjustment of one or more light sources 808 allows controller 804 to collect depth and angular position information at higher or lower resolutions based on the vehicle 100's travel speed. Alternatively, vehicle 100 can use small field-of-view cameras and large field-of-view cameras to detect different parts of the scene surrounding vehicle 100.

[0135] Furthermore, the process described herein, performed by controller 804, one or more light sources 808, and one or more cameras 812, can be used for both direct ToF calculations and / or indirect ToF calculations according to Equations 1 and 2 (these indirect ToF calculations involve using different phases of light emitted from one or more light sources 808 to calculate the distance to nearby objects). In some examples, controller 804 can be configured to perform either direct or indirect ToF calculations. Alternatively or additionally, controller 804 can switch between direct and indirect ToF calculations based on different characteristics of vehicle 100 and / or the scene surrounding vehicle 100. To implement indirect ToF calculations, controller 804 can control one or more light sources 808 to emit output light 816 with different phases for different pulses.

[0136] Figure 9 An embodiment of a light source operation 900 for performing ToF determination according to aspects of this disclosure is shown. The light source operation 900 can be implemented as described in the reference. Figures 1 to 8 The aspects of the vehicle 100 described, or may be referenced as follows. Figures 1 to 8The vehicle described is implemented in various ways. For example, the light source operation 900 may include, as referenced... Figure 8 The described light source 808. In some examples, the light source 808 includes a plurality of individual lamps 904 that work together to provide light emitted from the light source 808. As an example, the plurality of individual lamps 904 may be LEDs. Alternatively or alternatively, the plurality of individual lamps 904 may be halogen lamps. In both examples, the individual light source 808 may include multiple light sources (e.g., having a plurality of individual lamps 904).

[0137] In some examples, the controller of vehicle 100 (e.g., reference) Figure 8 The controller 804 described can independently control each individual lamp 904 of the light source 808. (See reference...) Figure 8 The controller can control the light source 808 to operate according to a first mode and / or a second mode. As mentioned above, the first mode can be referred to as the normal operating mode, in which the light source 808 provides a first function, such as (e.g., using visible light or light of visible wavelengths emitted from the light source 808) illuminating the scene around the vehicle 100. Accordingly, the controller can control each individual lamp 904 to provide this first function (e.g., scene illumination) as part of the first mode.

[0138] Alternatively, the vehicle's controller may control the light source 808 to operate according to a second mode. As previously described, the second mode may include the controller controlling the light source 808 to emit light that can be used to perform Time-of-Flight (ToF) determination and collect depth data of the scene around the vehicle 100 (e.g., to determine the distance from the vehicle 100 to any nearby objects). In some examples, the controller may operate according to both the first and second modes simultaneously. As part of the second mode, the controller may control a subset 908 of the individual lamps 904 to perform a third function to emit light with certain characteristics (e.g., light of a different type compared to the light emitted as part of the first mode, such as IR light or other types of non-visible light), enabling separate cameras or sensors to monitor and receive the light emitted from the subset 908 to generate a depth image of the scene around the vehicle 100. [Reference] Figure 10 The technology for monitoring and receiving light emitted from subset 908 is described in more detail by a camera or sensor.

[0139] In some examples, the controller can vary the number of individual lamps 904 pulsed within a subset 908 of light source 808 as part of the Time-of-Flight (ToF) determination and depth data generation described herein. For example, vehicle 100 can be activated by pulsed light applied to a small string of individual lamps 904 within subset 908, and more individual lamps 908 can be added until a range is met for performing ToF determination. Alternatively, controller 804 can control all individual lamps 904 of light source 808 to emit light for a second mode, wherein light source 808 alternates between each operating mode. In some examples, the individual lamps 904 in each light source 808 (e.g., and different light sources 808) can be operated independently of each other and can therefore be used to independently illuminate different areas around vehicle 100 for use with multiple passive sensors (e.g., cameras) placed at different locations around vehicle 100.

[0140] Figure 10 An embodiment of sensor operation 1000 for performing ToF determination according to aspects of this disclosure is shown. Sensor operation 1000 can be implemented as described in reference. Figures 1 to 9 The aspects of the vehicle 100 described, or may be referenced as follows. Figures 1 to 9 The vehicle-specific implementation is described. For example, sensor operation 1000 may include, as referenced... Figure 8 The camera 812 is described. Although discussed as a camera, camera 812 can generally be referred to as a sensor, as in the reference. Figures 1 to 3C As described.

[0141] In some examples, camera 812 includes multiple unit squares 1004, also referred to herein as pixels 1004, arranged in the pixel array of camera 812. Details of the unit pixel 1004 will be described later. For example, each unit pixel 1004 includes a photoelectric conversion element such as a photodiode, and circuitry for generating a pixel signal corresponding to a voltage value generated in the photoelectric conversion element, hereinafter referred to as pixel circuitry. Furthermore, pixel circuitry may include imaging signal generation circuitry. Each photoelectric conversion element may be associated with a corresponding pixel circuitry, or multiple photoelectric conversion elements may be associated with a common pixel circuitry.

[0142] In this example, multiple unit pixels 1004 are arranged in a two-dimensional lattice shape in a pixel array. The multiple unit pixels 1004 can be grouped into multiple pixel blocks or pixel groups, each pixel block or pixel group comprising a predetermined number of unit pixels. In the following text, the set of unit pixels arranged in the horizontal direction is referred to as a "row", and the set of unit pixels arranged in a direction orthogonal to the row is referred to as a "column".

[0143] The charge generated by each unit pixel 1004 corresponds to the amount of light received at the corresponding photoelectric conversion element. The unit pixels 1004 within the pixel array unit of the camera 812 can be disposed in one or more pixel groups 1008. For example, in Figure 10 In the configuration shown, the pixel array unit of camera 812 is composed of pixel group 1008, which includes a set of unit pixels 1004 that receive the wavelength components required to reconstruct color information from the scene. For example, in the case of reconstructing colors based on the three primary colors of red, green, and blue (RGB), in the pixel array unit of camera 812, optical color filter material can be deposited on the pixels according to a predetermined color filter array to control the arrival of light of the desired wavelength on the pixel surface. Specifically, unit pixels 1004 that receive red (R) light, unit pixels 1004 that receive green (G) light, and unit pixels 1004 that receive blue (B) light are arranged in pixel group 1008 according to a predetermined color filter array.

[0144] Examples of color filter array configurations include various arrays or pixel groups, such as a 2×2 pixel Bayer array, a 3×3 pixel color filter array used in X-Trans (trademarked) CMOS sensors (hereinafter also referred to as an "X-Trans (trademarked) array"), a 4×4 pixel quad Bayer array (also referred to as a "Quadra array"), and a 4×4 pixel color filter where a white RGB color filter is combined with a Bayer array (hereinafter also referred to as a "white RGB array"). Other possible 2×2 color filter array configurations include: Red, Clear, Clear, Blue (RCCB); Red, Clear, Clear, Clear (RCCC); Red, Yellow, Yellow, Cyan (RYYCy); and Red, Green, Blue, IR (RGBIR).

[0145] As shown, when employing a pixel group 1008 having unit pixels 1004 arranged in a color filter array and associated color filters, the unit pixels 1004 can be configured to form multiple Bayer arrays. When using a Bayer array as the color filter array configuration, in the pixel array of the camera 812, the basic pattern of the pixel group 1008, comprising a total of four 2×2 pixel unit pixels 1004, is repeated in both the column and row directions. For example, the basic pattern of the pixel group 1008 consists of a unit pixel 1012R including a color filter of red (R), a unit pixel 1012Gr including a color filter of green (Gr), a unit pixel 1012Gb including a color filter of green (Gb), and a unit pixel 1012B including a color filter of blue (B).

[0146] In some examples, the Time-of-Flight (ToF) pixel 1016 may be interspersed or included within the pixel array of the camera 812. The ToF pixel 1016 may be provided as a dedicated pixel for depth detection. The ToF pixel 1016 may include a filter for allowing light of a wavelength emitted by the light source 808 to pass through in order to detect distance to an object. For example, if one or more light sources 808 are capable of emitting infrared or near-infrared light, the ToF pixel 1016 may include a filter that allows infrared or near-infrared light to pass through while blocking other wavelengths of light. However, the example embodiments are not limited thereto, and other filters (e.g., a white filter) may be used depending on the wavelength of light detected for determining distance to an object. As will be understood, a pixel array including the ToF pixel 1016 is useful for simultaneously operating the light source 808 and the camera 812 in both a first and a second mode.

[0147] For reference Figure 8 As part of the second mode and third function described, the controller 804 of the vehicle 100 can control at least some pixels of the camera 812 (e.g., sensor) 1004 to operate in order to obtain information about the light received from a light source (e.g., as shown in the reference). Figures 8 to 9 The timing information of light generated by the light source 808 and reflected from objects or surfaces in the scene surrounding the vehicle 100.

[0148] Alternatively, the controller may control the 1004 pixels of the camera 812 according to, as referenced Figure 8 The first mode and second function described operate in a manner where pixel 1004 operates normally and captures the wavelength components required to reconstruct color information from the scene. Correspondingly, the controller can also control one or more pixels 1004 of camera 812 to function as ToF pixels when the second mode and third function are implemented or activated, and ToF pixel 1016 will then monitor and receive signals from a light source (e.g., as shown in the reference image). Figure 9 The light emitted from a subset 908 of the independent lamps 904 of the light source is described to assist in ToF determination for collecting depth data of the scene around the vehicle 100.

[0149] As described above, the ToF pixel 1016 can sense and reference... Figure 8 The light emitted from the light source (e.g., visible wavelength light) as part of the first mode is described as different from light of a different type, such as IR light or other types of non-visible light. Subsequently, the controller and / or ToF pixel 1016 can obtain timing information about the light received by the light source and reflected from objects or surfaces within the scene surrounding the vehicle 100, where this timing information is used for ToF determination and depth data generation (e.g., ToF depth image). That is, pixel 1004 within the pixel array of camera 812 receives light reflected from objects (e.g., as shown in the reference). Figure 8 The reflected light 824 is described. Subsequently, the controller 804 calculates the distance information of each ToF pixel 1016 based on the time elapsed between the emitted light and the received light, generates a depth image in which the distance to the object is represented by the depth value of each pixel, and outputs the depth image.

[0150] Figure 11 This is a flowchart 1100 illustrating aspects of the operation of a vehicle 100 according to an embodiment of this disclosure. For example, the controller of the vehicle 100 (e.g., as referenced) Figure 8 The controller 804 described can perform operations as shown in flowchart 1100 to assist in performing ToF determination using passive sensors (e.g., automotive cameras) and existing light sources on vehicle 100.

[0151] In operation 1105, the controller can operate in a first mode. In some examples, the first mode includes the controller controlling at least one light source of the vehicle 100 to provide a first function and controlling at least one camera (e.g., a passive sensor) of the vehicle 100 to provide a second function. The first function performed by the at least one light source may include the at least one light source illuminating the scene around the vehicle 100 using light of visible wavelengths emitted from the at least one light source. Additionally, the second function performed by the at least one camera may include the at least one camera capturing image data of the scene, wherein the image data is used to generate a color image of the scene.

[0152] In some embodiments, during operation 1110, the controller may determine to operate in a second mode. For example, the controller may determine this based on the vehicle 100 being in a specific vehicle control and / or operating mode (e.g., referring to...). Figure 1 The different levels of autonomy described are operated, the vehicle 100 performs a specific task (e.g., parking, reversing, turning, etc.), or the vehicle 100 will benefit from another triggering event that performs ToF calculations and collects depth data of the scene around the vehicle 100, to determine operation in the second mode.

[0153] In operation 1115, the controller can operate in a second mode. In some examples, the controller begins operating in the second mode based on the determination made in operation 1110. Alternatively or concurrently, the controller can operate continuously in both the first and second modes. In either case, the controller is capable of operating in both the first and second modes simultaneously. As part of the second mode, the controller can control at least one light source and at least one camera to provide a third function, wherein the first, second, and third functions are distinct from each other. For example, the third function may include the controller controlling at least one light source and at least one camera to generate depth data of the scene using light emitted from at least one light source and reflected back to at least one camera from the scene surrounding the vehicle 100 (e.g., objects from the scene), as referenced. Figures 8 to 10 As described, depth data is used to generate a depth image of the scene (e.g., a ToF depth image). Accordingly, the controller can transmit signaling to at least one light source and at least one camera to perform a third function as part of the second mode.

[0154] In operation 1120, the controller can generate depth data based on a third function performed by at least one light source and at least one camera while operating in the second mode. For example, the depth data is generated based on timing by the controller and / or camera of the time taken for light to be emitted from at least one light source, reflected from the scene, and received at at least one camera. This timing can then be used as part of the ToF calculations described herein (e.g., Equations 1 and 2 previously provided) to generate depth data and a depth image of the scene. In some examples, at least one light source includes multiple light sources (e.g., multiple LEDs or as referenced). Figure 9 The description mentions individual lights, with a controller controlling these multiple light sources to emit light with different properties. The depth data includes data generated based on the reflected light with different properties, allowing the depth image to be generated as an HDR depth image. For example, different properties may correspond to different light intensities, different frequencies, different wavelengths, etc. Additionally, at least one camera may include multiple cameras, and the controller can control at least one light source and multiple cameras to expand the field of view of the depth image.

[0155] Figure 12 This is a flowchart 1200 illustrating aspects of the operation of a vehicle 100 according to an embodiment of this disclosure. For example, the controller of the vehicle 100 (e.g., as referenced) Figure 8 The controller 804 described can perform the operations shown in flowchart 1200 to assist in performing Time-of-Flight (ToF) determination using passive sensors (e.g., automotive cameras) and existing light sources on vehicle 100. Additionally, some operations shown in flowchart 1200 are similar to those described in reference... Figure 11The operations described are the same or similar. For example, operations 1205, 1210, 1215, and 1225 in flowchart 1200 can correspond to the operations described in the reference. Figure 11 The flowchart 1100 describes operations 1105, 1110, 1115, and 1120.

[0156] In some examples, at operation 1220, in addition to the operations described with reference to flowchart 1100, the controller of vehicle 100 can control the radiation pattern of at least one light source to vary based on the vehicle's speed. For example, the controller can control the radiation pattern of at least one light source to narrow as the vehicle 100's speed increases (e.g., at higher speeds of vehicle 100) and widen as the vehicle 100's speed decreases (e.g., at lower speeds of vehicle 100). Such actuation or adjustment of at least one light source can enable the controller and / or at least one camera to collect depth and angular position information at higher resolution based on the vehicle 100's travel speed (e.g., as part of the depth data generated in scene 1225). Alternatively or concurrently, vehicle 100 can use a small field-of-view camera and a large field-of-view camera to detect different portions of the scene surrounding vehicle 100.

[0157] It can execute any of the steps, functions, and operations discussed in this article continuously and automatically.

[0158] Exemplary systems and methods of this disclosure have been described in relation to vehicle systems and electric vehicles. However, to avoid unnecessarily obscuring this disclosure, many known structures and devices have been omitted from the foregoing description. Such omissions should not be construed as a limitation on the scope of the claimed disclosure. Many specific details have been set forth to provide an understanding of this disclosure. However, it should be understood that this disclosure can be practiced in various ways beyond the specific details set forth herein.

[0159] Furthermore, while the exemplary embodiments shown herein illustrate various components of the system in combination, some components of the system may be located remotely, at distant locations within a distributed network such as a LAN and / or the Internet, or within a dedicated system. Therefore, it should be understood that the components of the system may be combined into one or more devices, such as servers, communication equipment, or co-located on specific nodes of a distributed network, such as analog and / or digital telecommunications networks, packet-switched networks, or circuit-switched networks. As will be understood from the foregoing description, and for computational efficiency reasons, the components of the system may be arranged anywhere within the distributed component network without affecting the operation of the system.

[0160] Furthermore, it should be understood that the various links connecting the elements can be wired or wireless links or any combination thereof, or any other known or subsequently developed element(s) capable of providing and / or transmitting data to and from the connected element. These wired or wireless links can also be secure links and can be capable of transmitting encrypted information. For example, the transmission medium used as a link can be any suitable carrier for electrical signals, including coaxial cables, copper wires, and optical fibers, and can take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0161] Although flowcharts have been discussed and illustrated with respect to specific event sequences, it should be understood that changes, additions, and omissions to this sequence may occur without materially affecting the operation of the disclosed embodiments, configurations, and aspects.

[0162] Many changes and modifications to this disclosure may be made. Some features of this disclosure may be provided without others.

[0163] In yet another embodiment, the systems and methods disclosed herein may be implemented in conjunction with a dedicated computer, a programmable microprocessor or microcontroller and one or more peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, hardwired electronic devices or logic circuits (such as discrete element circuits), programmable logic devices or gate arrays (such as PLDs, PLAs, FPGAs, PALs), a dedicated computer, any equivalent apparatus, etc. Generally, any one or more devices or apparatuses capable of implementing the methods shown herein may be used to implement various aspects of this disclosure. Exemplary hardware that may be used for this disclosure includes computers, handheld devices, telephones (e.g., cellular, internet, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. Furthermore, alternative software implementations may be constructed, including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing, to implement the methods described herein.

[0164] In yet another embodiment, the disclosed method can be readily implemented using software from an object-oriented or object-based software development environment that provides portable source code usable on a variety of computer or workstation platforms. Alternatively, the disclosed system can be implemented partially or entirely in hardware using standard logic circuitry or VLSI design. Whether to implement a system according to this disclosure using software or hardware depends on the speed and / or efficiency requirements of the system, the specific functionality, and the specific software or hardware system or microprocessor or microcomputer system used.

[0165] In yet another embodiment, the disclosed method can be implemented in part as software, which can be stored on a storage medium and executed on a programmed general-purpose computer, special-purpose computer, microprocessor, etc., in cooperation with a controller and memory. In these cases, the systems and methods of this disclosure can be implemented as programs embedded in a personal computer (such as applets, etc.). This can be a CGI script, a resource residing on a server or computer workstation, a routine embedded in a dedicated measurement system, system components, etc. The system can also be implemented by physically integrating the system and / or method into the software and / or hardware system.

[0166] While this disclosure describes components and functions implemented in embodiments with reference to specific standards and protocols, it is not limited to such standards and protocols. Other similar standards and protocols not mentioned herein exist and are considered to be included in this disclosure. Furthermore, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are regularly superseded by faster or more efficient equivalents with substantially the same functionality. Such alternative standards and protocols with the same functionality are considered to be equivalents included in this disclosure.

[0167] This disclosure includes, in various embodiments, configurations, and aspects, components, methods, processes, systems, and / or apparatuses substantially as depicted and described herein, including various embodiments, sub-combinations, and subsets thereof. Those skilled in the art, upon understanding this disclosure, will understand how to make and use the systems and methods disclosed herein. This disclosure includes, in several different embodiments, configurations, and aspects, providing apparatus and processes, for example, to improve performance, facilitate implementation, and / or reduce implementation costs, in the absence of matters not depicted and / or described herein, or in several different embodiments, configurations, or aspects thereof, in the absence of such matters that might have been used in previous apparatus or processes.

[0168] The foregoing discussion of this disclosure has been presented for purposes of illustration and description. The foregoing is not intended to limit this disclosure to the one or more forms disclosed herein. In, for example, the specific embodiments described above, various features of this disclosure are combined in one or more embodiments, configurations, or aspects for the purpose of simplification. Features of embodiments, configurations, or aspects of this disclosure may be combined in alternative embodiments, configurations, or aspects other than those discussed above. This approach to the disclosure should not be construed as reflecting an intention that the claimed disclosure requires more features than expressly recited in the claims. Rather, as reflected in the appended claims, the inventive aspect relies on fewer than all features of a single foregoing disclosed embodiment, configuration, or aspect. Therefore, the appended claims are hereby incorporated in this specific embodiment, wherein each claim relies on itself as an independent preferred embodiment of this disclosure.

[0169] Furthermore, while the description in this disclosure includes descriptions of one or more embodiments, configurations, or aspects, as well as certain variations and modifications, other changes, combinations, and modifications are also within the scope of this disclosure, for example, as may be within the skill and knowledge of one skilled in the art upon understanding this disclosure. It is intended to obtain the right to include alternative embodiments, configurations, or aspects to the permissible extent, including alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps of those claimed, regardless of whether such alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps are disclosed herein, and is not intended to publicly offer any patentable subject matter.

[0170] The embodiment includes a system comprising at least one light source providing a first function to a vehicle, at least one camera providing a second function to the vehicle, and a controller configured to operate in a first mode and a second mode. The first mode is a mode in which the controller is configured to control at least one light source to provide the first function and to control at least one camera to provide the second function. Alternatively, the second mode is a mode in which the controller is configured to control at least one light source and at least one camera to provide a third function to the vehicle. The first function, the second function, and the third function are distinct from each other.

[0171] In one aspect of the aforementioned system, the third function includes a controller controlling at least one light source and at least one camera to generate depth data of the scene using light emitted from at least one light source and reflected back from the scene surrounding the vehicle to at least one camera, wherein the depth data is used to generate a depth image of the scene.

[0172] In one aspect of the aforementioned system, the first function includes a controller controlling at least one light source to illuminate the scene using light of visible wavelengths emitted from the light source.

[0173] In one aspect of the aforementioned system, the second function includes a controller controlling at least one camera to capture image data of a scene, which is used to generate a color image of the scene.

[0174] The aforementioned system includes a controller that operates simultaneously in both a first mode and a second mode.

[0175] In the aforementioned system, at least one light source comprises multiple light sources, and a controller controls these multiple light sources to emit light with different properties. The depth data includes data generated based on reflected light with different properties, such that the depth image is an HDR depth image. For example, different properties correspond to different light intensities.

[0176] In the aforementioned system, at least one camera includes multiple cameras, and a controller controls at least one light source and multiple cameras to expand the field of view of the depth image.

[0177] Aspects of the aforementioned system include a controller that controls the radiation pattern of at least one light source to vary based on the vehicle's speed. For example, the controller controls the radiation pattern of at least one light source to narrow as the vehicle speed increases and to widen as the vehicle speed decreases.

[0178] The embodiment further includes a device comprising a controller configured to operate in a first mode and a second mode. The first mode is a mode in which the controller is configured to control at least one light source of the vehicle to provide a first function to the vehicle and to control at least one camera to provide a second function to the vehicle. Alternatively, the second mode is a mode in which the controller is configured to control at least one light source and at least one camera to provide a third function to the vehicle. The first, second, and third functions are distinct from each other.

[0179] In the aforementioned device, the third function includes a controller controlling at least one light source and at least one camera to generate depth data of the scene using light emitted from at least one light source and reflected from the scene around the vehicle back to at least one camera, wherein the depth data is used to generate a depth image of the scene.

[0180] In the aforementioned device, the first function includes a controller controlling at least one light source to illuminate the scene using light of visible wavelengths emitted from the light source.

[0181] In one aspect of the aforementioned device, the second function includes a controller controlling at least one camera to capture image data of a scene, which is used to generate a color image of the scene.

[0182] The aforementioned device includes a controller that operates simultaneously in both a first mode and a second mode.

[0183] In the aforementioned device, at least one light source comprises multiple light sources, and a controller controls the multiple light sources to emit light with different properties, wherein the depth data includes data generated based on reflected light with different properties, such that the depth image is an HDR depth image. For example, different properties correspond to different light intensities.

[0184] In the aforementioned device, at least one camera includes multiple cameras, and a controller controls at least one light source and multiple cameras to expand the field of view of the depth image.

[0185] Aspects of the aforementioned device include a controller that controls the radiation pattern of at least one light source to vary based on the vehicle's speed. For example, the controller controls the radiation pattern of at least one light source to narrow as the vehicle speed increases and to widen as the vehicle speed decreases.

[0186] The embodiment includes a vehicle comprising at least one light source providing a first function for the vehicle, at least one camera providing a second function for the vehicle, and a controller configured to operate in a first mode and a second mode. The first mode is a mode in which the controller is configured to control at least one light source to provide the first function and to control at least one camera to provide the second function. Alternatively, the second mode is a mode in which the controller is configured to control at least one light source and at least one camera to provide a third function for the vehicle. The first function, the second function, and the third function are distinct from each other.

[0187] In the aforementioned vehicle aspect, the third function includes a controller controlling at least one light source and at least one camera to generate depth data of the scene using light emitted from at least one light source and reflected from the scene surrounding the vehicle back to at least one camera, wherein the depth data is used to generate a depth image of the scene.

[0188] In the aforementioned vehicle aspect, the first function includes a controller controlling at least one light source to illuminate the scene using light of visible wavelengths emitted from the light source.

[0189] In the aforementioned vehicle aspect, the second function includes a controller controlling at least one camera to capture image data of a scene, which is used to generate a color image of the scene.

[0190] The aforementioned aspects of the vehicle include the controller operating simultaneously in both the first and second modes.

[0191] In the aforementioned vehicle aspect, at least one light source comprises multiple light sources, and a controller controls these multiple light sources to emit light with different properties, wherein the depth data includes data generated based on reflected light with different properties, such that the depth image is an HDR depth image. For example, different properties correspond to different light intensities.

[0192] In the aforementioned vehicle aspect, at least one camera includes multiple cameras, and a controller controls at least one light source and multiple cameras to expand the field of view of the depth image.

[0193] The aforementioned aspect of the vehicle includes a controller that controls the radiation pattern of at least one light source to vary based on the vehicle's speed. For example, the controller controls the radiation pattern of at least one light source to narrow as the vehicle speed increases and to widen as the vehicle speed decreases.

[0194] As described herein, any one or more aspects / embodiments are essentially disclosed herein.

[0195] Any one or more aspects / executives substantially disclosed herein may optionally be combined with any one or more other aspects / executives substantially disclosed herein.

[0196] As substantially disclosed herein, one or more means adapted to perform any one or more aspects / embodiments of the foregoing aspects / embodiments are provided.

[0197] The phrases “at least one,” “one or more,” “or,” and “and / or” are open-ended expressions that are both conjunction and disjunctive in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” “A, B, and / or C,” and “A, B, or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0198] The term "a" or "an" entity refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" are used interchangeably.

[0199] As used herein, the term "automatic" and its variations refer to any process or operation performed without significant human input, typically sequential or semi-sequential. However, a process or operation can be automatic if input is received prior to its execution, even if significant or insignificant human input is used in its execution. Human input is considered significant if it influences how the process or operation is performed. Human input that consents to the execution of a process or operation is not considered "significant."

[0200] Various aspects of this disclosure may take the form of a wholly hardware embodiment, a wholly software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as a “circuit,” “module,” or “system.” Any combination of one or more computer-readable media may be used. Computer-readable media may be computer-readable signal media or computer-readable storage media.

[0201] Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, IR, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, computer-readable storage media can be any tangible medium that may contain or store programs for use by or in connection with an instruction execution system, apparatus, or device.

[0202] Computer-readable signal media may include propagated data signals in which computer-readable program code is embedded, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and may transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code embedded on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0203] As used herein, the terms “determine,” “calculate,” “infer,” and their variations are used interchangeably and include any type of method, process, mathematical operation, or technique.

[0204] The term "electric vehicle" (EV), also referred to herein as an electrically driven vehicle, can be propelled by one or more electric motors or traction motors. Electric vehicles can be powered by electricity from an external source via a collector system, or they can contain batteries or generators to convert fuel into electricity. Electric vehicles generally include rechargeable energy storage systems (RESS) (also known as fully electric vehicles (FEVs)). Energy storage methods can include: chemical energy stored in the vehicle's onboard battery (e.g., in a battery electric vehicle or BEV), onboard kinetic energy storage devices (e.g., inertial wheels), and / or static energy (e.g., through onboard double-layer capacitors). Rechargeable onboard energy storage devices can take the form of batteries, double-layer capacitors, and inertial wheel energy storage devices.

[0205] The term "hybrid electric vehicle" refers to a vehicle that combines a conventional (usually fossil fuel-powered) powertrain with some form of electric propulsion. Most hybrid electric vehicles combine a conventional internal combustion engine (ICE) propulsion system with an electric propulsion system (hybrid vehicle drivetrain). In a parallel hybrid vehicle, both the ICE and the electric motor are connected to a mechanical transmission and can typically transmit power to drive the wheels simultaneously via the conventional transmission. In a series hybrid vehicle, only the electric motor drives the powertrain, and the smaller ICE acts as a generator to power the electric motor or recharge the battery. Power-split hybrid vehicles exhibit both series and parallel characteristics. A full hybrid vehicle, sometimes called a strong hybrid, is a vehicle that can operate solely on its engine, solely on its battery, or a combination of both. A mid-range hybrid vehicle is a vehicle that cannot be driven solely by its electric motor because the electric motor does not have sufficient power to propel the vehicle itself.

[0206] The term "rechargeable electric vehicle" or "REV" refers to a vehicle with an onboard rechargeable energy storage device, including electric vehicles and hybrid electric vehicles.

Claims

1. A system comprising: At least one light source, wherein the at least one light source is at least one of vehicle-mounted light sources on the vehicle, for providing a first function to the vehicle; At least one camera, said at least one camera being a passive sensor, for providing a second function for the vehicle; as well as A controller configured to operate in a first mode and a second mode, wherein the first mode is configured to control the at least one light source to provide the first function and control the at least one camera to provide the second function, and the second mode is configured to control the at least one light source and the at least one camera to synchronize, thereby constructing the at least one camera and the at least one light source into a time-of-flight camera to provide a third function for the vehicle. The first function, the second function, and the third function are different from each other.

2. The system as claimed in claim 1, wherein, The third function includes the controller controlling the at least one light source and the at least one camera to generate depth data of the scene using light emitted from the at least one light source and reflected back to the at least one camera from the scene around the vehicle, the depth data being used to generate a depth image of the scene.

3. The system as described in claim 2, wherein, The first function includes the controller controlling the at least one light source to illuminate the scene using light of visible wavelengths emitted from the light source.

4. The system as described in claim 3, wherein, The second function includes the controller controlling the at least one camera to capture image data of the scene, the image data being used to generate a color image of the scene.

5. The system as claimed in claim 1, wherein, The controller operates simultaneously in both the first mode and the second mode.

6. The system as claimed in claim 2, wherein, The at least one light source includes multiple light sources, and the controller controls the multiple light sources to emit light with different light intensities.

7. The system as claimed in claim 2, wherein, The controller controls the radiation pattern of the at least one light source to vary based on the speed of the vehicle.

8. The system of claim 7, wherein, The controller controls the radiation pattern of the at least one light source to narrow as the vehicle speed increases and to widen as the vehicle speed decreases.

9. An apparatus comprising: A controller configured to operate in a first mode and a second mode, wherein the first mode is a mode in which the controller is configured to perform the following operations: Controlling at least one light source of the vehicle to provide a first function to the vehicle, wherein the at least one light source is at least one of the vehicle-mounted light sources; and Control at least one camera to provide a secondary function for the vehicle, said at least one camera being a passive sensor. The second mode is the mode in which the controller is configured to perform the following operations: The at least one light source and the at least one camera are synchronized to configure the at least one camera and the at least one light source into a time-of-flight camera to provide a third function for the vehicle. The first function, the second function, and the third function are different from each other.

10. The device as claimed in claim 9, wherein, The third function includes the controller controlling the at least one light source and the at least one camera to generate depth data of the scene using light emitted from the at least one light source and reflected back to the at least one camera from the scene around the vehicle, the depth data being used to generate a depth image of the scene.

11. The device as claimed in claim 10, wherein, The first function includes the controller controlling the at least one light source to illuminate the scene using light of visible wavelengths emitted from the light source.

12. The device as claimed in claim 11, wherein, The second function includes the controller controlling the at least one camera to capture image data of the scene, the image data being used to generate a color image of the scene.

13. The device as claimed in claim 11, wherein, The controller operates simultaneously in both the first mode and the second mode.

14. The device as claimed in claim 12, wherein, The at least one light source includes multiple light sources, and the controller controls the multiple light sources to emit light with different light intensities.

15. The device as claimed in claim 12, wherein, The controller controls the radiation pattern of the at least one light source to narrow as the vehicle speed increases and to widen as the vehicle speed decreases.

Images