Vehicle monitoring systems and methods

Abstract

The disclosed system and method include: receiving an image, processing the image using a depth estimation module to generate a first depth map, generating a foreground mask of the image, patching a region of the image, generating a second depth map based on the patched region of the image, comparing the histograms of the first depth map and the histograms of the second depth map, and generating a proximity warning based on the comparison of these histograms.

Description

Technical Field

[0001] This disclosure relates generally to vehicle methods and systems, and more specifically to vehicle methods and systems for detecting and evaluating objects using camera data. Background Technology

[0002] Drivers often need to park their vehicles in unsafe locations. They rely on vehicle safety systems such as car alarms to help prevent or deter vandalism and theft. Modern vehicle safety systems depend on sensors to detect objects touching the vehicle or breaking windows. Parked or idle vehicles typically use battery-powered systems to power these safety systems. Modern vehicle safety systems are inaccurate in detecting potential threats and are inefficient in terms of power usage. What is needed are more accurate and energy-efficient vehicle safety systems. Summary of the Invention

[0003] One aspect of the present invention relates to a method for monitoring a vehicle, the method comprising receiving an image from an image sensor; processing the received image using a depth estimation module to generate a first depth map; processing the received image using a background modeling algorithm to generate one or more masks, a foreground mask and a background mask; generating a patched image using an image inpainting network, the patched image comprising the received image having patched regions associated with the one or more masks, the foreground mask and the background mask; processing the patched image using the depth estimation module to generate a second depth map; calculating a histogram of the first depth map and a histogram of the second depth map; determining one or more of the size and relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alarm, the alarm comprising data associated with the determined size and relative depth range of one or more foreground objects.

[0004] In one embodiment, the image is generated using a fisheye lens.

[0005] In one embodiment, the image sensor is mounted on the exterior of the vehicle.

[0006] In one embodiment, the method further includes processing the image using the depth estimation module and processing the image using the background modeling algorithm in parallel.

[0007] In one embodiment, the histogram of the first depth map is calculated based on the regions of the first depth map corresponding to one or more of the foreground and background masks.

[0008] In one embodiment, the method further includes using a mean filter to recover lost information from the histogram.

[0009] In one embodiment, the histogram of the second depth map is calculated based on the regions of the first depth map corresponding to one or more of the foreground and background masks.

[0010] In one embodiment, the size of the one or more foreground objects and one or more of the relative depth ranges are determined based on a comparison between the first depth map and the second depth map.

[0011] In one embodiment, the patched image reflects the background of the received image, wherein the one or more foreground objects are replaced with the generated background.

[0012] In one embodiment, the method further includes transmitting an alarm to a user equipment.

[0013] In one embodiment, the method further includes displaying the alarm.

[0014] In one embodiment, the method further includes determining, before generating the alarm, that one or more of the size and the relative depth range exceed a threshold.

[0015] In one embodiment, the method further includes repeating the method at regular intervals.

[0016] In one embodiment, the method further includes activating one or more sensors in response to determining one or more of the size of one or more foreground objects and the relative depth range.

[0017] In one embodiment, the method further includes applying one or more of the foreground mask and the background mask to the first depth map before calculating a histogram of the first depth map.

[0018] In one embodiment, the method further includes processing the patched image using the background modeling algorithm to generate a second or more masks among a foreground mask and a background mask.

[0019] In one embodiment, the method further includes applying one or more of the foreground mask and the background mask to the second depth map before calculating the histogram of the second depth map.

[0020] Another aspect of the present invention provides a user equipment, comprising: a processor; and a computer-readable storage medium storing computer-readable instructions, which, when executed by the processor, cause the processor to perform a method for monitoring a vehicle, the method comprising: receiving an image from an image sensor; processing the received image using a depth estimation module to generate a first depth map; processing the received image using a background modeling algorithm to generate one or more masks chosen from a foreground mask and a background mask; generating a patched image using an image inpainting network, the patched image including a received image having patched regions associated with the one or more masks chosen from the foreground mask and the background mask; processing the patched image using the depth estimation module to generate a second depth map; calculating a histogram of the first depth map and a histogram of the second depth map; determining one or more of the size and relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alarm, the alarm including data associated with the determined size and relative depth range of one or more foreground objects.

[0021] In one embodiment, the image is generated using a fisheye lens.

[0022] In another aspect, the present invention provides a computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being configured to perform a method for monitoring a vehicle when executed by a processor, the method comprising receiving an image from an image sensor; processing the received image using a depth estimation module to generate a first depth map; processing the received image using a background modeling algorithm to generate one or more masks, a foreground mask and a background mask; generating a patched image using an image inpainting network, the patched image comprising a received image having patched regions associated with the one or more masks, the foreground mask and the background mask; processing the patched image using the depth estimation module to generate a second depth map; calculating a histogram of the first depth map and a histogram of the second depth map; determining one or more of the size and relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alarm, the alarm comprising data associated with the determined size and relative depth range of one or more foreground objects. Attached Figure Description

[0023] Figure 1 A vehicle according to an embodiment of this disclosure is shown;

[0024] Figure 2A plan view of a vehicle according to at least some embodiments of this disclosure is shown;

[0025] Figure 3A This is a block diagram of an embodiment of the communication environment of a vehicle according to an embodiment of the present disclosure;

[0026] Figure 3B This is a block diagram of an embodiment of an internal sensor in a vehicle according to an embodiment of this disclosure;

[0027] Figure 4 An embodiment of a vehicle dashboard according to one embodiment of this disclosure is shown;

[0028] Figure 5 This is a block diagram of an embodiment of the vehicle's communication subsystem;

[0029] Figure 6 This is a block diagram of a computing environment associated with the embodiments presented herein;

[0030] Figure 7 It is a block diagram of a computing device associated with one or more components described herein;

[0031] Figure 8 This is a flowchart illustrating a method according to one or more embodiments described herein;

[0032] Figures 9A to 9G These are illustrations of images according to one or more embodiments described herein;

[0033] Figure 9H It is an illustration of a histogram according to one or more embodiments described herein; and

[0034] Figures 10A to 10F This is an illustration of a user interface according to one or more embodiments described herein. Detailed Implementation

[0035] Embodiments of this disclosure will be described in conjunction with vehicles, and in some embodiments with electric vehicles, rechargeable electric vehicles, and / or hybrid electric vehicles, as well as associated systems.

[0036] What is needed is an energy-efficient system that can determine whether a person or object is near a vehicle.

[0037] As described in this article, a vision-based system using one or more cameras on a vehicle may be able to detect moving objects (such as people), make one or more determinations about the size of the object and / or the proximity of the object to the vehicle, and in response, issue warnings or notifications, or perform one or more other tasks or functions.

[0038] In some embodiments, the system may be configured to detect moving objects using an onboard camera and, in response, activate one or more other cameras or sensors to collect additional information about the detected moving objects.

[0039] The following detailed description refers to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0040] Figure 1 A perspective view of a vehicle 100 according to an embodiment of this disclosure is shown. The electric vehicle 100 includes a front portion 110, a rear portion or rear section 120, a roof 130, at least one side portion 160, a chassis 140, and an interior portion 150. In any case, the vehicle 100 may include a frame 104 and one or more body panels 108 mounted or attached thereto. The vehicle 100 may include one or more internal components (e.g., components within the interior space 150 of the vehicle 100 or the user space), external components (e.g., components outside the interior space 150 of the vehicle 100 or the user space), a drive system, a control system, structural components, etc.

[0041] Although shown as a passenger car, it should be understood that the vehicle 100 described herein can include any means of transport or any type of transport vehicle designed to move one or more tangible objects, such as people, animals, goods, etc. The term "vehicle" does not require the means of transport to be moving or capable of moving. Typical vehicles may include, but are not limited to, passenger cars, trucks, motorcycles, buses, automobiles, trains, rail transport vehicles, ships, vessels, maritime transport vehicles, submarine transport vehicles, airplanes, space shuttles, aircraft, human-powered transport vehicles, etc.

[0042] In some embodiments, vehicle 100 may include a plurality of sensors, devices, and / or systems capable of assisting driving operations, such as automatic or semi-automatic control. Examples of various sensors and systems may include, but are not limited to, one or more of the following: cameras (e.g., standalone images, stereo images, combined images, etc.), infrared (IR) sensors, radio frequency (RF) sensors, ultrasonic sensors (e.g., transducers, transceivers, etc.), radio detection and ranging (RADAR) sensors (e.g., object detection sensors and / or systems), light detection and ranging (LiDAR) sensors and / or systems, ranging sensors and / or devices (e.g., encoders, etc.), orientation sensors (e.g., accelerometers, gyroscopes, magnetometers, etc.), navigation sensors and systems (e.g., GPS, etc.), and other ranging, imaging, and / or object detection sensors. Sensors may be disposed in the interior space 150 of vehicle 100 and / or on the exterior of vehicle 100. In some embodiments, sensors and systems may be disposed in one or more parts of vehicle 100 (e.g., frame 104, body panels, compartments, etc.). These sensors may include one or more fisheye cameras or other wide-angle cameras, which may be placed outside the vehicle 100 and configured to capture image data of the exterior of the vehicle 100.

[0043] In some embodiments, one or more image sensors may be placed around the exterior of the vehicle. The image sensors may be configured to capture image data depicting a scene outside the vehicle, as described herein. The image sensors may include, for example, fisheye lenses or other types of wide-angle lenses. The image sensors may be configured to capture images using fisheye lenses or other types of wide-angle lenses.

[0044] like Figure 1 As shown, vehicle 100 may include, for example, at least one of the following: a ranging and imaging system 112 (e.g., LiDAR, etc.), imaging sensors 116A, 116F (e.g., camera, IR, etc.), a radio object detection and ranging system sensor 116B (e.g., RADAR, RF, etc.), an ultrasonic sensor 116C, and / or other object detection sensors 116D, 116E. In some embodiments, the LiDAR system 112 and / or sensors may be mounted on the roof 130 of vehicle 100. In one embodiment, sensors may be located at least at the front 110, rear 120, or side 160 of vehicle 100. Among other things, sensors may be used to monitor and / or detect the positions of other vehicles, pedestrians, and / or other objects in or near vehicle 100. Although shown as being associated with one or more areas of vehicle 100, it should be understood that... Figure 1 and Figure 2Any of the sensors and systems 116A to 116K, 112 shown may be installed in, on, and / or around the vehicle 100 at any location, area, and / or zone of the vehicle 100.

[0045] Now for reference Figure 2 A plan view of a vehicle 100 according to an embodiment of this disclosure will be described. Specifically, Figure 2 A vehicle sensing environment 200 is illustrated, at least partially defined by sensors and systems 116A to 116K, 112 disposed within, above, and / or around a vehicle 100. Each sensor 116A to 116K may include an operational detection range R and an operational detection angle. The operational detection range R may define an effective detection limit or distance for the sensors 116A to 116K. In some cases, this effective detection limit may be defined as the distance from a portion of the sensors 116A to 116K (e.g., a lens, sensing surface, etc.) to a point in space offset from the sensors 116A to 116K. The effective detection limit may define a distance beyond which the sensing capability of the sensors 116A to 116K deteriorates, becomes inoperable, or becomes unreliable. In some embodiments, the effective detection limit may define a distance within which the sensing capability of the sensors 116A to 116K can provide accurate and / or reliable detection information. The operational detection angle may define at least one angle of the span of the sensors 116A to 116K, or at least one angle between a horizontal limit and / or a vertical limit. As can be understood, the operational detection limits and operational detection angles of sensors 116A to 116K together define the effective detection areas 216A to 216D of sensors 116A to 116K (e.g., effective detection area and / or volume, etc.).

[0046] In some embodiments, vehicle 100 may include a ranging and imaging system 112, such as LiDAR. The ranging and imaging system 112 may be configured to detect visual information in the environment surrounding vehicle 100. The visual information detected in the environment surrounding the ranging and imaging system 112 (e.g., via one or more sensors and / or a system processor, etc.) may be processed to generate a complete 360-degree view of the environment 200 surrounding the vehicle. The ranging and imaging system 112 may be configured to generate a changing 360-degree view of the environment 200 in real time, for example, while vehicle 100 is being driven. In some cases, the ranging and imaging system 112 may have an effective detection limit 204, which is a distance in 360 degrees outward from the center of vehicle 100. The effective detection limit 204 of the ranging and imaging system 112 defines an observation area 208 (e.g., region and / or volume, etc.) around vehicle 100. Any object falling outside the observation area 208 is in an undetected area 212 and will not be detected by the ranging and imaging system 112 of vehicle 100.

[0047] Sensor data and information can be collected by one or more sensors or systems 116A to 116K, 112 of vehicle 100 monitoring the vehicle's sensing environment 200. This information can be processed (e.g., via a processor, computer vision system, etc.) to determine targets (e.g., objects, signs, people, markings, roads, road conditions, etc.) within one or more detection zones 208, 216A to 216D associated with the vehicle's sensing environment 200. In some cases, information from multiple sensors 116A to 116K can be processed to form composite sensor detection information. For example, a first sensor 116A and a second sensor 116F can correspond to a first camera 116A and a second camera 116F aimed at the forward direction of travel of vehicle 100. In this example, images collected by cameras 116A, 116F can be combined to form stereo image information. This composite information can enhance the capabilities of individual sensors 116A to 116K by, for example, increasing the ability to determine the depth associated with targets in one or more detection zones 208, 216A to 216D. Similar image data can be collected by a rear-view camera (e.g., sensor 116G, 116H) targeting the vehicle 100 in the direction of rearward travel.

[0048] In some embodiments, multiple sensors 116A to 116K can be effectively combined to increase the sensing area and provide increased sensing coverage. For example, multiple RADAR sensors 116B disposed on the front 110 of a vehicle can be combined to provide a coverage area 216B spanning the entire front 110 of the vehicle. In some cases, multiple RADAR sensors 116B can cover a detection area 216B that includes one or more other sensor detection areas 216A. These overlapping detection areas can provide redundant sensing, enhanced sensing, and / or provide more sensing detail within a specific portion (e.g., area 216A) of a larger area (e.g., area 216B). Alternatively or additionally, the sensors 116A to 116K of the vehicle 100 can be arranged to produce full coverage via one or more sensing areas 208, 216A to 216D around the vehicle 100. In some areas, the sensing areas 216C of two or more sensors 116D, 116E can intersect at an overlap area 220. In some areas, the angles and / or detection limits of two or more sensing areas 216C, 216D (e.g., of two or more sensors 116E, 116J, 116K) may meet at a virtual intersection 224.

[0049] Vehicle 100 may include a plurality of sensors 116E, 116G, 116H, 116J, 116K disposed near the rear 120 of vehicle 100. These sensors may include, but are not limited to, imaging sensors, cameras, IR, radio object detection and ranging sensors, RADAR, RF, ultrasonic sensors, and / or other object detection sensors. Among other things, these sensors 116E, 116G, 116H, 116J, 116K can detect targets approaching or near the rear of vehicle 100. For example, another vehicle approaching the rear 120 of vehicle 100 may be detected by one or more of the ranging and imaging system (e.g., LiDAR) 112, the rear-view cameras 116G, 116H, and / or the rear-facing RADAR sensors 116J, 116K. As described above, images from the rear-view cameras 116G, 116H may be processed to generate a stereo view of targets visible to both cameras 116G, 116H (e.g., providing depth, etc., associated with the object or environment). As another example, vehicle 100 may be in motion, and one or more of the ranging and imaging system 112, forward-facing cameras 116A, 116F, forward-facing RADAR sensor 116B, and / or ultrasonic sensor 116C can detect targets in front of vehicle 100. This approach can provide critical sensor information to the vehicle control system at at least one of the aforementioned levels of autonomous driving.

[0050] Figure 3A and Figure 3B This is a block diagram of an embodiment of a communication system 300 for a vehicle 100 according to an embodiment of this disclosure. The communication system 300 may include one or more vehicle driving sensor and system 304, sensor processor 340, sensor data storage 344, vehicle control system 348, communication subsystem 350, control data 364, computing device 368, display device 372, and other components 374 that may be associated with the vehicle 100. These associated components may be electrically and / or communicatively coupled to each other via at least one bus 360. In some embodiments, one or more associated components may transmit and / or receive signals to at least one of a navigation source 356A, a control source 356B, or some other entity 356N via a communication network 352.

[0051] According to at least some embodiments of this disclosure, communication network 352 may include any type of known communication medium or set of communication media and may use any type of protocol, such as SIP, TCP / IP, SNA, IPX, AppleTalk, etc., to transmit messages between endpoints. Communication network 352 may include wired and / or wireless communication technologies. The Internet is an example of communication network 352, which constitutes an Internet Protocol (IP) network consisting of numerous computers, computing networks, and other communication devices located around the world, connected via numerous telephone systems and other means. Other examples of communication network 352 include, but are not limited to, standard Common Old-Style Telephone Systems (POTS), Integrated Services Digital Network (ISDN), Public Switched Telephone Network (PSTN), Local Area Networks (LANs) such as Ethernet, Token Ring networks, and / or similar networks, Wide Area Networks (WANs), virtual networks including but not limited to Virtual Private Networks (“VPNs”); the Internet, intranets, extranets, cellular networks, infrared networks; wireless networks (e.g., in the IEEE 802.9 protocol suite, known in the art). This refers to networks operating under any of the protocols and / or any other wireless protocols, as well as any other types of packet-switched or circuit-switched networks and / or any combination of these and / or other networks known in the art. Furthermore, it is understood that the communication network 352 is not necessarily limited to any one network type, but may include many different networks and / or network types. The communication network 352 may include multiple different communication media, such as coaxial cable, copper cable / wire, fiber optic cable, antennas for transmitting / receiving wireless messages, and combinations thereof.

[0052] The driving vehicle sensors and systems 304 may include at least one navigation sensor or system 308 (e.g., Global Positioning System (GPS), etc.), orientation sensor or system 312, ranging sensor or system 316, LiDAR sensor or system 320, RADAR sensor or system 324, ultrasonic sensor or system 328, camera sensor or system 332, infrared (IR) sensor or system 336, and / or other sensors or systems 338. These driving vehicle sensors and systems 304 can be combined with... Figure 1 and Figure 2 The described sensors and systems are similar to (if not identical to) 116A to 116K, 112.

[0053] Navigation sensor 308 may include one or more sensors having a receiver and an antenna, the sensor being configured to utilize a satellite-based navigation system 302, the navigation system including a navigation satellite network capable of providing geolocation and time information to at least one component of vehicle 100. Examples of navigation sensor 308 described herein may include, but are not limited to, at least one of the following: GLO TM Series of GPS and GLONASS combined sensors GPS15x TM Series of sensors GPS16x TM A series of sensors with high-sensitivity receivers and antennas. The GPS18x OEM series of high-sensitivity GPS sensors, the Dewetron DEWE-VGPS series of GPS sensors, the GlobalSat 1-Hz series of GPS sensors, and other industrial equivalent navigation sensors and / or systems can be used to perform navigation and / or geolocation functions using any known or future-developed standards and / or architectures.

[0054] Orientation sensor 312 may include one or more sensors configured to determine the orientation of vehicle 100 relative to at least one reference point. In some embodiments, orientation sensor 312 may include at least one pressure transducer, stress / strain gauge, accelerometer, gyroscope, and / or geomagnetic sensor. Examples of the orientation sensor 312 described herein may include, but are not limited to, at least one of the following: Bosch Sensortec BMX 160 series low-power absolute orientation sensor, Bosch Sensortec BMX055 9-axis sensor, Bosch Sensortec BMI055 6-axis inertial sensor, Bosch Sensortec BMI160 6-axis inertial sensor, Bosch Sensortec BMF055 9-axis inertial sensor (accelerometer, gyroscope, and magnetometer) with integrated Cortex M0+ microcontroller, Bosch Sensortec BMP280 absolute barometric pressure sensor, Infineon TLV493D-A1B6 3D magnetic sensor, Infineon TLI493D-W1B6 3D magnetic sensor, Infineon TL series 3D magnetic sensors, Murata Electronics SCC2000 series combined gyroscope sensor and accelerometer, Murata Electronics The SCC1300 series combines gyroscope sensors with accelerometers, other industrial equivalent orientation sensors, and / or systems that can perform orientation detection and / or determination functions using any known or future-developed standards and / or architectures.

[0055] The ranging sensor and / or system 316 may include one or more components configured to determine the change of position of vehicle 100 over time. In some embodiments, the ranging system 316 may utilize data from one or more other sensors and / or systems 304 to determine the position (e.g., distance, location, etc.) of vehicle 100 relative to a previously measured position of vehicle 100. Alternatively or additionally, the ranging sensor 316 may include one or more encoders, Hall speed sensors, and / or other measuring sensors / devices configured to measure wheel speed, rotation, and / or revolutions over time. Examples of the ranging sensor / system 316 described herein may include, but are not limited to, at least one of the following: Infineon TLE4924 / 26 / 27 / 28C high-performance speed sensor, Infineon TL4941plusC(B) single-chip differential Hall wheel speed sensor, Infineon TL5041plusC giant magnetoresistive (GMR) effect sensor, Infineon TL series magnetic sensors, EPC 25SP Accu-CoderPro. TM Incremental shaft encoders; EPC 30M compact incremental encoders employing advanced magnetic sensing and signal processing technology; EPC 925 absolute shaft encoders; EPC 958 absolute shaft encoders; EPC MA36S / MA63S / SA36S absolute shaft encoders; Dynapar... TM F18 commutation optical encoder, Dynapar TM The HS35R series phase array encoder sensors, other industrial equivalent ranging sensors and / or systems, can be used with any known or future-developed standard and / or architecture to perform changes in position change detection and / or determination functions.

[0056] LiDAR sensor / system 320 may include one or more components configured to use laser illumination to measure distance to a target. In some embodiments, LiDAR sensor / system 320 may provide 3D imaging data of the environment surrounding vehicle 100. This imaging data may be processed to generate a full 360-degree view of the environment surrounding vehicle 100. LiDAR sensor / system 320 may include a laser generator configured to generate multiple target-illuminating laser beams (e.g., laser channels). In some embodiments, the multiple laser beams may be aimed or directed at a rotating reflective surface (e.g., a mirror) and guided outward from LiDAR sensor / system 320 into the measurement environment. The rotating reflective surface may be configured to rotate continuously 360 degrees about an axis such that the multiple laser beams are guided within a full 360-degree range around vehicle 100. A photodiode receiver of LiDAR sensor / system 320 may detect when light emitted from the multiple laser beams into the measurement environment returns (e.g., reflected echoes) to LiDAR sensor / system 320. The LiDAR sensor / system 320 can calculate the distance from vehicle 100 to the illuminated target based on the time associated with the emission of light to the return of the detected light. In some embodiments, the LiDAR sensor / system 320 can generate more than 2 million points per second and has an effective operating range of at least 100 meters. Examples of the LiDAR sensor / system 320 described herein may include, but are not limited to, at least one of the following: LiDAR TM HDL-64E 64-channel LiDAR sensor, LiDAR TM HDL-32E 32-channel LiDAR sensor, LiDAR TM PUCK TM VLP-16 16-channel LiDAR sensor, Leica Geosystems Pegasus: Two mobile sensor platform LIDAR-Lite v3 measurement sensor, Quanergy M8 LiDAR sensor, Quanergy S3 solid-state LiDAR sensor The LeddarVU is a compact solid-state fixed-beam LiDAR sensor, other industrial equivalent LiDAR sensors and / or systems, and can be used with any known or future-developed standard and / or architecture to perform the detection of illuminated targets and / or obstacles in the environment surrounding vehicle 100.

[0057] RADAR sensor 324 may include one or more radio components configured to detect objects / targets in the environment of vehicle 100. In some embodiments, RADAR sensor 324 may determine, over time, the distance, position, and / or motion vector (e.g., angle, velocity, etc.) associated with a target. RADAR sensor 324 may include a transmitter configured to generate and emit electromagnetic waves (e.g., radio waves, microwaves, etc.) and a receiver configured to detect the returned electromagnetic waves. In some embodiments, RADAR sensor 324 may include at least one processor configured to interpret the returned electromagnetic waves and determine the positional characteristics of the target. Examples of RADAR sensor 324 as described herein may include, but are not limited to, at least one of the following: Infineon RASIC TM The RTN7735PL transmitter and RTN7745PL / 46PL receiver sensors, Autoliv ASP vehicle RADAR sensors, Delphi L2C0051TR 77GHz ESR electronically scanned RADAR sensors, Fujitsu Ten Ltd. automotive compact 77GHz 3D electronically scanned millimeter-wave RADAR sensors, other industrial equivalent RADAR sensors and / or systems can be used to perform radio target and / or obstacle detection in the environment surrounding a vehicle 100 using any known or future-developed standard and / or architecture.

[0058] The ultrasonic sensor 328 may include one or more components configured to detect objects / targets in the environment of the vehicle 100. In some embodiments, the ultrasonic sensor 328 may determine, over time, the distance, position, and / or motion vector (e.g., angle, velocity, etc.) associated with the target. The ultrasonic sensor 328 may include an ultrasonic transmitter and receiver, or transceiver, configured to generate and emit ultrasonic waves and interpret the return echoes of those waves. In some embodiments, the ultrasonic sensor 328 may include at least one processor configured to interpret the returned ultrasonic waves and determine the positional characteristics of the target. Examples of the ultrasonic sensor 328 described herein may include, but are not limited to, at least one of the following: Texas Instruments TIDA-00151 automotive ultrasonic sensor interface IC sensor, MB8450 ultrasonic proximity sensor ParkSonar TM-EZ ultrasonic proximity sensor, Murata Electronics MA40H1S-R open-structure ultrasonic sensor, Murata Electronics MA40S4R / S open-structure ultrasonic sensor, Murata Electronics MA58MF14-7N waterproof ultrasonic sensor, other industrial equivalent ultrasonic sensors and / or systems, and can use any known or future-developed standard and / or architecture to perform ultrasonic detection of targets and / or obstacles in the environment surrounding vehicle 100.

[0059] Camera sensor 332 may include one or more components configured to detect image information associated with the environment of vehicle 100. In some embodiments, camera sensor 332 may include a lens, a filter, an image sensor, and / or a digital image processor. One aspect of this disclosure is that multiple camera sensors 332 may be used together to generate stereo images, thereby providing depth measurements. Examples of camera sensors 332 as described herein may include, but are not limited to, at least one of the following: ON The system utilizes the MT9V024 global shutter VGA GS CMOS image sensor, Teledyne DALSAFalcon2 camera sensor, CMOSIS CMV50000 high-speed CMOS image sensor, other industrial equivalent camera sensors and / or systems, and can perform visual target and / or obstacle detection in the environment surrounding vehicle 100 using any known or future-developed standard and / or architecture.

[0060] Infrared (IR) sensor 336 may include one or more components configured to detect image information associated with the environment of vehicle 100. IR sensor 336 may be configured to detect targets in low-light, dark, or poorly lit environments. IR sensor 336 may include an IR emitting element (e.g., an IR light-emitting diode (LED) and an IR photodiode. In some embodiments, the IR photodiode may be configured to detect returned IR light with the same or approximately the same wavelength as emitted by the IR emitting element. In some embodiments, IR sensor 336 may include at least one processor configured to interpret the returned IR light and determine the positional characteristics of the target. IR sensor 336 may be configured to detect and / or measure the temperature associated with a target (e.g., an object, pedestrian, other vehicle, etc.). Examples of IR sensor 336 as described herein may include, but are not limited to, at least one of the following: a photodiode lead salt IR array sensor, a photodiode OD-850 near-infrared LED sensor, a photodiode SA / SHA727 steady-state IR emitter and IR detector. LS microbolometer sensor, TacFLIR 380-HD InSb MWIR FPA and HD MWIR thermal sensor The VOx 640x480 pixel detector sensor, DelphiIR sensor, other industrial equivalent IR sensors and / or systems, and perform IR visual target and / or obstacle detection in the environment surrounding vehicle 100 using any known or future-developed standard and / or architecture.

[0061] The vehicle 100 may also include one or more internal sensors 337. The internal sensors 337 can measure characteristics of the internal environment of the vehicle 100. These internal sensors 337 can be combined as follows: Figure 3B Described.

[0062] In some embodiments, the vehicle sensor and system 304 may include other sensors 338 and / or combinations of the aforementioned sensors. Alternatively or additionally, one or more of the aforementioned sensors may include one or more processors configured to process and / or interpret signals detected by the one or more sensors. In some embodiments, the processing of at least some sensor information provided by the vehicle sensor and system 304 may be performed by at least one sensor processor 340. Raw and / or processed sensor data may be stored in a sensor data memory 344 storage medium. In some embodiments, the sensor data memory 344 may store instructions used by the sensor processor 340 to process the sensor information provided by the sensor and system 304. In any case, the sensor data memory 344 may be a disk drive, an optical storage device, a solid-state storage device such as random access memory (“RAM”) and / or read-only memory (“ROM”), which may be programmable, flash-updatable, etc.

[0063] The vehicle control system 348 can receive processed sensor information from the sensor processor 340 and determine an aspect of the vehicle 100 to be controlled. Controlling an aspect of the vehicle 100 may include presenting information via one or more display devices 372 associated with the vehicle, sending commands to one or more computing devices 368 associated with the vehicle, and / or controlling the driving operations of the vehicle. In some embodiments, the vehicle control system 348 may correspond to one or more computing systems that control the driving operations of the vehicle 100 according to the aforementioned level of driving automation. In one embodiment, the vehicle control system 348 can operate the speed of the vehicle 100 by controlling the output signals to the accelerometer and / or braking system of the vehicle. In this example, the vehicle control system 348 may receive sensor data describing the environment surrounding the vehicle 100 and determine, based on the received sensor data, to adjust the acceleration, power output, and / or braking of the vehicle 100. The vehicle control system 348 may also control the steering and / or other driving functions of the vehicle 100.

[0064] The vehicle control system 348 can communicate in real time with driving sensors and system 304, thereby forming a feedback loop. Specifically, upon receiving sensor information describing the target conditions in the environment surrounding vehicle 100, the vehicle control system 348 can automatically change the driving operation of vehicle 100. Then, the vehicle control system 348 can receive subsequent sensor information describing any changes to the target conditions detected in the environment due to the changed driving operation. This continuous cycle of observation (e.g., via sensors, etc.) and action (e.g., selected control or non-control of vehicle operation, etc.) allows vehicle 100 to operate automatically in the environment.

[0065] In some embodiments, one or more components of the vehicle 100 (e.g., driving vehicle sensor 304, vehicle control system 348, display device 372, etc.) can communicate with one or more entities 356A to 356N via communication network 352 through communication subsystem 350 of the vehicle 100. Figure 5 An embodiment of the communication subsystem 350 is described in more detail. For example, the navigation sensor 308 may receive global positioning, location, and / or navigation information from the navigation source 356A. In some embodiments, and to name only a few examples, the navigation source 356A may be a Global Navigation Satellite System (GNSS) similar to (if not identical to) NAVSTAR GPS, GLONASS, EU Galileo, and / or BeiDou Navigation Satellite System (BDS).

[0066] In some embodiments, the vehicle control system 348 may receive control information from one or more control sources 356B. The control source 356B may provide vehicle control information, including automatic driving control commands, vehicle operation override control commands, etc. The control source 356B may correspond to an automated vehicle control system, a traffic control system, an administrative control entity, and / or some other control server. One aspect of this disclosure is that the vehicle control system 348 and / or other components of the vehicle 100 may exchange messages with the control source 356B via the communication network 352 and through the communication subsystem 350.

[0067] Information associated with controlling the driving operations of vehicle 100 can be stored in the control data memory 364 storage medium. The control data memory 364 can store instructions used by the vehicle control system 348 to control the driving operations of vehicle 100, historical control information, autonomous driving control rules, etc. In some embodiments, the control data memory 364 can be a disk drive, an optical storage device, a solid-state storage device such as random access memory (“RAM”) and / or read-only memory (“ROM”), which can be programmable, flash-updatable, and / or similar.

[0068] In some embodiments, the vehicle control system 348 may be configured to access data required for implementing a depth estimation module, a background modeling system, an image inpainting system, and for performing various image processing-related tasks as described herein, such as generating the described histogram. In addition to the mechanical components described herein, the vehicle 100 may also include multiple user interface devices. User interface devices receive human input and convert it into mechanical motion or electrical signals or stimuli. Human input may be one or more of the following: motion (e.g., body motion in two or three-dimensional space, body part motion, etc.), voice, touch, and / or physical interaction with components of the vehicle 100. In some embodiments, human input may be configured to control one or more functions of the vehicle 100 and / or systems of the vehicle 100 described herein. The user interface may include, but is not limited to, at least one graphical user interface of the following: a display device, a steering wheel or steering mechanism, a gear lever or button (e.g., including parking position, neutral position, reverse position, and / or drive position, etc.), an accelerator control pedal or mechanism, a brake control pedal or mechanism, a power control switch, a communication device, etc.

[0069] Figure 3BA block diagram illustrating an embodiment of internal sensors 337 of vehicle 100 is shown. These internal sensors 337 may be arranged into one or more groups, at least in part based on their functionality. For example, the interior space of vehicle 100 may include environmental sensors, one or more user interface sensors, and / or safety sensors. Alternatively or additionally, sensors associated with different devices within the vehicle (e.g., smartphones, tablets, laptops, wearable devices, etc.) may also be present.

[0070] Environmental sensors may include sensors configured to collect data relating to the interior environment of vehicle 100. Examples of environmental sensors may include, but are not limited to, one or more of the following: oxygen / air sensor 301, temperature sensor 303, humidity sensor 305, light / photoelectric sensor 307, and more. Oxygen / air sensor 301 may be configured to detect the quality or characteristics of the air in the interior space 150 of vehicle 100 (e.g., including the ratio and / or type of gases in the air within vehicle 100, hazardous gas levels, safe gas levels, etc.). Temperature sensor 303 may be configured to detect temperature readings of one or more objects, areas 216, and / or regions of vehicle 100. Humidity sensor 305 may detect the amount of water vapor present in the air within vehicle 100. Light / photoelectric sensor 307 may detect the amount of light present in vehicle 100. Furthermore, light / photoelectric sensor 307 may be configured to detect different levels of light intensity associated with light within vehicle 100.

[0071] User interface sensors may include sensors configured to collect data relating to one or more users (e.g., drivers and / or one or more passengers) in vehicle 100. As will be understood, user interface sensors may include sensors configured to collect data from zone 216 in one or more areas of vehicle 100. Examples of user interface sensors may include, but are not limited to, one or more of the following: infrared sensor 309, motion sensor 311, weight sensor 313, wireless network sensor 315, biometric sensor 317, camera (or image) sensor 319, audio sensor 321, and more.

[0072] Infrared sensor 309 can be used to measure IR light emitted from at least one surface in vehicle 100, a user, or other object. Among other things, infrared sensor 309 can be used to measure temperature, form images (especially under low light conditions), identify area 216, and even detect motion in vehicle 100.

[0073] Motion sensor 311 can detect the movement and / or motion of objects within vehicle 100. Optionally, motion sensor 311 can be used alone or in combination to detect movement. For example, when a passenger in the rear of vehicle 100 unbuckles their seatbelt and begins to move around within vehicle 100, the user may be operating vehicle 100 (e.g., while driving). In this example, the passenger's movement can be detected by motion sensor 311. In response to the detection of this movement and / or the direction associated with it, the passenger can be prevented from touching and / or engaging at least some vehicle control features. As understood, such movement / motion can be alerted to the user, allowing the user to take action to prevent the passenger from interfering with vehicle control. Optionally, the number of motion sensors in the vehicle can be increased to improve the accuracy of associating motion detected within vehicle 100.

[0074] Weight sensor 313 can be used to collect data related to objects and / or users in different areas of vehicle 100. In some cases, weight sensor 313 may be included in the seat and / or floor of vehicle 100. Optionally, vehicle 100 may include wireless network sensor 315. Sensor 315 may be configured to detect one or more wireless networks within vehicle 100. Examples of wireless networks may include, but are not limited to, those utilizing… Wi-Fi TM Wireless communication using ZigBee, IEEE 802.11, and other wireless technology standards. For example, a mobile hotspot can be detected within vehicle 100 via wireless network sensor 315. In this case, vehicle 100 can determine to share the detected mobile hotspot using and / or via one or more other devices associated with vehicle 100.

[0075] Biometric sensor 317 can be used to identify and / or record characteristics associated with a user. It is contemplated that biometric sensor 317 may include at least one of the following: an image sensor, an IR sensor, a fingerprint reader, a weight sensor, a pressure measuring element, a force transducer, a heart rate monitor, a blood pressure monitor, and the like described herein.

[0076] Camera sensor 319 can record still images, video, and / or combinations thereof. Camera sensor 319 can be used alone or in combination to identify objects, users, and / or other features within vehicle 100. Among other things, two or more camera sensors 319 can be used in combination to form stereoscopic and / or three-dimensional (3D) images. Stereoscopic images can be recorded and / or used to determine depth associated with objects and / or users within vehicle 100. Furthermore, the combined use of camera sensors 319 can determine complex geometry associated with features for user identification. For example, camera sensor 319 can be used to determine dimensions between various features of a user's face (e.g., depth / distance from the user's nose to the user's cheeks, linear distance between the centers of the user's eyes, and more). These dimensions can be used to verify, record, and even modify features used for user identification. Camera sensor 319 can also be used to determine movement associated with objects and / or users within vehicle 100. It should be understood that the number of image sensors used in vehicle 100 can be increased to provide greater dimensional accuracy and / or views of images detected within vehicle 100.

[0077] Audio sensor 321 can be configured to receive audio input from a user of vehicle 100. Audio input from the user can correspond to voice commands, conversations detected in vehicle 100, telephone calls made in vehicle 100, and / or other auditory expressions made in vehicle 100. Audio sensor 321 can include, but is not limited to, microphones and other types of acoustic-electric transducers or sensors. Optionally, internal audio sensor 321 can be configured to receive sound waves and convert them into equivalent analog or digital signals. Internal audio sensor 321 can be used to determine one or more locations associated with various sounds in vehicle 100. The location of a sound can be determined based on comparisons of volume levels, intensities, etc., between sounds detected by two or more internal audio sensors 321. For example, a first audio sensor 321 can be located in a first area of ​​vehicle 100, and a second audio sensor 321 can be located in a second area of ​​vehicle 100. If the first audio sensor 321 detects a sound at a first volume level, and the second audio sensor 321 detects a sound at a second, higher volume level in the second area of ​​vehicle 100, it can be determined that the sound is closer to the second area of ​​vehicle 100. As can be understood, the number of sound receivers used in vehicle 100 can be increased (e.g., more than two, etc.) to increase the accuracy of measurements around sound detection and the location or source of sound (e.g., via triangulation, etc.).

[0078] Safety sensors may include sensors configured to collect data relating to the safety of the user and / or one or more components of the vehicle 100. Examples of safety sensors may include, but are not limited to, one or more of the following: force sensor 325, mechanical motion sensor 327, orientation sensor 329, restraint sensor 331, and more.

[0079] Force sensor 325 may include one or more sensors configured within vehicle 100 to detect forces observed in vehicle 100. An example of force sensor 325 may include a force transducer that converts the measured force (e.g., force, weight, pressure, etc.) into an output signal. Mechanical motion sensor 327 may correspond to an encoder, accelerometer, damped mass, and the like. Optionally, mechanical motion sensor 327 may be adapted to measure gravity (i.e., G-force) observed within vehicle 100. Measuring the G-force observed within vehicle 100 can provide valuable information relating to acceleration, deceleration, impact, and / or forces that one or more users in vehicle 100 may have experienced. Orientation sensor 329 may include an accelerometer, gyroscope, magnetic sensor, etc., configured to detect orientation associated with vehicle 100.

[0080] Restraint sensor 331 may correspond to a sensor associated with one or more restraint devices and / or systems in vehicle 100. Seat belts and airbags are examples of restraint devices and / or systems. As can be understood, restraint devices and / or systems may be associated with one or more sensors configured to detect the state of the device / system. This state may include extension, engagement, retraction, disengagement, deployment, and / or other electrical or mechanical conditions associated with the device / system.

[0081] The associated device sensor 323 may include any sensor associated with a device in vehicle 100. As previously described, typical devices may include smartphones, tablets, laptops, etc. It is anticipated that vehicle control system 348 may employ the various sensors associated with these devices. For example, a typical smartphone may include image sensors, IR sensors, audio sensors, gyroscopes, accelerometers, wireless network sensors, fingerprint readers, and more. One aspect of this disclosure is that one or more of these associated device sensors 323 may be used by one or more subsystems of vehicle 100.

[0082] Figure 4An embodiment of an instrument panel 400 of vehicle 100 is shown. The instrument panel 400 of vehicle 100 includes: a steering wheel 410, a vehicle operation display 420 (e.g., configured to present and / or display driving data, such as speed, measured air resistance, vehicle information, entertainment information, etc.), one or more auxiliary displays 424 (e.g., configured to present and / or display information separate from the operation display 420, entertainment applications, movies, music, etc.), a head-up display 434 (e.g., configured to display any information previously described, including but not limited to, guidance information for a route to a destination, obstacle warning information for warning of potential collisions, or some or all of the main vehicle operation data such as speed, drag, etc.), a power management display 428 (e.g., configured to display data corresponding to the vehicle 100's power level, standby power, charging status, etc.), and an input device 432 (e.g., a controller, touchscreen, or other interface device configured to interface with one or more displays in the instrument panel or components of vehicle 100. The input device 432 may be configured as a joystick, mouse, touchpad, tablet computer, 3D gesture capture device, etc.). In some embodiments, the input device 432 may be used to manually manipulate a portion of the vehicle 100 into a charging position (e.g., move the charging pad to a desired separation distance, etc.).

[0083] While one or more displays of the dashboard 400 may be touchscreen displays, it should be understood that vehicle operation displays may be displays that cannot receive touch input. For example, an operation display 420 spanning the interior space centerline 404 and crossing first zone 408A and second zone 408B may be isolated from receiving input from touch, particularly input from passengers. In some cases, displays providing vehicle operation or critical system information and interfaces may be restricted from receiving touch input and / or configured as non-touchscreen displays. This type of configuration can prevent dangerous errors when providing touch input, where such input could lead to accidents or undesirable control.

[0084] In some embodiments, one or more displays of the dashboard 400 may be mobile devices and / or applications residing on mobile devices such as smartphones. Alternatively or additionally, any information described herein may be presented to one or more portions 420A to 420N of the operating display 420 or other displays 424, 428, 434. In one embodiment, one or more displays of the dashboard 400 may be physically detachable from or removable from the dashboard 400. In some cases, detachable displays may remain tethered to the dashboard.

[0085] Portions 420A to 420N of the operating display 420 can be dynamically reconfigured and / or resized to accommodate any information display described. Alternatively, the number of portions 420A to 420N used to visually present information via the operating display 420 can be dynamically increased or decreased as needed, and is not limited to the configuration shown.

[0086] Figure 5 Hardware diagrams are shown of communication components that may optionally be associated with vehicle 100 according to embodiments of this disclosure.

[0087] The communication component may include one or more wired or wireless devices, such as transceivers and / or modems, which not only allow communication between the various systems disclosed herein but also allow communication with other devices such as devices on a network and / or devices on a distributed network such as the Internet and / or in the cloud and / or with one or more other vehicles.

[0088] The communication subsystem 350 may also include vehicle-to-vehicle and vehicle-to-vehicle communication capabilities, such as hotspot and / or access point connections for any one or more of vehicle occupants and / or vehicle-to-vehicle communication.

[0089] Additionally, although not specifically shown, the communication subsystem 350 may include one or more communication links (which may be wired or wireless) and / or communication buses (managed by the bus manager 574), including one or more of the following: CANbus, OBD-II, ARCINC 429, Byteflight, CAN (Controller Area Network), D2B (Domestic Digital Bus), FlexRay, DC-BUS, IDB-1394, IEBus, I2C, ISO 9141-1 / -2, J1708, J1587, J1850, J1939, ISO 11783, Keyword Protocol 2000, LIN (Local Internetwork), MOST (Media-Oriented System Transmission), Multifunction Vehicle Bus, SMARTwireX, SPI, VAN (Vehicle Area Network), etc., or generally any communication protocol and / or standard (one or more).

[0090] Various protocols and communications can communicate wirelessly and / or via one or more transmission media, such as single wires, twisted pairs, fiber optics, IEEE 1394, MIL-STD-1553, MIL-STD-1773, power line communication, etc. (All of the above standards and protocols are incorporated herein by reference in their entirety.)

[0091] As discussed, the communication subsystem 350 enables communication between any vehicle-to-vehicle systems and subsystems and with non-co-located resources (such as those accessible via networks like the Internet).

[0092] In addition to well-known components (omitted for clarity), the communication subsystem 350 includes interconnecting elements, including one or more of the following: one or more antennas 504, interleaver / deinterleaver 508, analog front-end (AFE) 512, memory / storage device / cache 516, controller / microprocessor 520, MAC circuitry system 522, modulator / demodulator 524, encoder / decoder 528, multiple connectivity managers 534, 558, 562, 566, GPU 540, accelerometer 544, multiplexer / demultiplexer 552, transmitter 570, receiver 572, and additional radio components (such as Wi-Fi). Module 580, Wi-Fi / BT MAC module 584, (one or more) additional transmitters 588 and (one or more) additional receivers 592). The various components in device 350 are connected via one or more links / buses 5 (also not shown for clarity).

[0093] Device 350 may have one or more antennas 504 for wireless communication, such as multiple-input multiple-output (MIMO) communication and multi-user multiple-input multiple-output (MU-MIMO) communication. LTE, 4G, 5G, Near Field Communication (NFC), etc., and are generally used for any type of wireless communication. One or more antennas 504 may include, but are not limited to, one or more of the following: directional antennas, omnidirectional antennas, monopole antennas, patch antennas, loop antennas, microstrip antennas, dipole antennas, and any other antennas (one or more) suitable for communication transmission / reception. In an exemplary embodiment, MIMO transmission / reception may require specific antenna spacing. In another exemplary embodiment, MIMO transmission / reception can achieve spatial diversity, thereby allowing different channel characteristics at each antenna. In yet another embodiment, MIMO transmission / reception can be used to allocate resources to multiple users, for example, within vehicle 100 and / or in another vehicle.

[0094] One or more antennas 504 typically interact with an analog front-end (AFE) 512, which is necessary for the proper processing of received modulated signals and the conditioning of transmitted signals. The AFE 512 may functionally be located between the antenna and the digital baseband system to convert analog signals to digital signals for processing and vice versa.

[0095] Subsystem 350 may also include a controller / microprocessor 520 and a memory / storage device / cache 516. Subsystem 350 may interact with the memory / storage device / cache 516, which may store information and operations necessary for configuring and transmitting or receiving the information described herein. The memory / storage device / cache 516 may also be used in conjunction with the controller / microprocessor 520 to execute application programming or instructions, and for temporary or long-term storage of program instructions and / or data. As an example, memory / storage device / cache 520 may include computer-readable devices, RAM, ROM, DRAM, SDRAM, and / or other storage devices (one or more) and media.

[0096] The controller / microprocessor 520 may include a general-purpose programmable processor or controller for executing application programming or instructions associated with subsystem 350. Furthermore, the controller / microprocessor 520 may perform operations for configuring and transmitting / receiving information, as described herein. The controller / microprocessor 520 may include multiple processor cores and / or implement multiple virtual processors. Optionally, the controller / microprocessor 520 may include multiple physical processors. As an example, the controller / microprocessor 520 may include a specially configured application-specific integrated circuit (ASIC) or other integrated circuit, a digital signal processor (one or more), a controller, hardwired electronic or logic circuitry, a programmable logic device or gate array, a dedicated computer, etc.

[0097] Subsystem 350 may also include one or more transmitters 570, 588 and one or more receivers 572, 592, which can transmit and receive signals to and from other devices, subsystems, and / or other destinations using the one or more antennas 504 and / or links / buses. Subsystem 350 circuitry includes a media access control or MAC circuitry 522. MAC circuitry 522 provides control over access to the wireless medium. In an exemplary embodiment, MAC circuitry 522 may be arranged to contend for the wireless medium and configure frames or packets transmitted over wired / wireless media.

[0098] Subsystem 350 may optionally include a security module (not shown). This security module may contain information about, but not limited to, security parameters required to connect the device to one or more other devices or other available networks(s), and may include WEP or WPA / WPA-2 (optionally +AES and / or TKIP) secure access keys, network keys, etc. The WEP secure access key is a secure cipher used by the Wi-Fi network. Knowing this code allows the wireless device to exchange information with the access point and / or another device. Information exchange can be performed via encoded messages, where the WEP access code is typically selected by the network administrator. WPA is an additional security standard also used in conjunction with network connectivity, where encryption is stronger than WEP.

[0099] In some embodiments, the communication subsystem 350 further includes a GPU 540, an accelerometer 544, and Wi-Fi / BT / BLE ( The system includes a low-power PHY module 580 and a Wi-Fi / BT / BLE MAC module 584, as well as an optional wireless transmitter 588 and an optional wireless receiver 592. In some embodiments, the GPU 540 may be a graphics processing unit or visual processing unit including at least one circuit and / or chip that manipulates and modifies memory to accelerate the creation of images in a frame buffer for output to at least one display device. The GPU 540 may include one or more of the following: a display device connection port, a printed circuit board (PCB), a GPU chip, a metal-oxide-semiconductor field-effect transistor (MOSFET), memory (e.g., single data rate random access memory (SDRAM), double data rate random access memory (DDR) RAM, etc., and / or combinations thereof), auxiliary processing chips (e.g., processing video output capabilities, processing and / or other functions besides the GPU chip, etc.), capacitors, heat sinks, temperature control or cooling fans, motherboard connections, shielding, etc.

[0100] Various connectivity managers 534, 558, 562, and 566 manage and / or coordinate communication between subsystem 350 and one or more systems disclosed herein, as well as one or more other devices / systems. Connectivity managers 534, 558, 562, and 566 include a charging connectivity manager 534, a vehicle database connectivity manager 558, a remote operating system connectivity manager 562, and a sensor connectivity manager 566.

[0101] The charging connectivity manager 534 can not only coordinate the physical connectivity between vehicle 100 and the charging equipment / vehicle, but also communicate with one or more of the following: an electricity management controller, one or more third parties, and optionally a billing system(s). As an example, vehicle 100 can establish communication with the charging equipment / vehicle to: coordinate the interconnectivity between the two (e.g., by aligning the charging socket on the vehicle with the charger space on the charging vehicle), and optionally share navigation information. Once charging is complete, the amount of electricity supplied can be tracked and optionally forwarded to, for example, a third party for billing. In addition to managing the connectivity used for exchanging electricity, the charging connectivity manager 534 can also transmit information such as billing information to the charging vehicle and / or the third party. This billing information can be, for example, the vehicle owner, the vehicle's driver / occupant(s), company information, or any information that can generally be used to charge the appropriate entity for the electricity received.

[0102] The Vehicle Database Connectivity Manager 558 allows subsystems to receive and / or share information stored in the vehicle database. This information can be shared with other vehicle components / subsystems and / or other entities such as third parties and / or charging systems. This information can also be shared with one or more vehicle occupant devices, such as an app on a driver's mobile device used to track information about the vehicle and / or dealers or service / maintenance providers. Typically, any information stored in the vehicle database can optionally be shared with any one or more other devices optionally subject to any privacy or confidentiality constraints.

[0103] The remote operating system connectivity manager 562 facilitates communication between vehicle 100 and any one or more automated vehicle systems. This communication may include one or more of the following: navigation information, vehicle information, other vehicle information, weather information, occupant information, or any information generally related to the remote operation of vehicle 100.

[0104] The Sensor Connectivity Manager 566 facilitates communication between any one or more vehicle sensors (e.g., driving vehicle sensors and system 304, etc.) and any one or more other vehicle systems. The Sensor Connectivity Manager 566 can also facilitate communication between any one or more sensors and / or vehicle systems and any other destination (such as a service company, an application, or any destination that typically requires sensor data).

[0105] According to one exemplary embodiment, any communication discussed herein can be transmitted via one or more conductors used for charging. An exemplary protocol that can be used for these communications is power line communication (PLC). A PLC is a communication protocol that uses power lines to simultaneously carry data and alternating current (AC) power transmission or distribution. It is also known as power line carrier, power line digital subscriber line (PDSL), power communication, power line networking (PLN). In a DC environment within a vehicle, a PLC can be used in conjunction with a CAN bus, a LIN bus over power lines (DC-LIN), and a DC-BUS.

[0106] The communications subsystem may also optionally manage one or more identifiers, such as IP (Internet Protocol) addresses (one or more) associated with the vehicle, and one or more of these or other systems, subsystems, components, and / or devices. These identifiers may be used in conjunction with any one or more connectivity managers discussed herein.

[0107] Figure 6 A block diagram is shown of a computing environment 600 that can be used as a server, user computer, or other system provided and described herein. The computing environment 600 includes one or more user computers or computing devices, such as vehicle computing device 604, communication device 608, and / or more devices 612. Computing devices 604, 608, and 612 may include general-purpose personal computers (by way of example only, including those running various versions of Microsoft...). and / or Apple Personal computers and / or laptops running various commercially available operating systems. These computing devices 604, 608, and 612 can be any workstation computer running a UNIX-like operating system. They can also have any of a variety of applications, including, for example, database client and / or server applications and web browser applications. Alternatively, these computing devices 604, 608, and 612 can be any other electronic device capable of communicating via network 352 and / or displaying and navigating web pages or other types of electronic documents or information, such as thin client computers, internet-enabled mobile phones, and / or personal digital assistants. Although an exemplary computing environment 600 with two computing devices is shown, any number of user computers or computing devices can be supported.

[0108] The computing environment 600 may also include one or more servers 614, 616. In this example, server 614 is shown as a web server, and server 616 is shown as an application server. Web server 614 can be used to process requests for web pages or other electronic documents from computing devices 604, 608, 612. Web server 614 can run an operating system, including any of those discussed above and any commercially available server operating system. Web server 614 can also run a wide variety of server applications, including SIP (Session Initiation Protocol) servers, HTTP(s) servers, FTP servers, CGI servers, database servers, etc. Servers, etc. In some cases, network server 614 can publish available operations as one or more network services.

[0109] The computing environment 600 may also include one or more file and / or application servers 616, which, in addition to an operating system, may include one or more applications accessible to clients running on one or more computing devices 604, 608, 612. Servers 616 and / or 614 may be one or more general-purpose computers capable of executing programs or scripts in response to computing devices 604, 608, 612. As an example, servers 616, 614 may execute one or more network applications. Network applications may be implemented as one or more scripts or programs written in any programming language, such as... C Application servers 616 may also include C++, and / or any scripting language, such as Perl, Python, or TCL, and any combination of programming / scripting languages. One or more application servers may also include a database server, including but not limited to those that can access databases from... The commercially purchased database servers, these block servers, can handle requests from database clients running on computing devices 604, 608, and 612.

[0110] Web pages created by servers 614 and / or 616 can be forwarded to computing devices 604, 608, and 612 via network (file) servers 614 and 616. Similarly, network server 614 can receive web page requests, network service calls, and / or input data from computing devices 604, 608, and 612 (e.g., user computers, etc.) and can forward web page requests and / or input data to network (application) server 616. In another embodiment, server 616 can be used as a file server. Although for ease of description, Figure 6A separate network server 614 and file / application server 616 are shown; however, those skilled in the art will recognize that the functions described with respect to servers 614 and 616 can be performed by a single server and / or multiple dedicated servers, depending on the specific requirements and parameters of the implementation. Computer systems 604, 608, 612, network (file) server 614, and / or network (application) server 616 can be used as... Figures 1 to 6 The system, device or component described in the document.

[0111] The computing environment 600 may also include a database 618. Database 618 may reside in various locations. As an example, database 618 may reside on a local (and / or on) storage medium of one or more computers 604, 608, 612, 614, 616. Alternatively, the database may be located remotely from any or all of computers 604, 608, 612, 614, 616 and communicate with one or more of these computers (e.g., via network 352). Database 618 may reside in a storage area network (“SAN”) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to computers 604, 608, 612, 614, 616 may be stored locally on the respective computers and / or remotely, as appropriate. Database 618 may be a relational database, such as Oracle, suitable for storing, updating, and retrieving data in response to commands in SQL format.

[0112] Figure 7 An embodiment of a computer system 700 is illustrated, on which the aforementioned server, user computer, computing device, or other system or component may be deployed or executed. The computer system 700 is shown as including hardware elements electrically coupled via a bus 704. The hardware elements may include one or more central processing units (CPUs) 708; one or more input devices 712 (e.g., mouse, keyboard, etc.); and one or more output devices 716 (e.g., display devices, printers, etc.). The computer system 700 may also include one or more storage devices 720. As an example, the storage devices 720 may be disk drives, optical storage devices, solid-state storage devices such as random access memory (“RAM”) and / or read-only memory (“ROM”), which may be programmable, flash-updatable, and / or similar.

[0113] Computer system 700 may additionally include a computer-readable storage medium reader 724; a communication system 728 (e.g., a modem, network interface card (wireless or wired), infrared communication device, etc.); and working memory 736, which may include RAM and ROM devices as described above. Computer system 700 may also include a processing acceleration unit 732, which may include a DSP, a dedicated processor, and / or the like.

[0114] The computer-readable storage medium reader 724 can also be connected to computer-readable storage media, which together (and optionally, in conjunction with one or more storage devices 720) comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently containing computer-readable information. The communication system 728 can allow the exchange of data with the network and / or any other computer described above with respect to the computer environment described herein. Furthermore, as disclosed herein, the term "storage medium" can refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, magnetic core memory, disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information.

[0115] Computer system 700 may also include software elements shown as currently residing within working memory 736, including operating system 740 and / or other code 744. It should be understood that alternative embodiments of computer system 700 may have many variations different from those described above. For example, custom hardware and / or specific elements that may be implemented in hardware, software (including portable software such as applets), or both may also be used. Furthermore, connections to other computing devices, such as network input / output devices, may be employed.

[0116] The examples of processors 340 and 708 described herein may include, but are not limited to, at least one of the following: 800 and 801, featuring 4G LTE integration and 64-bit computing. 620 and 615, with 64-bit architecture A7 processor, M7 motion coprocessor, series, Core TM Series processors Series processors Atom TM Series processors, Intel Series processors i5-4670K and i7-4770K 22nm Haswell, i5-3570K 22nm Ivy Bridge, FX TM Series processors FX-4300, FX-6300 and FX-8350 32nm Vishera Kaveri processor, Texas Jacinto C6000 TM Automotive infotainment processor, Texas OMAP TM Automotive-grade mobile processors Cortex TM -M processor, Cortex-A and ARM926EJ-S TM Processors, other industrial equivalent processors; and can perform computing functions using any known or future-developed standards, instruction sets, libraries, and / or architectures.

[0117] The embodiments of this disclosure relate to systems and methods for performing other functions related to vehicle monitoring and object detection. While the description provided herein relates to monitoring vehicles, it should be understood that the same or similar methods can be used in other environments to implement functions that are additional to or replace monitoring. As described herein, using a single image sensor, a computer system can be enabled to detect objects, determine the size and / or proximity of the objects, and perform tasks in response to detection and determination, such as generating and / or displaying notifications.

[0118] Figure 8 This is a flowchart of an exemplary method 800 for monitoring a vehicle according to one or more embodiments of this disclosure. Method 800 can be performed by and / or using the vehicle described above.

[0119] Method 800 can be performed by a vehicle without a network connection, but offline status should not be considered necessary. It should be understood that the same or similar method 800 can be performed by a vehicle that is connected to a network, and certain steps can be offloaded from the vehicle to one or more network-connected devices, such as servers, personal computers, user equipment, etc.

[0120] In some embodiments, method 800 may be performed while the vehicle is parked, such as when the driver has finished driving and left the vehicle; however, it should be understood that some aspects of method 800 may be performed in other circumstances, such as while the vehicle is in use.

[0121] Method 800 can be executed by an embedded device in the vehicle, such as the CPU or DSP of the vehicle control system 348 described herein. In some embodiments, method 800 can be an algorithm executed on a DSP. For example, the DSP can perform fixed-point computation. The DSP can be an 8-bit quantized DSP. As described below, certain aspects of method 800 may involve compensating for or adjusting such fixed-point computation to reduce the negative aspects associated with fixed-point computation.

[0122] At the beginning 803 of method 800, the vehicle as described above may include one or more image sensors, such as cameras, attached to the exterior of the vehicle. The one or more image sensors may include wide-angle lenses, such as fisheye lenses. It should be understood that a wide-angle lens may not be necessary and the same or similar methods may be performed without using a wide-angle lens.

[0123] At 806, an image can be received from a first image sensor among one or more image sensors. The image from the image sensor can be received in the form of one or more video streams. For example, the image sensor can capture video in real time, and the images can be received in real time by a processor of the vehicle control system. In some embodiments, the processor can be configured to collect or receive images from the image sensor at regular intervals, such as once per second. Receiving images can include receiving video streams in real time. The video stream can be, for example, a YUV camera stream.

[0124] When image sensors are placed on the outside of a vehicle, the received images(s) can depict the area near the vehicle, such as a view of the area around the vehicle taken from the vehicle's perspective. For example, such as Figure 9A The image 900 shown can be captured by an image sensor with a fisheye lens. In the example image 900, a person 903 is shown standing near a vehicle 906, and due to the fisheye nature of the lens, a portion of the vehicle can be seen at the bottom of the image 900. Lines 909 and 912 on the image 900 indicate distances of 30 cm and 60 cm, respectively, and these lines are for illustrative purposes only.

[0125] In some embodiments, users can view images captured by an image sensor, such as via a display device on the vehicle itself or via a user device such as a smartphone. The processor can be configured to add lines 909, 912 to the captured image to provide a size reference for the user.

[0126] In step 809, after receiving an image, the processor can use a depth estimation module or a deep network to process the image to generate a depth map. The deep network can be configured to generate, for example, a depth map. Figure 9B The depth diagram shown is 915.

[0127] The depth estimation module or deep network may be a neural network trained using stereo images to detect the depth or distance of each pixel in a single non-stereo image. In some embodiments, the depth estimation module or deep network may be specifically trained using images captured from a lens that is the same as or similar to the lens used to perform method 800 (e.g., a fisheye lens). The depth estimation module or deep network may also be trained using training images captured by the same or similar image sensors, such as images captured by the same or similar image sensors on a vehicle that is the same as or similar to the vehicle used to perform method 800.

[0128] A depth estimation module or depth network can be configured to estimate or predict the distance from the sensor to the object represented in each pixel for a single RGB image. The depth estimation module or depth network can also be configured to output a depth map, where the depth or distance of each pixel is represented by color, grayscale, hue, or a number. For example, the output of the depth estimation module or depth network could be a grayscale image with gray shades representing the estimated distance to each pixel.

[0129] In 812, the processor can use a background modeling algorithm to process the received image to generate one or more foreground masks and one or more background masks. In some embodiments, these masks can be generated in parallel with performing depth estimation.

[0130] like Figure 9C As shown, a modified version 918 of the received image 900 can be created, such that any parts detected as foreground using a background modeling algorithm are removed from the image. Figure 9C In the example shown, Figure 9A The person 903 shown has been removed from the received image 900 to form a modified version 918. The foreground mask may include the shape of the person 903, and the background mask may include the modified version 918 of the received image 900.

[0131] In some embodiments, the background modeling algorithm can be used to generate a mask based not only on the received image but also on one or more additional images or frames received from the image sensor. For example, the background modeling algorithm can use the received image and one or more preceding or following frames from the image sensor to generate one or more foreground and / or background masks for the received image.

[0132] Background modeling algorithms can be configured to generate foreground or background masks for an image by comparing the image with one or more previous or subsequent frames to detect any moving objects. Moving objects can be identified as pixels that change between the image and one or more other images. Moving objects can be represented as foreground, while stationary objects can be represented as background. Once a pixel is detected as representing foreground, it can be removed, labeled, recorded, or otherwise identified as a foreground pixel. A foreground mask can include the set of pixels identified as foreground pixels. Similarly, a background mask can include the set of pixels identified as background pixels.

[0133] In 815, using a modified version of the received image created with a background modeling algorithm, an image insulated RGB image with a padded background can be created using an image insulated network 921, as shown. Figure 9D As shown. The image inpainting network can be configured to generate an RGB image 921 of the background by inpainting any foreground objects removed as described above. The image inpainting network can be, for example, a convolutional neural network trained to take an image with one or more missing or blank pixels as input and use context-awareness designed to estimate the content of the missing or blank pixels to fill in the missing or blank pixels. In this way, a vehicle control system processor can receive images from image sensors outside the vehicle, determine images containing foreground objects such as people, remove people from the images, and fill in the image background at the location of the person in the image, thereby creating an image as if the person were not in the received image.

[0134] At 818, the padded RGB image of the background can be fed into the same or different depth estimation module as described above to generate a depth map 924 of the padded RGB image 921 of the background, as shown. Figure 9E As shown. The depth map 924 generated based on the padded RGB image can be described as a depth map of the received image as if foreground objects such as people were not in the image. Therefore, the depth map 924 can include a depth map of the background or environment of the image sensor's field of view.

[0135] At 821, one or more second foreground or background masks can be generated using the padded RGB image of the background created in step 815, similar to the description above regarding step 812. Because the padded RGB image of the background essentially removes its foreground, the one or more second foreground masks can include masks with no pixels or very few pixels compared to the foreground mask created from the received image. The background mask created using the padded RGB image of the background can include the entire image or almost the entire image.

[0136] At 824, the first or more foreground (or background) masks created in step 812 above and applicable to the first depth map can be applied to the first depth map generated in step 809 to extract the foreground region of the first depth map created in step 809. Figure 9F Image 927 shows the extracted foreground region of the first depth map of the received image.

[0137] At step 827, one or more second masks generated in step 821 can be applied to the second depth map created above based on the padded RGB image in step 818. That is, one or more new foreground masks and / or one or more new background masks based on the padded RGB image can be applied to extract the foreground region of the second depth map. Using a new foreground mask based on the padded image can improve noise reduction in the computation used to estimate the final output. Figure 9G Image 930 shows the extracted foreground region of a depth map created from a patched RGB image of the background.

[0138] At point 830, after one or more masks have been applied as discussed with respect to steps 824 and 827, a histogram can be calculated for each of the depth maps based on the raw image received from the camera and the depth map based on the RGB background image. The histograms of each depth map can be plotted together as follows: Figure 9H The single chart 933 is shown. However, it should be understood that for vehicle control systems capable of analyzing data represented by histograms, an actual histogram illustration may not be necessary.

[0139] The histogram of the masking depth map of the received original image can be considered as the signal, while the masking depth map of the patched RGB background image can be considered as the reference.

[0140] The processor can be configured to determine whether an object is near the image sensor and therefore near the vehicle by subtracting the signal from a reference. The vertical y-axis of the histogram can represent the number of pixels, while the horizontal x-axis of the histogram can represent the estimated distance for each pixel.

[0141] In 833, method 800 may include determining one or more of the size and relative depth range of one or more foreground objects based on histograms of each of a depth map based on a depth map of an original image received from a camera and a depth map based on an RGB background image.

[0142] The processor can be configured to determine whether a minimum or threshold number of pixels exists within a specific distance range (e.g., between 30 cm and 60 cm) after subtracting a reference histogram from the signal histogram. Alternatively, the processor can be configured to determine whether the signal histogram includes a minimum or threshold number of pixels exceeding that of the reference histogram within the specific distance range (e.g., between 30 cm and 60 cm). The minimum or threshold number of pixels can be any value and can be set by the user or automatically.

[0143] If the threshold is not met, method 800 may include returning to step 806 and continuing to process newly received images, such as the next image in the video stream. If the threshold is met, method 800 may include generating a warning 836 as described below, and the method may continue processing additional images or may terminate at 839. In some embodiments, the method may continue at intervals and / or may be executed in response to the detection of a moving object.

[0144] In 836, based on one or more of the determined sizes and relative depth ranges of one or more foreground objects, the method may include generating one or more notifications. In some embodiments, the method may include generating warnings based on the range of detected objects. For example, the method may include generating different warnings when a person or object is within a proximity range of 30 cm to 60 cm.

[0145] Based on user settings, different notifications can be set for different ranges, such as if the object's range is less than 30 cm, between 30 cm and 60 cm, or greater than 60 cm. In this way, the vehicle control system can distinguish the range and generate different warnings for each range.

[0146] As an example, such warnings could include: not generating warnings for objects larger than 60 centimeters; generating basic-level warnings for objects between 30 and 60 centimeters; and generating higher-level warnings for objects within a proximity range of zero to 30 centimeters.

[0147] In some embodiments, basic-level warnings may include notifications on a smartphone app, while higher-level warnings may include audible sounds or other types of responses from the vehicle. Users can adjust such warnings and notifications.

[0148] In some embodiments, an alarm may be generated that includes data associated with the determined size and / or relative depth range of a detected object or person. Such data may include, for example, a written description, such as an explanation of the size and / or distance of the detected object. Alternatively, it may include an image of the object, such as a received image or an edited version of a received image.

[0149] For example, such as Figure 10A As shown, the user interface 1000 may include a display indicating that motion has been detected.

[0150] like Figure 10B As shown, the user interface 1003 may include a display indicating that motion has been detected within a specific distance (e.g., 60 cm).

[0151] like Figure 10C As shown, the user interface 1006 may include a display indicating that motion has been detected within a specific distance range (e.g., between 30 cm and 60 cm).

[0152] like Figure 10D and Figure 10E As shown, user interfaces 1009 and 1012 may include a display indicating that motion has been detected, and may include a summary of the results of the object detection and / or recognition system, such as whether identified and / or unidentified persons have been detected. Such user interfaces 1009 and 1012 may include identification of any identified persons or objects.

[0153] like Figure 10F As shown, the user interface 1015 may include a display indicating that motion has been detected, and may also include a display of images received from an image sensor.

[0154] In some embodiments, a warning or notification may be generated only at specific intervals or only when an object is first detected within a specific time range or distance range. In this way, for example, the movement of a person or object past a vehicle may result in the generation of only a single notification or a notification for each distance range of the person or object's movement.

[0155] In some embodiments, instead of generating a notification or in addition to generating a notification, the method may include activating one or more sensors in addition to activating the image sensor used in the method. For example, if motion is detected in a camera, that camera and / or other cameras may be activated and instructed to begin recording. For example, high-power sensors (such as LiDAR, high-definition image sensors, microphones, etc.) that are typically powered off when the vehicle is not in use may be powered on or otherwise activated in response to detecting an object at a specific distance using the methods described herein. Such a system enables efficient use of sensors, thereby extending battery life.

[0156] In some embodiments, after one or more other sensors are activated, these other sensors can be used to gather more information about the detected object. For example, a microphone can capture sound, a speaker can play noise, and a LiDAR sensor can scan the area to collect more data, etc.

[0157] In some embodiments, an object recognition system may be deployed to identify people or objects within an image. The generated warning may include information such as the detection of a person or object. Users may be able to set up whitelists to select which people and / or objects should not be considered threats and / or should not trigger warnings. If a vehicle owner is detected, the method may include activating features such as powering on the vehicle, unlocking, or opening a door.

[0158] In some embodiments, FastCV routines may be implemented for color correction, background subtraction, and other functions performed on the image used when performing the methods 800 described herein.

[0159] When method 800 is executed as an algorithm on a DSP using fixed-point computation, some information may be lost in one or more depth maps. In this case, a mean filter and / or a value filter can be applied to the histogram to recover the lost depth values.

[0160] When implemented by a DSP, this method may involve using a deep quantization network 842 or a deep network quantizer application to perform steps after receiving the raw image and before generating or computing the histogram. For example... Figure 8 As shown, steps 809, 812, 815, 818, 821, 824, and 827 can each be performed using the same or similar deep quantization network 842, and after step 830, a mean filter and / or a mean filter can be applied to account for fixed-point computation. Other processing can also be performed to reduce quantization noise.

[0161] It can execute any of the steps, functions, and operations discussed in this article continuously and automatically.

[0162] Exemplary systems and methods of this disclosure have been described with respect to vehicle systems and electric vehicles. However, to avoid unnecessarily obscuring this disclosure, many known structures and devices have been omitted from the foregoing description. This omission should not be construed as a limitation on the scope of the claimed disclosure. Many specific details have been set forth to provide an understanding of this disclosure. However, it should be understood that this disclosure can be practiced in various ways beyond the specific details set forth herein.

[0163] Furthermore, while the exemplary embodiments shown herein illustrate various components of the system in combination, some components of the system may be located remotely, at distant locations within a distributed network such as a LAN and / or the Internet, or within a dedicated system. Therefore, it should be understood that the components of the system may be combined into one or more devices, such as servers, communication equipment, or co-located on specific nodes of a distributed network, such as analog and / or digital telecommunications networks, packet-switched networks, or circuit-switched networks. As will be understood from the foregoing description, and for computational efficiency reasons, the components of the system may be arranged anywhere within the distributed component network without affecting the operation of the system.

[0164] Furthermore, it should be understood that the various links connecting the elements can be wired or wireless links or any combination thereof, or any other known or subsequently developed element(s) capable of providing and / or transmitting data to and from the connected element. These wired or wireless links can also be secure links and can be capable of transmitting encrypted information. For example, the transmission medium used as a link can be any suitable carrier for electrical signals, including coaxial cables, copper wires, and optical fibers, and can take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0165] Although flowcharts have been discussed and illustrated with respect to specific event sequences, it should be understood that changes, additions, and omissions to this sequence may occur without materially affecting the operation of the disclosed embodiments, configurations, and aspects.

[0166] Many changes and modifications to this disclosure may be made. Some features of this disclosure may be provided without providing others.

[0167] In yet another embodiment, the systems and methods disclosed herein may be implemented in conjunction with a dedicated computer, a programmable microprocessor or microcontroller and one or more peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, hardwired electronic devices or logic circuits (such as discrete element circuits), programmable logic devices or gate arrays (such as PLDs, PLAs, FPGAs, PALs), a dedicated computer, any equivalent apparatus, etc. Generally, any one or more devices or apparatuses capable of implementing the methods shown herein may be used to implement various aspects of this disclosure. Exemplary hardware that may be used for this disclosure includes computers, handheld devices, telephones (e.g., cellular, internet, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. Furthermore, alternative software implementations may be constructed, including but not limited to distributed processing or component / target distributed processing, parallel processing, or virtual machine processing, to implement the methods described herein.

[0168] In yet another embodiment, the disclosed method can be readily implemented using software from an object-oriented or object-based software development environment that provides portable source code usable on a variety of computer or workstation platforms. Alternatively, the disclosed system can be implemented partially or entirely in hardware using standard logic circuitry or VLSI design. Whether to implement a system according to this disclosure using software or hardware depends on the speed and / or efficiency requirements of the system, the specific functionality, and the specific software or hardware system or microprocessor or microcomputer system used.

[0169] In yet another embodiment, the disclosed method can be implemented in part as software, which can be stored on a storage medium and executed on a programmed general-purpose computer, special-purpose computer, microprocessor, etc., in cooperation with a controller and memory. In these cases, the systems and methods of this disclosure can be implemented as programs embedded in a personal computer (such as applets, etc.). This can be a CGI script, a resource residing on a server or computer workstation, a routine embedded in a dedicated measurement system, system components, etc. The system can also be implemented by physically integrating the system and / or method into the software and / or hardware system.

[0170] While this disclosure describes components and functions implemented in embodiments with reference to specific standards and protocols, it is not limited to such standards and protocols. Other similar standards and protocols not mentioned herein exist and are considered to be included in this disclosure. Furthermore, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are regularly superseded by faster or more efficient equivalents with substantially the same functionality. Such alternative standards and protocols with the same functionality are considered to be equivalents included in this disclosure.

[0171] This disclosure includes, in various embodiments, configurations, and aspects, components, methods, processes, systems, and / or apparatuses substantially as depicted and described herein, including various embodiments, sub-combinations, and subsets thereof. Those skilled in the art, upon understanding this disclosure, will understand how to make and use the systems and methods disclosed herein. This disclosure includes, in several different embodiments, configurations, and aspects, providing apparatus and processes, for example, to improve performance, facilitate implementation, and / or reduce implementation costs, in the absence of matters not depicted and / or described herein, or in several different embodiments, configurations, or aspects thereof, in the absence of such matters that might have been used in previous apparatus or processes.

[0172] The foregoing discussion of this disclosure has been presented for purposes of illustration and description. The foregoing is not intended to limit this disclosure to the one or more forms disclosed herein. In, for example, the specific embodiments described above, various features of this disclosure are combined in one or more embodiments, configurations, or aspects for the purpose of simplification. Features of embodiments, configurations, or aspects of this disclosure may be combined in alternative embodiments, configurations, or aspects other than those discussed above. This approach to the disclosure should not be construed as reflecting an intention that the claimed disclosure requires more features than expressly recited in the claims. Rather, as reflected in the appended claims, the inventive aspect relies on fewer than all features of a single foregoing disclosed embodiment, configuration, or aspect. Therefore, the appended claims are hereby incorporated in this specific embodiment, wherein each claim relies on itself as an independent preferred embodiment of this disclosure.

[0173] Furthermore, while the description in this disclosure includes descriptions of one or more embodiments, configurations, or aspects, as well as certain variations and modifications, other changes, combinations, and modifications are also within the scope of this disclosure, for example, as may be within the skill and knowledge of one skilled in the art upon understanding this disclosure. It is intended to obtain the right to include alternative embodiments, configurations, or aspects to the permissible extent, including alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps of those claimed, regardless of whether such alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps are disclosed herein, and is not intended to publicly offer any patentable subject matter.

[0174] An embodiment includes a method for monitoring a vehicle, the method comprising: receiving an image from an image sensor; processing the received image using a depth estimation module to generate a first depth map; processing the received image using a background modeling algorithm to generate one or more masks, a foreground mask and a background mask; generating a patched image using an image inpainting network, the patched image including the received image having patched regions associated with the one or more masks, the foreground mask and the background mask; processing the patched image using the depth estimation module to generate a second depth map; calculating a histogram of the first depth map and a histogram of the second depth map; determining one or more of the size and relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alarm, the alarm including data associated with the determined size and relative depth range of one or more foreground objects.

[0175] Aspects of the above method include, wherein the image is generated using a fisheye lens.

[0176] Aspects of the above method include, wherein the image sensor is mounted on the exterior of the vehicle.

[0177] The above-described methods include, in particular, the parallel processing of the image using the depth estimation module and the background modeling algorithm.

[0178] Aspects of the above method include, wherein the histogram of the first depth map is calculated based on the regions of the first depth map corresponding to one or more of the foreground and background masks.

[0179] Aspects of the above method include, wherein the histogram of the second depth map is calculated based on the regions of the first depth map corresponding to one or more of the foreground and background masks.

[0180] Aspects of the above method include determining the size of the one or more foreground objects and one or more of the relative depth ranges based on a comparison between the first depth map and the second depth map.

[0181] Aspects of the above method include, wherein the repaired image reflects the background of the received image, and wherein the one or more foreground objects are replaced with the generated background.

[0182] The methods described above include, but are not limited to, transmitting alarms to user equipment.

[0183] The methods described above include, but are not limited to, displaying the alarm.

[0184] Aspects of the above method include, further comprising, determining, before generating the alarm, that one or more of the size and the relative depth range exceed a threshold.

[0185] The above methods include, but are not limited to, repeating the methods at regular intervals.

[0186] Aspects of the above method include, further comprising, activating one or more sensors in response to determining one or more of the size and the relative depth range of one or more foreground objects.

[0187] An embodiment includes a user equipment comprising: a processor; and a computer-readable storage medium storing computer-readable instructions, which, when executed by the processor, cause the processor to perform a method for monitoring a vehicle, the method comprising: receiving an image from an image sensor; processing the received image using a depth estimation module to generate a first depth map; processing the received image using a background modeling algorithm to generate one or more masks chosen from a foreground mask and a background mask; generating a patched image using an image inpainting network, the patched image including a received image having patched regions associated with the one or more masks chosen from the foreground mask and the background mask; processing the patched image using the depth estimation module to generate a second depth map; calculating a histogram of the first depth map and a histogram of the second depth map; determining one or more of the size and relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alarm including data associated with the determined size and relative depth range of one or more foreground objects.

[0188] The aforementioned user equipment includes aspects wherein the image is generated using a fisheye lens.

[0189] The aforementioned user equipment includes, wherein the image sensor is mounted on the exterior of the vehicle.

[0190] The aforementioned user equipment includes, wherein the method further includes parallel processing of the image using the depth estimation module and processing of the image using the background modeling algorithm.

[0191] The aforementioned aspects of the user equipment include, wherein the histogram of the first depth map is calculated based on the regions of the first depth map corresponding to one or more of the foreground and background masks.

[0192] The aforementioned aspects of the user equipment include, wherein the histogram of the second depth map is calculated based on the regions of the first depth map corresponding to one or more of the foreground and background masks.

[0193] The aforementioned user equipment includes, wherein, based on a comparison of the first depth map and the second depth map, determining the size of the one or more foreground objects and one or more of the relative depth ranges.

[0194] The aforementioned aspects of the user equipment include, wherein the patched image reflects the background of the received image, and wherein the one or more foreground objects are replaced with the generated background.

[0195] The aforementioned aspects of the user equipment include, wherein the method further includes transmitting an alarm to the user equipment.

[0196] The aforementioned user equipment includes various aspects, wherein the method further includes displaying the alarm.

[0197] The aforementioned aspects of the user equipment include, wherein the method further includes, prior to generating the alarm, determining that one or more of the size and the relative depth range exceed a threshold.

[0198] The aforementioned user equipment includes various aspects, wherein the method further includes repeating the method at regular intervals.

[0199] The aforementioned aspects of the user equipment include, wherein the method further includes, in response to determining one or more of the dimensions of one or more foreground objects and the relative depth range, activating one or more sensors.

[0200] An embodiment includes a computer program product comprising: a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being configured to perform a method for monitoring a vehicle when executed by a processor, the method comprising: receiving an image from an image sensor; processing the received image using a depth estimation module to generate a first depth map; processing the received image using a background modeling algorithm to generate one or more masks chosen from a foreground mask and a background mask; generating a patched image using an image inpainting network, the patched image including a received image having patched regions associated with the one or more masks chosen from the foreground mask and the background mask; processing the patched image using the depth estimation module to generate a second depth map; calculating a histogram of the first depth map and a histogram of the second depth map; determining one or more of the size and relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alarm including data associated with the determined size and relative depth range of one or more foreground objects.

[0201] The aforementioned computer program product includes aspects wherein the image is generated using a fisheye lens.

[0202] The aforementioned computer program product includes aspects wherein the image is generated using a fisheye lens.

[0203] The aforementioned computer program product includes, wherein the image sensor is mounted on the exterior of the vehicle.

[0204] The aforementioned computer program product includes aspects wherein the method further includes parallel execution of processing the image using the depth estimation module and processing the image using the background modeling algorithm.

[0205] The aforementioned computer program product includes a histogram of the first depth map calculated based on regions of the first depth map corresponding to one or more of the foreground and background masks.

[0206] The aforementioned computer program product includes a histogram of the second depth map calculated based on regions of the first depth map corresponding to one or more of the foreground and background masks.

[0207] The aforementioned computer program product includes determining, based on a comparison of the first depth map and the second depth map, the size of the one or more foreground objects and one or more of the relative depth ranges.

[0208] The aforementioned computer program product includes aspects in which the patched image reflects the background of the received image, and wherein the one or more foreground objects are replaced with the generated background.

[0209] The aforementioned computer program product includes various aspects, wherein the method further includes transmitting an alarm to a user device.

[0210] The aforementioned computer program product includes various aspects, wherein the method further includes displaying the alarm.

[0211] The aforementioned computer program product includes, wherein the method further includes, determining that one or more of the size and the relative depth range exceed a threshold before generating the alarm.

[0212] The aforementioned computer program product includes various aspects, wherein the method further includes repeating the method at regular intervals.

[0213] The aforementioned computer program product includes, wherein the method further includes, in response to determining one or more of the dimensions of one or more foreground objects and the relative depth range, activating one or more sensors.

[0214] Any one or more aspects / executives substantially disclosed herein may optionally be combined with any one or more other aspects / executives substantially disclosed herein.

[0215] A device or apparatus is adapted to perform any one or more aspects / embodiments as substantially disclosed herein in the foregoing aspects / embodiments.

[0216] The phrases “at least one,” “one or more,” “or,” and “and / or” are open-ended expressions that are both conjunction and disjunctive in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” “A, B, and / or C,” and “A, B, or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0217] The term "a" or "an" entity refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" are used interchangeably.

[0218] As used herein, the term "automatic" and its variations refer to any process or operation performed without significant human input, typically sequential or semi-sequential. However, a process or operation can be automatic if input is received prior to its execution, even if significant or insignificant human input is used in its execution. Human input is considered significant if it influences how the process or operation is performed. Human input that consents to the execution of a process or operation is not considered "significant."

[0219] Various aspects of this disclosure may take the form of a wholly hardware embodiment, a wholly software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as a “circuit,” “module,” or “system.” Any combination of one or more computer-readable media may be used. Computer-readable media may be computer-readable signal media or computer-readable storage media.

[0220] Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include the following: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, computer-readable storage media can be any tangible medium that can contain or store programs for use by or in connection with an instruction execution system, apparatus, or device.

[0221] Computer-readable signal media may include propagated data signals in which computer-readable program code is embedded, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and may transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code embedded on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0222] As used herein, the terms “determine,” “calculate,” “infer,” and their variations are used interchangeably and include any type of method, process, mathematical operation, or technique.

[0223] The term "electric vehicle" (EV), also referred to herein as an electrically driven vehicle, can be propelled by one or more electric motors or traction motors. Electric vehicles can be powered by electricity from an external source via a collector system, or they can contain batteries or generators to convert fuel into electricity. Electric vehicles generally include rechargeable energy storage systems (RESS) (also known as fully electric vehicles (FEVs)). Energy storage methods can include: chemical energy stored in the vehicle's onboard battery (e.g., in a battery electric vehicle or BEV), onboard kinetic energy storage devices (e.g., inertial wheels), and / or static energy (e.g., through onboard double-layer capacitors). Rechargeable onboard energy storage devices can take the form of batteries, double-layer capacitors, and inertial wheel energy storage devices.

[0224] The term "hybrid electric vehicle" refers to a vehicle that combines a conventional (usually fossil fuel-powered) powertrain with some form of electric propulsion. Most hybrid electric vehicles combine a conventional internal combustion engine (ICE) propulsion system with an electric propulsion system (hybrid vehicle drivetrain). In a parallel hybrid vehicle, both the ICE and the electric motor are connected to a mechanical transmission and can typically transmit power to drive the wheels simultaneously via the conventional transmission. In a series hybrid vehicle, only the electric motor drives the powertrain, and the smaller ICE acts as a generator to power the electric motor or recharge the battery. Power-split hybrid vehicles exhibit both series and parallel characteristics. A full hybrid vehicle, sometimes called a strong hybrid, is a vehicle that can operate solely on its engine, solely on its battery, or a combination of both. A mid-range hybrid vehicle is a vehicle that cannot be driven solely by its electric motor because the electric motor does not have sufficient power to propel the vehicle itself.

[0225] The term "rechargeable electric vehicle" or "REV" refers to a vehicle with an onboard rechargeable energy storage device, including electric vehicles and hybrid electric vehicles.

Claims

1. A method of monitoring a vehicle, the method comprising: receiving an image from an image sensor; processing the received image with a depth estimation module to generate a first depth map; processing the received image with a background modeling algorithm to generate one or more of a foreground mask and a background mask; generating a patched image with an image patching network, the patched image comprising the received image with patched regions associated with the one or more of the foreground mask and the background mask; processing the patched image with the depth estimation module to generate a second depth map; computing a histogram of the first depth map and a histogram of the second depth map; determining one or more of a size and a relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alert comprising data associated with the one or more of the size and the relative depth range of the one or more foreground objects. The image is generated with a fisheye lens.

2. The method of claim 1, wherein, The image sensor is mounted outside the vehicle.

3. The method of claim 1, wherein, 4. The method of claim 1, further comprising processing the image with the depth estimation module is performed in parallel with processing the image with the background modeling algorithm. The histogram of the first depth map is computed based on regions of the first depth map corresponding to the one or more of the foreground mask and the background mask.

5. The method of claim 1, wherein, 6. The method of claim 1, further comprising recovering missing information from the histograms using a mean filter. The histogram of the second depth map is computed based on regions of the first depth map corresponding to the one or more of the foreground mask and the background mask.

7. The method of claim 1, wherein, The one or more of the size and the relative depth range of the one or more foreground objects is determined based on a comparison of the first depth map and the second depth map.

8. The method of claim 1, wherein, The patched image reflects a background of the received image, wherein the one or more foreground objects are replaced with the generated background.

9. The method of claim 1, wherein, 10. The method of claim 1, further comprising transmitting the alert to a user device.

11. The method of claim 1, further comprising displaying the alert.

12. The method of claim 1, further comprising determining that the one or more of the size and the relative depth range exceed a threshold value prior to generating the alert.

13. The method of claim 1, further comprising repeating the method at intervals.

14. The method of claim 1, further comprising activating one or more sensors in response to determining the one or more of the size and the relative depth range of one or more foreground objects.

15. The method of claim 1, further comprising applying the one or more of the foreground mask and the background mask to the first depth map prior to computing the histogram of the first depth map. ​ 16. The method of claim 15, further comprising processing the inpainted image with the background modeling algorithm to generate a second one or more of a foreground mask and a background mask.

17. The method of claim 16, further comprising applying the second one or more of the foreground mask and the background mask to the second depth map prior to computing a histogram of the second depth map.

18. A user device comprising: a processor; and a computer readable storage medium having computer readable instructions stored thereon, which when executed by the processor, cause the processor to perform a method of monitoring a vehicle, the method comprising: receiving an image from an image sensor; processing the received image with a depth estimation module to generate a first depth map; processing the received image with a background modeling algorithm to generate one or more of a foreground mask and a background mask; generating an inpainted image with an image inpainting network, the inpainted image comprising the received image with inpainted regions associated with the one or more of the foreground mask and the background mask; processing the inpainted image with the depth estimation module to generate a second depth map; computing a histogram of the first depth map and a histogram of the second depth map; determining one or more of a size and a relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alert, the alert comprising data associated with the one or more of the size and the relative depth range of the one or more foreground objects determined. the image is generated with a fisheye lens.

19. The user equipment of claim 18, wherein, 20. A computer program product comprising: a non-transitory computer readable storage medium having computer readable program code embodied thereon, the computer readable program code configured to perform, when executed by a processor, a method of monitoring a vehicle, the method comprising: receiving an image from an image sensor; processing the received image with a depth estimation module to generate a first depth map; processing the received image with a background modeling algorithm to generate one or more of a foreground mask and a background mask; generating an inpainted image with an image inpainting network, the inpainted image comprising the received image with inpainted regions associated with the one or more of the foreground mask and the background mask; processing the inpainted image with the depth estimation module to generate a second depth map; computing a histogram of the first depth map and a histogram of the second depth map; determining one or more of a size and a relative depth range of one or more foreground objects based on the histogram of the first depth map and the histogram of the second depth map; and generating an alert, the alert comprising data associated with the one or more of the size and the relative depth range of the one or more foreground objects determined. ​

Images