Perspective viewing system for vehicle

Abstract

A see-through vision system for a vehicle includes a look-around system having an externally facing camera. An externally facing camera is in communication with the controller. The driver monitoring system includes a driver monitoring sensor configured to monitor a viewing direction of a driver. The controller includes a memory storing a three-dimensional model of the vehicle, a simultaneous localization and mapping (SLAM) module, a surround field module, and a data link for connection to the wearable vision system. The controller also stores instructions for identifying a viewing direction of the driver using the driver monitoring system, identifying at least one opaque portion of the vehicle in the driver's field of view using the viewing direction and a three-dimensional model of the vehicle, a cropped live video of the exterior field of view of the vehicle is generated at the location of the at least one opaque portion using the surround field of view module.

Description

Technical Field

[0001] This topic relates to vehicles, and more specifically, to a system for visualizing the external environment through opaque parts of a vehicle. Background Technology

[0002] Operating a vehicle requires the operator to be aware of their surroundings. However, some parts of the vehicle are opaque or otherwise obscured, preventing the operator from seeing the entire environment. Vehicles address this shortcoming by using mirrors, outward-facing cameras, vision systems, and the like. However, such systems still provide insufficient information or observation to the operator because they display information about the surroundings on screens located away from where the operator might be looking, positions that are static and potentially unintuitive within the vehicle.

[0003] Therefore, it is desirable to provide a system for visually observing the external environment around a vehicle from the position of a vehicle operator, combining the visible area and the obstructed area in a continuous and natural manner. Summary of the Invention

[0004] In one exemplary embodiment, the perspective vision system for a vehicle includes a surround-view system having a plurality of outward-facing cameras. Each of the outward-facing cameras communicates with a controller. The driver monitoring system includes at least one driver monitoring sensor configured to monitor the driver's viewing direction. The at least one driver monitoring sensor communicates with the controller. The controller includes a non-transitory memory storing a 3D model of the vehicle, a Simultaneous Localization and Mapping (SLAM) module, an surround-view module, and a data link for connecting to the wearable vision system. The controller also stores instructions for: using the driver monitoring system to identify the driver's viewing direction; using the viewing direction and the 3D model of the vehicle to identify at least one opaque portion of the vehicle in the driver's field of vision; using the surround-view module to generate a cropped live video of the external view of the vehicle at the location of the at least one opaque portion; and using the data link to export the generated cropped live video to the wearable vision system.

[0005] In addition to one or more features described herein, the at least one driver monitoring sensor includes an inward-facing camera that defines a field of view including the driver's head.

[0006] In addition to one or more features described herein, the SLAM module is configured to use simultaneous localization and mapping algorithms to determine the position and orientation of the wearable vision system relative to the vehicle, at least in part based on the driver’s viewing direction.

[0007] In addition to one or more features described herein, the SLAM module is configured to receive at least one of sensor information and image information from the wearable vision system via a data link, and the SLAM module is configured to determine the position and orientation of the wearable vision system relative to the vehicle based at least in part on the driver’s viewing direction and at least one of the sensor information and image information from the wearable vision system.

[0008] In addition to one or more of the features described in this paper, the simultaneous localization and mapping algorithms are executed specifically on the controller.

[0009] In addition to one or more features described herein, image information includes image feeds captured from cameras set on the wearable vision system.

[0010] In addition to one or more features described herein, image information also includes object identifiers of objects in the image feed.

[0011] In addition to one or more features described herein, the SLAM module is also configured to determine the position and orientation of the wearable vision system relative to the vehicle based at least in part on the recognition of objects in the image feed.

[0012] In addition to one or more features described herein, the three-dimensional model of the vehicle defines the relative positions and dimensions of the vehicle's structural elements and determines which structural elements of the vehicle are opaque.

[0013] In addition to one or more features described herein, the vehicle also includes a set of vehicle operation sensors configured to identify vehicle speed, vehicle position, hazard identification, warning signals, driving instructions, and driving direction.

[0014] In addition to one or more features described herein, the cropped live video of the vehicle’s external view at a location with at least one opaque portion generated using the surround view module includes annotations of the cropped live view with at least one of vehicle driving speed, pedestrian detection information and hazard identification, warning signals, and driving instructions.

[0015] In another exemplary embodiment, a method for generating a perspective view of a vehicle includes: using a vehicle controller to determine the position and orientation of a wearable vision system within the vehicle. The vehicle controller operates a Simultaneous Localization and Mapping (SLAM) algorithm based at least in part on sensor information from a vehicle driver monitoring system. The method determines at least one opaque portion of the vehicle in the driver's field of view based on the position and orientation of the wearable vision system within the vehicle and a three-dimensional model of the vehicle. The method generates a cropped video feed of the vehicle's external view at the location of the at least one opaque portion of the vehicle and outputs the cropped video feed to the wearable vision system.

[0016] In addition to one or more features described herein, the method includes displaying a cropped video feed to the driver using at least one screen of a wearable vision system.

[0017] In addition to one or more of the features described in this paper, the Simultaneous Localization and Mapping (SLAM) algorithm is performed only on the vehicle controller.

[0018] In addition to one or more features described herein, the method also includes receiving image information from a wearable vision system, wherein the determined position and orientation of the wearable vision system within the vehicle are also based on the image information from the wearable vision system.

[0019] In addition to one or more features described herein, the received image information includes image feeds captured from cameras set on the wearable vision system.

[0020] In addition to one or more features described herein, image information also includes object identifiers of objects in the image feed.

[0021] In addition to one or more features described herein, the three-dimensional model of the vehicle defines the relative positions and dimensions of the vehicle's structural elements and determines which structural elements of the vehicle are opaque.

[0022] In addition to one or more features described herein, generating a cropped video feed of the vehicle's external view includes disabling the cropped video feed in response to identifying that the vehicle's speed exceeds a threshold.

[0023] In addition to one or more features described herein, generating a cropped video feed for the vehicle’s external view includes adding a display of the vehicle’s driving parameters to the external view, wherein the driving parameters include at least one of driving speed, navigation system orientation, hazard detection, and pedestrian detection.

[0024] The above-described features and advantages, as well as other features and advantages, of this disclosure will become apparent when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0025] Other features, advantages, and details appear by way of example only in the following detailed description, which is described in detail with reference to the accompanying drawings, wherein:

[0026] Figure 1 It is a vehicle that includes a perspective observation system;

[0027] Figure 2 It is stored in the vehicle's control system. Figure 1 A 3D model of the vehicle;

[0028] Figure 3 yes Figure 1 Example operation of the perspective observation system; and

[0029] Figure 4 It is used to implement Figure 1 The system architecture of the in-vehicle perspective observation system. Detailed Implementation

[0030] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. It should be understood that throughout the drawings, corresponding reference numerals denote the same or corresponding parts and features. As used herein, the term "module" refers to processing circuitry that may include application-specific integrated circuits (ASICs), electronic circuitry, processor (shared, dedicated, or group) and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality.

[0031] As used herein, the term controller refers to a dedicated controller that includes a processor and memory, a general-purpose system processor and memory that stores software modules for implementing a particular control process, a network of interconnected processors and memory configured to collaboratively implement one or more control subroutines, or any similar configuration of processors and memory capable of implementing the control operations described herein.

[0032] As used in this article, wearable vision systems refer to any wearable display, including smart glasses, systems equipped with augmented reality, etc.

[0033] In a typical example, the Vehicle Perspective Visualization System (STVS) utilizes the vehicle's processor and computing power to perform a simultaneous localization and mapping (SLAM) process to determine the position and orientation of the wearable vision system relative to the vehicle. The vehicle processor compares the wearable vision system's position and orientation with a 3D model of the vehicle and determines which portions of the wearable vision system's screen are oriented towards the opaque parts of the vehicle structure. Based on this determination, the vehicle processor causes the screen to display an image of the vehicle's exterior, excluding the opaque parts of the vehicle structure, at the identified portions of the screen.

[0034] By using this architecture, while wearing the wearable vision system, the vehicle operator is provided with a field of view comparable to that available when the vehicle is not opaque, effectively eliminating blind spots. Furthermore, by using the vehicle processor to implement augmented reality features, STVS allows augmented reality features, including SLAM-based visualization, to be implemented on any number of wearable visualization systems, even when the wearable visualization system lacks the internal hardware and software for augmented reality implementation. Moreover, STVS allows some implementations of the visualization system to be hardware-agnostic and presents the same perspective vehicle visualization to any user, regardless of the manufacturer or model of the wearable vision system they may be using.

[0035] According to an exemplary embodiment, Figure 1 Vehicle 10 is shown. Vehicle 10 includes an assembly of outward-facing imaging sensors 20, each defining a field of view 22 away from vehicle 10. Although in Figure 1 The example shown includes four different imaging sensors 20, but it should be understood that the vehicle 10 may include a sufficient number and any number of outward-facing imaging sensors 20 to generate a full surround view of the vehicle 10.

[0036] Each of the imaging sensors 20 provides the sensed image to the controller 30. The controller 30 includes the vehicle portion 40 of the STVS architecture 300 (in... Figure 4 (See detailed diagram) and surround view generation module 50. Controller 30 may be a dedicated STVS system controller including memory and processor, a sub-part of a general controller, a controller network configured to communicate with each other and cooperate to implement the STVS architecture 300 surround view generation module 50 and vehicle part 40, or any similar computer controller configuration including memory and processor and arranged to implement the system described herein.

[0037] Also connected to controller 30 is a wearable vision system 60, such as a pair of smart glasses. The wearable vision system can be connected to controller 30 directly, via wired connection 62, via indirect wired connection through one or more additional vehicle systems 10, or via wireless connection 64. The wireless connection can be any wireless connection protocol, including but not limited to Bluetooth.

[0038] At least one interior-facing camera 70 defines a field of view 72 including the vehicle operator and a wearable device 60 worn by the vehicle operator. In some instances, additional interior-facing cameras and other sensor types may be incorporated into the cabin 14 of the vehicle 10. The interior-facing camera 70, along with any other available interior sensors, is connected to the controller 30 and provides sensed information to the driver tracking module 42 of the controller 30. In this example, the wearable vision system includes positioning sensors and / or the ability to generate additional information, which is provided to the driver tracking module 42 but is not required for STVS operation. The driver tracking module 42 combines the interior cabin 14 sensed information to track the driver of the vehicle 10, specifically the driver's gaze or field of view direction and position.

[0039] Continue to refer to Figure 1 Vehicle 10, Figure 2 A three-dimensional model 16 of the vehicle 10 is shown, which is included in the memory of the controller 30 or another memory accessible by the controller 30. Figure 3 The image shows a wearable vision system 60 as a pair of smart glasses worn by the driver 66 when the driver 66 is looking at the rear 307 of the vehicle 10.

[0040] A 3D model 16 of vehicle 10 provides the relative positions and dimensions of the structural elements of vehicle 10. These structural elements define which parts of vehicle 10 can be seen through (e.g., windows) and which parts are opaque. By positioning the wearable vision system 60 within the 3D model 16 and using the driver monitoring features 42 of the vehicle portion 40 of the STVS architecture 300 and the SLAM module to determine the orientation of the wearable vision system 60, the controller 30 is able to identify which portions 304 of the external field of view will be observed, excluding the opaque portions of vehicle 10 between the wearable vision system and the outside. This portion 304 of the external field of view is then presented to the driver 66 using a screen, projection, or any form of presentation capability supported by the wearable vision system in the wearable vision system 60. Depending on the specifications of the wearable vision system 60 and the calculated opaque region of interest, the portion 304 of the external field of view may be superimposed on a full-screen image, or it may be the only portion of the screen that includes the image, while the rest of the screen is transparent. In another example, any alternative configuration for displaying the external field of view portion 304 to the driver 66 using the wearable vision system 60 while allowing the driver 66 to see the visible portion can be implemented by the controller 30.

[0041] Continue to refer to Figures 1 to 3 , Figure 4 An example architecture 300 for implementing an STVS system including vehicle portion 40 and wearable vision system portion 310 (collectively referred to as STVS portions 40, 310) is shown, where lines indicate data connections and arrows indicate the general expected direction of data flow. In some examples, data on all connections may be bidirectional, and the general flow direction is the direction of data flow across most of the data flow through said connections. STVS portions 40, 310 are interconnected via data link 306. Data link 306 may be wireless, wired, or a combination of wireless and wired. In some examples, the data link may also include indirect communication links via one or more common vehicle systems, including but not limited to connections via a common vehicle controller, vehicle communication bus, or any similar connection.

[0042] The vehicle portion 40 of architecture 300 includes onboard sensors 320 (e.g., an interior-facing camera 70), a central SLAM processing module 322, driving information 324 received from controller 30 and / or other vehicle systems, a 3D model 16, a dataset 326 identifying opaque and non-opaque portions of the vehicle 3D model 16, a dataset 328 identifying any user-set data, a dataset 330 identifying mandatory viewing areas of the vehicle 10 (e.g., blind spots during lane changes), a vehicle surround view system (SVS 332), and a blocked view output 334 configured to output portions of the external view blocked by the opaque portions of the vehicle 10.

[0043] Wearable vision system portion 310 includes a dataset 336 identifying user settings, a screen 338 or other image display (generally referred to as screen 338), and a received video stream 340 including data elements defining where the received video stream 340 should be positioned on the screen. These features 336, 338, and 340 cooperate to output live video 342 on screen 338. In some examples, wearable vision system portion 310 includes internal sensors 344 (e.g., position and orientation sensors) and / or image recognition software 346. In systems including image recognition software 346, the software 346 is configured to identify one or more objects in recorded images provided by a camera on wearable vision system 60.

[0044] During the operation of the STVS architecture 300, the central SLAM hub 322 receives data from the vehicle-mounted sensors 320 and any image recognition 346 performed by the wearable vision system 310 from any available internal sensors 344 of the wearable vision system 310. The SLAM hub 322 then uses the received information to perform a simultaneous localization and mapping process that identifies the position and orientation of the wearable vision system 60 within the vehicle 10. The position and orientation of the wearable vision system 60 are provided to the vehicle-mounted SVS 332. Additionally, the position and orientation are processed using a 3D model 16 and a dataset 326, which identifies the visible and invisible areas of the dataset 330. The dataset 330 is provided to the vehicle-mounted SVS 332. The vehicle-mounted SVS 332 also receives driving information 324, including vehicle speed, direction, and other detected driving characteristics.

[0045] Using the received data, the vehicle-mounted SVS 332 generates a 360-degree live video feed 333, and the live video feed 333 is annotated with driving information 335. The vehicle-mounted SVS 332 also generates a cropped live video 334, which is a feed positioned only to fill the opaque areas in the desired viewing area.

[0046] The cropped live video feed 334 is provided as a video stream 340 and is supplied to the wearable vision system display 338. The wearable vision system 60 then outputs the live video stream 342 on a corresponding screen and provides the driver 66 with a field of view 304 of the external environment via the wearable vision device 60. Figure 3 ).

[0047] By performing the calculation of the projected area requirement around vehicle 10 within the vehicle rather than within the wearable vision system, the STVS architecture 300 reduces the computational requirements of the wearable vision system, thereby improving performance and increasing compatibility with a wider range of wearable vision systems. Furthermore, some implementations support adding video annotations calculated by the vehicle's surround-view video system 332 without increasing the computational requirements of the wearable vision system.

[0048] Using this architecture 300, the vehicle 10 becomes an augmented reality processing device, performing the different computations required by standard augmented reality devices at the controller 30 while using the wearable vision system 60 as a display.

[0049] In some examples, the in-vehicle SVS 332 can also provide image correction and alignment adjustments to the video stream generated by the imaging sensor 20, so that the video stream captured by the imaging sensor 20 matches and aligns with the actual surrounding environment of the vehicle 10. This operation aligns the images from the imaging sensor 20, resulting in seamless image stitching. In a further embodiment, when the wearable vision system 310 includes processing hardware and software capable of performing device-specific image corrections, an additional level of image correction can be performed on the cropped live video in the wearable vision system 310.

[0050] The term “a” does not indicate a limitation of quantity, but rather that at least one of the referenced items is present. Unless the context clearly indicates otherwise, the term “or” means “and / or”. Throughout the specification, reference to “aspect” means that a particular element described in connection with that aspect (e.g., a feature, structure, step, or characteristic) is included in at least one aspect described herein and may or may not be present in other aspects. Furthermore, it should be understood that the described elements may be combined in any suitable manner in the aspects.

[0051] When a component, such as a layer, film, region, or substrate, is referred to as being “on” another component, it can be directly on the other component, or there may be intermediate components. Conversely, when a component is referred to as being “directly” on another component, there are no intermediate components.

[0052] Unless otherwise stated herein, all test standards are the most recent standards effective up to the filing date of this application, or, if priority is claimed, the most recent standards effective up to the filing date of the earliest priority application in which the test standards appear.

[0053] Unless otherwise defined, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0054] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A perspective vision system for vehicles, comprising: A surround view system comprising a plurality of outward-facing cameras, each of which communicates with a controller; A driver monitoring system, the driver monitoring system including at least one driver monitoring sensor configured to monitor the driver's direction of observation, the at least one driver monitoring sensor communicating with the controller; The controller includes a non-transitory memory for storing a 3D model of the vehicle, a simultaneous localization and mapping (SMR) module, an surround view module, and a data link for connecting to a wearable vision system; and The controller also stores instructions for the following operations: using the driver monitoring system to identify the driver's viewing direction, using the viewing direction and the vehicle's 3D model to identify at least one opaque part of the vehicle in the driver's field of vision, using the surround view module to generate a cropped live video of the vehicle's external view at the location of at least one opaque part, and using a data link to export the generated cropped live video to the wearable vision system.

2. The vehicle according to claim 1, wherein, The at least one driver monitoring sensor includes an inward-facing camera that defines a field of view including the driver's head.

3. The vehicle according to claim 1, wherein, The simultaneous localization and mapping (SLT) module is configured to use a SLT algorithm to determine the position and orientation of the wearable vision system relative to the vehicle, at least in part, based on the driver’s viewing direction.

4. The vehicle according to claim 3, wherein, The simultaneous localization and mapping (SMR) module is further configured to receive at least one of sensor information and image information from the wearable vision system via the data link, and the SMR module is configured to determine the position and orientation of the wearable vision system relative to the vehicle based at least in part on the driver’s viewing direction and at least one of the sensor information and image information from the wearable vision system.

5. The vehicle according to claim 4, wherein, The image information includes images captured from a camera mounted on the wearable vision system.

6. The vehicle according to claim 5, wherein, The image information also includes the object identifier of the object in the image feed.

7. The vehicle according to claim 6, wherein, The simultaneous localization and mapping module is further configured to determine the position and orientation of the wearable vision system relative to the vehicle based at least in part on the identification of objects in the image feed.

8. The vehicle of claim 3, wherein the simultaneous localization and mapping algorithm is executed specifically on the controller.

9. The vehicle of claim 1, wherein the three-dimensional model of the vehicle defines the relative positions and dimensions of the structural elements of the vehicle, and defines which structural elements of the vehicle are opaque.

10. The vehicle of claim 1, wherein the vehicle further comprises a set of vehicle operation sensors configured to identify vehicle speed, vehicle position, hazard detection, warning signals, driving instructions, and driving direction, and wherein the cropped live video of the external view of the vehicle generated at the location of the at least one opaque portion using the surround view module includes annotations of the cropped live view with the vehicle speed, pedestrian detection information, and at least one of hazard detection, warning signals, and driving instructions.

Images