Semantic Segmentation Using Driver Attention Information

Abstract

A method for creating a trained semantic segmentation network model and operating a vehicle using the model. An example method includes: an external vision camera configured to capture an image representing an artificial representation of the driver's view, a driver-facing camera configured to capture the driver's eye movements, and an electronic controller. The electronic controller is configured to: receive the image from the camera; calibrate the image of the driver's eye movements using the artificial driver view; create a pixel-weighted heatmap of the calibrated image; create a trained semantic segmentation neural network model and a trained attention neural network model using the pixel-weighted heatmap and the artificial driver view; and operate the vehicle using the trained semantic segmentation neural network model and the trained attention neural network model.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 786,711, filed on December 31, 2018, the entire content of which is hereby incorporated by reference in its entirety. Technical field

[0003] Embodiments relate, among other things, to obtaining training data for an attention neural network model and using the trained model for autonomous driving functions. Background art

[0004] Modern vehicles include autonomous or semi - autonomous driving functions that utilize semantic segmentation network models. These segmentation network models are trained, for example, with respect to object identification and labels assigned to each pixel within a defined object. Summary of the invention

[0005] The importance of pixels is ignored during the training of segmentation network models, and each pixel is treated equally. In fact, pixels have different relevance, and some pixels should be considered more important than others. Although segmentation network models are known, training a segmentation network model in combination with the use of weighted pixels to represent different importance is not available, or not implemented in the context of the use of driver eye tracking.

[0006] Embodiments described herein provide, among other things, a system and method for using a semantic segmentation model that is trained with acquired data to incorporate importance - based pixel weighting.

[0007] One embodiment provides a system for creating a trained semantic segmentation neural network model and a trained attention neural network model to operate a vehicle. The system includes: an outside view camera, a driver - facing camera, and one or more electronic controllers. The one or more electronic controllers are configured to: receive images from the outside view camera and receive images of driver eye movements from the driver - facing camera. The one or more electronic controllers are configured to: use the images from the outside view camera to calibrate the images of driver eye movements to create a calibration image, and the calibration image is used to create a pixel - weighted heatmap for the calibration image. The one or more electronic controllers are further configured to: use the images from the outside view camera to create a trained semantic segmentation neural network model and use the pixel - weighted heatmap to create a trained attention neural network model. The one or more electronic controllers are configured to: use the trained semantic segmentation neural network model and the trained attention neural network model to operate the vehicle.

[0008] Another embodiment provides a method for creating a trained semantic segmentation neural network model and a trained attention neural network model to operate a vehicle. The method includes: receiving, via one or more electronic controllers, an image from an external vision camera and an image of a driver's eye movement from a driver-facing camera. The method further includes: calibrating, via the one or more electronic controllers, the image of the driver's eye movement using the image from the external vision camera to create a calibrated image for creating a pixel-weighted heatmap of the calibrated image. The method further includes: creating, via the one or more electronic controllers, a trained semantic segmentation neural network model using the image from the external vision camera, and creating a trained attention neural network model using the pixel-weighted heatmap. The method further includes: operating the vehicle using the trained semantic segmentation neural network model and the trained attention neural network model via the one or more electronic controllers.

[0009] Other aspects, features, and embodiments will become apparent by considering the detailed description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of a system for improving the acquisition of training data for a semantic segmentation model according to one embodiment.

[0011] Figure 2 is according to one embodiment Figure 1 of a first electronic controller of the system.

[0012] Figure 3 is according to one embodiment Figure 1 of a second electronic controller of the system.

[0013] Figure 4 is according to one embodiment Figure 1 of an external vision camera of the system.

[0014] Figure 5 is according to one embodiment Figure 1 of a driver-facing camera of the system.

[0015] Figure 6 is according to one embodiment of using Figure 2 an image calibration engine of an electronic controller to receive inputs from Figure 1 and Figure 2 cameras and create a pixel-weighted heatmap based on the inputs.

[0016] Figure 7 is according to one embodiment of using Figure 3Flowchart of a method for a training apparatus of an attention neural network model of an electronic controller to merge a pixel-weighted heatmap and an exteroceptive image using a cost function used in training the attention neural network model.

[0017] Figure 8 is according to one embodiment using Figure 3 Flowchart of a method for a training apparatus of a semantic segmentation neural network model of an electronic controller to merge an exteroceptive image during training of the semantic segmentation neural network model.

[0018] Figure 9 Block diagram of a system for operating an autonomous vehicle using a semantic segmentation classifier according to one embodiment.

[0019] Figure 10 is using Figure 9 Flowchart of a method for a vehicle operation engine of a third electronic controller to receive an exteroceptive image from a second exteroceptive camera and process the image through a trained model to operate the vehicle.

[0020] Figure 11 is in Figure 6 Conceptual diagram of a labeled heatmap created in the method for use in training an attention neural network model. Detailed Description

[0021] Before explaining any embodiments in detail, it is to be understood that the present disclosure is not intended in its application to be limited to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. Embodiments can have other configurations and can be practiced or carried out in various ways.

[0022] Various embodiments can be implemented using a plurality of hardware- and software-based devices, as well as a plurality of different structural components. Additionally, embodiments can include hardware, software, and electronic components or modules which, for discussion purposes, can be illustrated and described as if most components were implemented only in hardware. However, one of ordinary skill in the art and upon reading of this detailed description will recognize that, in at least one embodiment, the electronic-based aspects of the present invention can be implemented in software (e.g., software stored on a non-transitory computer-readable medium) executable by one or more processors. For example, the "control units" and "controllers" described in the specification can include one or more electronic processors, one or more memory modules including a non-transitory computer-readable medium, one or more communication interfaces, one or more application-specific integrated circuits (ASICs), and various connections (e.g., a system bus or one or more networks) connecting the various components.

[0023] Figure 1FIG. illustrates a vehicle system 100 for calibrating an image outside the vehicle using captured driver eye movements for training an attention neural network model. The system 100 includes a vehicle 105. Although the vehicle 105 is illustrated as a four-wheel vehicle, it can cover various types and designs of vehicles. For example, the vehicle 105 can be a car, motorcycle, truck, bus, semi-trailer truck, and others. In the illustrated example, the vehicle 105 includes a number of hardware components, including a first electronic controller 110, a first exterior camera 115, and a driver-facing camera 120. The first electronic controller 110 is communicatively connected to the first exterior camera 115, the driver-facing camera 120, and a second electronic controller 125 using one or more connections (e.g., a direct link and network connections (e.g., Controller Area Network or CAN bus)). Wired and wireless connections are possible.

[0024] The first exterior camera 115 is configured to capture an image outside the vehicle. The driver-facing camera 120 is configured to capture driver eye movements. The first exterior camera 115 and the driver-facing camera 120 provide data (images) to the first electronic controller 110 of the system 100.

[0025] In the illustrated example, the second electronic controller 125 is located outside the vehicle 105 and is communicatively connected to the first electronic controller 110 via one or more communication links and in this example via a wireless connection. However, the second electronic controller 125 can be located within the vehicle system 100 and connected via a direct connection or via a vehicle network or bus.

[0026] Figure 2 is Figure 1 A block diagram of the first electronic controller 110 of the system. The first electronic controller 110 includes a number of electrical and electronic components that provide power, operation control, and protection to the components and modules within the first electronic controller 110. The first electronic controller 110 includes, among other things, a first electronic processor 205 (e.g., a programmable electronic microprocessor, microcontroller, or similar device) and a first memory 210. The first memory 210 is, for example, a non-transitory machine-readable memory. The first electronic controller also includes a first communication interface 215.

[0027] The first electronic processor 205 is communicatively coupled to a first memory 210 and a first communication interface 215. The first memory 210 includes an image calibration engine 220. The image calibration engine 220 is, for example, software or a set of computer-readable instructions that calibrates inputs from the first exterior camera 115 and the driver-facing camera 120. After the inputs are calibrated, the first electronic controller 110 is configured to provide the calibrated inputs and the inputs from the first exterior camera 115 to the second electronic controller 125. In other embodiments, the image calibration engine 220 may be located within the second electronic controller 125. In embodiments where the image calibration engine 220 is located within the second electronic controller 125, the first electronic controller 110 provides the inputs from the first exterior image 115 and the driver-facing camera 120 directly to the second electronic controller 125. The first electronic processor 205, in cooperation with software stored in the first memory 210 (e.g., the software described above), and the first communication interface 215 are configured to implement one or more of the methods described herein.

[0028] The first electronic controller 110 may be implemented in a number of separate controllers (e.g., programmable electronic controllers), each configured to perform a specific function or sub-function. Additionally, the first electronic controller 110 may include sub-modules that include additional electronic processors, memories, or application specific integrated circuits (ASICs) for handling communication functions, signal processing, and applications of the methods listed below. In other embodiments, the first electronic controller 110 includes additional, fewer, or different components.

[0029] Figure 3 is a block diagram of a second electronic controller 125 of the vehicle 105. The second electronic controller 125 is generally similar to the first electronic controller 110. Accordingly, all details of its architecture and connections to other components will not be described. The second electronic controller 125 includes, among other things, a second electronic processor 305 (e.g., a programmable electronic microprocessor, microcontroller, or similar device) and a second memory 310. The second memory 310 is, for example, a non-transitory machine-readable memory. The second electronic controller 125 also includes a second communication interface 315.

[0030] The second electronic processor 305 is communicatively connected to a second memory 310 and a second communication interface 315. The second memory 310 includes an attention neural network model trainer 320 and a semantic segmentation neural network model trainer 325. The attention neural network model trainer 320 is, for example, software or a set of computer-readable instructions that receives inputs from the first electronic controller 110 and uses the received inputs to train an attention neural network model to create a trained attention neural network model. The semantic segmentation network model trainer 325 is, for example, software or a set of computer-readable instructions that receives inputs from the first electronic controller 110 and uses the received inputs to train a semantic segmentation neural network model to create a trained semantic segmentation neural network model. In the illustrated example, the inputs are from the first electronic controller 110 located in the vehicle 105. However, in other embodiments, multiple vehicles may send inputs to the second electronic controller 125.

[0031] Like the first electronic controller 110, the second electronic controller 125 may be implemented in a number of separate controllers, each configured to perform a specific function or sub-function. Additionally, the second electronic controller 125 may include sub-modules that include additional electronic processors, memories, or application specific integrated circuits (ASICs) for handling communication functions, signal processing, and application of the methods listed below. Further, the second electronic controller 125 includes additional, fewer, or different components than those shown.

[0032] Figure 4 is a block diagram of the first exterior vision camera 115 of the vehicle 105. The first exterior vision camera 115 includes, among other things, a third communication interface 405, a first image signal processor 410, and a first lens (len) and image sensor assembly 415. In the system 100, the camera is a front facing camera that is configured in such a way that the provided images are similar to the images of a driver's view through the windshield outside of the vehicle 105.

[0033] Figure 5 is a block diagram of the driver-facing camera 120 of the vehicle 105. The driver-facing camera 120 includes, among other things, a fourth communication interface 505, a second image signal processor 510, and a second lens and image sensor assembly 515. As the name implies, the driver-facing camera 120 faces the driver and is configured to track the driver's eye movements. As will be explained in more detail, the tracked eye movements are used to calibrate the images from the first exterior vision camera 115.

[0034] The images captured by the first exterior vision camera 115 and the driver-facing camera 120 are provided to the image calibration engine 220.Figure 6 Illustrated is an example method 600 of an image calibration engine 220. The image calibration engine 220 controls the calibration of an image and creates a calibrated heatmap. Although currently only using labels that classify objects within an image and predictions of which label each pixel is associated with to train a semantic segmentation network model, doing so creates a semantic segmentation model in which each pixel or group of pixels within an object has equal relevance. However, the human mind does not process each object in the field of view with equal relevance. Therefore, it is beneficial to observe driver eye movements relative to the image captured by the first exterior vision camera 115, and a second level of relevance or interest information is provided for pixels within an object contained within the image captured by the first exterior vision camera 115.

[0035] At step 605, a first electronic processor 205 receives a first image from the first exterior vision camera 115. The first image serves as an artificial representation of the driver's view. At step 610, a second image of the driver's eye movement is received. Next, calibration is performed to determine which pixels correspond to the driver's eye movement (step 615). The calibration of pixels to eye movement is achieved through pixel weighting based on the driver's focus of gaze (i.e., where the driver is looking). Known techniques for determining a person's focus of gaze can be used in the described embodiments. Regarding weighting, in an example, if the driver is looking at a pedestrian crossing the road, the pedestrian is weighted more heavily than a bird flying in the sky. In one instance, based on an image of the driver's eye movement, a weight is assigned to each pixel in the image from the exterior vision camera. Then, the calibrated image is processed by the first electronic processor 205 to create a pixel-weighted heatmap (step 620). In one example, the pixel-weighted heatmap indicates the weight by the darkness of the color. The darker the color on the heatmap, the heavier the weight of the corresponding pixel. In other embodiments, the weighting of pixels is represented in a different form. For example, one form can be the number location of the pixel with the corresponding weight value. Once the image is calibrated, a representation of the calibrated image (e.g., a heatmap) is provided to the second electronic controller 125 (step 625). In one embodiment, the representation of the calibrated image can be provided to the second electronic controller 125 after it is created. However, in other embodiments, the calibrated image can be stored in a memory storage location and transferred to the second electronic controller 125 at a later time.

[0036] Figure 7Illustrates an example method 700 of an attention neural network model trainer 320. At step 705, the attention neural network model trainer 320 receives a pixel-weighted heatmap and an exteroceptive image captured by the first exteroceptive camera 115. As noted, currently labels are used to classify pixels in an object group and the pixel weights are not considered to train a semantic segmentation model. In an example embodiment, the pixel-weighted heatmap is used as a classification label and the exteroceptive image is used as an input. Training can be performed by using a cross-entropy cost function. The second electronic processor 305 trains the model using a cost function that incorporates the pixel weights represented by the pixel-weighted heatmap (step 710).

[0037] Figure 8 Illustrates an example method 800 of a semantic segmentation neural network model trainer 325. At step 805, the semantic segmentation neural network model trainer 325 receives an exteroceptive image from the first exteroceptive camera 115. The second electronic processor 305 trains the model using a cross-entropy cost function that incorporates the exteroceptive image (step 810). Although a cross-entropy cost function is used in one example, in other embodiments, different cost functions can be used. The trained semantic segmentation neural network model provides a mechanism for analyzing an image and classifying segmentation features in the image.

[0038] After a trained semantic segmentation neural network model and a trained attention neural network model have been created, a vehicle with autonomous driving capabilities can incorporate these models into autonomous and semi-autonomous driving systems and can operate the vehicle based on these models. Figure 9 Illustrates an automatic vehicle system 900 for operating an autonomous vehicle. In the illustrated example, the automatic vehicle system 900 includes an architecture similar to that of Figure 1 vehicle 105. However, the automatic vehicle system 900 does not have a driver-facing camera. The hardware components within the automatic vehicle system 900 include a third electronic controller 905 and a second exteroceptive camera 910. The third electronic controller 905 is communicatively coupled to the second exteroceptive camera 910.

[0039] The second external vision camera 910 is configured to capture images of the exterior of the autonomous vehicle. The second external vision camera 910 provides data (images) to the third electronic controller 905 of the system 900. The third electronic controller 905 includes, among other things, a third electronic processor 915 (e.g., a programmable electronic microprocessor, microcontroller, or similar device) and a third memory 920. The communication connection between the third electronic processor 915 and the third memory 920 is similar to the connection between the electronic processor and the memory described above. The third memory 920 is, for example, a non-transitory machine-readable memory. The third memory 920 includes a vehicle operation engine 930. The vehicle operation engine 930 is, for example, software or a set of computer-readable instructions that processes the input from the second external vision camera 910 and provides the generated image features to the autonomous and semi-autonomous driving systems. The third electronic processor 915, in cooperation with the software stored in the third memory 920 (e.g., the software described above), is configured to implement one or more of the methods described herein.

[0040] Figure 10 An example method 1000 of the vehicle operation engine 930 is illustrated, where the vehicle operation engine 930 controls autonomous and semi-autonomous driving systems including so-called Advanced Driver Assistance Systems (ADAS). Examples of ADAS include lane detection and lane keeping systems, forward collision warning and mitigation systems, object detection systems, and free space detection systems. At step 1005, an external vision image captured by the second external vision camera 910 is received from the third electronic controller 905. Then, the external vision image is processed by a trained semantic segmentation neural network model trained in method 800. Processing the external vision image using the semantic segmentation neural network model extracts segmentation features (step 1010). In one example, the extraction of the segmentation features identifies the presence of the vehicle. The external vision image is also processed by a trained attention neural network model trained in method 700. A pixel-weighted heatmap is created from processing the external vision image using the attention neural network model (step 1015). The pixel-weighted heatmap represents a prediction made by the attention neural network model based on training from driver eye movement data, which indicates the importance of the pixels in the image. After the pixel-weighted heatmap and the segmentation features have been obtained, these features and the heatmap are concatenated (step 1020). Next, the concatenated heatmap and segmentation features are processed using a modified cost function of a segmentation classifier (step 1025). In some embodiments, the cost function of the segmentation classifier can be modified to accept these two inputs without concatenating the inputs. The segmentation classifier uses a cross-entropy cost function that incorporates the pixel weighting represented by the pixel-weighted heatmap and the segmentation features. An example cross-entropy cost function is as follows:

[0041] (Equation 1)

[0042] Using a pixel-weighted heatmap, Equation 1 is changed as follows:

[0043]

[0044] Here, w pixel is the weight associated with each pixel, which is determined based on the weighting performed as part of the association of the gaze information from the driver-facing camera 120 with the image from the first external camera 115 determined by the calibration engine 220 in method 600. Next, the vehicle is operated according to the processing of the concatenated segmentation features and the pixel-weighted heatmap using the segmentation classifier.

[0045] Although a cross-entropy cost function is used in one example, in other embodiments, different cost functions can be used and changed to incorporate the pixel weighting determined by the image calibration engine 220.

[0046] As previously described, the pixel weighting determined by image calibration is presented as a pixel-weighted heatmap. Figure 11 is a view of a pixel-weighted heatmap 1100 according to one embodiment. In the example provided, the degree of shading of the region indicates the weight of the pixels in that region. For example, the non-shaded region 1105 has a very low weight or no weight, while the darkest shaded region 1110 has the highest weight. In the illustrated example, pixels are grouped into regions of the assigned weight. However, the pixels in the group may have actual weights within the tolerance of the assigned weight (e.g., within 1% of the assigned weight). In other embodiments, there are no regions, and the weighting of the pixels is the actual value received from calibrating the driver's eye movement using the exteroceptive image.

[0047] By adding weights to the pixels, autonomous and semi-autonomous driving functions are improved, for example, by ignoring irrelevant objects and acting based on higher-correlation objects (e.g., braking, steering, or accelerating).

[0048] Although a specific order of steps is indicated in the methods illustrated in Figures 6 - 8 and Figure 10 , the timing, sequence, and inclusion of steps can vary, where appropriate, without departing from the purpose and advantages of the provided examples.

[0049] Accordingly, the embodiments described herein provide, among other things, a system and method for capturing an image and calibrating the image to create a representation of a calibrated image, the representation of the calibrated image indicating correlation-based pixel weighting. The various features and advantages of the embodiments are set forth in the following claims.

Claims

1. A system for creating a trained semantic segmentation neural network model and a trained attention neural network model to operate a vehicle, the system comprising: An external vision camera; A driver-facing camera; One or more electronic controllers configured to: Receive an image from the external vision camera; Receive an image of the driver's eye movement from the driver-facing camera; Calibrate the image of the driver's eye movement using the image from the external vision camera by assigning a weight to each pixel in the image from the external vision camera based on the image of the driver's eye movement to create a calibrated image; Create a pixel-weighted heatmap of the calibrated image; Create a trained semantic segmentation neural network model using the image from the external vision camera; Create a trained attention neural network model using the pixel-weighted heatmap and the image from the external vision camera; And Operate the vehicle using the trained semantic segmentation neural network model and the trained attention neural network model.

2. The system according to claim 1, wherein A first electronic controller receives an image from the external vision camera; receives the image of the driver's eye movement from the driver-facing camera; calibrates the image of the driver's eye movement using the image from the external vision camera to create a calibrated image; and creates a pixel-weighted heatmap of the calibrated image, and wherein a second electronic controller receives the pixel-weighted heatmap of the calibrated image and the image from the external vision camera.

3. The system according to claim 2, wherein, The second electronic controller is configured to: create a trained semantic segmentation neural network model and a trained attention neural network model using a cost function that incorporates the pixel-weighted heatmap.

4. The system according to claim 3, wherein, The second electronic controller is further configured to: train a semantic segmentation classifier using a cost function that incorporates the pixel-weighted heatmap.

5. The system according to claim 2, wherein The second electronic controller is configured to: receive the pixel-weighted heatmap from multiple vehicles.

6. The system according to claim 1, wherein, The external vision camera is a forward-facing camera.

7. The system according to claim 1, wherein Creating a trained attention neural network model includes: using the pixel-weighted heatmap as a classification label and using the image from the external vision camera as an input.

8. A method for creating a trained semantic segmentation neural network model and a trained attention neural network model to operate a vehicle, the method comprising: Receiving an image from an external vision camera via one or more electronic controllers; Receiving an image of the driver's eye movement from a driver-facing camera via the one or more electronic controllers; Calibrating the image of the driver's eye movement using the image from the external vision camera to create a calibrated image by assigning a weight to each pixel in the image from the external vision camera based on the image of the driver's eye movement via the one or more electronic controllers; Creating a pixel-weighted heatmap of the calibrated image via the one or more electronic controllers; Creating a trained semantic segmentation neural network model using the image from the external vision camera via the one or more electronic controllers; Create a trained attention neural network model via the one or more electronic controllers, using the pixel-weighted heatmap and the image from the external vision camera; and Operate a vehicle via the one or more electronic controllers, using the trained semantic segmentation neural network model and the trained attention neural network model.

9. The method according to claim 8, wherein Receive an image from the external vision camera; Receive an image of the driver's eye movement from the driver-facing camera; Calibrate the image of the driver's eye movement using the image from the external vision camera to create a calibrated image; and Creating the pixel-weighted heatmap of the calibrated image is performed via a first electronic controller, and wherein the method further includes providing the pixel-weighted heatmap of the calibrated image to a second electronic controller via the first electronic controller.

10. The method according to claim 9, wherein Operating the vehicle via the one or more electronic controllers, using the trained semantic segmentation neural network model and the trained attention neural network model is performed via the second electronic controller.

11. The method according to claim 10, further comprising: Train a semantic segmentation classifier via the second electronic controller, using a cost function that incorporates the pixel-weighted heatmap.

12. The method according to claim 9, further comprising: Receive from a plurality of vehicles that provide pixel-weighted heatmaps.

13. The method according to claim 8, wherein, Creating the trained attention neural network model includes: using the pixel-weighted heatmap as a classification label and using the image from the external vision camera as an input.

14. A system for operating a vehicle using a trained semantic segmentation neural network model and a trained attention neural network model created using the method according to any one of claims 8-13, the system including: An external vision camera; One or more electronic controllers configured to: Receive an image from the external vision camera; Process the image from the external vision camera using the trained semantic segmentation neural network model to extract segmentation features; Process the image from the external vision camera using the trained attention neural network model to create a pixel-weighted heatmap; Concatenate the segmentation features with the pixel-weighted heatmap; Process the concatenated segmentation features and pixel-weighted heatmap using a segmentation classifier; and Operate the vehicle based on the processing of the concatenated segmentation features and pixel-weighted heatmap using the segmentation classifier.

15. The system according to claim 14, wherein Processing the concatenated segmentation features and pixel-weighted heatmap using the segmentation classifier includes: using a modified cost function that incorporates the pixel-weighted heatmap.

16. A computer program product having instructions that, when executed, cause a computing device to perform the method according to any one of claims 8-13.

Images