Information processing device, information processing system, and information processing method

Abstract

To improve stability of control of multiple devices.SOLUTION: An information processor includes: a first OS that controls the execution of a first application that controls the operation of a first device installed in a vehicle; a second OS that controls the execution of a second application that controls the operation of a second device installed in the vehicle and whose safety requirement level is higher than the safety requirement level of the first device; and a hypervisor 70 that runs on a processor and controls the execution of the first OS and the second OS.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] Patent Document 1 discloses a technology that selects the display device that the driver is looking at from among multiple display devices installed in a vehicle, and displays information necessary for driving on the screen of the selected display device.

[0003] The display control device disclosed in Patent Document 1 includes a gaze detection unit that detects the direction in which the driver is looking based on an image of the driver's face captured by a capture unit, and a selection unit that selects, from among a plurality of display devices, the display device at which the driver is looking based on gaze information indicating the direction in which the gaze is directed, as the display device on which to display information necessary for driving.

[0004] According to this display control device, the information necessary for driving is displayed on the display device toward which the driver is directing his or her gaze, allowing the driver to check the information necessary for driving while grasping the situation around the vehicle toward which the driver is directing his or her gaze. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-249567 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the conventional technology disclosed in Patent Document 1, when a first application that controls a speedometer or the like, which has a high level of safety requirements, and a second application that controls a device such as a navigation device or gaze detection unit, which has a lower level of safety requirements than the display device, run on the same OS (Operating System), a failure in the second application may cause a failure in the OS, which may result in unstable operation of the first application.

[0007] Non-limiting embodiments of the present disclosure contribute to providing an information processing device, an information processing system, and an information processing method that can improve the stability of the operations of multiple applications. [Means for solving the problem]

[0008] An information processing device according to one embodiment of the present disclosure includes a first OS that controls the execution of a first application that governs the operation of a first device mounted in a vehicle, a second OS that controls the execution of a second application that governs the operation of a second device mounted in the vehicle and having a safety requirement level higher than the safety requirement level of the first device, and a hypervisor that runs on a processor and controls the execution of the first OS and the second OS.

[0009] An information processing system according to an embodiment of the present disclosure includes the above-described information processing device, a first device, and a second device.

[0010] An information processing method according to one embodiment of the present disclosure includes a step of a first OS controlling the execution of a first application that controls the operation of a first device mounted in a vehicle, a step of a second OS controlling the execution of a second application that controls the operation of a second device mounted in the vehicle and having a safety requirement level higher than the safety requirement level of the first device, and a step of a hypervisor running on a processor controlling the execution of the first OS and the second OS. Effect of the Invention

[0011] According to an embodiment of the present disclosure, it is possible to construct an information processing device, an information processing system, and an information processing method that can improve the stability of the operations of multiple applications.

[0012] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings, in which such advantages and / or benefits are provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of which are provided to obtain one or more identical features. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing a configuration example of an information processing system 100 according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a diagram for explaining a configuration example of an information processing system 100 configured by virtualizing multiple different guest OSs on hardware. [Diagram 3] A diagram showing an example of building a hypervisor layer on the hardware and building two guest OS layers on top of that. [Figure 4] A diagram showing an example of building a host OS on hardware and then building a hypervisor on top of that [Diagram 5] 1 is a flowchart for explaining the operation of the information processing system 100 according to an embodiment of the present disclosure. [Figure 6A] Flowchart for explaining display position selection processing [Figure 6B] Flowchart for explaining display position selection processing [Figure 7] A sequence chart for explaining the operation of the information processing system 100 when no failure occurs in the guest OS 101 of the gaze detection device 10 or the like. [Figure 8] FIG. 13 is a diagram for explaining the operation of the information processing system 100 when a failure occurs in the guest OS 30A1 of the display device 30A. [Figure 9] 9 is a sequence chart illustrating the operation in FIG. 8 . [Figure 10] FIG. 13 is a diagram for explaining the operation of the information processing system 100 when a failure occurs in the guest OS 30A1 of the display device 30A. [Figure 11] 11 is a sequence chart illustrating the operation in FIG. 10 . [Figure 12] 1 is a sequence chart for explaining the operation of the information processing system 100 when a failure occurs in the guest OS 101 of the gaze detection device 10. [Figure 13] 1 is a sequence chart for explaining an example of the operation of the information processing system 100 when the hypervisor 70 monitors each guest OS. [Figure 14] 1 is a sequence chart illustrating an example of the operation of the information processing system 100 when the monitoring guest OS 90 monitors the state of each guest OS. [Figure 15] 1 is a sequence chart showing an example of an operation when each guest OS monitors the status of the other guest OS in the information processing system 100. [Figure 16] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing system 100 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functions are denoted by the same reference numerals, and redundant description will be omitted.

[0015] (Embodiment) First, the background that led to the creation of the embodiments of the present disclosure will be described.

[0016] For example, in the known technology described in Patent Document 1, when a first application that controls a speedometer or the like, which has a high level of safety requirements, and a second application that controls a device such as a navigation device or gaze detection unit, which has a lower level of safety requirements than the display device, run on the same OS (Operating System), a failure in the second application may cause a failure in the OS, which may result in unstable operation of the first application.

[0017] For example, if a failure occurs in an application that controls a gaze detection device with a low safety requirement level, making it impossible to control the gaze detection device, the failure may spread to other applications, causing them to also lose control of a speedometer, which has a high safety requirement level.

[0018] Therefore, it is desirable to improve the stability of the operations of a plurality of applications.

[0019] Fig. 1 is a diagram illustrating an example of a configuration of an information processing system 100 according to an embodiment of the present disclosure. A vehicle 200 illustrated in Fig. 1 includes the information processing system 100, a gaze detection device 10 that sequentially detects the gaze of an occupant of the vehicle 200, and a surroundings monitoring device 20 that monitors objects present around the vehicle.

[0020] Each of the gaze detection device 10 and the surroundings monitoring device 20 is an example of an information processing device according to the present disclosure.

[0021] The occupants include the driver and passengers of the vehicle 200. The vehicle 200 is a passenger car, a freight car, a passenger van, etc. In the embodiment of the present disclosure, a configuration example of an information processing system 100 that detects the line of sight of the driver of the vehicle 200 will be described.

[0022] (Information processing system 100) The information processing system 100 includes a plurality of display devices 30A to 30C and a display control device 50.

[0023] Hereinafter, when there is no need to distinguish between the multiple display devices 30A to 30C, they will be referred to as "display device 30."

[0024] Each of the display device 30 and the display control device 50 is an example of an information processing device according to the present disclosure.

[0025] (Gaze detection device 10) The gaze detection device 10 is a device that detects the gaze direction of a driver of the vehicle 200 and outputs gaze information indicating the direction in which the driver's gaze is directed. The gaze detection device 10 is an example of a first device according to the present disclosure.

[0026] The gaze detection device 10 includes an image capturing unit 11, an image processing unit 12, a gaze detection unit 13, and a gaze information output unit .

[0027] (Image capture section 11) The image capturing unit 11 is a camera that captures images of the inside of the vehicle, etc. The image capturing unit 11 includes an image capturing element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

[0028] The image capturing unit 11 is provided at a position where it can capture an image of the driver's face, captures an image including the driver's face, and outputs data of the captured image (hereinafter referred to as image data) to the image processing unit 12 successively.

[0029] Positions where the driver's face can be photographed include, for example, the dashboard of the vehicle 200, the instrument panel at the driver's seat, the ceiling of the vehicle 200, and the like.

[0030] The number of image capturing units 11 is not limited to one, and a plurality of image capturing units may be provided in the vehicle.

[0031] (Image processing unit 12) When the image data is input, the image processing unit 12 obtains a binary image by binarizing the image captured by the image capturing unit 11. The image processing unit 12 inputs the binary image data (hereinafter referred to as binary image data) to the gaze detection unit 13.

[0032] (Gaze detection unit 13) The gaze detection unit 13 detects the gaze based on the binary image data. For example, the gaze detection unit 13 identifies the eyeball area of ​​the driver by analyzing the binary image data, and obtains the eyeball center coordinates from the curvature of the eyeball shape.

[0033] Furthermore, the gaze detection unit 13 identifies the iris (pupil) region from the eyeball region to obtain the iris center coordinates, and determines the direction from the eyeball center coordinates to the iris center coordinates as the gaze direction.

[0034] Furthermore, gaze detection unit 13 specifies a straight line extending from the starting point in the gaze direction, starting from the center position of the eyeball, as the gaze, and transmits gaze information indicating the specified gaze to gaze information output unit 14.

[0035] (Gaze information output unit 14) The line-of-sight information output unit 14 outputs the line-of-sight information transmitted from the line-of-sight information output unit 14 to the information input unit 51 of the display control device 50. The configuration of the information input unit 51 will be described in detail later.

[0036] The image processing unit 12, the gaze detection unit 13, and the gaze information output unit 14 are an example of a first application according to the present disclosure.

[0037] (Periphery monitoring device 20) The periphery monitoring device 20 is a device for detecting objects existing around the vehicle, and is an example of a first device according to the present disclosure.

[0038] The surroundings monitoring device 20 includes a surroundings image capturing unit 21, an image processing unit 22, an object detection sensor 23, an object detection unit 25, and a detection information output unit .

[0039] (Peripheral image capture unit 21) The surrounding image capturing unit 21 is an omnidirectional camera, a panoramic camera, or the like, that captures images of the surroundings of the vehicle. Hereinafter, the images of the surroundings of the vehicle will be referred to as "surrounding images." The surrounding image capturing unit 21, like the image capturing unit 11 of the gaze detection device 10, includes an imaging element such as a CCD or a CMOS.

[0040] The surrounding images are, for example, images of the scenery in front of the vehicle 200, the scenery on the sides of the vehicle 200, the scenery behind the vehicle 200, etc. The scenery includes, for example, objects present on the roadway on which the vehicle 200 is traveling. The objects present on the roadway include, for example, automobiles, buildings, structures (advertisements, road signs, traffic lights, telegraph poles, etc.), people, animals, etc.

[0041] The surrounding image photographing unit 21 is disposed in a position where it can photograph the scenery outside the vehicle, for example. Such positions include the front grill, side mirrors, ceiling, rear bumper, etc. The surrounding image photographing unit 21 inputs data of the photographed surrounding image (hereinafter referred to as surrounding image data) to the image processing unit 22.

[0042] (Image processing unit 22) The image processing unit 22 performs image processing for detecting a preset object to be detected from among objects present around the vehicle 200, based on the surrounding image data. Hereinafter, the preset object to be detected will be referred to as a "target object."

[0043] The object is, for example, a passenger car, a motorcycle, a bus, a taxi, a bicycle, a pedestrian, an animal, etc. The image processing is, for example, edge detection processing, etc. The image processing unit 22 inputs the image of the object to the object detection unit 25.

[0044] (Object detection sensor 23) The object detection sensor 23 is a sensor for detecting the position of an object present around the vehicle, and is, for example, a sonar.

[0045] For example, a sonar emits a pulsed sound wave of a certain frequency, and when the pulsed sound wave hits an object around the vehicle, it is reflected, and part of the pulsed sound wave returns to the sonar. The longer the distance from the vehicle to the object, the longer it takes for the pulsed sound wave to return to the sonar.

[0046] Using this relationship, sonar estimates the distance to an object by emitting a pulsed sound wave and measuring the round-trip time it takes for the echo to return to the sonar after being reflected by the object, and then detects the object's position relative to the vehicle based on the estimated distance.

[0047] Furthermore, the object detection sensor 23 inputs object position information, which is information indicating the position of the detected object, to the target object detection unit 25 .

[0048] (Object detection unit 25) The object detection unit 25 performs, for example, the following determination based on the image of the object and the object position information.

[0049] (1) The object detection unit 25 determines that there is no risk of collision, for example, when the rear end of the preceding vehicle to be detected is present in front of the vehicle and the distance from the rear end to the front end of the vehicle is greater than a predetermined value (for example, several meters to several tens of meters) that changes depending on the relative speed.

[0050] (2) The object detection unit 25 determines that there is a collision risk, for example, when the rear end of the preceding vehicle to be detected is present in front of the vehicle and the distance from the rear end to the front end of the vehicle is less than the above-mentioned specified value.

[0051] In this way, the object detection unit 25 detects an object that may collide with the vehicle, generates object detection information that is information indicating that an object that may collide with the vehicle has been detected, and transmits it to the detection information output unit 26.

[0052] (Detection information output unit 26) The detection information output unit 26 outputs the object detection information transmitted from the object detection unit 25 to the information input unit 54 of the display control device 50. The configuration of the information input unit 54 will be described in detail later.

[0053] The image processing unit 22, the object detection unit 25, and the detection information output unit 26 are an example of a first application or a second application according to the present disclosure.

[0054] (Display control device 50) The display control device 50 is a device that generates an attention-attracting image to be provided to an occupant based on the line-of-sight information and the object detection information, and transmits the generated attention-attracting image to the display device 30. The display control device 50 is an example of a second device according to the present disclosure.

[0055] The attention drawing image is an example of information necessary for driving. For example, the attention drawing image is an image for drawing the attention of a passenger that an object that may collide with the vehicle 200 is present around the vehicle. For example, the attention drawing image is created based on object detection information transmitted from the surroundings monitoring device 20.

[0056] The attention-calling image may be a symbol such as "!" that can call the attention of the occupant, or a letter such as "Danger of collision." The attention-calling image may also be a combination of these.

[0057] The display control device 50 includes an information input unit 51, a memory unit 52, a first selection unit 57, an information input unit 54, an attention-calling image creation unit 55, an image information output unit 56, the first selection unit 57, and a second selection unit 58.

[0058] The information input unit 51, the first selection unit 57, the information input unit 54, the attention image creation unit 55, the image information output unit 56, the first selection unit 57, and the second selection unit 58 are an example of a second application according to the present disclosure.

[0059] (Information input unit 51) The information input unit 51 receives the line-of-sight information transmitted from the line-of-sight information output unit 14 and transfers it to a first selection unit 57 and a second selection unit 58 .

[0060] (Storage unit 52) The storage unit 52 stores, for example, coordinate data, an identification ID (Identifier) ​​of the display device 30, a gaze detection history, an information display history, etc. The details of this information will be described later.

[0061] (First selection unit 57) For example, when the gaze information output unit 14 outputs the gaze information at a predetermined cycle, the first selection unit 57 determines that the gaze detection device 10 is detecting the gaze of the driver normally.

[0062] In this case, the first selection unit 57 selects, from among the multiple display devices 30, a display device 30 that is present in the direction of the line of sight, as the display device on which to display the attention calling image.

[0063] Specifically, the first selection unit 57 reads out the coordinate data stored in the storage unit 52, and specifies the display device 30 present in the driver's line of sight with the front of the in-vehicle meter panel as the reference position. The coordinate data is data representing the position of each display surface of the multiple display devices 30 within the vehicle.

[0064] The first selection unit 57 selects the identified display device 30 as the display device on which the attention calling image is to be displayed.

[0065] The first selection unit 57 reads out the identification ID corresponding to the selected display device 30, for example, from the storage unit 52, and transmits it to the image information output unit 56. The configuration of the image information output unit 56 will be described in detail later.

[0066] (Second selection unit 58) For example, when the gaze information output unit 14 does not output the gaze information at a predetermined cycle, the second selection unit 58 determines that the gaze detection device 10 is in a state in which it cannot normally detect the gaze of the driver.

[0067] Examples of conditions in which the gaze cannot be detected normally include a condition in which the lens of the image capture unit 11 becomes foggy and the driver's gaze cannot be detected, a condition in which a bug occurs in the software that controls the gaze detection device 10 and the driver's gaze cannot be detected, and a condition in which the gaze is obstructed by glasses or the like.

[0068] In this way, if gaze information is not output from the gaze information output unit 14, the attention-warning image, which is information necessary for driving, cannot be displayed on the display device 30 where the driver's gaze is directed, and this may result in a decrease in driving safety.

[0069] As a countermeasure to this, the display control device 50 according to an embodiment of the present disclosure is configured to estimate, among a plurality of display devices 30, the display device 30 which is most likely to be directed by the driver's gaze, based on the historical information described below, and to display a warning image on the estimated display device 30.

[0070] Specifically, the second selection unit 58 selects, based on the historical information, from among the multiple display devices 30, the display device that is likely to be directed by the driver's gaze as the display device on which to display the attention warning image, and thereby causes the selected display device 30 to display the attention warning image.

[0071] The history information includes, for example, gaze detection history, information display history, and the like.

[0072] The gaze detection history may include, for example, the following information: (1) The cumulative time that the gaze was directed toward the display device 30 selected by the first selection unit 57 in the past. (2) The number of times that the gaze has been directed to the display device 30 selected by the first selection unit 57 in the past. (3) Information indicating the display device 30 to which the gaze was directed most recently, among the display devices 30 previously selected by the first selection unit 57.

[0073] Examples of the information display history include the following information: (1) The cumulative time that the attention-drawing image has been displayed on the display device 30 selected by the first selection unit 57 in the past. (2) The number of times the attention-calling image has been displayed on the display device 30 selected by the first selection unit 57 in the past. (3) Information indicating the display device 30 that displayed the attention drawing image most recently among the display devices 30 previously selected by the first selection unit 57.

[0074] These pieces of history information may be preset in the second selection unit 58, or may be stored in the storage unit 52.

[0075] (Information input unit 54) The information input unit 54 receives the object detection information output by the detection information output unit 26 and transfers it to the attention-attracting image creation unit 55 .

[0076] (Warning Image Creation Department 55) The attention-calling image generating unit 55 generates the attention-calling image described above based on the object detection information, and inputs it to the image information output unit 56.

[0077] (Image information output unit 56) The image information output unit 56 outputs the attention-attracting image created by the attention-attracting image creation unit 55 to the display device 30 selected by the first selection unit 57 or the second selection unit 58.

[0078] Specifically, when the image information output unit 56 receives the identification ID of the display device 30 transmitted from the first selection unit 57 or the second selection unit 58, it identifies the display device 30 corresponding to the identification ID by referring to the table information set in the memory unit 52 or the image information output unit 56.

[0079] Then, the image information output unit 56 outputs the attention-calling image created by the attention-calling image creation unit 55 to the identified display device 30.

[0080] (Display device 30) The display device 30A includes an information input unit 31, an original image generating unit 32, an image superimposing unit 33, and an image display unit 34. The display device 30A is an example of a second device according to the present disclosure.

[0081] The display device 30A is, for example, a center display provided in the vehicle interior.

[0082] The display device 30B has the same configuration as the display device 30A. The display device 30B is, for example, an in-vehicle meter panel provided in front of the driver's seat. The display device 30B is an example of a second device according to the present disclosure.

[0083] The display device 30C has a similar configuration to the display device 30A. The display device 30C is, for example, a head-up display. The display device 30C is an example of a second device according to the present disclosure.

[0084] The types of display device 30 provided on vehicle 200 are not limited to these, and may include, for example, an electronic mirror (e.g., an electronic rearview mirror or an electronic side mirror) that combines an on-board camera and a display.

[0085] Furthermore, the number of display devices 30 provided in the vehicle 200 is not limited to three, but may be two, or may be four or more.

[0086] (Information input unit 31) The information input unit 31 inputs the attention drawing image output by the image information output unit 56 and transfers it to the image superimposition unit 33 .

[0087] (Original image generator 32) The original image generating unit 32 generates original image data and inputs it to the image superimposing unit 33. The original image data is data of an image (original image) to be displayed on the image display device 34 when there is no object that may be collided with around the vehicle.

[0088] Specifically, the original image displayed on the image display unit 34 of the display device 30A is, for example, a map displayed on a navigation screen, an image for operating an audio device, or the like.

[0089] Moreover, the original images displayed on the image display unit 34 of the display device 30B are, for example, images of a speedometer, an oil pressure gauge, a fuel gauge, and the like.

[0090] Moreover, the original image displayed on the image display unit 34 of the display device 30C is, for example, an image showing the vehicle speed, an image showing the vehicle traveling direction, and the like.

[0091] (Image superimposition unit 33) The image superimposing unit 33 generates superimposed image data representing an image in which the attention calling image is superimposed on the original image, based on the original image and the attention calling image, and inputs the generated superimposed image data to the image display device .

[0092] (Image display unit 34) The image display device 34 generates a superimposed image based on the superimposed image data, and displays the superimposed image on the screen of the image display device 34 .

[0093] The information input unit 31, the original image generation unit 32, the image superimposition unit 33, and the image display unit 34 are an example of a second application according to the present disclosure.

[0094] According to the display device 30 configured in this manner, for example, when the driver of the vehicle 200 is gazing at the on-board meter panel while traveling and the distance to the vehicle ahead becomes narrower, a warning image is superimposed on the original image on the on-board meter panel toward which the driver is looking.

[0095] Therefore, even if the driver is not looking ahead, the driver can be notified early that the vehicle 200 may collide with a nearby object, allowing the driver to take action to avoid the collision, such as sudden braking.

[0096] Next, with reference to FIG. 2, a configuration example of an information processing system 100 configured by virtualizing a plurality of different guest OSs on hardware will be described.

[0097] FIG. 2 is a diagram illustrating an example of the configuration of an information processing system 100 that is configured by virtualizing a plurality of different guest OSs on hardware.

[0098] The information processing system 100 includes hardware 60, a hypervisor 70, a gaze detection device 10, a surroundings monitoring device 20, a display control device 50, and a plurality of display devices 30.

[0099] The multiple display devices 30 include a first display device and a second display device mounted on the vehicle 200.

[0100] The hardware 60 includes a processor having a central processing unit (CPU) or an electronic control unit (ECU), neither of which is shown, and provides an execution environment for a plurality of computer programs.

[0101] The hardware 60 may be configured to include one processor, or may be configured to include multiple processors. The multiple processors may configure the information processing system 100 according to the present embodiment.

[0102] A virtualization layer of a hypervisor 70 is created on the hardware 60. The hypervisor 70 is virtualization software that runs on the hardware 60 and controls the execution of a plurality of guest OSs 101 to 30C1.

[0103] The multiple guest OSs 101 to 30C1 are an example of a first OS or a second OS according to the present disclosure.

[0104] A plurality of different types of guest OSs 101 to 30C1 can be virtualized and installed on the hardware 60 by the hypervisor 70. In other words, a plurality of virtual devices (virtual machines) can be realized on one device.

[0105] Examples of hypervisor-type virtualization software include "COQOS Hypervisor SDK" and "QNX Hypervisor." These can be used in this embodiment, but the virtualization software is not limited to these.

[0106] The guest OS 101 is an OS that controls the execution of an application for generating gaze detection information by the gaze detection device 10. The guest OS 101 is an example of a first OS according to the present disclosure.

[0107] The application is an example of the first application or the second application according to the present disclosure. For example, Automotive Safety Integrity Level (ASIL)-A is applied to the application. Details of ASIL will be described later.

[0108] The guest OS 201 is an OS that controls the execution of an application that generates object detection information by the perimeter monitoring device 20. The guest OS 201 is an example of a first OS according to the present disclosure.

[0109] The application is an example of the first application or the second application according to the present disclosure. For example, ASIL-B is applied to the application.

[0110] The guest OS 501 is an OS that controls the execution of an application for generating image information by the display control device 50. The guest OS 501 is an example of a first OS or a second OS according to the present disclosure.

[0111] The application is an example of the first application or the second application according to the present disclosure. For example, ASIL-B is applied to the application.

[0112] The guest OS 30A1 is an OS that controls the execution of an application that displays an image on the display device 30 A. The guest OS 30A1 is an example of a second OS according to the present disclosure.

[0113] The application is an example of a second application according to the present disclosure, and is, for example, subject to ASIL-B.

[0114] The guest OS 30C1 is an OS that controls the execution of an application that displays an image on the display device 30C. The guest OS 30C1 is an example of a second OS according to the present disclosure.

[0115] The application is an example of a second application according to the present disclosure. For example, ASIL-QM is applied to the application.

[0116] The aforementioned ASIL is an index that specifies the safety requirements and safety measures for automobiles, as stipulated in ISO26262, a functional safety standard for the automotive industry.

[0117] ASIL has five safety levels: D, C, B, A, and QM, with the safety standards being D, C, B, A, and QM in descending order.

[0118] For example, airbags, anti-lock braking systems, and power steering are at the highest risk of failure, so these mechanisms are subject to ASIL-D, the most stringent requirements for ensuring safety.

[0119] In addition, ASIL-A applies to backlights, ASIL-B applies to headlights, brake lights, etc., and ASIL-C applies to cruise control.

[0120] In this case, the guest OS's safety standards may be, for example, in the order of highest to lowest, guest OS 30A1, guest OS 501, and guest OS 30C1.

[0121] In the information processing system 100 according to the embodiment of the present disclosure, the hypervisor 70 can operate multiple virtual machines in parallel.

[0122] Therefore, for example, even if a failure occurs in one of the applications of multiple display devices 30, the operation of the display device 30 with a high safety requirement level can be continued without the above-mentioned failure affecting the applications of the other display devices 30 that are not experiencing the failure.

[0123] Furthermore, even if a failure occurs in the applications of the gaze detection device 10, the surroundings monitoring device 20, and the display control device 50, the operation of the display device 30A can be continued without the above-mentioned failure affecting the applications of the display device 30A that are not experiencing a failure.

[0124] The configuration of the information processing system 100 is not limited to the example configuration of FIG. 2, and may be configured as shown in FIGS.

[0125] Figure 3 shows an example of building a hypervisor layer on hardware and then building two guest OS layers on top of that, and Figure 4 shows an example of building a host OS on hardware and then building a hypervisor on top of that.

[0126] The host OS is an OS that controls the execution of applications for controlling the driving of a vehicle, for example.

[0127] Next, the operation of the information processing system 100 will be described with reference to FIGS.

[0128] FIG. 5 is a flowchart for explaining the operation of the information processing system 100 according to the embodiment of the present disclosure.

[0129] The image capturing unit 11 starts capturing an image of the driver's face (step S11). The captured image is transmitted to the line of sight detection unit 13 via the image processing unit 12.

[0130] After the process of step S11, the gaze detection unit 13 detects the direction in which the driver's gaze is directed based on the received image (step S12).

[0131] Gaze information, which is the result of gaze detection, is transmitted to first selection unit 57 and second selection unit 58 via gaze information output unit 14 and information input unit 51. The processing of first selection unit 57 and second selection unit 58 will be described later.

[0132] In parallel with the process of step S12, the information input unit 54 receives the object detection information transmitted by the detection information output unit 26, and transfers the object detection information to the attention-attracting image creation unit 55 (step S13).

[0133] After the process of step S13, the attention-attracting image creating unit 55 creates an attention-attracting image based on the object detection information (step S14).

[0134] After the process of step S14, the attention calling image creating unit 55 transmits the created attention calling image to the image information output unit 56. Thereafter, a display position selection process shown in step S15 is executed.

[0135] The display position selection process shown in step S15 will be described with reference to FIGS. 6A and 6B.

[0136] 6A and 6B are flowcharts for explaining the display position selection process.

[0137] The display control device 50 determines whether or not line-of-sight information is being output (step S21).

[0138] If the line of sight information has not been output (step S21, NO), the process proceeds to step S22.

[0139] In step S22, a process is executed to determine whether the gaze detection function is active or has been stopped by an operation by the driver or the like.

[0140] Specifically, when the gaze information is not output in the above-mentioned step S21, the display control device 50 determines whether the gaze detection function of the gaze detection device 10 is in an on state or an off state (step S22).

[0141] The on state is a state in which the gaze detection function of the gaze detection device 10 is active, and the off state is a state in which the gaze detection function of the gaze detection device 10 is inactive.

[0142] The driver can select whether the gaze detection function is on or off. For example, when the gaze detection function is set to on, a signal indicating that the gaze detection function is working is transmitted from the gaze detection device 10 to the display control device 50.

[0143] When the gaze detection function is set to off, the gaze detection device 10 transmits to the display control device 50 a signal indicating that the gaze detection function is stopped.

[0144] Based on these signals, the display control device 50 can determine whether the gaze detection function is working or whether the gaze detection function has been stopped by an operation by the driver or the like.

[0145] When the gaze detection function of the gaze detection device 10 is in the OFF state (step S22, NO), the display control device 50 determines that the gaze detection function has been turned off by the driver and therefore the gaze information is not being output. In this case, the process of step S30 is executed and the display of the attention-calling image is continued. The process of step S30 will be described later.

[0146] When the gaze detection function of the gaze detection device 10 is on (step S22, YES), the display control device 50 determines that the gaze information is not being output because the gaze cannot be detected normally.

[0147] In this case, the driver is notified that gaze detection is not being performed even though the gaze detection function is on (step S23).

[0148] The notification method may be, for example, displaying a message such as "line of sight not detected" on the display device 30, or playing a sound from a speaker in the vehicle.

[0149] The notification method may be to flash a lamp in the vehicle or to vibrate the steering wheel, seat, etc.

[0150] At this time, a message may be displayed on the display device 30 to prompt the driver to check whether the camera is fogged up, or whether the driver's face is not covered by a mask or the like.

[0151] After step S23, the second selection unit 58 selects a display device 30 (step S24). As a result, an attention calling image is displayed on the selected display device 30 (step S25).

[0152] The processes of steps S24 and S25 may be executed between the completion of the process of step S21 and the start of the process of step S22.

[0153] After the process of step S25, the display control device 50 determines whether or not an operation to turn off the gaze detection function has been performed (step S26).

[0154] For example, when an operation to turn off the gaze detection function is performed (step S26, YES), the display control device 50 notifies the driver that the gaze detection function has been turned off by displaying a message on the display device 30 asking "Do you want to set the gaze detection function to off?", and then executes processing from step S31 onwards.

[0155] Furthermore, if an operation to turn off the gaze detection function is not performed despite the message being displayed on the display device 30 (step S26, NO), the display control device 50 repeats the processes from step S23 onwards.

[0156] Note that the display control device 50 may automatically or manually restart the gaze detection device 10, for example, instead of performing the process of step S26.

[0157] By restarting the gaze detection device 10, the software may operate normally and the gaze may be detected.

[0158] If gaze information cannot be obtained even after restarting the gaze detection device 10, the display control device 50 generates a message to inform the driver of this fact and displays it on the display device 30. This makes it possible to inform the driver that gaze detection is difficult even after restarting the device.

[0159] The display control device 50 may forcibly turn off the gaze detection function after notifying the driver that the gaze cannot be detected even after rebooting.

[0160] Returning to step S21, if the line of sight information has been output (step S21, YES), the process of step S27 is executed.

[0161] In step S21, when the line-of-sight information is output, the display control device 50 determines whether or not the line of sight is directed to any one of the plurality of display devices 30 (step S27).

[0162] If the line of sight is directed to any one of the plurality of display devices 30 (step S27, YES), the first selection unit 57 executes the process of step S .

[0163] The first selection unit 57 selects the display device 30 to which the line of sight is directed (step S28).

[0164] When the display device 30 is selected in step S28, the image information output unit 56 transmits an attention-calling image to the selected display device 30. As a result, the attention-calling image is superimposed on the original image displayed on the display device 30 to which the line of sight is directed (step S29).

[0165] After that, the display control device 50 determines, for example, whether or not the driver has completed an action corresponding to the attention-calling image displayed on the display device 30 (step S30).

[0166] For example, when an obstacle avoidance operation is performed after the display control device 50 displays the message "Watch out for collision" on the head-up display, the display control device 50 determines whether the collision avoidance operation has been completed based on the direction of steering wheel rotation, the amount of steering wheel rotation, the amount of brake pedal depression, the amount of accelerator pedal depression, etc.

[0167] If the collision avoidance operation has not been completed (step S30, NO), the display control device 50 repeats the processes from step S29 onwards.

[0168] When the collision avoidance operation is completed (step S30, YES), the display control device 50 determines that the action corresponding to the attention drawing image is completed, and erases the attention drawing image (step S31).

[0169] The process of step S31 is not limited to this, and may be the following process.

[0170] For example, after displaying a confirmation message "Were you able to avoid the collision?" on a touch-panel center display, if the driver touches the confirmation message, the display control device 50 may determine that the action corresponding to the warning image has been completed and erase the warning image.

[0171] Returning to step S27, if the line of sight is not directed to any of the plurality of display devices 30 (step S27, NO), the process of step S16 shown in FIG. 5 is executed.

[0172] In step S16 shown in FIG. 5, if the processing up to step S15 has not been completed (step S16, NO), the display control device 50 repeats the processing from step S13 onwards.

[0173] In step S16 shown in FIG. 5, if the processing up to step S15 is completed (step S16, YES), the display control device 50 ends the series of processing.

[0174] In addition, when the driver's gaze is not directed at any of the multiple display devices 30, the first selection unit 57 may be configured to identify the display device 30 closest to the gaze as the display device 30 located in the direction in which the driver's gaze is directed.

[0175] The following describes information transmitted between the hypervisor and multiple guest OSs.

[0176] First, with reference to FIG. 7, an operation for displaying an attention-calling image on the display device 30A when no failure occurs in the guest OS 101 of the gaze detection device 10 or the like will be described.

[0177] FIG. 7 is a sequence chart for explaining the operation of the information processing system 100 when no failure occurs in the guest OS 101 of the gaze detection device 10 or the like.

[0178] When the hypervisor 70 receives image information transmitted from a peripheral device (eg, the image capturing unit 11) (step S41), the hypervisor 70 transmits the received image information to the guest OS 101 of the gaze detection device 10 (step S42).

[0179] The application that controls the gaze detection device 10 on the guest OS 101 that receives the image information generates gaze information, and the guest OS 101 transmits the gaze information to the hypervisor 70 (step S43).

[0180] The hypervisor 70 transmits the received line-of-sight information to the guest OS 501 of the display control device 50 (step S44).

[0181] When the hypervisor 70 receives the surrounding video information transmitted from a peripheral device (eg, the surrounding image capturing unit 21) (step S45), it transmits the received surrounding video information to the guest OS 201 of the surrounding monitoring device 20 (step S46).

[0182] The application that controls the surroundings monitoring device 20 on the guest OS 201 that has received the surroundings video information generates object detection information, and the guest OS 201 transmits the object detection information to the hypervisor 70 (step S47).

[0183] The hypervisor 70 transmits the received object detection information to the guest OS 501 of the display control device 50 (step S48).

[0184] Then, the application that controls the display control device 50 on the guest OS 501 creates an attention-calling image to be displayed on, for example, the image display unit 34 of the display device 30A based on the object detection information. Then, the guest OS 501 transmits the attention-calling image to the hypervisor 70 (step S49).

[0185] The hypervisor 70 transmits the received attention image to the guest OS 30A1 of the display device 30A (step S50).

[0186] Then, the application for controlling the display device 30A of the guest OS 30A1 generates superimposed image information based on the received attention-calling image, and the guest OS 30A1 transmits the superimposed image information to the hypervisor 70 (step S51).

[0187] The hypervisor 70 transmits the received superimposed image information to a peripheral device (for example, the image display unit 34 of the display device 30A) (step S52).

[0188] As a result, a superimposed image in which the attention calling image is superimposed on the original image is displayed on the image display unit 34 of the display device 30A. Note that examples of the superimposed image will be described later.

[0189] Next, with reference to FIG. 8 and FIG. 9, an operation will be described in which a guest OS failure occurs in some of the devices and then the guest OS is restored under the control of the hypervisor 70. FIG.

[0190] Here, a case will be described in which a failure occurs in the guest OS 30A1 of the display device 30A present in the driver's line of sight, and the guest OS 30A1 is subsequently restored under the control of the hypervisor 70.

[0191] Fig. 8 is a diagram for explaining the operation of the information processing system 100 when a failure occurs in the guest OS 30A1 of the display device 30A. Fig. 9 is a sequence chart for explaining the operation in Fig. 8.

[0192] When the hypervisor 70 receives an attention warning image transmitted, for example, from the guest OS 501 corresponding to the display control device 50 shown in FIG. 7, the hypervisor 70 transmits the received attention warning image to the guest OS 30A1 corresponding to the display device 30A located in the direction of the driver's line of sight 80 (step S61).

[0193] When the guest OS 30A1 receives the attention-calling image, as shown in the upper diagram of FIG. 8, the display device 30A generates a superimposition image based on the received attention-calling image.

[0194] Then, the application that controls the display device 30A on the guest OS 30A1 displays the generated superimposed image on the image display unit 34 of the display device 30A (step S62). For example, ASIL-B is applied to this application.

[0195] In addition, the application that controls the display device 30B on the guest OS 30B1 of the display device 30B that has not received the attention image displays only the original image on the image display unit 34 of the display device 30B (step S63). For example, ASIL-C is applied to the application.

[0196] The hypervisor 70 transmits a monitor signal for monitoring the state of each guest OS such as the guest OS 30A1 at regular intervals, for example, to each guest OS (step S64).

[0197] Then, when a response signal to the monitoring signal is not returned from each guest OS, such as the guest OS 30A1, within a certain period, the hypervisor 70 determines that a failure has occurred in the guest OS (step S65).

[0198] For example, if a failure occurs in the guest OS 30A1, the hypervisor 70 transmits a reset signal to the guest OS 30A1 (step S66).

[0199] When the guest OS 30A1 receives the reset signal, the guest OS 30A1 performs a recovery process from the failure based on the reset signal.

[0200] When the failure is resolved, the guest OS 30A1 transmits a recovery notification indicating that the failure has been resolved to the hypervisor 70 (step S67).

[0201] The hypervisor 70 that has received the recovery notification transmits the warning image again to the guest OS 30A1 (step S68).

[0202] When the guest OS 30A1 receives the attention-attracting image, the image superimposing unit 33 of the display device 30A present in the direction in which the driver's line of sight 80 is directed generates a superimposed image based on the attention-attracting image.

[0203] Then, as shown in the lower diagram of Figure 8, the application controlling the display device 30A of the guest OS 30A1 causes the generated superimposed image to be displayed again on the image display unit 34 of the display device 30A located in the direction of the driver's line of sight 80 (step S69).

[0204] Next, referring to Figures 10 and 11, we will explain the operation when a failure occurs in the guest OS of one of the display devices and then the gaze is moved to another display device before the recovery of the guest OS is completed.

[0205] Here, we will explain a case where a failure occurs in guest OS 30A1 of display device 30A, which is located in the direction of the driver's line of sight 80, and then the line of sight 80 moves from display device 30A to display device 30B before recovery of guest OS 30A1 is completed.

[0206] Fig. 10 is a diagram for explaining the operation of the information processing system 100 when a failure occurs in the guest OS 30A1 of the display device 30A. Fig. 11 is a sequence chart for explaining the operation in Fig. 10.

[0207] In the following, a description of processes similar to those shown in FIG. 9 will be omitted, and processes different from those shown in FIG. 9 will be described.

[0208] Assume that after a failure occurs in guest OS 30A1, and before hypervisor 70 sends a reset signal to guest OS 30A1, the driver's line of sight 80 moves from display device 30A to display device 30B, as shown in the upper diagram of FIG. 10.

[0209] In this case, the destination of the attention calling image is not the display device 30A but the display device 30B located in the direction of the current line of sight 80 of the driver.

[0210] Therefore, when the hypervisor 70 receives the attention calling image transmitted from the guest OS 501 of the display control device 50, the hypervisor 70 transmits the received attention calling image to the guest OS 30B1 of the display device 30B (step S70).

[0211] When the guest OS 30B1 receives the attention-calling image, the image superimposing unit 33 of the display device 30B generates a superimposed image based on the received attention-calling image.

[0212] Then, the application that controls the display device 30B on the guest OS 30B1 causes the image display unit 34 of the display device 30B to display the superimposed image, as shown in the lower diagram of FIG. 10 (step S71).

[0213] Thereafter, when the guest OS 30A1 corresponding to the display device 30A is restored by the reset signal received in step S66, the application controlling the display device 30A on the guest OS 30A1 causes the original image generated by the original image generation unit 32 of the display device 30A to be displayed on the image display unit 34 of the display device 30A (step S72).

[0214] Note that, in Figures 8 to 11, an example of operation when a failure occurs in the guest OS 30A1 of the display device 30A has been described. Below, with reference to Figure 12, an example of operation when a failure occurs in the guest OS 101 of the gaze detection device 10 having a low safety requirement level will be described.

[0215] In the following, a description of processes similar to those shown in FIG. 7 will be omitted, and processes different from those shown in FIG. 7 will be described.

[0216] FIG. 12 is a sequence chart for explaining the operation of the information processing system 100 when a failure occurs in the guest OS 101 of the gaze detection device 10. In FIG.

[0217] When the hypervisor 70 receives image information transmitted from a peripheral device (eg, the image capturing unit 11) (step S41), the hypervisor 70 transmits the received image information to the guest OS 101 of the gaze detection device 10 (step S42).

[0218] The hypervisor 70 transmits a monitor signal for monitoring the state of each guest OS such as the guest OS 30A1 at regular intervals, for example, to each guest OS (step S80).

[0219] If a response signal to the monitoring signal is not returned from a guest OS such as the guest OS 30A1 within a certain period, the hypervisor 70 determines that a failure has occurred in the guest OS (step S81).

[0220] For example, when a failure occurs in the guest OS 101 of the gaze detection device 10, the hypervisor 70 transmits a reset signal to the guest OS 101 (step S82).

[0221] Upon receiving the reset signal, the guest OS 101 performs a recovery process from the failure based on the reset signal.

[0222] When the failure is recovered, the guest OS 101 transmits a recovery notification indicating that the failure has been recovered to the hypervisor 70 (step S83).

[0223] The hypervisor 70 that has received the recovery notification transmits the image information again to the guest OS 101 (step S84).

[0224] The guest OS 101 that receives the image information generates line-of-sight information and transmits it to the hypervisor 70 (step S43).

[0225] The hypervisor 70 transmits the received line-of-sight information to the guest OS 501 of the display control device 50 (step S44).

[0226] Next, the monitoring process of each guest OS will be described with reference to FIGS.

[0227] (Guest OS monitoring process example 1) FIG. 13 is a sequence chart for explaining an example of the operation of the information processing system 100 when the hypervisor 70 monitors each guest OS.

[0228] In steps S81 to S89, the hypervisor 70 performs a monitoring process for each guest OS.

[0229] Specifically, first, the hypervisor 70 transmits a monitor signal to each guest OS. Then, the hypervisor 70 receives a response from each guest OS to the monitor signal, thereby monitoring the state of each guest OS.

[0230] It is assumed that a failure occurs in the guest OS 30A1 after a monitor signal is sent to the guest OS 30A1 in step S89.

[0231] In this case, the hypervisor 70 transmits a reset signal to the guest OS 30A1 (step S90).

[0232] When the guest OS 30A1 is restored by the reset signal, the guest OS 30A1 transmits a restoration notification indicating that the failure has been restored to the hypervisor 70 (step S91).

[0233] (Guest OS monitoring process example 2) As a modified example, the monitoring of each guest OS may be performed by a monitoring guest OS 90 included in the information processing system 100 instead of by the hypervisor 70.

[0234] Here, the monitoring guest OS 90 is an example of a third OS according to the present disclosure. The monitoring guest OS 90 can control the execution of a predetermined monitoring application that detects the occurrence of a failure related to any of the guest OSs. The monitoring application is an example of a third application that detects the occurrence of a failure related to any of the first OS and the second OS.

[0235] The monitoring guest OS 90 may be provided in the display control device 50, or in another device in the information processing system 100.

[0236] FIG. 14 is a sequence chart for explaining an example of the operation of the information processing system 100 when the monitoring guest OS 90 monitors the state of each guest OS.

[0237] In FIG. 14, for example, it is assumed that the display control device 50 includes a monitoring guest OS 90 in addition to a guest OS 501.

[0238] As shown in FIG. 14, in steps S81 to S89, the monitoring guest OS 90 performs a monitoring process for each guest OS.

[0239] Specifically, first, the monitoring guest OS 90 transmits a monitoring signal to each guest OS, and then receives a response from each guest OS to the monitoring signal, thereby monitoring the state of each guest OS.

[0240] When the occurrence of a failure in any of the guest OSes is detected, the monitoring guest OS 90 controls the execution of an application that generates a reset signal for restoring the guest OS.

[0241] In the example shown in FIG. 14, it is assumed that a fault occurs in the guest OS 30A1 after step S89.

[0242] In this case, the monitoring guest OS 90 transmits a reset signal to the guest OS 30A1 (step S90).

[0243] When the guest OS 30A1 is restored by the reset signal, the guest OS 30A1 transmits a restoration notification indicating that the failure has been restored to the monitoring guest OS 90 (step S91).

[0244] According to this modification, since the monitoring guest OS 90 performs the monitoring process, it is possible to reduce the processing load on the hypervisor 70. In addition, the program of the hypervisor 70 can be simplified.

[0245] (Guest OS monitoring process example 3) In another example, the monitoring of each guest OS may involve each guest OS monitoring the status of the other guest OS.

[0246] FIG. 15 is a sequence chart showing an example of operation when each guest OS in the information processing system 100 monitors the status of the other guest OS.

[0247] In the example shown in FIG. 15, the multiple guest OSs 30A1 to 30A1 can control the execution of applications that monitor each other's status.

[0248] As shown in FIG. 15, in steps S101 to S107, the guest OSs 30A1 to 30A101 each monitor the status of the other.

[0249] For example, the guest OS 30A1 controls the execution of an application that monitors the state of the guest OS 201 and an application that monitors the state of the guest OS 101.

[0250] Furthermore, the guest OS 201 controls the execution of an application that monitors the state of the guest OS 101 and an application that monitors the state of the guest OS 30A1.

[0251] Then, for example, if a fault occurs in the guest OS 30A1, the guest OS 201 detects the fault and sends a reset signal to the guest OS 30A1 (step S108).

[0252] When the guest OS 30A1 is restored by the reset signal, the guest OS 30A1 transmits a restoration notification indicating that the failure has been restored to the guest OS 201 (step S109).

[0253] According to this example, even if a monitoring guest OS 90 dedicated to monitoring is not prepared, the status of each guest OS can be monitored by improving an existing guest OS.

[0254] In steps S110 to S113, the same processes as those in steps S101 to S107 are performed.

[0255] (effect) As described above, the information processing device of this embodiment includes a first OS that controls the execution of a first application that governs the operation of a first device mounted in the vehicle, a second OS that controls the execution of a second application that governs the operation of a second device mounted in the vehicle and having a safety requirement level higher than the safety requirement level of the first device, and a hypervisor that runs on a processor and controls the execution of the first OS and the second OS.

[0256] With this configuration, the hypervisor 70 can operate multiple guest OSs in parallel. Therefore, if a failure occurs in one of the multiple guest OSs, the scope of impact of the failure can be limited to the display device corresponding to that OS, thereby preventing the display devices corresponding to the other OSs among the multiple guest OSs from all becoming inoperable at the same time.

[0257] Therefore, for example, even if a failure occurs in the OS corresponding to the gaze detection device 10 having a low safety requirement level, the display control device 50 can continue normal processing by other guest OSs, thereby preventing the display device 30B having a high safety requirement level from becoming inoperable.

[0258] In addition, for example, even if the display of a warning image on the screen of a navigation device is interrupted, the warning image can be displayed on the in-vehicle meter panel or other device to which the driver's gaze has moved, thereby contributing to safe driving by the driver.

[0259] Next, a hardware configuration example of the information processing system 100 will be described with reference to Fig. 16. Fig. 16 is a diagram illustrating a hardware configuration example of the information processing system 100 according to an embodiment of the present disclosure.

[0260] The vehicle 200 includes, as an on-vehicle device 210, a gaze detection device 10, a surroundings monitoring device 20, a display control device 50, a display device 30A, a display device 30B, a display device 30C, and an ECU 220.

[0261] The ECU 220 includes an auxiliary storage device 230A, a memory device 230B, a CPU 230C, and an interface device 230D. These constitute an information processing device 230, and are connected to each other via a bus line 230E.

[0262] The auxiliary storage device 230A is a hard disk drive (HDD), flash memory, or the like that stores files and data necessary for processing by the ECU 220. When an instruction to start a program is received, the memory device 230B reads out the program from the auxiliary storage device 230A and stores it.

[0263] The CPU 230C executes the programs stored in the memory device 230B, and realizes various functions of the ECU 220 in accordance with the programs.

[0264] The interface device 230D is, for example, an interface that connects the CPU 230C to the gaze detection device 10, the surroundings monitoring device 20, and the like.

[0265] In the embodiment of the present disclosure, an example configuration has been described in which an attention-warning image is created using object detection information output from the perimeter monitoring device 20. However, the attention-warning image may also be created using vehicle information obtained from various sensors installed in the vehicle 200.

[0266] The vehicle information includes, for example, door opening / closing status information, tire pressure information, engine water temperature information, and remaining driving distance information.

[0267] The attention-calling image created based on the door open / close state information is, for example, a message saying "The door is open."

[0268] If the door is closed after such a message is displayed on the display device 30, the display control device 50 determines in the processing of the above-mentioned step S30 that the action corresponding to the warning image has been completed, and erases the warning image in the processing of the above-mentioned step S31.

[0269] The warning image created based on the tire pressure information may be, for example, a message such as "Air pressure exceeds the proper range" or "Air pressure is low."

[0270] After such a message is displayed on the display device 30, if the air pressure is adjusted to the appropriate value, the display control device 50 determines in the processing of the above-mentioned step S30 that the action corresponding to the warning image has been completed, and erases the warning image in the processing of the above-mentioned step S31.

[0271] The warning image created based on the engine water temperature information is, for example, a message saying "Possibility of overheating."

[0272] If the engine is inspected after such a message is displayed on the display device 30, the display control device 50 determines in the processing of step S30 described above that the action corresponding to the warning image has been completed, and erases the warning image in the processing of step S31 described above.

[0273] The warning image created based on the remaining driving distance information is, for example, a message saying "10km remaining driving distance."

[0274] If fuel is replenished after such a message is displayed on the display device 30, the display control device 50 determines in the processing of step S30 described above that the action corresponding to the warning image has been completed, and erases the warning image.

[0275] The second selection unit 58 may be configured as follows.

[0276] The second selection unit 58 may be configured to select, when line-of-sight information is not being output, a preset display device 30 as a display device to which the line of sight of the occupant is more likely to be directed than other display devices.

[0277] For example, by setting a head-up display as the display device 30 that displays a warning image when gaze information is not being output, the warning image can be automatically displayed on the head-up display when gaze information is not being output.

[0278] With this configuration, even if gaze information is not being output, a warning image can be displayed on a display device 30 that has been preset as a display device 30 that is likely to be the target of the occupants' gaze, thereby improving driving safety.

[0279] The image information output unit 56 may be configured as follows.

[0280] When the second selection unit 58 selects a display device 30 which is more likely to be directed by the occupant's gaze than the other display devices 30, the image information output unit 56 may be configured to generate image information indicating that the occupant should shift their gaze from a first display device among the multiple display devices 30 which has not been selected by the second selection unit 58 to the second display device selected by the second selection unit 58, and output the image information to the first display device.

[0281] In this case, for example, when no line-of-sight information is being output, the image information output unit 56 generates a message saying "See the head-up display."

[0282] Then, the image information output unit 56 displays the message on the center display and also displays an attention-calling image on the head-up display.

[0283] By displaying the message "See the head-up display" on the center display in this way, even if the driver's gaze cannot be detected while he or she is looking at the center display, the message on the center display makes the driver aware that an attention-calling image is being displayed on the head-up display. Therefore, the driver can quickly look at the head-up display.

[0284] This allows the driver to quickly take action in response to the attention-calling image, further improving driving safety.

[0285] The second selection unit 58 may be configured to select a terminal device that is portable by the occupant as the display device 30 to which the occupant's gaze is more likely to be directed than other display devices 30.

[0286] For example, after a terminal device such as a smartphone and an in-vehicle device mounted on the vehicle 200 complete pairing by performing short-range wireless communication with each other via Bluetooth (registered trademark) or the like, if the terminal device is selected by the second selection unit 58, the image information output unit 56 transfers an attention-calling image to the terminal device selected by the second selection unit 58. This makes it possible to display an attention-calling image on the terminal device even when the terminal device is used as a navigation device.

[0287] In addition, the display control device 50 may be configured to, for example, determine that if an action corresponding to the warning image is not completed after displaying a warning image for a certain period of time, the driver is ignoring the warning image or the action is not required, and erase the warning image on the display device 30.

[0288] By erasing the attention-calling image, only the original image is displayed on the display device 30, making it easier for the driver to recognize the original image, and thus making it possible to further improve driving safety.

[0289] In addition, the display control device 50 may use information obtained from a sensor that detects objects around the vehicle to identify the position of an object such as a motorcycle, and display a warning image on the display device 30 that is positioned between that position and the driver.

[0290] For example, when vehicle 200 is traveling toward an intersection and detects a motorcycle approaching vehicle 200 from the front left of vehicle 200, display control device 50 causes a message such as "Risk of collision" to flash on the center display located to the front left of the driver.

[0291] This makes it possible to inform the driver that an object may collide with the vehicle 200. At this time, the display control device 50 may simultaneously output voice guidance to call attention to the situation. This allows the driver to take action to avoid the collision, such as sudden braking, thereby further improving the safety of driving.

[0292] It should be noted that, for example, the following aspects are also understood to fall within the technical scope of the present disclosure.

[0293] (1) An information processing device includes a first OS that controls the execution of a first application that governs the operation of a first device mounted in a vehicle, a second OS that controls the execution of a second application that governs the operation of a second device mounted in the vehicle and having a safety requirement level higher than the safety requirement level of the first device, and a hypervisor that runs on a processor and controls the execution of the first OS and the second OS.

[0294] (2) The first device is a gaze detection device that detects the direction in which an occupant of the vehicle is looking, and the second device is a display device that is identified based on the direction of the gaze detected by the gaze detection device.

[0295] (3) The display device is a device that displays an attention-grabbing image to inform the occupants of the presence of an object detected by a detection device that detects objects present in the vicinity of the vehicle.

[0296] (4) The hypervisor detects the occurrence of a failure related to either the first OS or the second OS.

[0297] (5) When the hypervisor detects the occurrence of the failure, it performs processing on the OS in which the failure has occurred to restore the OS.

[0298] (6) The computer further includes a third OS that controls the execution of a third application that detects the occurrence of a failure related to either the first OS or the second OS.

[0299] (7) When the third application detects the occurrence of the failure, it performs a process on the OS in which the failure has occurred to restore the OS.

[0300] (8) The above information processing system includes the above information processing device, a first device, and a second device.

[0301] (9) An information processing method includes the steps of: a first OS controlling the execution of a first application that controls the operation of a first device mounted in a vehicle; a second OS controlling the execution of a second application that controls the operation of a second device mounted in the vehicle and having a safety requirement level higher than the safety requirement level of the first device; and a hypervisor executing on a processor controlling the execution of the first OS and the second OS.

[0302] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or amended examples within the scope described in the present disclosure, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the above-mentioned embodiments may be arbitrarily combined within the scope of the present disclosure.

[0303] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims may include various modifications and alterations of the specific examples exemplified in the present disclosure. [Industrial Applicability]

[0304] An embodiment of the present disclosure is suitable for a display control device and an information processing system. [Explanation of symbols]

[0305] 10. Line of Sight Detection Device 11 Image capture section 12 Image processing section 13. Gaze detection unit 14. Gaze information output unit 20 Periphery monitoring device 21 Peripheral image capture unit 22 Image processing section 23 Object detection sensor 25 Object detection unit 26 Detection information output section 30 Display device 30A display device 30B Display device 30C display device 31 Information input section 32 Original image generation unit 33 Image Overlay Unit 34 Image display section 50 Display control device 51 Information input section 52 Storage section 54 Information input section 55 Warning Image Creation Department 56 Image information output section 57 First Selection Section 58 Second Selection Section 60 Hardware 70 Hypervisor 80 line of sight 100 Information Processing Systems 200 vehicles 210 Onboard equipment 220 ECU 230 Information processing equipment 230A Auxiliary storage device 230B Memory device 230C CPU 230D Interface Device 230E Bus Line

Claims

1. a first OS that controls execution of a first application that governs the operation of a first device mounted in a vehicle; a second OS that is mounted on the vehicle and controls execution of a second application that governs an operation of a second device having a safety requirement level higher than that of the first device; a fourth OS that controls execution of a fourth application that governs the operation of a third device mounted in the vehicle; a hypervisor that runs on a processor and controls execution of the first OS, the second OS, and the fourth OS; the first device is a gaze detection device that detects a direction in which a gaze of an occupant of the vehicle is directed, the second device and the third device are display devices identified based on the direction of the line of sight detected by the line of sight detection device, the second device is a center display, The third device is a head-up display. Information processing device.

2. The information processing device according to claim 1 , wherein the display device is a device that displays an attention-calling image for notifying the occupant of the presence of an object detected by a detection device that detects an object existing around the vehicle.

3. The information processing apparatus according to claim 1 , wherein the hypervisor detects an occurrence of a failure related to either the first OS or the second OS.

4. The information processing apparatus according to claim 3 , wherein when the hypervisor detects the occurrence of the failure, the hypervisor performs a process for restoring the OS in which the failure has occurred.

5. 4. The information processing apparatus according to claim 1, further comprising a third OS that controls execution of a third application that detects the occurrence of a failure related to either the first OS or the second OS.

6. The information processing apparatus according to claim 5 , wherein the third application, when detecting the occurrence of the failure, performs a process for restoring the OS in which the failure has occurred.

7. The vehicle further includes a fifth OS that controls execution of a fifth application that controls operation of a fourth device having a safety requirement level higher than the safety requirement level of the first device, When the occurrence of a failure related to the second OS is detected while the second device is displaying an attention image, the hypervisor transmits the attention image to the fifth OS, and the fourth device displays the attention image. The information processing device according to any one of claims 1 to 6.

8. 8. An information processing system comprising: the information processing device according to claim 1; the first device; the second device; and the third device.

9. A step of controlling, by a first OS, execution of a first application that governs an operation of a first device mounted in a vehicle; a step of controlling, by a second OS, execution of a second application that controls an operation of a second device that is mounted on the vehicle and has a safety requirement level higher than the safety requirement level of the first device; a fourth OS controlling execution of a fourth application that controls an operation of a third device mounted in the vehicle; a hypervisor executing on a processor controlling execution of the first OS, the second OS, and the fourth OS; the first device is a gaze detection device that detects a direction in which a gaze of an occupant of the vehicle is directed, the second device and the third device are display devices identified based on the direction of the line of sight detected by the line of sight detection device, the second device is a center display, The third device is a head-up display. Information processing methods.

Images