Information processing method and information processing device

Abstract

In an information processing device according to an embodiment of this disclosure, a first OS, which is a real-time OS that is activated when a target vehicle is activated, acquires a video obtained by photographing the periphery of the target vehicle by a camera provided in the target vehicle, and displays the acquired video on at least one of a first display and a second display.

Description

Information processing method and information processing device

[0001] The present invention relates to an information processing method and an information processing device.

[0002] Conventionally, there has been known a technique for controlling the display output of each of a plurality of displays in a vehicle equipped with the plurality of displays. For example, Patent Document 1 listed below discloses an image control device that, when a plurality of camera applications that output images captured by cameras equipped in the vehicle are simultaneously operating, gives priority to displaying the screen of a camera application that displays an image from a camera dedicated to reversing.

[0003] Japanese Patent Application Laid-Open No. 2014-229066

[0004] However, in the above-described conventional technology, when an application program that displays a car navigation image displays an image captured by a camera (e.g., an image captured by a camera specifically designed for backing up), the following problem may occur: For example, in a situation where the vehicle is to be backed up immediately after being started, if the application program that displays the car navigation image starts up slowly, the image captured by the camera specifically designed for backing up cannot be displayed at the appropriate time.

[0005] In one aspect, the present invention has been made in consideration of the above circumstances, and its purpose is to provide an information processing method and an information processing device that can display image data captured by a camera on a display as soon as the vehicle is started.

[0006] In order to solve the above-mentioned problem, an information processing method according to one aspect of the present invention is an information processing method that causes a computer to execute a process for controlling the display output of each of a first display and a second display provided in a target vehicle, wherein in the computer, a first OS, which is a real-time OS (Operating System) that controls the display output of each of the first display and the second display, is started when the target vehicle is started, and a second OS that controls the display output of the first display completes its startup after the startup of the first OS is completed, and the first OS that has completed its startup executes the steps of acquiring video of the surroundings of the target vehicle captured by an imaging device provided in the target vehicle, and displaying the acquired video on at least one of the first display and the second display.

[0007] According to the present invention, it is possible to provide an information processing method and an information processing device that can quickly display an image captured by a camera on a display after a vehicle is started.

[0008] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments and is not to be construed as limiting the scope of the invention.

[0009] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in the present embodiment are described in natural language, more specifically, they are specified using pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

[0010] §1 Application Example FIG. 1 is a block diagram showing a schematic configuration of a target vehicle TV equipped with an information processing device (information processing device 1) according to this embodiment. The target vehicle TV is an example of the "target vehicle" of the present invention. As illustrated in FIG. 1, the target vehicle TV includes, for example, the information processing device 1, a first display 2, a second display 3, a control system ECU (Electronic Control Unit) 4, and an imaging device 5. The first display 2 is an example of the "first display" of the present invention, and the second display 3 is an example of the "second display" of the present invention.

[0011] The first display 2 and the second display 3 are each a display device (display panel) whose display output is controlled by the information processing device 1, and are configured, for example, as a liquid crystal display or an organic EL display. The first display 2 and the second display 3 are each connected to the information processing device 1 via, for example, a controller area network (CAN) or other in-vehicle LAN, and communicate with each other (transmit and receive information). Through this communication, content (images, information) controlled by the information processing device 1 is displayed on the first display 2 and the second display 3, respectively.

[0012] The first display 2 is provided, for example, near a so-called center console. The first display 2 may be configured as a touch panel display that can accept touch operations. For example, an image (navigation image IN) showing navigation information NI generated by the navigation application 122 is displayed on the first display 2. In this embodiment, the "application program" may be abbreviated as "app." Also, in FIG. 2, the "application" is written as "app."

[0013] The navigation application 122 is an application program that realizes a "car navigation system Sys that provides navigation services such as route guidance to a destination to a passenger in a target vehicle TV." For example, the navigation application 122 generates navigation information NI including route information (map information) to a destination and outputs the generated navigation information NI. The navigation application 122 will be described in detail later.

[0014] Furthermore, for example, a video (captured video CV) of the surroundings of the target vehicle TV captured by the imaging device 5 provided in the target vehicle TV is displayed on the first display 2. In particular, the information processing device 1 may display, among the captured videos CV, a rear captured video BCV, which is a video CV captured by the imaging device 5 of the area behind the target vehicle TV, only on the first display 2. Control of the display output related to the captured video CV (particularly the rear captured video BCV) will be described in detail later.

[0015] So far, the navigation image IN and the captured video CV have been given as examples of information (content) displayed on the first display 2, but content other than these may also be displayed on the first display 2. For example, the first display 2 may display various menu screens, television broadcasts, information about music being played in the vehicle on the target vehicle TV, and the like.

[0016] The second display 3 is provided, for example, near the front of the driver of the target vehicle TV. The second display 3 may be a so-called meter display. The second display 3 displays an image (meter image IM) showing information (driving information TI) about the state (particularly, the driving state) of the target vehicle TV, such as the driving speed of the target vehicle TV, the engine rotation speed (rpm), the mileage, and the remaining fuel level. The meter image IM may further show information about the safety of the target vehicle TV (including the occupants of the target vehicle TV), such as whether or not the occupants of the target vehicle TV are wearing seat belts, information required by laws and regulations, various warnings, and the like.

[0017] Although examples of information (content) displayed on each of the first display 2 and the second display 3 have been described above, the information displayed on each of the first display 2 and the second display 3 is not limited to what has been described above. Furthermore, the information processing device 1, which controls the display output of each of the first display 2 and the second display 3, can display information displayed on the first display 2 on the second display 3, or information displayed on the second display 3 on the first display 2. For example, the information processing device 1 may display at least one of the navigation image IN and the captured video CV on the second display 3, or the meter image IM on the first display 2. However, in this embodiment, it is desirable that the rear captured video BCV of the captured video CV is not displayed on the second display 3, but is displayed only on the first display 2.

[0018] The control system ECU 4 is a system (electronic control computer) that controls the driving of the target vehicle TV, and controls, for example, the acceleration, steering, and deceleration (braking) of the target vehicle TV. In the example shown in FIG. 1 , the control system ECU 4 outputs information (driving information TI) related to the state (particularly the driving state) of the target vehicle TV, such as the driving speed of the target vehicle TV, engine rotation speed (rpm), mileage, and remaining fuel, to the information processing device 1. The information processing device 1 uses the driving information TI acquired from the control system ECU 4 to display a meter image IM, for example, on the second display 3. The control system ECU 4 and the information processing device 1 may be connected, for example, via the above-mentioned in-vehicle LAN, and may communicate with each other (transmit and receive information).

[0019] The imaging device 5 is provided in the target vehicle TV and captures images of the surroundings of the target vehicle TV. The imaging device 5 may be, for example, a camera (video camera) equipped with an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), an ultrasonic camera, or an infrared camera. The target vehicle TV may be provided with multiple imaging devices 5. For example, imaging devices 5 may be disposed in the front grille, under the left and right door mirrors, and near the rear bumper of the target vehicle TV. In the example shown in FIG. 1 , the imaging device 5 outputs images (captured images CV) of the surroundings of the target vehicle TV to the information processing device 1. The information processing device 1 displays the captured images CV acquired from the imaging device 5 on, for example, the first display 2. The imaging device 5 and the information processing device 1 may be connected, for example, via the above-mentioned in-vehicle LAN, and may communicate with each other (transmit and receive information).

[0020] The information processing device 1 controls the display output of each of the first display 2 and the second display 3 provided in the target vehicle TV. In the example shown in Fig. 1 , the information processing device 1 displays, for example, a navigation image IN and a captured video CV on the first display 2. The information processing device 1 also displays, for example, a meter image IM on the second display 3. The information processing device 1 displays the meter image IM and images (information, content) other than the meter image IM, such as the navigation image IN and the captured video CV, and in particular controls the display output (for example, controls switching and updating the content to be displayed).

[0021] As will be described in detail later, an environment in which multiple operating systems (OS) can operate is created in the information processing device 1, and an environment in which at least a first OS 110 and a second OS 120 can operate is created. The first OS 110 is a real-time OS that executes processes that require real-time performance, and in particular, executes control of display output that requires real-time performance. In contrast, the second OS 120 is a general-purpose OS that executes processes that require less real-time performance than the processes executed by the first OS 110 (e.g., control of display output).

[0022] The first OS 110 is started when the target vehicle TV is started. For example, when a start switch (also referred to as a "main power switch" or "ignition switch") for starting the target vehicle TV is turned on, the target vehicle TV (for example, the BCM (Body Control Module) of the target vehicle TV) starts the first OS 110. When the target vehicle TV is started, the target vehicle TV may start the first OS 110 by outputting a first start request to the information processing device 1. The method for turning on the start switch of the target vehicle TV may be, for example, an engine key method in which the target vehicle TV is started by the occupant turning the key of the target vehicle TV inserted into the keyhole, or a push-start method in which the target vehicle TV is started by the occupant pressing a button-shaped start switch.

[0023] The second OS 120, like the first OS 110, is started when the target vehicle TV is started. When the target vehicle TV is started, the target vehicle TV may start the second OS 120 by outputting a second startup request to the information processing device 1. The target vehicle TV may output the first startup request before the second startup request. Furthermore, the target vehicle TV may output the first startup request and the second startup request simultaneously (almost simultaneously). The second OS 120 may be started by the first OS 110. For example, the first OS 110 started by the first startup request may start the second OS 120.

[0024] In this embodiment, the timing at which the second OS 120 completes booting is later than the timing at which the first OS 110 completes booting. In other words, the second OS 120 completes booting after the first OS 110 completes booting. Here, "(OS) booting is completed" means that application programs provided in the OS (in this embodiment, each of the first OS 110 and the second OS 120) become executable (operable).

[0025] In the information processing device 1, the control of the display output of the captured video CV is executed by the first OS 110, not the second OS 120. The first OS 110 is a real-time OS that is activated when the target vehicle TV is activated. Therefore, the information processing device 1 can promptly display the captured video CV captured by the imaging device 5 (camera) on at least one of the first display 2 and the second display 3 after the target vehicle TV is activated. In other words, for example, when the second OS 120 controls the display output of the captured video CV, such control becomes possible after the activation of the second OS 120 is completed. The activation of the second OS 120 is completed after the activation of the first OS 110 is completed. Therefore, by having the first OS 110, which completes its startup before the second OS 120 completes its startup, control the display output of the captured video CV, the information processing device 1 can display the captured video CV more quickly than when the second OS 120 controls the display output of the captured video CV. In particular, by having the first OS 110, which is a real-time OS that is started when the target vehicle TV is started, control the display output of the captured video CV, the information processing device 1 can display the captured video CV more quickly after the target vehicle TV is started. The information processing device 1, whose overview has been described above, will be described in detail below using Figures 2 and 3.

[0026] 2 is a schematic diagram illustrating an example of the configuration of the information processing device 1 according to this embodiment. As shown in FIG. 2, the information processing device 1 according to this embodiment includes a control unit 10, a storage unit 12, a display output interface 14, a communication interface 16, and a video input interface 18.

[0027] The control unit 10 includes a CPU (Central Processing Unit) 130, which is a hardware processor, and is configured to execute information processing based on programs and various data. In addition to the CPU 130, the control unit 10 may also include a RAM (Random Access Memory) and a ROM (Read Only Memory), not shown. The CPU 130 is an example of a processor resource.

[0028] The storage unit 12 is an example of a memory resource, and is configured, for example, by a hard disk drive, a solid state drive, etc. In this embodiment, the storage unit 12 stores various types of information such as an information processing program PG.

[0029] The information processing program PG is a program for causing the information processing device 1 to execute information processing (FIG. 3) described below that controls the display output of each of the first display 2 and the second display 3 provided in the target vehicle TV. The information processing program PG includes a series of instructions for the information processing.

[0030] The display output interface 14 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to each of the first display 2 and the second display 3. The type of the display output interface 14 may be selected appropriately depending on the type of each of the first display 2 and the second display 3 to be connected. In the illustrated example, the information processing device 1 uses the display output interface 14 to output, to each of the first display 2 and the second display 3, signals indicating content (images) to be displayed on each of the first display 2 and the second display 3. In other words, the information processing device 1 controls the display output of each of the first display 2 and the second display 3 via the display output interface 14.

[0031] The communication interface 16 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. As described above, the communication interface 16 may also be an interface for performing communication via a CAN or other in-vehicle LAN. The type and number of communication interfaces 16 may be selected appropriately depending on the type and number of connected external devices (devices other than the information processing device 1). The information processing device 1 may use this communication interface 16 to perform data communication with other information processing devices via a network. In the illustrated example, the information processing device 1 uses the communication interface 16 to acquire, from the control system ECU 4, driving information TI related to the status (e.g., driving status) of the target vehicle TV, such as the driving speed, engine rotation speed (rpm), mileage, and remaining fuel level of the target vehicle TV. The information processing device 1 may also use the communication interface 16 to communicate with a GPS (Global Positioning System) or the like to acquire location information indicating the location of the target vehicle TV.

[0032] The video input interface 18 is, for example, a USB port, a dedicated port, or the like, and is an interface for connecting to the imaging device 5. The type and number of the video input interfaces 18 may be selected appropriately depending on the type and number of the imaging devices 5 to be connected. In the illustrated example, the information processing device 1 uses the video input interface 18 to acquire, from the imaging device 5, captured video CV of the area around the target vehicle TV.

[0033] The information processing device 1 may include an input device, such as a mouse or keyboard, for input. The information processing device 1 may also include an output device, such as a speaker, for output. An operator, such as a user, can operate the information processing device 1 by using the input and output devices. Furthermore, the information processing device 1 may also include a drive device, such as a CD drive or DVD drive, for reading various information, such as programs, stored in a storage medium. Such a storage medium is a medium that stores information, such as programs, through electrical, magnetic, optical, mechanical, or chemical action so that a computer or other device, machine, etc., can read the various information, such as the stored programs. The information processing program PG may be stored in, for example, a storage medium. The information processing device 1 may acquire the information processing program PG from the storage medium.

[0034] Note that, with regard to the specific hardware configuration of the information processing device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the processor resource may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), or the like. The storage unit 12 may be configured with RAM and ROM included in the control unit 10. The information processing device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the information processing device 1 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a PC (Personal Computer), or the like.

[0035] The control unit 10 of the information processing device 1 loads the information processing program PG stored in the storage unit 12 onto the RAM. Then, the control unit 10 causes the CPU 130 to interpret and execute instructions included in the information processing program PG loaded onto the RAM, thereby controlling each component.

[0036] An environment in which the first OS 110 and the second OS 120 can operate is created in the control unit 10, and the first OS 110 and the second OS 120 each share processing. The first OS 110 and the second OS 120 operate on, for example, a hypervisor 118. The hypervisor 118 is virtualization software that runs on, for example, the CPU 130 and controls the execution of the first OS 110 and the second OS 120. In the example shown in FIG. 2 , the hypervisor 118 is provided as a function of the first OS 110. The first OS 110 and the second OS 120 are connected to each other so that they can communicate with each other.

[0037] However, the method of operating the first OS 110 and the second OS 120 in the information processing device 1 is not limited to the example shown in FIG. 2 . The hypervisor 118 may be a host type or a bare metal type. For example, the CPU 130 may be a multi-core processor having multiple processor cores, with the first OS 110 assigned (placed) on one processor core and the second OS 120 assigned (placed) on another processor core. The first OS 110 and the second OS 120 may share processing responsibilities by communicating with each other between the processor core assigned to the first OS 110 and the processor core assigned to the second OS 120.

[0038] The first OS 110 is a real-time OS that controls the display output of the first display 2 and the second display 3. An example of a real-time OS that can be used as the first OS 110 is QNX (registered trademark). In contrast, the second OS 120 is a general-purpose OS that controls the display output of the first display 2. Examples of general-purpose OS that can be used as the second OS 120 include Android (registered trademark) OS, iOS, Windows (registered trademark), and Linux (registered trademark). The first OS 110 and the second OS 120 each execute various application programs to implement various functional units in software.

[0039] In this embodiment, the first OS 110 and the second OS 120 are each started when the target vehicle TV is started. For example, when the target vehicle TV is started, the target vehicle TV outputs a first startup request to the information processing device 1 to start the first OS 110, and outputs a second startup request to the information processing device 1 to start the second OS 120. The second OS 120, which is a general-purpose OS, executes multiple processes, including user interface processing and multimedia processing, and various applications (software) required to execute these processes are implemented. Therefore, the startup of the second OS 120 takes longer than the startup of the first OS 110, which is a real-time OS that executes only "specific processes requiring real-time performance." Therefore, in the information processing device 1, the startup of the second OS 120 is completed after the startup of the first OS 110 is completed.

[0040] The first OS 110, which is a real-time OS, executes processes requiring real-time performance, such as controlling the display output of information related to the driving and safety of the target vehicle TV (including the safety of the occupants of the target vehicle TV). For example, the first OS 110 controls the display output of information (content) related to the status (e.g., driving status) of the target vehicle TV and the safety of the target vehicle TV (including the occupants of the target vehicle TV). Specifically, the first OS 110 controls the display output of driving information TI, i.e., controls the display output of a meter image IM representing the driving information TI. In addition to controlling the display output of the meter image IM, the first OS 110 also controls the display output of the captured video CV. In the illustrated example, the first OS 110 includes a meter image generation application 112, a captured video processing application 114, and an area control unit 116 as functional units that execute processes requiring real-time performance.

[0041] The meter image generation application 112 controls the display output of a meter image IM indicating the state of the target vehicle TV (e.g., driving conditions such as driving speed, engine rotation speed (rpm), mileage, and remaining fuel). For example, the meter image generation application 112 first acquires driving information TI from the control system ECU 4 via the communication interface 16. Then, the meter image generation application 112 generates a meter image IM using the acquired driving information TI. Specifically, the meter image generation application 112 generates a meter image IM indicating the state of the target vehicle TV indicated by the driving information TI (e.g., driving conditions such as driving speed, engine rotation speed (rpm), mileage, and remaining fuel). The meter image generation application 112 outputs the generated meter image IM to at least one of the first display 2 and the second display 3 via the display output interface 14, and displays the meter image IM on at least one of the first display 2 and the second display 3. For example, the meter image generation application 112 outputs the meter image IM to the second display 3 via the display output interface 14 , and displays the meter image IM on the second display 3 .

[0042] The meter image generation application 112 needs to notify an occupant (e.g., a driver) of the target vehicle TV of the status of the target vehicle TV (e.g., driving status such as driving speed, engine rotation speed (rpm), mileage, and remaining fuel) in almost real time. Therefore, the meter image generation application 112 needs to update the meter image IM at an extremely short interval (e.g., a interval of about one tenth of a second). Therefore, in the information processing device 1, the meter image generation application 112 that controls the display output of the meter image IM is provided in the first OS 110, which is a real-time OS. That is, in the information processing device 1, the first OS 110, which is a real-time OS, displays the meter image IM, and, for example, the first OS 110 displays the meter image IM on the second display 3. The first OS 110, which is a real-time OS (particularly, the meter image generation application 112 provided in the first OS 110), can achieve the real-time characteristics and high display reliability required for controlling the display output of the meter image IM.

[0043] The captured video processing application 114 controls the display output of captured video CV captured by the imaging device 5 provided in the target vehicle TV of the surroundings of the target vehicle TV. For example, the captured video processing application 114 first acquires the captured video CV from the imaging device 5 via the video input interface 18. The captured video processing application 114 outputs the acquired captured video CV to at least one of the first display 2 and the second display 3 via the display output interface 14, and displays the captured video CV on at least one of the first display 2 and the second display 3. For example, the captured video processing application 114 displays the captured video CV on the first display 2.

[0044] As described above, the captured video processing application 114 that controls the display output of the captured video CV is an application program executed by the first OS 110, which is a real-time OS that is started when the target vehicle TV is started. That is, in the information processing device 1, the first OS 110, which is a real-time OS that is started when the target vehicle TV is started, acquires the captured video CV and displays the acquired captured video CV on at least one of the first display 2 and the second display 3. Therefore, the first OS 110, which is a real-time OS that is started when the target vehicle TV is started (particularly, the captured video processing application 114 provided in the first OS 110), can display the captured video CV promptly after the target vehicle TV is started. That is, the information processing device 1 can display the captured video CV on at least one of the first display 2 and the second display 3 promptly after the target vehicle TV is started.

[0045] An occupant (e.g., a driver) of the target vehicle TV can easily check the surroundings of the target vehicle TV from the captured video CV displayed by the captured video processing application 114. In particular, the rear captured video BCV, which is the captured video CV of the area behind the target vehicle TV captured by the imaging device 5, allows the occupant of the target vehicle TV to easily check the presence of obstacles, pedestrians, etc. behind the target vehicle TV, for example, when the target vehicle TV is backing up.

[0046] As described above, the captured video CV is displayed by the captured video processing application 114 on at least one of the first display 2 and the second display 3. However, it is desirable that the rear captured video BCV of the captured video CV is displayed only on the first display 2. In other words, it is desirable that the captured video processing application 114 does not display the rear captured video BCV of the captured video CV on the second display 3, but displays it only on the first display 2.

[0047] Here, the meter image IM is displayed on, for example, the second display 3, and the second display 3 is provided, for example, near the front of the driver. Therefore, the steering wheel of the target vehicle TV is disposed, for example, between the second display 3 and the driver. In this case, the steering wheel may obstruct the view of the target vehicle TV occupants, such as the driver, making it difficult for them to view (confirm) the content and information displayed on the second display 3. Furthermore, for example, when the driver is operating the steering wheel, the driver's arms, hands, etc. may obstruct the view of the target vehicle TV occupants, including the driver, making it difficult for them to view the content and information displayed on the second display 3. Therefore, the information processing device 1 (particularly, the first OS 110) uses separate displays for displaying the meter image IM and the captured image CV (particularly, the rear captured image BCV). For example, the first OS 110 displays the meter image IM on the second display 3, and displays the captured video CV (particularly, the rear captured video BCV) on the first display 2. Through this processing, the information processing device 1 can prevent the occurrence of a situation in which the steering wheel, the driver's arms, hands, etc. get in the way, making it difficult for the occupants of the target vehicle TV, including the driver, to check the captured video CV (particularly, the rear captured video BCV).

[0048] The area control unit 116 adjusts the display areas of the content (images) controlled by each of the first OS 110 and the second OS 120. As described above, the first OS 110 controls the display output of each of the first display 2 and the second display 3, and the second OS 120 controls the display output of the first display 2. That is, in this embodiment, the display output of the first display 2 is controlled by each of the first OS 110 and the second OS 120. Therefore, the area control unit 116 adjusts the control of the display output of the first display 2 by the first OS 110 and the control of the display output of the first display 2 by the second OS 120. For example, the first OS 110 displays the captured video CV, and the second OS 120 displays the navigation image IN on the first display 2. Therefore, the area control unit 116 adjusts (controls) the display area on the first display 2 for, for example, the captured image CV whose display output is controlled by the first OS 110 and the navigation image IN whose display output is controlled by the second OS 120.

[0049] In this embodiment, the area control unit 116 prioritizes "control of the display output of the first display 2 by the first OS 110" over "control of the display output of the first display 2 by the second OS 120." When there is a conflict between "control of the display output of the first display 2 by the first OS 110" and "control of the display output of the first display 2 by the second OS 120," the area control unit 116 prioritizes the former over the latter. An example of a conflict between "control of the display output of the first display 2 by the first OS 110" and "control of the display output of the first display 2 by the second OS 120" is when the first OS 110 and the second OS 120 attempt to display different images (contents) on the first display 2 at the same time.

[0050] For example, when the first OS 110 attempts to display a certain image (content) on the first display 2 at a certain timing, and the second OS 120 attempts to display another image on the first display 2 at a certain timing, the area control unit 116 may perform the following control. That is, the area control unit 116 may hide the "image that the second OS 120 attempts to display on the first display 2 at a certain timing," and display only the "image that the first OS 110 attempts to display on the first display 2 at a certain timing" on the first display 2 at the certain timing. That is, the area control unit 116 may (1) display only the image that the first OS 110 attempts to display on the first display 2 at the certain timing, and (2) not display the image that the second OS 120 attempts to display on the first display 2 at the certain timing on the first display 2. The area control unit 116 may display on the first display 2 the "image that the second OS 120 is attempting to display on the first display 2" after the "image that the first OS 110 is attempting to display on the first display 2" has disappeared.

[0051] Furthermore, area control unit 116 may enlarge an image that first OS 110 is attempting to display on first display 2 and display it on the first display 2, and may reduce an image that second OS 120 is attempting to display on first display 2 and display it on the first display 2. For example, area control unit 116 may make the display area on first display 2 for the image that first OS 110 is attempting to display on first display 2 larger than the display area on first display 2 for the image that second OS 120 is attempting to display on first display 2.

[0052] As described above, the first OS 110 controls the display output of content (images), such as meter images IM and captured video CV, that should be notified to the occupants (e.g., the driver) of the target vehicle TV in nearly real time. In other words, the content whose display output is controlled by the first OS 110 is highly necessary to be notified to the occupants of the target vehicle TV, and in particular, is information that needs to be notified to the occupants of the target vehicle TV in nearly real time. In contrast, as will be described in detail later, the second OS 120 controls the display output of content that has lower requirements for real-timeness, such as navigation images IN, than the content whose display output is controlled by the first OS 110. In other words, the content whose display output is controlled by the second OS 120 has a lower need to be notified to the occupants of the target vehicle TV (particularly, the need for real-time notification) than the content whose display output is controlled by the first OS 110. Therefore, the area control unit 116 prioritizes "control of the display output of the first display 2 by the first OS 110" over "control of the display output of the first display 2 by the second OS 120." Through this control, the area control unit 116 can display on the first display 2 content whose display output is controlled by the first OS 110 (e.g., captured video CV) in priority over content whose display output is controlled by the second OS 120 (e.g., navigation image IN). The area control unit 116 can display (1) content whose display output is controlled by the first OS 110 and for which "the necessity of notifying the occupants of the target vehicle TV is high, particularly the necessity of notifying the occupants of the target vehicle TV in nearly real time" on the first display 2 in preference to (2) content whose display output is controlled by the second OS 120 and for which "the necessity of notifying the occupants of the target vehicle TV (particularly the necessity of notifying in real time) is lower than that of content whose display output is controlled by the first OS 110" on the first display 2. For example, even if content is being displayed on the first display 2 by the second OS 120, the area control unit 116 displays "content whose display output is controlled by the first OS 110" on the first display 2 in preference to such content.Therefore, the area control unit 116 can prioritize and display on the first display 2 "content whose display output is controlled by the first OS 110," which is assumed to have higher requirements for real-time performance than "content whose display output is controlled by the second OS 120."

[0053] When the first OS 110 and the second OS 120 attempt to display different images (contents) on the first display 2 at the same time, the area control unit 116 displays, for example, only the "image that the first OS 110 is attempting to display" on the first display 2 and hides the "image that the second OS 120 is attempting to display." Then, when the "image that the first OS 110 is attempting to display (to display) on the first display 2" is gone, the area control unit 116 displays the "image that the second OS 120 is attempting to display," which had been hidden, on the first display 2. In other words, the area control unit 116 may switch between displaying and hiding the "image that the second OS 120 is attempting to display" on the first display 2 depending on whether or not the "image that the first OS 110 is attempting to display (to display) on the first display 2" is present. By this control, the area control unit 116 can display the "image that the second OS 120 is trying to display" on the first display 2 as soon as the "image that the first OS 110 is displaying (or trying to display) on the first display 2" is gone.

[0054] The second OS 120, which is a general-purpose OS, executes processes that require versatility over real-time performance, such as user interface processing and multimedia processing. The user interface processing includes, for example, processing for accepting operations from the occupants of the target vehicle TV, processing for executing operations corresponding to the accepted operations (e.g., processing for generating information corresponding to the accepted operations), and processing for outputting the execution results (e.g., processing for outputting the generated information). In other words, the user interface processing includes processing related to input from the occupants of the target vehicle TV and processing related to output to the occupants of the target vehicle TV. Furthermore, the multimedia processing includes, for example, processing for communicating information such as text, images, audio, and music using digital data, and processing for managing security in such communications. In the illustrated example, the second OS 120 includes a navigation application 122 as a functional unit that executes processes that require versatility over real-time performance (e.g., user interface processing, multimedia processing, etc.).

[0055] The navigation application 122 is an application program that realizes the car navigation system Sys and provides navigation information NI, including route information to a destination, to an occupant of the target vehicle TV. For example, the navigation application 122 communicates with a GPS or the like via the communication interface 16 to acquire location information of the target vehicle TV. The navigation application 122 also accepts input from an occupant of the target vehicle TV, for example, a user operation specifying a destination. The navigation application 122 generates navigation information NI, including route information (map information) to the destination, based on the location information of the target vehicle TV and the destination specified by the user operation. The navigation information NI may include road traffic information, congestion information, and the like in addition to the route information. The navigation application 122 then controls the output of the generated navigation information NI. For example, the navigation application 122 may generate a navigation image IN indicating the navigation information NI and control the display output of the generated navigation image IN or the audio output of the navigation information NI. That is, the navigation application 122 processes inputs (for example, reception of a user operation to specify a destination) and outputs (for example, display of a navigation image IN showing navigation information NI) related to the car navigation system Sys.

[0056] In this embodiment, the navigation application 122 outputs a navigation image IN showing navigation information NI to at least one of the first display 2 and the second display 3 via the display output interface 14, and displays the navigation image IN on at least one of the first display 2 and the second display 3. For example, the navigation application 122 outputs the navigation image IN to the first display 2 via the display output interface 14, and displays the navigation image IN on the first display 2. The navigation image IN is a so-called navigation image, and shows the navigation information NI including the above-mentioned route information. The navigation image IN may show road traffic information, congestion information, etc. in addition to the route information.

[0057] As described above, the navigation application 122 is an application program that realizes "a car navigation system Sys that provides navigation services such as route guidance to a destination to a passenger in a target vehicle TV," and is executed by the second OS 120. The second OS 120 is a general-purpose OS. That is, in the information processing device 1, the second OS 120, which is a general-purpose OS, processes input and output related to the car navigation system Sys.

[0058] The second OS 120 (navigation application 122) performs input processing related to the car navigation system Sys, for example, by accepting a user operation related to the car navigation system Sys and executing an operation corresponding to the accepted user operation. Specifically, the second OS 120 accepts a user operation specifying a destination for the car navigation system Sys and generates navigation information NI corresponding to the accepted user operation. For example, the second OS 120 communicates with a GPS or the like and generates the navigation information NI from the location information of the target vehicle TV acquired through such communication and the destination specified by the accepted user operation.

[0059] Furthermore, the second OS 120 (navigation application 122) controls, for example, the output of the generated navigation information NI as an output process related to the navigation application 122. Specifically, the second OS 120 generates a navigation image IN showing the navigation information NI, controls the display output of the generated navigation image IN, and controls the audio output of the navigation information NI.

[0060] Generally, a general-purpose OS is superior to a real-time OS in user interface processing and multimedia processing. Processing related to the car navigation system Sys includes user interface processing and multimedia processing, such as accepting user operations, communicating with a GPS or the like, generating navigation information NI, and outputting the generated navigation information NI. Processing related to the car navigation system Sys may further include processing related to images such as a GUI (Graphical User Interface) included in a navigation image IN showing the navigation information NI, for example.

[0061] In the information processing device 1, the second OS 120, which is a "general-purpose OS superior to a real-time OS in user interface processing and multimedia processing," executes "input and output processing related to the car navigation system Sys," including various user interface processing and multimedia processing. Therefore, the information processing device 1 realizes the following effect by having the second OS 120, which is a general-purpose OS, process the "input and output related to the car navigation system Sys" instead of the first OS 110, which is a real-time OS. That is, the information processing device 1 can easily and efficiently process the "input and output related to the car navigation system Sys," including various user interface processing and multimedia processing, using the second OS 120, which is a general-purpose OS.

[0062] The processing performed by the second OS 120 is not limited to "input processing and output processing related to the car navigation system Sys." The second OS 120 may further perform user interface processing and multimedia processing other than "input and output related to the car navigation system Sys." In addition to input and output related to the car navigation system Sys, the second OS 120 may also perform user interface processing and multimedia processing, such as output of various menu screens, television broadcasts, and information about music being played in the target vehicle TV. The second OS 120 may also perform communication processing with devices external to the target vehicle TV (e.g., vehicles other than the target vehicle TV, roadside infrastructure, etc.).

[0063] As described above, the navigation application 122 realizes the car navigation system Sys by performing processes such as accepting user operations, communicating with a GPS or the like, generating navigation information NI corresponding to the accepted user operations, generating a navigation image IN showing the navigation information NI, and controlling the display output of the navigation image IN. Here, user operations related to the car navigation system Sys can take various forms. For example, if the first display 2 is configured as a touch panel display, user operations related to the car navigation system Sys may be performed as touch operations on the touch panel display. User operations related to the car navigation system Sys may also be performed as pressing operations on physical operation switches, buttons, etc. Furthermore, if the target vehicle TV has a gesture recognition function that applies image analysis technology or the like, user operations related to the car navigation system Sys may also be performed as gestures for the gesture recognition function. Similarly, if the target vehicle TV has a voice analysis function, user operations related to the car navigation system Sys may also be performed as speech (voice input) for the voice analysis function. As shown in these examples, the interfaces (user interfaces) that accept user operations related to the car navigation system Sys are becoming extremely diverse. To accommodate this diversity, the second OS 120 (navigation application 122) that processes inputs (acceptance of user operations) related to the car navigation system Sys is equipped with multiple applications (software). Therefore, it is expected that the startup of the second OS 120 will take longer than the startup of the first OS 110. In other words, it is expected that the startup of the second OS 120 will be completed after the startup of the first OS 110 is completed.

[0064] Therefore, in the information processing device 1, the first OS 110, not the second OS 120, controls the display output of the captured video CV. In other words, in the information processing device 1, the real-time OS (first OS 110) that is started when the target vehicle TV is started controls the display output of the captured video CV, rather than the general-purpose OS (second OS 120) that is expected to take time to start up. For example, the first OS 110 displays the captured video CV on the first display 2 on which the navigation image IN is displayed by the second OS 120. In other words, on the first display 2 provided in the target vehicle TV, the captured video CV is displayed by the first OS 110, and the navigation image IN is displayed by the second OS 120. Of the content (information) displayed on the first display 2, the display output of the navigation image IN is controlled by the second OS 120, and the display output of the captured video CV is controlled by the first display 2. By configuring this, the information processing device 1 achieves the following effects. That is, the information processing device 1 can display the captured video CV on the first display 2 immediately after the target vehicle TV is started up, by using the first OS 110, which is a real-time OS that is started up when the target vehicle TV is started up. Also, the information processing device 1 can flexibly and efficiently process input (e.g., reception of user operations) and output (e.g., display of a navigation image IN) related to the car navigation system Sys, by using the second OS 120, which is a general-purpose OS.

[0065] §3 Operation Example Figure 3 is a flowchart showing an example of the processing procedure of the information processing device 1 according to this embodiment. The processing procedure described below is an example of the processing procedure of an information processing method PM that causes a computer (in this embodiment, the information processing device 1) to execute the process of "controlling the display output of each of the first display 2 and the second display 3 provided in the target vehicle TV." However, the processing procedure described below is merely an example, and each step may be modified as much as possible. Furthermore, steps in the processing procedure described below may be omitted, replaced, or added as appropriate depending on the embodiment.

[0066] When the target vehicle TV is started, for example, the target vehicle TV (for example, the BCM of the target vehicle TV) starts the first OS 110 and the second OS 120. In the illustrated example, the first OS 110 is started by a first start-up request (from the target vehicle TV), and the second OS 120 is started by a second start-up request (from the target vehicle TV). In the illustrated example, the first start-up request is output before the second start-up request, but the first start-up request and the second start-up request may be output simultaneously (almost simultaneously). Furthermore, the first OS 110 started by the first start-up request may start the second OS 120.

[0067] (Step S110) The first OS 110 is started in response to the start-up request (first start-up request), that is, the first OS 110 performs start-up processing (step S110). When the start-up processing (processing for making application programs provided in the first OS 110 executable (operable)) is completed, the first OS 110 proceeds to step S120.

[0068] (Steps S120 to S140) After booting, the first OS 110 executes an initialization process for in-vehicle communication (step S120). For example, in the first OS 110, the meter image generation application 112 executes a preparation process for acquiring the driving information TI from the control system ECU 4 via the communication interface 16. Then, the first OS 110 (particularly, the meter image generation application 112) acquires the driving information TI from the control system ECU 4. Thereafter, the first OS 110 (particularly, the meter image generation application 112) executes a meter image generation process (step S130), and generates a meter image IM using, for example, the acquired driving information TI. The first OS 110 (the meter image generation application 112) displays the meter image IM generated in step S130 on the second display 3 (step S140). For example, the first OS 110 (meter image generation application 112 ) outputs the meter image IM to the second display 3 via the display output interface 14 , and displays the meter image IM on the second display 3 .

[0069] (Steps S150 to S170) The first OS 110 (particularly, the captured video processing application 114) executes input processing for the captured video CV (step S150), i.e., acquires the captured video CV. For example, the first OS 110 (captured video processing application 114) acquires the captured video CV from the imaging device 5 via the video input interface 18. The first OS 110 (captured video processing application 114) then executes output processing for the captured video CV acquired in step S150 (step S160) and displays the captured video CV on the first display 2 (step S170). For example, the first OS 110 (captured video processing application 114) outputs the captured video CV to the first display 2 via the display output interface 14, and displays the captured video CV on the first display 2. In particular, the first OS 110 (captured video processing application 114 ) displays the rear captured video BCV of the captured video CV only on the first display 2 , not on the second display 3 .

[0070] When the target vehicle TV reverses, the target vehicle TV may output an image display request to the first OS 110 (particularly, the captured image processing application 114). For example, when detecting an "operation for reversing the target vehicle TV (for example, an operation of shifting the shift lever into reverse)" by the driver of the target vehicle TV, the target vehicle TV may output an image display request to the first OS 110. Upon receiving the image display request, the first OS 110 (particularly, the captured image processing application 114) may display, on the first display 2, a rear captured image BCV, which is an image CV captured by the imaging device 5 behind the target vehicle TV. In other words, the information processing device 1 may automatically display the rear captured image BCV on the first display 2 when the target vehicle TV reverses. With this control, when the target vehicle TV is backing up, the occupants (e.g., the driver) of the target vehicle TV can easily check the presence of obstacles, pedestrians, etc. behind the target vehicle TV by using the rear captured image BCV displayed on the first display 2 without performing any special operations.

[0071] Furthermore, the information processing device 1 achieves the following effect by displaying the rearward captured image BCV on the first display 2, which is provided near the center console, for example, instead of on the second display 3, which is provided near the front of the driver, for example: The information processing device 1 can prevent the occurrence of a situation in which the steering wheel, the driver's arms, hands, etc. get in the way and make it difficult for the occupants of the target vehicle TV, including the driver, to check the rearward captured image BCV.

[0072] (Step S210) The second OS 120 is started in response to the start-up request (second start-up request), that is, the second OS 120 performs start-up processing (step S210). When the start-up processing (processing for making application programs provided in the second OS 120 executable (operable)) is completed, the second OS 120 proceeds to step S220.

[0073] (Steps S220 to S230) After the second OS 120 has completed booting, it executes a navigation image generation process (step S220). That is, the navigation application 122 executed by the second OS 120 after booting completes generates a navigation image IN. For example, the second OS 120 (navigation application 122) communicates with a GPS or the like via the communication interface 16 to acquire location information of the target vehicle TV. The second OS 120 (navigation application 122) also accepts, for example, a user operation specifying a destination. The second OS 120 (navigation application 122) generates navigation information NI including route information (map information) to the destination based on the location information of the target vehicle TV and the destination specified by the user operation. The second OS 120 (navigation application 122) then generates a navigation image IN showing the navigation information NI. The second OS 120 (navigation application 122) displays the navigation image IN generated in step S220 on the first display 2 (step S230). For example, the second OS 120 (navigation application 122) outputs the navigation image IN to the first display 2 via the display output interface 14, and displays the navigation image IN on the first display 2.

[0074] As described above, in the information processing device 1, the process of controlling the display output of the captured video CV is executed by the first OS 110 (particularly, the captured video processing application 114), which is a real-time OS that is started when the target vehicle TV is started. Therefore, the information processing device 1 can display the captured video CV promptly after the target vehicle TV is started, and in particular, can display the captured video CV before the startup of the second OS 120 is completed. In other words, the information processing device 1 can display the captured video CV on the first display 2, which displays the navigation image IN, etc., without waiting for the startup of the "second OS 120, which is a general-purpose OS that processes input and output related to, for example, the car navigation system Sys." In the example shown in FIG. 3 , the captured video CV is displayed on the first display 2 before the "startup process of the second OS 120" in step S210 is completed. The information processing device 1 can display the captured video CV more quickly than when the second OS 120, which is a general-purpose OS, executes the process of controlling the display output of the captured video CV (i.e., the second OS 120 controls the display output of the captured video CV after the startup process is completed). In particular, the information processing device 1 can display the captured video CV on, for example, the first display 2 quickly after the target vehicle TV is started up by using the first OS 110, which is a real-time OS that is started up when the target vehicle TV is started up.

[0075] [Features] As described above, the information processing device 1 according to this embodiment includes a control unit 10 on which a first OS 110 and a second OS 120 operate. The first OS 110 is activated when the target vehicle TV is activated, and controls the display output of each of the first display 2 and the second display 3. The second OS 120 completes its activation after the activation of the first OS 110 is completed, and controls the display output of the first display 2. The first OS 110 is a real-time OS that acquires captured video CV of the surroundings of the target vehicle TV captured by the imaging device 5 provided in the target vehicle TV, and displays the acquired captured video CV on at least one of the first display 2 and the second display 3.

[0076] The information processing method PM according to this embodiment is an information processing method that causes a computer (e.g., an information processing device 1) to execute a process for controlling the display output of each of the first display 2 and the second display 3 provided in the target vehicle TV. In the computer (information processing device 1), a first OS 110, which is a real-time operating system (OS) that controls the display output of each of the first display 2 and the second display 3, is started when the target vehicle TV is started. In the computer (information processing device 1), a second OS 120 that controls the display output of the first display 2 completes its startup after the first OS 110 has completed its startup. The first OS 110, which has completed its startup, executes steps S150 and S170 illustrated in FIG. 3 . In step S150, the first OS 110 acquires a captured video CV of the surroundings of the target vehicle TV captured by an imaging device 5 provided in the target vehicle TV. In step S170, the first OS 110 displays the captured video CV acquired in step S150 on at least one of the first display 2 and the second display 3.

[0077] According to this configuration, the information processing device 1 (information processing method PM) can promptly display the captured video CV captured by the imaging device 5 (camera) on at least one of the first display 2 and the second display 3 after the target vehicle TV is started. That is, for example, when the second OS 120 controls the display output of the captured video CV, such control becomes possible after the startup of the second OS 120 is completed. The startup of the second OS 120 is completed after the startup of the first OS 110 is completed. Therefore, by the first OS 110, which completes its startup before the startup of the second OS 120 is completed, controlling the display output of the captured video CV, the information processing device 1 (information processing method PM) can display the captured video CV more quickly than when the second OS 120 controls the display output of the captured video CV. In particular, the first OS 110, which is a real-time OS that is started when the target vehicle TV is started, controls the display output of the captured video CV, so that the information processing device 1 (information processing method PM) can display the captured video CV promptly after the target vehicle TV is started.

[0078] §4 Modifications Although the embodiments of the present invention have been described above in detail, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiment, and descriptions of similar points to those in the above embodiment are omitted where appropriate. The following modifications can be combined as appropriate.

[0079] In the above embodiment, an example has been described in which the information processing device 1 and the control system ECU 4 are configured as separate computers. However, the configuration of the information processing device 1 according to this embodiment is not limited to this example and may be determined appropriately depending on the embodiment. For example, the information processing device 1 and the control system ECU 4 may be integrated into one computer. Furthermore, at least one of the information processing device 1 and the control system ECU 4 may be configured as multiple computers.

[0080] REFERENCE SIGNS LIST 1...information processing device 1, 2...first display (first display), 3...second display (second display), 5...imaging device, 10...controller, 110...first OS, 120...second OS, BCV...rear captured image (image captured behind the target vehicle), CV...captured image (image), IM...meter image, PM...information processing method, Sys...car navigation system, TV...target vehicle

Claims

1. An information processing method that causes a computer to execute a process for controlling the display output of each of a first display and a second display equipped in a target vehicle, wherein the computer executes the following steps: a first OS, which is a real-time OS (Operating System) that controls the display output of each of the first display and the second display, is started when the target vehicle is started; a second OS that controls the display output of the first display completes its startup after the startup of the first OS is completed; and the first OS that has completed its startup executes the following steps: acquiring video of the surroundings of the target vehicle taken by an imaging device equipped in the target vehicle; and displaying the acquired video on at least one of the first display and the second display.

2. The information processing method according to claim 1, wherein control of the display output of the first display by the first OS takes priority over control of the display output of the first display by the second OS.

3. The information processing method described in claim 1 or 2, wherein the first OS displays the image captured behind the target vehicle from the image captured by the imaging device around the target vehicle on the first display without displaying it on the second display.

4. An information processing method according to claim 1 or 2, wherein the second OS is a general-purpose OS that processes input and output related to a car navigation system that provides information, including route information to a destination, to occupants of the target vehicle.

5. The information processing method according to claim 1 or 2, wherein the first OS displays a meter image indicating the driving state of the target vehicle on the second display.

6. An information processing device comprising: a control unit on which a first OS, which is started when a target vehicle is started and which controls the display output of each of a first display and a second display, and a second OS, which completes its startup after the startup of the first OS is completed and controls the display output of the first display, operates; wherein the first OS is a real-time OS, and the first OS acquires video of the surroundings of the target vehicle captured by an imaging device provided in the target vehicle, and displays the acquired video on at least one of the first display and the second display.

Images