Information processing device

Abstract

To display, on an on-vehicle display, contents associated with a possibility that an object actually crosses a road.SOLUTION: An information processing device comprises a control part that when a sensing part senses an object existing in a running direction of a vehicle running on a road and a positional relation between the vehicle and the object is in a predetermined state and when it is assumed that the object crosses the road, displays an image of the object showing the object, on a position where an actual position of the object is reflected in a road image showing the road displayed on a display part in the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device. [Background technology]

[0002] Patent Document 1 discloses a technique that enables an occupant to clearly see an object that exists outside the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2020 / 031912 publication Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 1 displays object-related information indicating the object on a HUD (Head-Up Display), but there is room for improvement in the content displayed on the in-vehicle display to allow occupants to recognize objects outside the vehicle.

[0005] Therefore, an object of the present disclosure is to provide an information processing device that can display, on an in-vehicle display, content related to the possibility that an object will actually cross a road. [Means for solving the problem]

[0006] The information processing device according to claim 1 includes a control unit that, when a detection unit detects an object present in the traveling direction of a vehicle traveling on a road, the positional relationship between the vehicle and the object is in a predetermined state, and it is expected that the object will cross the road, displays an object image showing the object at a position that reflects the actual position of the object in a road image showing the road displayed on a display unit inside the vehicle. and when it is no longer expected that the object will cross the road after the object image is displayed, the control unit erases the object image from the display unit. .

[0007] In the information processing device according to claim 1, when an object is detected by the detection unit, the positional relationship between the vehicle and the object is in a predetermined state, and it is expected that the object will cross the road, the control unit displays an object image in a position that reflects the actual position of the object on the road image. As a result, the information processing device displays the object image on the condition that it is expected that the object will cross the road, and therefore it is possible to display content related to the possibility that the object will actually cross the road on the in-vehicle display. In the information processing device according to claim 1, the control unit erases the object image from the display unit when it is no longer expected that the object will cross the road after displaying the object image. This allows the information processing device to reduce the amount of information displayed on the display unit when it is no longer expected that the object will cross the road.

[0008] The information processing device according to claim 2 is the information processing device according to claim 1, wherein the object is expected to cross the road when the actual direction of the object faces the road.

[0009] In the information processing device according to claim 2, the case where an object is expected to cross a road is when the actual orientation of the object is facing the road. This allows the information processing device to determine the possibility that the object will cross the road based on the actual orientation of the object, and control whether or not to display the object image.

[0010] The information processing device according to claim 3 is the information processing device according to claim 2, wherein the control unit displays the object image that reflects the actual orientation of the object.

[0011] In the information processing device according to claim 3, the control unit displays an object image that reflects the real orientation of the object. This allows the occupant to grasp the real orientation of the object by looking at the display unit.

[0014] Claim 4 The information processing device according to the present invention comprises: 3 In any one of the above, the control unit displays a vehicle image showing the vehicle at a position on the road image that reflects the actual position of the vehicle.

[0015] Claim 4In the information processing device according to the present invention, the control unit displays the vehicle image in a position on the road image that reflects the actual position of the vehicle. This allows the occupant of the information processing device to understand the positional relationship between the vehicle and an object by looking at the display unit. [Effects of the Invention]

[0016] As described above, the information processing device according to the present disclosure can display, on an in-vehicle display, content related to the possibility that an object will actually cross a road. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 2 is a block diagram showing the hardware configuration of a vehicle. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a vehicle. [Figure 3] 1 is a first flowchart showing the flow of a control process. [Figure 4] 10 is a second flowchart showing the flow of the control process. [Figure 5] FIG. 10 is a first explanatory diagram showing an example of a display displayed on a display unit while the vehicle is traveling. [Figure 6] FIG. 2 is a second explanatory diagram showing an example of a display displayed on the display unit while the vehicle is traveling. DETAILED DESCRIPTION OF THE INVENTION

[0018] A vehicle 10 according to this embodiment will be described. The vehicle 10 may be any of an engine vehicle, a hybrid vehicle, and an electric vehicle, but in this embodiment, as an example, the vehicle 10 is an engine vehicle.

[0019] Fig. 1 is a block diagram showing the hardware configuration of a vehicle 10. As shown in Fig. 1, the vehicle 10 includes an on-board device 20, an ECU (Electronic Control Unit) 30, and on-board equipment 40. The on-board device 20 is an example of an "information processing device."

[0020] The vehicle-mounted device 20 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage unit 24, an in-vehicle communication I / F (Interface) 25, an input / output I / F 26, and a wireless communication I / F 27. The CPU 21, the ROM 22, the RAM 23, the storage unit 24, the in-vehicle communication I / F 25, the input / output I / F 26, and the wireless communication I / F 27 are connected to each other via an internal bus 28 so as to be able to communicate with each other.

[0021] The CPU 21 is a central processing unit that executes various programs and controls each part. That is, the CPU 21 reads programs from the ROM 22 or the storage unit 24 and executes the programs using the RAM 23 as a work area. The CPU 21 controls each of the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 22 or the storage unit 24.

[0022] The ROM 22 stores various programs and various data. The RAM 23 serves as a working area for temporarily storing programs or data.

[0023] The storage unit 24 is configured with a storage device such as an eMMC (embedded Multi Media Card) or a UFS (Universal Flash Storage), and stores various programs and various data. The storage unit 24 stores an information processing program 24A for causing the CPU 21 to execute control processing, which will be described later.

[0024] The in-vehicle communication I / F 25 is an interface for connecting to the ECU 30. The interface uses a communication standard based on the CAN protocol. The in-vehicle communication I / F 25 is connected to an external bus 29. Although not shown in the figure, multiple ECUs 30 are provided for each function of the vehicle 10.

[0025] The input / output I / F 26 is an interface for communicating with an in-vehicle device 40 mounted on the vehicle 10 .

[0026] The on-vehicle devices 40 are various devices mounted on the vehicle 10. The vehicle 10 includes, as examples of the on-vehicle devices 40, a detection unit 40A and a display unit 40B.

[0027] The detection unit 40A is a detection device capable of detecting the state of the vehicle 10, including the inside and outside of the vehicle. The detection unit 40A includes, for example, a camera, a vehicle speed sensor, a millimeter wave sensor, and a GPS (Global Positioning System) device.

[0028] The camera serving as the detection unit 40A is, for example, an imaging device that captures images using an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera is provided at the front of the vehicle 10 and captures images ahead of the vehicle. The images captured by the camera are used, for example, to recognize the distance between the vehicle and a preceding vehicle traveling ahead of the vehicle, the lane, and objects present ahead of the vehicle. The images captured by the camera are stored in the storage unit 24. The camera may be configured as an imaging device for other purposes, such as a drive recorder and an ADAS (Advanced Driver Assistance System). The camera may also be connected to the in-vehicle device 20 via the ECU 30 (for example, a camera ECU).

[0029] The vehicle speed sensor serving as the detection unit 40A is a sensor for detecting the vehicle speed of the vehicle 10, and is provided on, for example, a wheel.

[0030] The millimeter wave sensor serving as the detection unit 40A is a sensor for detecting an object present in front of the vehicle, i.e., in the traveling direction of the vehicle 10, and is provided at least at the front of the vehicle 10. The millimeter wave sensor is used to detect an object by transmitting a transmission wave ahead of the vehicle 10 and receiving a reflected wave from the object ahead. Here, the object is, for example, an object that may pose a danger to the traveling of the vehicle 10, such as a pedestrian, a bicycle, a wall, or an oncoming vehicle. Note that the object may be registered in advance in the vehicle 10, or may be registered by input by the occupant. Furthermore, it is desirable that the object can be updated, such as by deletion or addition, by the in-vehicle device 20 automatically or based on input by the occupant. Hereinafter, in this embodiment, the object is assumed to be a pedestrian, as an example.

[0031] The GPS device serving as the detection unit 40A is a device that detects the current location of the vehicle 10. The GPS device includes an antenna (not shown) that receives signals from GPS satellites. The GPS device may be connected to the vehicle-mounted device 20 via a car navigation system that is connected to the ECU 30 (for example, a multimedia ECU).

[0032] The display unit 40B is an in-vehicle display for displaying an operation suggestion related to a function of the vehicle 10, an image related to an explanation of the function, etc. The display unit 40B is provided, for example, on a meter panel.

[0033] The wireless communication I / F 27 is a wireless communication module for communicating with the outside world, and uses communication standards such as 5G, LTE, and Wi-Fi (registered trademark).

[0034] Next, a description will be given of the functional configuration of the vehicle 10. Fig. 2 is a block diagram showing an example of the functional configuration of the vehicle 10.

[0035] 2, the CPU 21 of the vehicle-mounted device 20 has, as functional components, an acquisition unit 21A and a control unit 21B. Each functional component is realized by the CPU 21 reading and executing an information processing program 24A stored in the storage unit 24.

[0036] The acquisition unit 21A periodically acquires detection information detected by the detection unit 40A while the vehicle 10 is traveling. The detection information includes, for example, an image captured by a camera, and detection results detected by a vehicle speed sensor, a millimeter wave sensor, and a GPS device.

[0037] The control unit 21B controls the display content of the display unit 40B based on the detection information acquired by the acquisition unit 21A. For example, while the vehicle 10 is traveling, the control unit 21B can display, on the display unit 40B, a vehicle image 50 showing the vehicle 10, a road image 52 showing the road on which the vehicle 10 is traveling, and an object image 54 showing an object (see FIGS. 5 and 6).

[0038] 3 is a first flowchart showing the flow of a control process in which the vehicle-mounted device 20 controls the display content of the display unit 40B. The control process is performed by the CPU 21 reading the information processing program 24A from the storage unit 24, expanding it into the RAM 23, and executing it. As an example, the control process shown in FIG. 3 is performed periodically while the vehicle 10 is traveling.

[0039] 3, the CPU 21 acquires the detection information detected by the detection unit 40A, and then the CPU 21 proceeds to step S11.

[0040] In step S11, the CPU 21 displays the vehicle image 50 and the road image 52 on the display unit 40B based on the detection information acquired in step S10. Specifically, the CPU 21 displays the vehicle image 50 in a position on the road image 52 that reflects the actual position of the vehicle 10. For example, if the vehicle 10 is traveling close to the left side of the road, the CPU 21 displays the vehicle image 50 at the left edge of the road image 52. Then, the CPU 21 proceeds to step S12.

[0041] In step S12, the CPU 21 determines whether the display condition is met based on the detection information acquired in step S10. If the CPU 21 determines that the display condition is met (step S12: YES), the CPU 21 proceeds to step S13. On the other hand, if the CPU 21 does not determine that the display condition is met (step S12: NO), the CPU 21 terminates the control process. In this embodiment, the CPU 21 determines that the display condition is met when the detection unit 40A detects an object present in the traveling direction of the vehicle 10, the positional relationship between the vehicle 10 and the object is in a predetermined state, and it is expected that the object will cross the road. Here, the positional relationship between the vehicle 10 and the object is in a predetermined state when the relative time to collision (TTC), which is calculated by dividing the relative speed between the vehicle 10 and the object, is equal to or less than a predetermined value. The CPU 21 calculates the TTC between the vehicle 10 and the object using the distance and relative speed between the vehicle 10 and the object as detection results of the millimeter wave sensor, which is the detection unit 40A. In addition, when it is assumed that the object is crossing the road, the actual orientation of the object is toward the road. CPU 21 uses a known image recognition technique to determine the orientation of the object's body from the image captured by the camera serving as detection unit 40A.

[0042] In step S13, CPU 21 displays object image 54 on display unit 40B based on the detection information acquired in step S10. Specifically, CPU 21 displays object image 54 at a position that reflects the actual position of the object on road image 52. For example, if the actual position of the object is outside the road, CPU 21 displays object image 54 at a position that indicates outside the road on road image 52. Then, CPU 21 ends the control process.

[0043] Fig. 4 is a second flowchart showing the flow of the control process. As an example, the control process shown in Fig. 4 is executed periodically when the object image 54 is displayed on the display unit 40B.

[0044] 4, the CPU 21 acquires the detection information detected by the detection unit 40A, and then the CPU 21 proceeds to step S21.

[0045] In step S21, the CPU 21 determines whether or not the object is not expected to cross the road based on the detection information acquired in step S20. If the CPU 21 determines that the object is not expected to cross the road (step S21: YES), the process proceeds to step S22. On the other hand, if the CPU 21 does not determine that the object is not expected to cross the road (step S21: NO), the control process ends. If the object is not expected to cross the road, the actual orientation of the object outside the road is not facing the road. The CPU 21 uses known image recognition technology to determine the position and body orientation of the object from an image captured by a camera, which is the detection unit 40A.

[0046] In step S22, the CPU 21 erases the object image 54 from the display unit 40B, and then the CPU 21 ends the control process.

[0047] Next, a display example displayed on the display unit 40B as a result of the control process shown in FIG. 3 or 4 being performed by the vehicle-mounted device 20 will be described.

[0048] Fig. 5 is a first explanatory diagram showing an example of a display displayed on the display unit 40B while the vehicle 10 is traveling. Specifically, Fig. 5 shows an example of a display on the display unit 40B when the vehicle 10 is traveling on a straight road.

[0049] 5 displays a vehicle image 50 showing a vehicle 10 traveling in the direction of arrow S, and a road image 52 showing a straight road along which the vehicle 10 is traveling. As an example, the vehicle image 50 is a substantially rectangular shape. In addition, in the road image 52, the area between two straight lines is inside the road, and the area outside the two straight lines is outside the road.

[0050] 5 displays an object image 54 in addition to the vehicle image 50 and the road image 52, in response to the CPU 21 determining that the display conditions are met. The object image 54 is displayed on the forward side of the vehicle 10 in the traveling direction, outside the road to the right of the vehicle image 50 in the drawing, reflecting the actual position of the object.

[0051] As an example, the object image 54 is configured with a circular frame 54A and an icon 54B displayed within the frame 54A and indicating the type of object. In this embodiment, a plurality of types of icons 54B are provided, and an icon 54B selected by the CPU 21 based on the acquired detection information is displayed within the frame 54A. In FIG. 5, an icon 54B indicating a pedestrian is displayed.

[0052] Here, CPU 21 displays object image 54 that reflects the actual orientation of the object. As an example, display unit 40B shown in Fig. 5 displays the body of a pedestrian represented by icon 54B facing the road, because CPU 21 has determined from the detection information that the body of the object is facing the road (e.g., left side in the figure).

[0053] Fig. 6 is a second explanatory diagram showing a display example displayed on display unit 40B while vehicle 10 is traveling. Specifically, Fig. 6 shows a display example on display unit 40B after a predetermined time has elapsed since the display example shown in Fig. 5 was displayed.

[0054] Here, the CPU 21 changes the vehicle image 50 and the road image 52 in accordance with changes in the actual position of the vehicle 10. As a result, in Fig. 6, the position of the vehicle image 50 changes from Fig. 5 to the upper side in the figure as the vehicle 10 travels straight along a straight road. Furthermore, although not shown, when the vehicle 10 enters a curved road from a straight road, the CPU 21 changes the two straight lines in the road image 52 into two curved lines.

[0055] Furthermore, CPU 21 changes the position of object image 54 in accordance with changes in the actual position of the object. For example, although not shown, when an object enters a road, CPU 21 displays object image 54 within the road indicated by road image 52.

[0056] 6, CPU 21 erases object image 54 from display unit 40B upon determining that the object is no longer expected to cross the road. As an example, CPU 21 erases object image 54 because it has determined from the image captured by the camera serving as detection unit 40A that the object's body is not facing the road.

[0057] As described above, in the vehicle-mounted device 20, the CPU 21, as a function of the control unit 21B, displays the object image 54 at a position that reflects the actual position of the object on the road image 52 when an object is detected by the detection unit 40A, the positional relationship between the vehicle 10 and the object is in a predetermined state, and it is expected that the object will cross the road. As a result, the vehicle-mounted device 20 displays the object image 54 on the condition that it is expected that the object will cross the road, and therefore it is possible to display content related to the possibility that the object will actually cross the road on the in-vehicle display.

[0058] Furthermore, when an object is expected to cross the road, the actual orientation of the object is facing the road. As a result, the vehicle-mounted device 20 can determine the possibility that the object will cross the road based on the actual orientation of the object, and control whether or not to display the object image 54.

[0059] Furthermore, in the vehicle-mounted device 20, the CPU 21, as a function of the control unit 21B, displays an object image 54 that reflects the real orientation of the object. This allows the vehicle-mounted device 20 to allow the occupant to grasp the real orientation of the object by looking at the display unit 40B.

[0060] Furthermore, in the vehicle-mounted device 20, the CPU 21, as a function of the control unit 21B, erases the object image 54 from the display unit 40B when it is no longer expected that the object will cross the road after displaying the object image 54. This allows the vehicle-mounted device 20 to reduce the amount of information displayed on the display unit 40B when it is no longer expected that the object will cross the road.

[0061] Furthermore, in the vehicle-mounted device 20, the CPU 21, as a function of the control unit 21B, displays the vehicle image 50 at a position that reflects the actual position of the vehicle 10 on the road image 52. This allows the occupant of the vehicle-mounted device 20 to understand the positional relationship between the vehicle 10 and an object by looking at the display unit 40B.

[0062] (others) In the above embodiment, the in-vehicle device 20 is an example of an information processing device, but the present invention is not limited thereto. An external device such as a server not mounted on the vehicle 10 may also be an example of an information processing device, or a combination of the in-vehicle device 20 and the external device may also be an example of an information processing device. For example, when the combination of the in-vehicle device 20 and the external device is an example of an information processing device, at least some of the functional configurations of the CPU 21 of the in-vehicle device 20 shown in Fig. 2 may be performed by the CPU of the external device. In this case, the control process shown in Fig. 3 or 4 is executed by one processor of the CPU 21 of the in-vehicle device 20 or the CPU of the external device, or by a combination of multiple processors of the CPU 21 of the in-vehicle device 20 and the CPU of the external device.

[0063] In the above embodiment, the detection unit 40A includes a camera, a vehicle speed sensor, a millimeter wave sensor, and a GPS device, but the configuration of the detection unit 40A is not limited to this. For example, the detection unit 40A includes a millimeter wave sensor as a detection device that detects an object present in the traveling direction of the vehicle 10. However, instead of or in addition to the millimeter wave sensor, the detection unit 40A may include a LIDAR (Laser Imaging Detection and Ranging) to detect an object.

[0064] In the above embodiment, the display unit 40B is provided on the meter panel, but the location of the display unit 40B inside the vehicle is not limited to this. For example, the display unit 40B may be provided on the instrument panel or configured as a HUD that projects an image onto the display surface of the front windshield.

[0065] In the above embodiment, the display unit 40B is a display device provided in the vehicle 10, but this is not limiting. A portable terminal such as a smartphone of a passenger may be installed inside the vehicle and used as the display unit 40B.

[0066] In the above embodiment, the case where the positional relationship between the vehicle 10 and the object is in a predetermined state is defined as the case where the TTC between the vehicle 10 and the object is equal to or less than a predetermined value, but this is not limiting. For example, other cases, such as the case where the distance between the vehicle 10 and the object is equal to or less than a predetermined value, may also be defined as the case where the positional relationship between the vehicle 10 and the object is in a predetermined state.

[0067] In the above embodiment, the case where an object is expected to cross the road is the case where the actual orientation of the object is facing the road, but this is not limited to this. For example, the case where an object is expected to cross the road may be other cases, such as the case where the object is raising its hand.

[0068] In the above embodiment, the actual orientation of the object reflected in the object image 54 is determined appropriately depending on the type of object. For example, in the above embodiment, the object is a pedestrian, and the orientation of the pedestrian's body is reflected in the object image 54. However, if the object is a bicycle, the orientation of the bicycle's tires may be reflected in the object image 54.

[0069] In the above embodiment, the case where it is not expected that an object will cross the road is a case where an object outside the road is not actually facing the road, but this is not limited to this. For example, the case where it is not expected that an object will cross the road may be another case, such as a case where an object outside the road remains in place for a predetermined period of time or more.

[0070] In the above embodiment, the control processing executed by the CPU 21 after reading the software (program) may be executed by various processors other than a CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and application-specific integrated circuits (ASICs), which are dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processing. The control processing may be executed by one of these processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0071] In the above embodiment, the information processing program 24A is pre-stored (installed) in the storage unit 24, but the present invention is not limited to this. The information processing program 24A may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 24A may also be downloaded from an external device via a network. [Explanation of symbols]

[0072] 10 vehicles 20 In-vehicle device (information processing device) 21B Control section 40A detection unit 40B Display section 50 vehicle images 52 Road Images 54 Object images

Claims

1. a control unit that, when a detection unit detects an object present in the traveling direction of a vehicle traveling on a road, the positional relationship between the vehicle and the object is in a predetermined state, and it is expected that the object will cross the road, displays an object image of the object at a position that reflects the actual position of the object in a road image that shows the road displayed on a display unit inside the vehicle; the control unit, after displaying the object image, erases the object image from the display unit when it is no longer expected that the object will cross the road. Information processing device.

2. When the object is expected to cross the road, the actual orientation of the object faces the road. The information processing device according to claim 1 .

3. the control unit displays the object image reflecting the actual orientation of the object. The information processing device according to claim 2 .

4. the control unit displays a vehicle image showing the vehicle at a position on the road image that reflects the actual position of the vehicle. The information processing device according to claim 1 .

Images