Information processing device, information processing method and information processing system

Abstract

The present technology relates to an information processing device, an information processing method, and an information processing system, which enable a user to easily identify the position of a sensor in a poor state among sensors mounted on a vehicle. Position-related information related to a relative position or direction with respect to a vehicle is obtained, and a composite image is displayed, wherein a poor state image indicating the position of a sensor in a poor state among sensors mounted on a vehicle is composited onto a vehicle image reflecting the vehicle based on the position-related information. The present technology can be applied, for example, to enable a user to identify a sensor in a poor state among sensors mounted on a vehicle.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and an information processing system, and in particular to an information processing device, an information processing method, and an information processing system that enable a user to easily identify the position of a sensor that is in a bad condition among sensors mounted on a vehicle, for example. Background Art

[0002] In recent years, in order to assist driving such as automatic braking, many sensors such as cameras have gradually been installed on the outside of four-wheeled vehicles.

[0003] Dirt easily adheres to the sensors installed outside the vehicle, and the dirt adhered to the sensors will hinder the sensing of the sensors, so the dirt adhered to the sensors needs to be cleaned.

[0004] For example, Patent Document 1 proposes a technology for notifying a user of the contamination status of a sensor by displaying an image captured by a camera serving as a sensor on a portable terminal.

[0005] Prior art literature

[0006] Patent Document 1: Japanese Patent Publication No. 2012-106557

[0007] When an image captured by a camera as a sensor is displayed on a portable terminal, the user can recognize that there is dirt on the camera, but it is difficult to recognize the location (position on the car) where the camera with dirt is mounted. Furthermore, when the location of the camera with dirt is unknown, it is difficult for the user to clean the dirt attached to the sensor. Summary of the invention

[0008] In view of such a situation, an object of the present technology is to enable a user to easily recognize the position of a sensor in a vehicle that is in a poor condition due to dirt or the like.

[0009] The information processing device of the present technology includes: a position-related information acquisition unit, which acquires position-related information related to the relative position or direction relative to a vehicle; and a display control unit, which displays a composite image, wherein the composite image is an image in which an image indicating the position of a sensor in a poor state among the sensors mounted on the vehicle is synthesized onto a vehicle image reflecting the vehicle based on the position-related information.

[0010] The information processing method of the present technology includes: obtaining position-related information related to the relative position or direction of a vehicle; and displaying a composite image, wherein the composite image is an image of a poor condition that is synthesized, based on the position-related information, onto a vehicle image reflecting the vehicle, representing the position of a sensor in a poor condition among the sensors mounted on the vehicle.

[0011] The information processing system of the present technology includes: a sensor mounted on a vehicle; a poor condition detection unit for detecting a poor condition of the sensor; a display unit for displaying an image; a position-related information acquisition unit for acquiring position-related information related to a relative position or direction relative to the vehicle; and a display control unit for displaying a composite image, wherein the composite image is an image in which a poor condition image representing a position of a sensor mounted on the vehicle that is in a poor condition is synthesized onto a vehicle image reflecting the vehicle based on the position-related information.

[0012] In the information processing device, information processing method and information processing system of the present technology, position-related information related to the relative position or direction with respect to the vehicle is obtained, and a composite image is displayed. The composite image is an image in which a poor condition image representing the position of a sensor in a poor condition among the sensors mounted on the vehicle is synthesized onto a vehicle image reflecting the vehicle based on the position-related information.

[0013] In addition, the information processing device may be an independent device or an internal component constituting a device.

[0014] In addition, the information processing device can be realized by executing the program on a computer. The program can be provided by being transmitted via a transmission medium or stored in a storage medium.

[0015] According to the present technology, a user can easily recognize the position of a sensor that is in a bad condition among sensors mounted on a vehicle.

[0016] In addition, the effects described here are not limitative, and any effects described in the present invention may be used. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing an overview of one embodiment of a vehicle control system as an information processing system to which the present technology is applied.

[0018] Figure 2 It is a block diagram showing an example of the electrical configuration of the vehicle 1 .

[0019] Figure 3 2 is a block diagram showing an example of the electrical configuration of the external terminal 40 .

[0020] Figure 4 A diagram for explaining AR (Augmented Reality) display as a first display example of a composite image.

[0021] Figure 5 A diagram for explaining VR (Virtual Reality) display as a second display example of a composite image.

[0022] Figure 6 A diagram for explaining a bird's-eye view display as a third display example of a composite image.

[0023] Figure 7 1 is a diagram for explaining a control example of the equipment of the vehicle 1 in synchronization with the notification process.

[0024] Figure 8 1 is a flowchart for explaining the notification process of the vehicle control device 10 .

[0025] Fig. 9 1 is a flowchart for explaining the notification process of the external terminal 40 .

[0026] Fig.10 This is a diagram showing a display example of a poor condition image including an image showing the location of dirt.

[0027] Fig.11 The sensor 30 having a light emitting portion i A cross-sectional view and a plan view of a configuration example.

[0028] Fig.12 This is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied.

[0029] Description of Reference Numerals

[0030] 1 vehicle, 10 vehicle control device, 201, 202 vehicle-mounted lights, 301 to 305 sensors, 40 off-vehicle terminal, 111 display unit, 112 input unit, 113 information processing unit, 114, 115, 141 communication unit, 142 information processing unit, 143 input unit, 144 camera, 145 display unit, 151 image processing unit, 152 position-related information acquisition unit, 153 image generation unit, 154 display control unit, 155 control unit, 200 optical sensor, 201 lens, 202 substrate, 203 light-emitting unit, 901 bus, 902 CPU, 903 ROM, 904 RAM, 905 hard disk, 906 output unit, 907 input unit, 908 communication unit, 909 driver, 910 input / output interface, 911 removable storage medium DETAILED DESCRIPTION

[0031] <One embodiment of an information processing system to which the present technology is applied>

[0032] Figure 1 This is a diagram showing an overview of one embodiment of a vehicle control system as an information processing system to which the present technology is applied.

[0033] exist Figure 1 In the embodiment, the vehicle control system includes a vehicle 1 and an off-vehicle terminal 40 .

[0034] The vehicle 1 is, for example, a four-wheeled passenger car, and has a built-in vehicle control device 10. The vehicle 1 is equipped with vehicle-mounted lights 201 and 202 as light-emitting parts equipped on the vehicle 1 such as headlights, taillights, LEDs (Light Emitting Diodes) and headlights, and sensors 301, 302, 303, 304, and 305 such as cameras, etc.

[0035] The vehicle control device 10 receives the signal from the sensor 30 i (In this embodiment, i=1, 2, 3, 4, 5) The vehicle control device 10 performs various controls for driving assistance, etc. In addition, the vehicle control device 10 controls turning on and off the vehicle-mounted lamps 201 and 202, etc.

[0036] Furthermore, the vehicle control device 10 and the external terminal 40 perform wireless communication to exchange various information.

[0037] For example, the vehicle control device 10 receives information from the sensor 30 i Sensor information, detection sensor 30 i The state of the sensor is not good, and the sensor 30 including the state of the sensor is not good i The vehicle control device 10 receives the position-related information and the cleaning completion notification sent from the external terminal 40, such as the position of the vehicle 1 and the like.

[0038] The vehicle control device 10 controls the sensor 30 in a poor state according to the position-related information from the external terminal 40. i Car lights 20 j (In this embodiment, j=1, 2) Turn on the light. In addition, the vehicle control device 10 turns on the vehicle-mounted light 20 according to the cleaning completion notification from the off-vehicle terminal 40. j Turn off the lights.

[0039] The off-vehicle terminal 40 is an information processing device such as a portable terminal that can be carried by a user (eg, the driver of the vehicle 1 ) such as a smartphone, and performs wireless communication with the vehicle control device 10 to exchange various information.

[0040] For example, the off-board terminal 40 obtains position-related information related to the relative position or direction of the off-board terminal 40 with respect to the vehicle and transmits it to the vehicle control device 10. In addition, the off-board terminal 40 uses the sensor 30 as a sensor according to the user's operation. i A cleaning completion notification indicating that the cleaning of the poor state, such as dirt, is completed is sent to the vehicle control device 10.

[0041] Furthermore, the off-board terminal 40 receives the state failure information transmitted from the vehicle control device 10. The off-board terminal 40 generates a sensor 30 indicating a state failure based on the state failure information from the vehicle control device 10. i The bad condition image of the position is obtained, and the bad condition image is synthesized on the vehicle image reflecting the vehicle 1 based on the position association information. Then, the off-vehicle terminal 40 displays the synthesized image in which the bad condition image is synthesized on the vehicle image on the display unit 145.

[0042] exist Figure 1 In the example, a composite image in which the poor state image is composited (overlapped) onto the vehicle image is displayed on the off-vehicle terminal 40, indicating that the sensor 30 on the vehicle 1 is in a poor state. i A blackened circular image at the position of the camera and a dialog box image containing the message "This camera is dirty with mud" are used as poor status images.

[0043] As above, in Figure 1 In the vehicle control system of the present invention, the vehicle image reflecting the vehicle 1 is synthesized with the sensor 30 mounted on the vehicle 1 indicating that the state is not good. i Therefore, the user can easily identify the sensor 301 to 305 mounted on the vehicle 1 that is in a bad state by carrying the off-vehicle terminal 40 outside the vehicle and viewing the synthesized image. i location.

[0044] Here, a sensor 30 is mounted outside the vehicle. i In the vehicle 1, the vehicle control device 10 detects that the sensor 30 i If the sensor is dirty or in a bad condition, the user needs to go outside the vehicle to clean the sensor 30. i For example, the vehicle 1 is not equipped with an automatic cleaning sensor 30 using water, cleaning fluid, air, etc. i If the cleaning system is installed, or even if the cleaning system is installed, but the cleaning system fails to clean the sensor 30 i If there is dirt on the sensor, the user needs to go outside the vehicle to clean the sensor 30 i .

[0045] Clean the sensor 30 when the user goes outside the car i When the user needs to identify the dirty sensor 30 i The position on the vehicle 1 and the dirt condition are detected as required. i In the case where the dirt information is only displayed on the in-vehicle display of the vehicle 1, the user needs to look at the display and remember the sensor 30 with dirt. iThe location and condition of the dirt, and go outside the car to find the sensor with dirt 30 i In addition, after cleaning the dirty sensor 30 i Afterwards, the user needs to return from outside the vehicle to the inside of the vehicle to confirm whether the vehicle control device 10 determines that the sensor 30 i The dirt is removed and the sensor 30 i Is there a problem? That is, is the sensor 30 i The state of dirt is not detected.

[0046] Furthermore, it is envisioned that in the future, when the vehicle 1 is realized as an autonomous vehicle, in order to improve the robustness of sensing, the sensor 30 mounted on the vehicle 1 i The number of sensors increases, and the sensor 30 i It is also expected that in the era of popularization of self-driving cars, there will be more opportunities to use vehicles that are not owned by oneself, such as automatically dispatched taxis and shared cars. In this way, in the case where the vehicle 1 is not owned by oneself, it is expected that the user who uses the vehicle 1 will not be able to master the sensor 30. i Position and sensor 30 on the vehicle 1 being carried i The sensor with dirt is found. i It will be time-consuming.

[0047] Therefore, the following method can be considered: for example, by placing the sensor 30 close to the dirt i The position of the lamp (for example, the vehicle lamp 20 j etc.) or near a dirty sensor 30 i The sensor emits a sound at the location of the dirt, allowing the user to identify the dirty sensor 30 i location.

[0048] However, as described above, the sensor 30 mounted on the vehicle 1 i When the number of sensors increases, there are multiple sensors 30 near the lamp. i In this case, even if the sensor 30 is close to the dirt i The light at the position of the sensor flashes, which makes it difficult for the user to easily identify the sensor 30 that is dirty. i location.

[0049] Therefore, in Figure 1 In the vehicle control system of the present invention, as described above, a composite image is displayed on the portable off-vehicle terminal 40, wherein the composite image is a composite image of the sensor 30 mounted on the vehicle 1 indicating a poor state. iThe image of the poor condition of the position is synthesized onto the image reflecting the vehicle 1. In this case, the user can easily identify the poor condition sensor 301 to 305 having dirt or the like among the sensors 301 to 305 mounted on the vehicle 1 by carrying the off-vehicle terminal 40 outside the vehicle and viewing the synthesized image. i location.

[0050] In addition, as the vehicle 1, in addition to a four-wheeled passenger car, agricultural vehicles such as tractors, construction vehicles such as excavators, tanks, and other arbitrary vehicles can also be used. In addition, instead of the vehicle 1, an unmanned aerial vehicle, an airplane, a robot, and other arbitrary devices equipped with sensors can be used.

[0051] Furthermore, the sensor 30 i The sensor is not in good condition. In addition to the adhesion of dirt such as garbage, insects, mud, etc. (obstructions that hinder sensing), the sensor 30 i Failure and any other poor state. i The condition is not good and dirt (adhesion) is used.

[0052] In addition, as the sensor 30 i , cameras (including stereo cameras), ranging sensors (ToF (Time of Flight) sensors, LIDAR (Light Detection and Ranging) sensors, millimeter wave sensors, etc.), acceleration sensors, temperature sensors, and any other sensors can be used.

[0053] Moreover, in Figure 1 In the embodiment, although five sensors 301 to 305 are mounted on the vehicle 1, the sensor 30 mounted on the vehicle 1 i The number is not limited to 5.

[0054] As a method of (bidirectional) wireless communication between the vehicle control device 10 and the external terminal 40 , for example, any method such as a wireless LAN (Local Area Network) and Bluetooth (registered trademark) may be adopted.

[0055] Furthermore, the wireless communication between the vehicle control device 10 and the external terminal 40 may be performed point-to-point or via a network such as the Internet.

[0056] In addition, the off-vehicle terminal 40 may be implemented by, for example, a dedicated terminal, or may be implemented by executing an application on a general-purpose information processing device such as a smartphone. Furthermore, the off-vehicle terminal 40 may also be configured as a detachable component of the vehicle control device 10 .

[0057] <Electrical Configuration Example of Vehicle 1>

[0058] Figure 2 Yes means Figure 1 A block diagram of an example of the electrical configuration of the vehicle 1 is shown.

[0059] like Figure 1 As described in FIG. 1 , the vehicle 1 includes a vehicle control device 10 , vehicle-mounted lights 201 and 202 , and sensors 301 to 305 such as cameras.

[0060] The vehicle control device 10 includes a display unit 111 , an input unit 112 , an information processing unit 113 , and communication units 114 and 115 .

[0061] The display unit 111 displays various information under the control of the information processing unit 113 .

[0062] The input unit 112 is operated by the user, and supplies input information corresponding to the user's operation to the information processing unit 113. When the input information is supplied from the input unit 112 to the information processing unit 113, the information processing unit 113 performs processing corresponding to the input information from the input unit 112.

[0063] In addition, the display unit 111 and the input unit 112 may be integrally configured by using, for example, a touch panel or the like.

[0064] The information processing unit 113 performs control of each component constituting the vehicle control device 10 and other various processes.

[0065] The communication unit 114 and the vehicle lamp 20 j 、Sensor 30 i Wired communication between etc.

[0066] For example, the communication unit 114 transmits the vehicle light control information supplied from the information processing unit 113 to the vehicle light 20. j . Car lights 20 j The light is turned on, flashed, or turned off, or the direction and intensity of light emission are changed according to the light control information from the communication unit 114 .

[0067] In addition, for example, the communication unit 114 receives the data from the sensor 30 i The transmitted sensor information is supplied to the information processing unit 113. i The sensor information includes: i The sensing result (for example, a photographic image having RGB (Red, Green, Blue) obtained by a camera as pixel values ​​and a distance image having a distance obtained by a distance measuring sensor as pixel values) and the like) as the sensor 30 iThe information processing unit 113 receives information from the sensor 30 of the communication unit 114. i The sensor information is detected as sensor 30 i The condition of the sensor is poor, the dirt is 30 i The information processing unit 113 generates information indicating whether there is dirt or not. i The poor status information includes information indicating that the sensor 30 i The sensor position information of the position (on the vehicle 1 ) is supplied to the communication unit 115 and transmitted to the external terminal 40 .

[0068] The communication unit 115 performs wireless communication with the external terminal 40 and the like.

[0069] For example, the communication unit 115 receives the sensor 30 supplied from the information processing unit 113. i The poor status information is sent to the off-vehicle terminal 40.

[0070] Furthermore, for example, the communication unit 115 receives the position-related information and the cleaning completion notification transmitted from the external terminal 40 , and supplies the information to the information processing unit 113 .

[0071] The position-related information sent from the off-vehicle terminal 40 is information related to the relative position or direction of the off-vehicle terminal 40 with respect to the vehicle 1, and the information processing unit 113 calculates the relative position of the off-vehicle terminal 40 with respect to the vehicle 1 (hereinafter simply referred to as the relative position) based on the position-related information. That is, when the position-related information indicates the relative position of the off-vehicle terminal 40 with respect to the vehicle 1, the information processing unit 113 adopts the relative position indicated by the position-related information as the relative position of the off-vehicle terminal 40. In addition, when the position-related information indicates the relative direction of the off-vehicle terminal 40 with respect to the vehicle 1, the information processing unit 113 obtains, for example, the position of the relative direction indicated by the position-related information at a specified distance from the vehicle 1 as the relative position of the off-vehicle terminal 40. Then, the information processing unit 113 controls the vehicle lights 20 based on the relative position of the off-vehicle terminal 40, etc. j The vehicle light control information is supplied to the communication unit 114 and sent to the vehicle light 20 j , thereby controlling the vehicle lights 20 j .

[0072] The cleaning completion notification sent from the off-vehicle terminal 40 is a dirt sensor 30. i When the information processing unit 113 receives the cleaning completion notification from the communication unit 115, the information processing unit 113 repeats: i The sensor information is detected as sensor 30i The poor condition information is generated by detecting the presence or absence of dirt in the poor condition.

[0073] <Example of electrical configuration of the external terminal 40>

[0074] Figure 3 Yes means Figure 1 A block diagram of an example of the electrical configuration of the off-vehicle terminal 40.

[0075] The external terminal 40 includes a communication unit 141 , an information processing unit 142 , a camera 144 , and a display unit 145 .

[0076] The communication unit 141 performs wireless communication with the vehicle control device 10 and the like.

[0077] For example, the communication unit 141 transmits the position-related information and the cleaning completion notification supplied from the information processing unit 142 (control unit 155) to the vehicle control device 10. In addition, the communication unit 141 receives the poor status information transmitted from the vehicle control device 10 (communication unit 115) and supplies it to the information processing unit 142 (control unit 155).

[0078] The information processing unit 142 performs control of each component constituting the vehicle control device 10 and other various processes.

[0079] The information processing unit 142 includes an image processing unit 151 , a position-related information acquisition unit 152 , an image generation unit 153 , a display control unit 154 , and a control unit 155 .

[0080] The image captured by the camera 144 is supplied to the image processing unit 151. When the image captured by the camera 144 is a vehicle image reflecting the vehicle 1, the image processing unit 151 supplies the vehicle image, that is, the captured image, to the position-related information acquisition unit 152 and, if necessary, to the display control unit 154.

[0081] Furthermore, the image processing unit 151 generates a CG (Computer Graphics) image of the vehicle 1 based on the photographic image which is the vehicle image, and supplies the vehicle image as the CG image to the display control unit 154 .

[0082] The position-related information acquisition unit 152 uses the vehicle image, i.e., the photographic image, etc., from the image processing unit 151, and obtains the position-related information related to the relative position or direction of the off-vehicle terminal 40 relative to the vehicle 1 by inference, for example based on SLAM (Simultaneous Localization and Mapping), and supplies it to the image generation unit 153 and the control unit 155.

[0083] Here, the position-related information acquisition unit 152 acquires the position-related information of the external terminal 40 based on SLAM, but the position-related information of the external terminal 40 may be acquired by any other method.

[0084] For example, the position-related information of the off-vehicle terminal 40 can be inferred from the shape of the vehicle 1 reflected in the vehicle image, i.e., the photographic image. In addition, a predetermined mark can be set on the vehicle 1, and the position-related information of the off-vehicle terminal 40 can be inferred based on the mark on the vehicle 1 reflected in the vehicle image, i.e., the photographic image. Moreover, a highly directional radio wave is transmitted from the vehicle 1, and the position-related information of the off-vehicle terminal 40 can be inferred based on the reception status of the radio wave in the off-vehicle terminal 40. In addition, when the vehicle 1 and the off-vehicle terminal 40 are equipped with a GPS (Global Positioning System) function, the position-related information of the off-vehicle terminal 40 can be inferred based on the positions of the vehicle 1 and the off-vehicle terminal 40 obtained by the GPS function. Moreover, when the sensor 30 of the vehicle 1 i When infrared rays for distance measurement are emitted or when an LED emitting visible light is provided on the vehicle 1 , the position-related information of the external terminal 40 can be estimated based on the reception condition of the infrared rays or visible light in the external terminal 40 .

[0085] The above is a method for calculating the position-related information in the off-vehicle terminal 40, but the position-related information of the off-vehicle terminal 40 may be calculated in the vehicle 1 instead of in the off-vehicle terminal 40. Then, the position-related information calculated in the vehicle 1 is sent from the vehicle 1 to the off-vehicle terminal 40, and the off-vehicle terminal 40 receives and obtains the position-related information from the vehicle 1.

[0086] In the vehicle 1, for example, the sensor 30 is a camera. i , a user carrying the off-vehicle terminal 40 and outside the vehicle is photographed, and by identifying the user reflected in the photographic image obtained by the photography, the user's position-related information can be calculated as the position-related information of the off-vehicle terminal 40. In addition, for example, a human sensor is used as a sensor 30 i According to the sensing result of the human sensor, the user's position-related information can be calculated as the position-related information of the off-vehicle terminal 40. In addition, a highly directional radio wave can be sent from the off-vehicle terminal 40, and the position-related information of the off-vehicle terminal 40 can be calculated (estimated) according to the reception condition of the radio wave in the vehicle 1. In addition, when the off-vehicle terminal 40 can emit infrared rays or visible light, the position-related information of the off-vehicle terminal 40 can be calculated according to the light reception condition of the infrared rays or visible light in the vehicle 1.

[0087] Furthermore, by combining a plurality of the above-described methods to calculate the position-related information of the external terminal 40 , it is possible to improve robustness.

[0088] The position information of the external terminal 40 is supplied from the position information acquisition unit 152 to the image generation unit 153, and the poor condition information transmitted from the vehicle control device 10 is supplied from the control unit 155 to the image generation unit 153. The image generation unit 153 generates an image indicating that the sensor 30 is dirty based on the position information from the position information acquisition unit 152 and the poor condition information from the control unit 155. i The image of the poor status of the position is supplied to the display control unit 154.

[0089] The display control unit 154 performs display control to supply various images to the display unit 145 for display. In addition, the display control unit 154 generates a composite image by synthesizing the poor condition image from the image generation unit 153 with the vehicle image of the photographic image or the CG vehicle image supplied from the image processing unit 151. Then, the display control unit 154 supplies the composite image to the display unit 145 for display.

[0090] The control unit 155 performs control of each component of the information processing unit 142 and other various processes.

[0091] For example, the control unit 155 generates a cleaning completion notification based on the operation of the input unit 143, and transmits it to the communication unit 141. In addition, the control unit 155 transmits the position-related information supplied from the position-related information acquisition unit 152 to the communication unit 141. Furthermore, the control unit 155 supplies the poor state information transmitted from the vehicle control device 10 and supplied from the communication unit 141 to the image generation unit 153.

[0092] The input unit 143 is operated by the user, and supplies input information corresponding to the user's operation to the control unit 155 .

[0093] The camera 144 captures an image of a subject such as the vehicle 1 , for example, and supplies a captured image obtained by the capture to the image processing unit 151 .

[0094] The display unit 145 displays an image such as a synthesized image supplied from the display control unit 154 in accordance with the display control of the display control unit 154 .

[0095] In addition, the input unit 143 and the display unit 145 may be integrally configured using, for example, a touch panel or the like.

[0096] Depend on Figure 2 Vehicle 1 and Figure 3 The off-board terminal 40 is composed of Figure 1In the vehicle control system of the present invention, for example, when the vehicle control device 10 of the vehicle 1 is started and the vehicle 1 is parked in a safe place, the user is informed of the sensor 30. i As the predetermined time, for example, when the user approaches the vehicle 1 (when entering the specified range of the vehicle 1), when the user unlocks the lock (key) of the vehicle 1 from outside the vehicle, when the user starts the engine of the vehicle 1, etc.

[0097] In addition, in order to make sufficient power available in the vehicle control device 10, the notification process is preferably performed when the engine of the vehicle 1 is started. However, when the vehicle 1 is an electric car, since sufficient power can be used in the vehicle control device 10 even if the engine of the vehicle 1 is not started, the notification process may be performed when the engine of the vehicle 1 is not started.

[0098] Furthermore, even if a composite image is displayed on the off-vehicle terminal 40 while the vehicle 1 is traveling, the user driving the vehicle 1 cannot see the composite image displayed on the off-vehicle terminal 40 , so the composite image is not displayed on the off-vehicle terminal 40 while the vehicle 1 is traveling.

[0099] When the vehicle control device 10 ( Figure 2 ) starts notification processing, the sensor 30 i The sensor 30 is obtained as i The sensor 30 detects the dirt condition of the dirt in the bad state and sends it in the sensor information. i The sensor information sent is received by the communication unit 114 and supplied to the information processing unit 113. i The method for obtaining the dirt condition may be any method. i The dirt status of the sensor 30 can be obtained at any time in addition to when the notification process starts. For example, when the vehicle 1 is driving or parked, the dirt status of the sensor 30 can be obtained. i The dirt condition is stored in the vehicle control device 10 and the sensor 30 i Then, in the subsequent notification process, the sensor 30 stored in the memory can be used. i The dirt condition.

[0100] The information processing unit 113 detects the sensor 30 that is in a bad state and is contaminated based on the sensor information (the contamination status included in the sensor information) from the communication unit 114. i (Detection sensor 30 i When the sensor detects dirt, i In the case ofi Information indicating that the sensor 30 is dirty is displayed on the display unit 111. In addition, the information processing unit 113 i The position and condition (status) of the dirt are displayed on the display unit 111.

[0101] In addition, the information processing unit 113 processes the sensor 30 i The information processing unit 113 generates information indicating whether there is dirt or not. i The poor status information includes information indicating that the sensor 30 i The sensor position information of the position is supplied to the communication unit 115 and sent to the external terminal 40.

[0102] The user in the vehicle 1 sees the display on the display unit 111 and recognizes that the sensor 30 i If there is dirt on the sensor, perform a 30 i cleaning.

[0103] Clean the dirty sensor 30 i When the user takes the external terminal 40 out of the vehicle, the user uses the external terminal 40 to check whether the sensor 30 is dirty. i Therefore, the user goes outside the vehicle, points the display unit 145 of the smartphone serving as the off-vehicle terminal 40 toward himself, and takes a photo of the vehicle 1 through the camera 144 provided on the opposite side of the display unit 145 (the back side if the side with the display unit 145 is the front side).

[0104] At the off-board terminal 40 ( Figure 3 ), when the notification process starts, the camera 144 starts taking pictures.

[0105] Thus, the camera 144 captures a photographic image reflecting the vehicle 1 seen from the position of the external terminal 40 (the user's position) and supplies it to the image processing unit 151. The image processing unit 151 supplies the photographic image from the camera 144 as a vehicle image to the position-related information acquisition unit 152.

[0106] Furthermore, the image processing unit 151 supplies the photographic image as the vehicle image from the camera 144 to the display control unit 154. Alternatively, the image processing unit 151 generates a CG of the vehicle 1 as seen from the position of the off-vehicle terminal 40 based on the photographic image as the vehicle image from the camera 144, and supplies the vehicle image as the CG to the display control unit 154.

[0107] The display control unit 154 displays the vehicle image from the image processing unit 151 on the display unit 145 .

[0108] On the other hand, the position-related information acquisition unit 152 uses the photographic image as the vehicle image from the image processing unit 151 to acquire position-related information related to the relative position or direction of the off-vehicle terminal 40 with respect to the vehicle 1 by estimation, and supplies it to the image generation unit 153 and the control unit 155. The position-related information supplied to the control unit 155 is transmitted to the vehicle control device 10 via the communication unit 141, and is used for the vehicle-mounted lamp 20 as needed. j control, etc.

[0109] In addition, at the off-board terminal 40 ( Figure 3 ), the communication unit 141 receives the poor state information sent from the communication unit 115 of the vehicle control device 10, and supplies it to the control unit 155. The control unit 155 supplies the poor state information from the communication unit 141 to the image generation unit 153.

[0110] The image generation unit 153 generates an image of the sensor 30 indicating that there is dirt based on the position-related information from the position-related information acquisition unit 152 and the poor state information from the control unit 155. i The image of the poor status of the position is supplied to the display control unit 154.

[0111] When the display control unit 154 is supplied with the bad state image from the image generation unit 153, it generates a composite image by synthesizing the bad state image from the image generation unit 153 with the vehicle image (photographic image or CG) reflecting the vehicle 1 seen from the position of the off-vehicle terminal 40 supplied from the image processing unit 151. Then, the display control unit 154 supplies the composite image to the display unit 145 for display.

[0112] The synthesized image is a composite image of the sensor 30 indicating a poor state, which is synthesized on the vehicle image reflecting the vehicle 1 as seen from the position of the off-vehicle terminal 40, that is, the position of the user. i Therefore, the user can easily identify the sensor 301 having dirt and being in a bad state among the sensors 301 to 305 mounted on the vehicle 1 by looking at the synthesized image. i That is, the user can view the composite image displayed on the display unit 145 of the off-vehicle terminal 40 without having to view the dirty sensor 30 displayed on the display unit 111 of the vehicle control device 10. i The sensor 30 can be identified as having dirt on it by returning to the vehicle. i As a result, user convenience can be improved.

[0113] Dirty sensor detected 30 i The user is located close to the sensor 30 i And clean up. End of sensor 30 iThe user who wants to clean the equipment operates the input unit 143 in order to notify the end of cleaning. The control unit 155 sends the cleaning end notification to the communication unit 141 according to the operation of the input unit 143.

[0114] The cleaning completion notification sent by the communication unit 141 is received by the communication unit 115 of the vehicle control device 10 and supplied to the information processing unit 113. The information processing unit 113 detects the sensor 30 that is in a bad state and has dirt based on the cleaning completion notification and the sensor information from the communication unit 114. i Then, when the sensor 30 detects dirt i In the case of, repeat the same processing as the above case.

[0115] On the other hand, in the case where the sensor 30 does not detect dirt i In this case, the notification processing of the vehicle control device 10 and the external terminal 40 is terminated.

[0116] In addition, here, the cleaning completion notification is notified to the information processing unit 113 of the vehicle control device 10 by operating the input unit 143 of the off-vehicle terminal 40, but in addition to this, the cleaning completion notification can also be notified to the information processing unit 113 by operating the input unit 112 of the vehicle control device 10.

[0117] In addition, in addition to generating a composite image by synthesizing a poor condition image on a vehicle image reflecting the vehicle 1 as viewed from the position of the off-vehicle terminal 40, the display control unit 154 can also generate a composite image by synthesizing a poor condition image on a bird's-eye view image of the vehicle 1 and display it on the display unit 145.

[0118] <Display example of composite image>

[0119] Figure 4 A diagram for explaining AR (Augmented Reality) display as a first display example of a composite image.

[0120] In the AR display, a composite image is displayed in which the poor condition image is composited with the photographed image which is the vehicle image of the vehicle 1 photographed by the camera 144 from the position of the external terminal 40 .

[0121] exist Figure 4 In the example, the sensor 30 on the vehicle 1 is shown as being dirty. i A circular image of the position and a dialog image recording the information "This camera is dirty with mud" are displayed as a poor condition image, and a composite image in which the poor condition image is synthesized on the photographic image as a vehicle image is displayed on the display unit 145.

[0122] in addition, Figure 4 The composite image displayed on the display unit 145 is shown when the user carrying the external terminal 40 is located behind the vehicle 1 and slightly behind the left side (toward the front) of the vehicle 1 .

[0123] According to the AR display, a vehicle image of the vehicle 1 as seen from the position of the user (the off-vehicle terminal 40 carried by the user) and the sensor 30 indicating that there is dirt are displayed. i The image of the poor condition of the location is displayed, so the user can easily identify the sensor 30 with dirt. i location.

[0124] Figure 5 A diagram for explaining VR (Virtual Reality) display as a second display example of a composite image.

[0125] In the VR display, a composite image is displayed which is a composite image in poor condition synthesized on CG, which is generated based on the vehicle image photographed by camera 144 from the position of external terminal 40 and using a 3D (dimensional dimension) model, etc., as a vehicle image reflecting the vehicle 1 seen from the position of external terminal 40.

[0126] exist Figure 5 In the example, the sensor 30 on the vehicle 1 is shown as being dirty. i A circular image of the position and a dialog image containing the message "This camera is dirty with mud" are displayed as a poor condition image, and a composite image in which the poor condition image is synthesized on the CG as a vehicle image is displayed on the display unit 145.

[0127] In addition, Figure 5 In, with Figure 4 , and represents a composite image displayed on the display unit 145 when the user carrying the external terminal 40 is located behind the vehicle 1 and slightly behind the left side of the vehicle 1 .

[0128] According to the VR display, similarly to the AR display, the vehicle image of the vehicle 1 as viewed from the position of the user (the external terminal 40 carried by the user) and the sensor 30 indicating that there is dirt are displayed. i The image of the poor condition of the location is displayed, so the user can easily identify the sensor 30 with dirt. i location.

[0129] Here, AR display and VR display are similar in that a composite image is displayed by combining the poor condition image with the vehicle image of the vehicle 1 as viewed from the position of the off-vehicle terminal 40. However, the vehicle image displayed in AR is a photographic image taken by the camera 144, while the vehicle image displayed in VR is CG, and AR display and VR display are different in this point.

[0130] Figure 6 A diagram for explaining a bird's-eye view display as a third display example of a composite image.

[0131] In the bird's-eye view display, a composite image is displayed in which the poor condition image and the relative position image indicating the relative position of the external terminal 40 are composited into a vehicle image (hereinafter also referred to as a bird's-eye view image) overlooking the vehicle 1 .

[0132] exist Figure 6 In the example, the sensor 30 on the vehicle 1 is shown as being dirty. i A composite image in which the poor status image and the relative position image are synthesized onto the overhead image is displayed on the display unit 145, and a circular image indicating the relative position of the off-vehicle terminal 40 and information "current position" indicating that its relative position is the user's position are used as relative position images.

[0133] As the bird's-eye view image, for example, an image of a bird's-eye view of the vehicle 1 for use in a parking assistance system or the like can be used.

[0134] According to the bird's-eye view display, a bird's-eye view image of the vehicle 1 and a sensor 30 indicating dirt are displayed. i The poor state image of the position of the sensor 30 and the relative position image indicating the relative position (current position) of the user (the external terminal 40 carried by the user) are displayed, so that the user can easily recognize the dirty sensor 30 i location.

[0135] The AR display, VR display, and bird's-eye view display described above can be switched as needed.

[0136] For example, AR display, VR display, and bird's-eye view display can be switched according to user operations.

[0137] In addition, switching between AR display, VR display, and bird's-eye view display can be performed as follows.

[0138] For example, from the user's position, that is, the position of the off-board terminal 40, the dirty sensor 30 cannot be seen. i In this case, an overhead display can be performed on the off-vehicle terminal 40.

[0139] In addition, from the position of the external terminal 40, the dirty sensor 30 can be seen. i In this case, AR display or VR display can be performed on the external terminal 40.

[0140] In this case, when the dirty sensor 30 is not visible from the position of the off-vehicle terminal 40, i When the user moves to the sensor 30 where dirt can be seen, the bird's-eye view display can be used to encourage the user to move to the sensor 30 where dirt can be seen. i Then, when the user moves to the sensor 30 where dirt can be seen i After the location is located, the user can easily identify the dirty sensor 30 through AR display or VR display. i location.

[0141] In addition, in the off-vehicle terminal 40, for example, AR display or VR display can be performed according to the brightness of the vehicle image photographed by the camera 144. For example, when the average brightness of the vehicle image is above a threshold, AR display is performed, and when the average brightness of the vehicle image is not above the threshold, VR display can be performed.

[0142] For example, at night, the vehicle image photographed by the camera 144 is dark, and the vehicle 1 reflected in the vehicle image is difficult to visually recognize. In this case, even if a composite image is synthesized with a poor state image on the vehicle image, the vehicle 1 reflected in the composite image is difficult to visually recognize.

[0143] Therefore, when the vehicle image photographed by the camera 144 is dark, for example, when the average value of the brightness of the vehicle image is not above the threshold, the vehicle 1 reflected in the composite image can be prevented from being difficult to visually recognize by performing VR display of the CG of the vehicle 1.

[0144] In addition, here, the brightness of the vehicle image photographed by camera 144 (whether the vehicle image is dark) is determined based on the average value of the luminance of the vehicle image, but in addition, for example, the brightness of the vehicle image photographed by camera 144 can also be determined based on the light intensity measured by a illuminance meter.

[0145] In addition, as the poor state image for generating the composite image, in addition to the sensor 30 indicating the presence of dirt, the i The dirty sensor 30 may also be used to obtain an image of the position of the dirty sensor 30. i An arbitrary GUI (Graphical User Interface) that displays the location in an easily visible manner.

[0146] <Control of Equipment of Vehicle 1 Synchronized with Notification Processing>

[0147] Figure 7 1 is a diagram for explaining an example of control of the devices of the vehicle 1 in synchronization with the notification process.

[0148] In the vehicle control device 10, the control of the equipment of the vehicle 1 is performed synchronously with (in parallel with) the notification process so as to assist the user to easily recognize the sensor 30 that is dirty. i location.

[0149] To help users easily identify the dirty sensor 30 i The control of the equipment of the vehicle 1 by the vehicle control device 10 is called auxiliary control. In the vehicle control device 10, auxiliary control can be performed based on the position association information of the off-vehicle terminal 40 sent from the off-vehicle terminal 40 and the surrounding brightness, etc.

[0150] For example, in the vehicle control device 10, according to the surrounding brightness, when the surrounding brightness is dark, Figure 7 As shown, the information processing unit 113 controls the dirt sensor 30 through the communication unit 114. i Nearby vehicle lights 20 j (Dirty sensor 30 i The surrounding vehicle lights 20 j ), thereby making the vehicle-mounted lamp 20 j In this case, for example, at night, due to the dirty sensor 30 i Nearby vehicle lights 20 j Turn on the light so the user can find the sensor with dirt 30 i and auxiliary dirty sensor 30 i cleaning operations.

[0151] Furthermore, in the vehicle control device 10, the communication unit 115 receives the position-related information of the external terminal 40 transmitted from the external terminal 40 and supplies it to the information processing unit 113. The information processing unit 113 calculates the relative position of the external terminal 40 based on the position-related information from the communication unit 115. The information processing unit 113 uses the relative position of the external terminal 40 to determine whether the dirty sensor 30 is visible from the position of the external terminal 40, i.e., the position of the user. i Then, when the ambient brightness is low and the dirty sensor 30 is visible from the position of the external terminal 40, i In the case of the above, the information processing unit 113 can control the sensor 30 with dirt i Nearby vehicle lights 20 j Make it turn on the light.

[0152] In addition, it is possible to determine whether the surroundings are dark based on the brightness (luminance) of the image captured by the camera 144 or the light intensity measured by an illuminometer.

[0153] In addition, the information processing unit 113 can control the vehicle-mounted lamp 20 according to the relative position of the external terminal 40. j Make it flash or turn on the light.

[0154] For example, the user (carrying the off-board terminal 40) moves away from the dirty sensor 30. i In the case of a dirt sensor 30, the information processing unit 113 can control i Nearby vehicle lights 20 j In this case, according to the flashing vehicle lights 20 j , users can identify the dirty sensor 30 i The approximate location of .

[0155] In addition, for example, the user is located on the sensor 30 where there is dirt. i If the sensor 30 is dirty, the information processing unit 113 controls the sensor 30 to i Nearby vehicle lights 20 j , so that it turns on the light at a higher light intensity. In this case, for example, the sensor 30 that recognizes the dirt i The approximate location of the user is close to the dirty sensor 30 i When the sensor is dirty, i The area around the sensor is illuminated brightly, so the user can check if there is dirt on the sensor 30 i Sensors for dirt conditions and assistance at night 30 i cleaning operations.

[0156] In addition, the information processing unit 113 causes the vehicle-mounted lamp 20 j When the light is turned on or flashing, the vehicle-mounted light 20 may be turned on or flashed according to the user's operation or when the cleaning completion notification is received. j Turn off the lights.

[0157] <Notification of handling>

[0158] Figure 8 Yes Description Figure 2 Flowchart of notification processing of the vehicle control device 10.

[0159] In step S11, the information processing unit 113 of the vehicle control device 10 receives information from each sensor 30 via the communication unit 114. i Get sensor 30 i The sensor information is transferred to step S12.

[0160] In step S12, the information processing unit 113 uses each sensor 30 i Sensor information to determine whether there is a dirty sensor 30 i Here, the information processing unit 113 that performs the processing of step S12 may function as a poor state detection unit that detects the state of the sensor 30. i The dirt is in a bad state.

[0161] In step S12, it is determined that there is no sensor 30 that is dirty. i In this case, the process transfers to step S13, the information processing unit 113 generates poor state information indicating that there is no dirt that is in a poor state, and sends it from the communication unit 115 to the off-vehicle terminal 40, thereby ending the notification process of the vehicle control device 10.

[0162] On the other hand, in step S12, the sensor 30 that is determined to be dirty i In the case of the above, the process transfers to step S14, and the information processing unit 113 generates poor state information indicating that there is dirt, which is a poor state. Furthermore, the information processing unit 113 includes the sensor 30 indicating that there is dirt in the poor state information. i The sensor position information of the respective positions is transmitted from the communication unit 115 to the external terminal 40, and the processing is transferred from step S14 to step S15.

[0163] In step S15, the information processing unit 113 determines whether the cleaning completion notification has been received from the off-vehicle terminal 40. In step S15, if it is determined that the cleaning completion notification has not been received, the process moves to step S16, and the information processing unit 113 controls the vehicle-mounted lamp 20 through the communication unit 114 as needed and based on the location-related information of the off-vehicle terminal 40. j , for example, making the sensor 30 with dirt i Nearby vehicle lights 20 j Then, the process returns from step S16 to step S15, and the same process is repeated hereinafter.

[0164] In step S15, if it is determined that the cleaning end notification has been received, that is, for example, the user has touched the sensor 30 that is dirty. i After cleaning, if the off-vehicle terminal 40 is operated to notify a cleaning completion notification and the cleaning completion notification is transmitted from the off-vehicle terminal 40 to the vehicle control device 10, the process proceeds to step S17.

[0165] In step S16, the vehicle lamp 20 j When the light is on, in step S17, the information processing unit 113 controls the vehicle light 20 via the communication unit 114.j The light is turned off and the process returns to step S11. i cleaning, as the poor state of the dirt is removed (temporarily assumed so), so that the already turned on the car lamp 20 j Turn off the lights.

[0166] Then, in step S11, as described above, the information processing unit 113 obtains the information of each sensor 30 i The sensor information is as follows, in step S12, until it is determined that there is no dirty sensor 30 i Until then, repeat the same process.

[0167] Fig. 9 Yes Description Figure 3 Flowchart of notification processing of the off-vehicle terminal 40.

[0168] In step S21, the control unit 155 of the off-vehicle terminal 40 obtains the poor state information from the vehicle control device 10, and the process transfers to step S22. That is, the communication unit 141 waits for the poor state information to be sent from the vehicle control device 10, receives the poor state information, and supplies it to the control unit 155. The control unit 155 obtains the poor state information supplied from the communication unit 141 in this way.

[0169] In step S22, the control unit 155 uses the state failure information from the vehicle control device 10 obtained in step S21 to determine whether there is a dirty sensor 30. i .

[0170] In step S22, it is determined that the sensor 30 is dirty. i In the case of the condition that the poor state information from the vehicle control device 10 indicates that there is dirt and includes the sensor 30 with the dirt i In the case of the sensor position information, the control unit 155 supplies the poor state information from the vehicle control device 10 to the image generating unit 153, and the processing is transferred to step S23.

[0171] In step S23 , control unit 155 determines whether input unit 143 has been operated to notify the end of cleaning.

[0172] In step S23, when it is determined that the input unit 143 has not been operated to notify the end of cleaning, the processing is transferred to step S24, the position-related information acquisition unit 152 acquires the position-related information of the external terminal 40, supplies it to the control unit 155 and the image generation unit 153, and the processing is transferred to step S25.

[0173] In step S25, the image generation unit 153 generates an image of the sensor 30 indicating that there is dirt based on the position-related information from the position-related information acquisition unit 152 and the poor state information from the control unit 155. i The image of the poor status of the position is supplied to the display control unit 154.

[0174] The display control unit 154 generates a composite image by combining the poor condition image from the image generation unit 153 with the vehicle image supplied from the image processing unit 151 or the vehicle image overlooking the vehicle 1. Then, the display control unit 154 supplies the composite image to the display unit 145 for display.

[0175] Furthermore, the control unit 155 transmits the position-related information from the position-related information acquisition unit 152 from the communication unit 141 to the vehicle control device 10 , and the process returns from step S25 to step S23 .

[0176] Here, as described above, the position-related information transmitted from the communication unit 141 to the vehicle control device 10 is Figure 8 It is used in the process of step S16.

[0177] On the other hand, if it is determined in step S23 that input unit 143 has been operated to notify the cleaning completion notification, the process moves to step S26, control unit 155 transmits the cleaning completion notification from communication unit 141 to vehicle control device 10, and the process returns to step S21.

[0178] In step S21 , as described above, the state failure information is acquired from the vehicle control device 10 , and the process proceeds to step S22 .

[0179] Then, in step S22, it is determined that there is no sensor 30 that is dirty. i In the case where the poor status information from the vehicle control device 10 indicates that there is no dirt, the notification processing of the external terminal 40 is completed.

[0180] <Display of a poor condition image including an image showing the location of dirt>

[0181] Fig.10 This is a diagram showing a display example of a poor condition image including an image showing the location of dirt.

[0182] For example, sensor 30 i In the case of a larger camera, the sensor 30 i When a portion of the lens of the camera is dirty, the position of the dirt (on the lens) can be detected by the vehicle control device 10, and the sensor 30 can be included in the poor state information. iThe dirt position information of the camera's dirt position (range).

[0183] In this case, the off-board terminal 40 ( Figure 3 ) can generate an image including the sensor 30 indicating that there is dirt. i The image of the dirt location is in poor condition.

[0184] Fig.10 The display example of the composite image is shown in which the poor condition image including the sensor 30 indicating that there is dirt is composited onto the vehicle image of the vehicle 1 as viewed from the position of the off-vehicle terminal 40. i Image of the dirt location.

[0185] exist Fig.10 In the middle, it means there is dirt on the sensor 30 i A circular image of the position of the camera and a dialog image containing the message "This camera is dirty with mud" are displayed as poor status images. Fig.10 In the simulation, the sensor is 30 i An image in which a shadow representing dirt is included in an image of a camera, and a poor state image including the dirt position image is displayed as a dirt position image indicating the position of the dirt.

[0186] As described above, when the dirt position image is displayed, the user can easily recognize the sensor 30 where the dirt is. i Which part is soiled?

[0187] In addition, in the dirt position image, the position of the dirt can be indicated by, for example, changing the color of the image or making it flash.

[0188] In addition, the user cleans the dirty sensor 30 i When the dirt is to be removed, the display of the poor condition image in the off-vehicle terminal 40 may be changed to the display of an image indicating that the dirt has been removed.

[0189] exist Fig.10 In the example, as an image showing that the dirt has disappeared, the sensor 30 showing that there was dirt is displayed. i and a dialog box image containing the message "The dirt has fallen off". Fig.10 In the figure, as an image showing that the dirt has disappeared, the sensor 30 which is free of dirt is shown. i The camera simulates the image.

[0190] <Sensor 30 having a light emitting portion i Example of configuration>

[0191] Fig.11 The sensor 30 having a light emitting portion i A cross-sectional view and a plan view of a configuration example.

[0192] In the vehicle 1, not limited to the sensor 30 i The vehicle lights 20 are equipped at the positions around (nearby positions) j . In sensor 30 i There is no vehicle light around the j In case of dirt on the sensor 30 i The nearest vehicle light 20 j Even if Figure 7 Turn on the light as described in the instructions, or it may be possible to move away from the dirty sensor 30 i Car lights 20 j In this way, the sensor 30 meters away from the dirt i Car lights 20 j Turning on the light does not help the user to easily identify the dirty sensor 30 i ’s position will become a hindrance.

[0193] Therefore, the sensor 30 mounted on the vehicle 1 i , a sensor 30 having a light emitting portion may be used i .

[0194] exist Fig.11 In the sensor 30 i For example, it is a camera and has an optical sensor 200 composed of an image sensor, etc. A lens 201 is provided on the light receiving surface side of the optical sensor 200. In addition, a circular substrate 202 is provided around the optical sensor 200, and a plurality of light emitting units 203 are provided on the substrate 202 in a manner surrounding the optical sensor 200 (lens 201).

[0195] Here, the light emitting unit 203 may be composed of an LED or the like. Fig.11 In the embodiment, eight light emitting units 203 are provided, but in the sensor 30 i The number of the light emitting units 203 provided above may be one or a plurality other than eight.

[0196] As described above, in the sensor 30 i When the light emitting unit 203 is provided, the vehicle lamp 20 j , can make the sensor with dirt 30 i The light emitting unit 203 is turned on or flashes. i When the light emitting unit 203 is turned on or flashes, the sensor 30 having dirt is iSurrounding vehicle lights 20 j Compared with turning on the light or flashing, it is easier for users to identify the sensor 30 that is dirty. i more accurate location.

[0197] In addition, the poor condition image and the synthesized image may be generated not only on the off-vehicle terminal 40 , but also on the vehicle control device 10 and a computer on the cloud, and supplied to the off-vehicle terminal 40 .

[0198] <Help for computers using this technology>

[0199] Next, the above-mentioned series of processing by the vehicle control device 10 and the external terminal 40 can be performed by hardware or software. When the series of processing is performed by software, the program constituting the software is installed on a general-purpose computer or the like.

[0200] Fig.12 This is a block diagram showing a configuration example of one embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0201] The program can be stored in advance in the hard disk 905 and the ROM 903 which are storage media built in the computer.

[0202] Alternatively, the program may be recorded (stored) in a removable storage medium 911 driven by the drive 909. Such a removable storage medium 911 may be provided as so-called package software. Here, as the removable storage medium 911, for example, there are a floppy disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, a semiconductor memory, and the like.

[0203] In addition, in addition to being installed on the computer from the removable storage medium 911 as described above, the program can also be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 905. That is, the program can be transmitted to the computer wirelessly from a download site via an artificial satellite used for digital satellite broadcasting, or can be transmitted to the computer via a network such as a LAN (Local Area Network) or the Internet in a wired manner.

[0204] The computer has a built-in CPU (Central Processing Unit) 902 , and the CPU 902 is connected to an input / output interface 910 via a bus 901 .

[0205] When a user operates the input unit 907 and a command is input to the CPU 902 through the input / output interface 910, the program stored in the ROM (Read Only Memory) 903 is executed. Alternatively, the CPU 902 loads the program stored in the hard disk 905 into the RAM (Random Access Memory) 904 for execution.

[0206] Thus, the CPU 902 performs the processing of the flowchart or the processing according to the configuration of the block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 or sends it from the communication unit 908 through the input / output interface 910 as needed, and stores it in the hard disk 905 or the like.

[0207] The input unit 907 is composed of a keyboard, a mouse, a microphone, etc. The output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, and the like.

[0208] Here, in this specification, the processing performed by the computer according to the program is not necessarily performed in the time series of the order described as the flowchart. That is, the processing performed by the computer according to the program includes processing executed in parallel or individually (for example, parallel processing or processing according to the object).

[0209] Furthermore, the program may be processed by one computer (processor) or by a plurality of computers in a distributed manner, or the program may be transmitted to a remote computer for execution.

[0210] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in their respective housings and connected via a network, as well as a device housing multiple modules in one housing, are both systems.

[0211] In addition, the embodiment of the present technology is not limited to the above-mentioned embodiment, and various changes can be made within the scope not departing from the gist of the present technology.

[0212] For example, the present technology may adopt a cloud computing configuration in which one function is shared among a plurality of devices via a network and the devices jointly process the functions.

[0213] In addition, each step described in the above flowchart may be executed on a single device or may be executed in a shared manner on a plurality of devices.

[0214] Furthermore, when a plurality of processes are included in one step, the plurality of processes included in the one step may be executed in a shared manner on a plurality of devices in addition to being executed on a single device.

[0215] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may exist.

[0216] In addition, the present technology can adopt the following configuration

[0217] <1>

[0218] An information processing device, comprising:

[0219] a position-related information acquisition unit that acquires position-related information related to a relative position or direction with respect to the vehicle; and

[0220] The display control unit displays a composite image in which a poor condition image indicating a position of a sensor in poor condition among sensors mounted on the vehicle is composited onto a vehicle image reflecting the vehicle based on the position association information.

[0221] <2>

[0222] The information processing device described in <1>, wherein the display control unit displays the composite image, wherein the composite image is an image that is composited, based on the position association information, an image indicating a relative position of the information processing device with respect to the vehicle and the poor condition image onto an image of the vehicle overlooking the vehicle.

[0223] <3>

[0224] The information processing device described in <1>, wherein the display control unit displays the composite image, wherein the composite image is an image obtained by composite the poor condition image onto the vehicle image of the vehicle viewed from the position of the information processing device based on the position association information.

[0225] <4>

[0226] The information processing device described in <3>, wherein the vehicle image is a CG (Computer Graphics) image of the vehicle viewed from the position of the information processing device or a photographic image of the vehicle photographed by a camera from the position of the information processing device.

[0227] <5>

[0228] The information processing device described in <1>, wherein the display control unit can be switched to:

[0229] A bird's-eye view display, displaying a composite image in which an image indicating a relative position of the information processing device with respect to the vehicle and the poor condition image are composited onto an image of the vehicle overlooking the vehicle;

[0230] VR (Virtual Reality) display, displaying the synthesized image obtained by synthesizing the poor condition image onto a CG (Computer Graphics) image of the vehicle as viewed from the position of the information processing device;

[0231] An AR (Augmented Reality) display displays the synthesized image in which the poor condition image is synthesized onto a photographic image of the vehicle taken by a camera from the position of the information processing device.

[0232] <6>

[0233] The information processing device described in <5>, wherein the display control unit,

[0234] performing the bird's-eye view display when the sensor in a bad state is not visible from the position of the information processing device;

[0235] The VR display or the AR display is performed when the sensor in a bad state is visible from the position of the information processing device.

[0236] <7>

[0237] The information processing device described in <5>, wherein the display control unit performs the VR display or the AR display according to the brightness of the photographic image.

[0238] <8>

[0239] The information processing device according to any one of <1> to <7>, wherein the vehicle turns on a light emitting unit around the sensor in the poor condition.

[0240] <9>

[0241] The information processing device described in <8>, wherein the vehicle turns off the light emitting unit when the sensor in the bad state is no longer in a bad state.

[0242] <10>

[0243] The information processing device described in <8> or <9>, wherein the sensor includes the light emitting unit.

[0244] <11>

[0245] The information processing device described in any one of <1> to <10>, wherein the sensor is a camera.

[0246] <12>

[0247] The information processing device described in any one of <1> to <11>, wherein the poor state means that the sensor is dirty.

[0248] <13>

[0249] The information processing device described in <12>, wherein the poor state image includes an image showing a position of the dirt on the sensor where the dirt is present.

[0250] <14>

[0251] The information processing device described in any one of <1> to <13>, wherein the information processing device is a portable terminal.

[0252] <15>

[0253] An information processing method, comprising:

[0254] Obtaining position-related information related to the relative position or direction of the vehicle; and

[0255] A composite image is displayed, wherein a poor condition image indicating a position of a sensor that is poorly conditioned among sensors mounted on the vehicle is composited onto a vehicle image reflecting the vehicle based on the position association information.

[0256] <16>

[0257] An information processing system comprising:

[0258] Sensors, mounted on vehicles;

[0259] a poor state detection unit, for detecting a poor state of the sensor;

[0260] A display unit, which displays images;

[0261] a position-related information acquisition unit that acquires position-related information related to a relative position or direction with respect to the vehicle; and

[0262] The display control unit displays a composite image in which a poor condition image indicating a position of a sensor in poor condition among sensors mounted on the vehicle is composited onto a vehicle image reflecting the vehicle based on the position association information.

Claims

1. An information processing device, characterized in that include: a position-related information acquisition unit, which acquires position-related information related to the relative position or direction of the information processing device with respect to the vehicle; as well as a display control unit that causes a composite image to be displayed on a display unit, wherein the composite image is obtained by: generating a poor state image based on the position association information, the poor state image indicating the position of a sensor in a poor state among sensors mounted on the vehicle, and composite the poor state image onto a vehicle image reflecting the vehicle, wherein the display control unit switches between bird's-eye view display, virtual reality display and augmented reality display for displaying the composite image, and The display control unit performs the bird's-eye view display when the sensor in a poor condition is not visible from the position of the information processing device, and performs the virtual reality display or the augmented reality display when the sensor in a poor condition is visible from the position of the information processing device.

2. The information processing device according to claim 1, characterized in that The display control unit displays the composite image on the display unit, wherein the composite image is an image in which an image indicating a relative position of the information processing device with respect to the vehicle and the poor condition image are composited onto the vehicle image overlooking the vehicle based on the position association information.

3. The information processing device according to claim 1, characterized in that The display control unit causes the display unit to display the composite image, the composite image being an image in which the poor condition image is composited onto the vehicle image of the vehicle viewed from the position of the information processing device based on the position association information.

4. The information processing device according to claim 3, characterized in that: The vehicle image is a computer image of the vehicle viewed from the position of the information processing device or a photographic image of the vehicle photographed by a camera from the position of the information processing device.

5. The information processing device according to claim 1, characterized in that The bird's-eye view display displays the synthesized image in which the image indicating the relative position of the information processing device with respect to the vehicle and the poor condition image are synthesized on the vehicle image in which the vehicle is viewed from above; The virtual reality display displays the composite image that composites the poor condition image onto a computer image of the vehicle as viewed from the position of the information processing device; The augmented reality display shows the synthesized image in which the poor condition image is synthesized onto a photographic image of the vehicle taken by a camera from the position of the information processing device.

6. The information processing device according to claim 5, characterized in that: The display control unit performs the virtual reality display or the augmented reality display according to the brightness of the photographic image.

7. The information processing device according to claim 1, characterized in that: The vehicle turns on a light emitting unit around the sensor in the bad condition.

8. The information processing device according to claim 7, characterized in that: The vehicle turns off the light emitting unit when the state of the sensor in the bad state is resolved.

9. The information processing device according to claim 7, characterized in that: The sensor includes the light emitting portion.

10. The information processing device according to claim 1, characterized in that The sensor is a camera.

11. The information processing device according to claim 1, characterized in that: The bad state means that the sensor is dirty.

12. The information processing device according to claim 11, characterized in that: The bad state image includes an image indicating the location of the dirt on the sensor where the dirt is present.

13. The information processing device according to claim 1, characterized in that: The information processing device is a portable terminal.

14. An information processing method of an information processing device, characterized in that include: Obtaining position-related information related to the relative position or direction of the information processing device relative to the vehicle; as well as causing a synthesized image to be displayed on a display unit, the synthesized image being obtained by: generating a poor state image based on the position association information, the poor state image indicating the position of a sensor in a poor state among sensors mounted on the vehicle, and synthesizing the poor state image onto a vehicle image reflecting the vehicle, The information processing method switches between bird's-eye view display, virtual reality display and augmented reality display for displaying the composite image, and performs the bird's-eye view display when the sensor in a poor condition is not visible from the position of the information processing device, and performs the virtual reality display or the augmented reality display when the sensor in a poor condition is visible from the position of the information processing device.

15. An information processing system, characterized in that include: Sensors, mounted on vehicles; a poor state detection unit, for detecting a poor state of the sensor; A display unit, which displays images; as well as An information processing device, the information processing device comprising: a position-related information acquisition unit that acquires position-related information related to a relative position or direction of the information processing device with respect to the vehicle; and a display control unit, causing a composite image to be displayed on the display unit, wherein the composite image is obtained by: generating a poor state image based on the position association information, the poor state image indicating the position of a sensor in a poor state among sensors mounted on the vehicle, and composite the poor state image onto a vehicle image reflecting the vehicle, wherein the display control unit switches between bird's-eye view display, virtual reality display and augmented reality display for displaying the composite image, and The display control unit performs the bird's-eye view display when the sensor in a poor condition is not visible from the position of the information processing device, and performs the virtual reality display or the augmented reality display when the sensor in a poor condition is visible from the position of the information processing device.

Images