Signal processing device, signal processing method, and signal processing system

Abstract

This disclosure relates to a signal processing device, a signal processing method, and a signal processing system which make it possible to improve the operability and safety of remote control. In the present invention, the current position of a moving body and a proximate object present in the vicinity of the moving body are predicted on the basis of position information for the moving body, the speed and steering angle of the moving body, and information about the proximate object, which are received via communication with the moving body, while accounting for the delay associated with the communication, and the display method is switched according to the delay time, and a display image for displaying the prediction results is created and displayed. The present invention can be applied to a remote control system.

Description

Signal processing device, signal processing method, and signal processing system

[0001] The present disclosure relates to a signal processing device, a signal processing method, and a signal processing system, and in particular to a signal processing device, a signal processing method, and a signal processing system that, when operating a vehicle from a remote location, enables safe remote operation by appropriately displaying information about the surroundings of the vehicle transmitted from the vehicle while taking into account communication delay times.

[0002] 2. Description of the Related Art Development of autonomous driving technologies for vehicles, ships, aircraft, and other moving objects is progressing, and in the near future, fully autonomous vehicles that do not require a driver will be realized.

[0003] However, as a step before fully autonomous driving is realized, it is expected that remote control, which allows moving objects to be controlled from a remote location, will become widespread.

[0004] For example, when a vehicle serving as a moving body is remotely controlled, the vehicle's own position is sensed in the vehicle, and the information on the vehicle's own position that is the sensing result is transmitted to an operation terminal operated by a driver in a remote location.

[0005] The operating terminal receives and presents the vehicle position information transmitted from the vehicle, and based on the presented vehicle position information, the driver inputs steering information, which is then transmitted to the vehicle.

[0006] The vehicle receives the steering information transmitted from the operation terminal and drives based on the received steering information, thereby achieving remote control.

[0007] Incidentally, it is known that delays occur in communication between an operating terminal and a vehicle to be operated.

[0008] Therefore, when the vehicle's own position is received by the operating terminal, the vehicle has already moved by the delay time, so the vehicle's own position information presented to the driver is position information from the past by the delay time, which could lead to a decrease in operability.

[0009] Therefore, a technology has been proposed in which the current position of the vehicle to be operated is estimated based on the vehicle's own position information up to the last moment and the delay time related to communication, taking into account movement during the delay time, and the current position is displayed superimposed on the transmitted own vehicle position information (see Patent Document 1).

[0010] JP 2010-061346 A

[0011] However, while it is possible to estimate and display the vehicle's position taking delay time into account, the positions of vehicles and pedestrians around the vehicle being controlled are not displayed taking delay time into account, which means there is a risk of misidentifying the positions of surrounding vehicles and pedestrians and resulting in contact.

[0012] The present disclosure has been made in consideration of such circumstances, and in particular, when operating a vehicle from a remote location, it is possible to appropriately present position information of the vehicle and surrounding objects transmitted from the vehicle while taking into account communication delay times.

[0013] A signal processing device and a signal processing system according to one aspect of the present disclosure are a signal processing device and a signal processing system that include a prediction unit that predicts the current position of a surrounding object, which is an object present in the vicinity of a mobile body, based on information about the surrounding object received through communication with the mobile body, taking into account a delay time associated with the communication.

[0014] A signal processing method according to one aspect of the present disclosure is a signal processing method including a step of predicting the current position of a surrounding object, which is an object present in the vicinity of a moving body, based on information about the surrounding object received through communication with the moving body, taking into account a delay time associated with the communication.

[0015] In one aspect of the present disclosure, based on information about surrounding objects that are objects present around a mobile body and received through communication with the mobile body, the current position of the surrounding objects is predicted taking into account the delay time associated with the communication.

[0016] 18 is a diagram illustrating an overview of the present disclosure. FIG. 19 is a diagram illustrating a configuration example of a first embodiment of a remote control system of the present disclosure. FIG. 19 is a diagram illustrating a configuration example of an operator terminal of FIG. 2. FIG. 29 is a diagram illustrating vehicle position information. FIG. 29 is a diagram illustrating a configuration example of the vehicle of FIG. 2. FIG. 29 is a diagram illustrating a display method of a host vehicle. FIG. 30 is a diagram illustrating a display method of surrounding vehicles. FIG. 31 is a diagram illustrating a display method for recognizing that a surrounding vehicle is decelerating. FIG. 32 is a flowchart illustrating remote control processing by the remote control system of FIG. 2. FIG. 33 is a flowchart illustrating the host vehicle position prediction processing of FIG. 34. FIG. 35 is a diagram illustrating a host vehicle position prediction method. FIG. 36 is a flowchart illustrating the host vehicle position display method determination processing of FIG. 35. FIG. 37 is a flowchart illustrating the display viewpoint determination processing of FIG. 36. FIG. 38 is a flowchart illustrating the display information determination processing of FIG. 37. FIG. 39 is a flowchart illustrating the peripheral object prediction processing of FIG. 38. FIG. 40 is a flowchart illustrating the peripheral object display method determination processing of FIG. 40. FIG. 41 is a diagram illustrating an application example of a bird's-eye view. FIG. 42 is a flowchart illustrating the display viewpoint determination processing when the application example of the bird's-eye view of FIG. 41 is used. FIG. 43 is a diagram illustrating an example of a case where a warning display is displayed in the peripheral object display method. 27 is a flowchart illustrating a peripheral object display method decision process when the warning display of FIG. 20 is performed. FIG. 28 is a flowchart illustrating a peripheral object display method decision process when a delay time factor is used for judgment. FIG. 29 is a diagram illustrating an application example of a display method for a host vehicle. FIG. 30 is a diagram illustrating an application example of a display method for peripheral vehicles. FIG. 31 is a diagram illustrating an application example of a display method for peripheral vehicles. FIG. 32 is a diagram illustrating an application example of a display method for peripheral vehicles. FIG. 33 is a diagram illustrating an example configuration of a second embodiment of a remote control system of the present disclosure. FIG. 34 is a flowchart illustrating remote control processing by the remote control system of FIG. 27. FIG. 35 is a diagram illustrating an example configuration as a modified example of the second embodiment of the remote control system of the present disclosure. FIG. 36 is a flowchart illustrating remote control processing by the remote control system of FIG. 29. FIG. 37 is a diagram illustrating an application example of a warning display using V2X communication in the remote control system of FIG. 29. FIG. 38 is a diagram illustrating an application example of a peripheral object display method using V2X communication in the remote control system of FIG. 29. FIG. 39 is a diagram illustrating an example configuration of a general-purpose computer.

[0017] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0018] Hereinafter, embodiments for carrying out the present technology will be described. The description will be made in the following order: 1. Overview of the present disclosure 2. First embodiment 3. Second embodiment 4. Modification of the second embodiment 5. Example of execution by software

[0019] <<1. Overview of the Present Disclosure>> The present disclosure provides, in particular, a method for remotely controlling a vehicle, whereby information about the surroundings of the vehicle transmitted from the vehicle can be presented while taking communication delays into consideration. Therefore, an overview of the present disclosure will be first described.

[0020] Note that the technology disclosed herein is not limited to vehicles, but can be applied to all remotely controlled moving bodies, such as ships and aircraft, but here we will use the example of remotely controlling a vehicle as a moving body for the explanation.

[0021] Development of autonomous driving technology is progressing, and in the near future, fully autonomous vehicles that do not require a driver will be realized. However, as a preliminary step before autonomous driving becomes widespread, it is expected that vehicles will be monitored and controlled by a remote operator. In this case, communication delays may cause difficulties in remote control and monitoring.

[0022] For example, as shown in the upper part of FIG. 1, consider a case where a vehicle 1 is remotely controlled by an operation terminal 2 operated by a driver at a remote location.

[0023] In the upper part of FIG. 1, the vehicle 1 captures images of the surroundings using a camera or the like and transmits the captured images to the operation terminal 2 as vehicle surroundings information.

[0024] When the operation terminal 2 acquires the vehicle surroundings information transmitted from the vehicle 1, an image of the surroundings of the vehicle 1 included in the vehicle surroundings information is presented to the driver. The driver refers to the image of the surroundings of the vehicle 1 presented on the operation terminal and inputs vehicle steering information for steering the vehicle 1. The operation terminal 2 transmits the vehicle steering information input by the driver to the vehicle 1.

[0025] Then, when the vehicle 1 receives the vehicle steering information from the operation terminal 2, the vehicle 1 realizes driving based on the received vehicle steering information. Through this series of processes, a driver at a remote location can operate the operation terminal 2 to remotely control the vehicle 1.

[0026] However, delays generally occur in communication between the vehicle 1 and the operation terminal 2 .

[0027] Therefore, for example, when an image is captured by vehicle 1 as vehicle surroundings information, transmitted to operation terminal 2, and received by operation terminal 2, when the received image is presented, there will be a delay relative to the timing at which the image was transmitted, and the presented image will be an image from the past by the delay time.

[0028] More specifically, after the vehicle 1 captures an image of the area ahead in the direction of travel, it is transmitted to the operating terminal 2, and by the time the image VL shown in the upper left of Figure 1 is presented on the operating terminal 2, an image from the past by the delay time has already been presented.

[0029] At the timing when the image VL shown in the upper left part of FIG. 1 is presented on the operation terminal 2, for example, an image VC shown in the upper right part of FIG. 1 is being captured.

[0030] More specifically, due to a communication delay, the oncoming vehicle CL was located at the far side of the crosswalk PC at the time the image VL was captured, but the oncoming vehicle CL moves forward by the delay time that occurs during transmission from the vehicle 1 to the operation terminal 2. Therefore, at the time the image VL is presented, the oncoming vehicle CL has actually moved onto the crosswalk PC, as shown in the image VC.

[0031] In other words, even though the surrounding conditions of vehicle 1 are as shown in image VC, the driver will input vehicle steering information while being presented with a past image VL that corresponds to the delay time related to communication.

[0032] As a result, the delay reduces the driver's operability, and since steering information is input based on oncoming vehicles or pedestrians that are actually in different positions, there is a risk of contact with oncoming vehicles or nearby pedestrians, which could reduce safety.

[0033] Therefore, in the present disclosure, the position of an oncoming vehicle that has moved by the delay time is estimated based on past vehicle surroundings information, and the estimated result is superimposed and displayed, allowing the actual position of the oncoming vehicle at the time the vehicle surroundings information is presented to be recognized.

[0034] More specifically, for example, as shown in image VF in the lower left of Figure 1, in addition to the oncoming vehicle CL obtained from vehicle surroundings information, the position to which the oncoming vehicle will move by the delay time is estimated, and the estimated result is superimposed and displayed as frame CF.

[0035] This means that even if there is a communication delay between the vehicle 1 and the operating terminal 2, the operator can recognize the position of the oncoming vehicle at the current timing, taking into account the communication delay, simply by looking at the image VF presented as the vehicle surroundings information, thereby improving operability and safety in remote control.

[0036] <<2. First Embodiment>> Next, with reference to FIG. 2, a configuration example of a remote control system for controlling a vehicle from a remote location to which the technology of the present disclosure is applied will be described.

[0037] The remote control system 11 in FIG. 2 is made up of operator terminals 32-1 to 32-n operated by operators 31-1 to 31-n, respectively, who are operators, a network 33, and vehicles 34-1 to 34-m.

[0038] In the following, when there is no particular need to distinguish between the operators 31-1 to 31-n, the operator terminals 32-1 to 32-n, and the vehicles 34-1 to 34-m, they will be referred to simply as the operator 31, the operator terminal 32, and the vehicle 34, and the same applies to other configurations.

[0039] The vehicle 34 is intended to be remotely controlled based on steering information supplied from the operator terminal 32, but it may also be intended to be a vehicle that can be directly controlled by a driver with a driver on board, or a vehicle that can be fully autonomously driven, and be set in a remotely controlled mode.

[0040] The vehicle 34 acquires information on its own position, steering angle, speed, and surrounding objects, and transmits this information as vehicle information to the operator terminal 32 via the network.

[0041] The vehicle information is information used to estimate the current position of the vehicle and surrounding objects, taking into account delay times associated with communication with the operator terminal 32. Among these, information on the vehicle's position, steering angle, and speed at the time the vehicle information is acquired is required to estimate the current position of the vehicle.

[0042] The information on the surrounding objects may include, for example, images of the periphery of the vehicle 34 captured by a camera or the like, or distance images captured by a depth sensor or the like. In other words, the information on the surrounding objects is information that enables the recognition of the positions and distances of objects around the vehicle 34. For this reason, the information on the surrounding objects may include not only images and distance images captured by a camera, but also sensing results from an ultrasonic sensor, radar, or the like.

[0043] When the operator terminal 32 acquires the vehicle information transmitted from the vehicle 34 via the network 33 , it measures the delay time involved in the communication between itself and the vehicle 34 .

[0044] The operator terminal 32 estimates the position of the vehicle 34 and objects around the vehicle 34 based on the vehicle information and delay time, and generates a display image of the area around the vehicle 34 to be used for remote control, such as by superimposing the estimated result on the original position information that does not take into account the delay contained in the vehicle information, and presents it to the operator 31.

[0045] At this time, the operator terminal 32 switches the presentation method based on the delay time and presents information about the vehicle and its surroundings to the operator 31.

[0046] In addition, the presentation of the display image around the vehicle 34 used for remote control is intended to be an image that can be seen when the operator 31 gets into the driver's seat of the vehicle 34, displayed on a display device that constitutes the operator terminal 32, such as a display, but is not limited to this.

[0047] That is, the display image of the surroundings of the vehicle 34 used for remote control may be displayed, for example, on an HMD (Head Mounted Display), or the operator 31 may wear the HMD and the display image may be switched depending on changes in the line of sight.

[0048] The operator 31 inputs steering information for steering the vehicle 34 based on the display image required for remote control presented on the operator terminal 32 .

[0049] The operator terminal 32 transmits the steering information input by the operator 31 to the vehicle 34 via the network 33 .

[0050] When the vehicle 34 acquires the steering information transmitted from the operator terminal 32 via the network 33, the vehicle 34 controls its traveling based on the acquired steering information.

[0051] The operator terminal 32 can be equipped with various types of operation input units. In Fig. 2, an example is shown in which the operation input unit 32a of the operator terminal 32-1 is configured with a keyboard and a mouse. Also, an example is shown in which the operation input unit 32b of the operator terminal 32-2 is configured with a steering wheel. An example is shown in which the operation input unit 32c of the operator terminal 32-3 is configured with a touch panel that also functions as a display unit.

[0052] The operation input units 32a to 32c are merely examples, and may have other configurations as long as they can be operated by the operator 31 and allow steering information to be input.

[0053] Furthermore, the vehicles 34-1 to 34-m operated by each of the operator terminals 32-1 to 32-n do not necessarily have to be one-to-one, and one operator terminal 32 may remotely control the vehicle 34 to be operated by switching between them.

[0054] With the above configuration, the display image presented to the operator 31 is switched depending on the delay time related to communication, and the position of the vehicle 34 is estimated taking into account the delay time related to communication, and the estimated result is superimposed and displayed.

[0055] As a result, the operator 31 can visually recognize the effects of delays in communication, which makes it possible to improve the operability and safety of remote control.

[0056] <Configuration Example of Operator Terminal> Next, a configuration example of the operator terminal 32 will be described with reference to FIG.

[0057] The operator terminal 32 is a terminal device consisting of a computer running on an OS (Operating System) such as Windows (registered trademark), iOS (registered trademark), Linux (registered trademark), or Android (registered trademark), and is composed of a control unit 51, an input unit 52, an output unit 53, a memory unit 54, a communication unit 55, a drive 56, and a removable storage medium 57, which are connected to each other via a bus 58 and can send and receive data and programs.

[0058] The control unit 51 is composed of a processor and a memory, and controls the overall operation of the operator terminal 32. The control unit 51 also includes a steering information acquisition unit 71, a delay time acquisition unit 72, a vehicle information acquisition unit 73, a current position prediction unit 74, a display image creation unit 75, and a steering information supply unit 76.

[0059] The steering information acquisition unit 71 acquires the operation content when the operator 31 operates the input unit 52 to input steering information. The input unit 52 has a configuration corresponding to the operation input units 32a to 32c in FIG. 2 .

[0060] The delay time acquisition unit 72 compares the time information attached to the vehicle information transmitted from the vehicle 34 when it is transmitted with the current time information, thereby measuring and acquiring the delay time according to the state of the network 33. Note that this time information may be, for example, time information obtained from satellite waves used to acquire location information.

[0061] The vehicle information acquisition unit 73 controls the communication unit 55 to receive vehicle information sent from the vehicle 34, and stores the self-position information of each vehicle 34 contained in this vehicle information and the position information of vehicles and pedestrians around the vehicle 34 estimated based on the surrounding object information in association with time information in the position information storage unit 92.

[0062] The information stored in the location information storage unit 92 is, for example, information such as that shown in Figure 4, and includes an ID that individually identifies the target vehicle or pedestrian, type, measurement time, and x, y, z, and yaw information that serve as location information.

[0063] The type is a classification of vehicles or pedestrians, such as the subject vehicle, medium-sized vehicle, large vehicle, motorcycle, bicycle, pedestrian, etc. The subject vehicle is the vehicle 34 that transmitted the vehicle information, and the others are classifications of vehicles and pedestrians detected as peripheral object information.

[0064] For this classification, the vehicle information acquisition unit 73 may perform semantic segmentation processing based on the captured image, or may use a trained DNN (Deep Neural Network) or the like. Furthermore, other information may be added to the information accumulated in the position information accumulation unit 92. For example, roll and pitch information may be added in addition to yaw as posture information.

[0065] The current position prediction unit 74 reads out a prediction method stored in a prediction method storage unit 91 stored in the storage unit 54, and predicts the current position of the vehicle based on the steering angle, speed, and self-position information transmitted as vehicle information. The prediction method stored in the prediction method storage unit 91 is, for example, an arithmetic formula or program that makes it possible to estimate the position when the steering angle, speed, self-position information, etc. are input.

[0066] In addition, the current position prediction unit 74 predicts the current position based on information on the positions of vehicles and pedestrians around the vehicle 34 for multiple frames, which is stored in the position information storage unit 92 in association with time information.

[0067] The display image creation unit 75 generates a display image to be presented to the operator 31 based on the vehicle information acquired by the vehicle information acquisition unit 73, and outputs the generated image to the display unit 81 of the output unit 53 for display.

[0068] At this time, the display image generating unit 75 generates a display image by switching the display method in accordance with the delay time.

[0069] The display image generated by switching the display method according to the delay time will be described in detail later.

[0070] The steering information supply unit 76 controls the communication unit 55 to transmit the steering information acquired by the steering information acquisition unit 71 to the vehicle 34 to be operated, and supplies the steering information.

[0071] The input unit 52 is composed of input devices such as a keyboard, mouse, steering wheel, touch panel, etc., through which the operator 31 inputs steering information, and supplies the control unit 51 with various signals corresponding to the input steering information.

[0072] The output unit 53 is controlled by the control unit 51 and includes a display unit 81 and an audio output unit 82. The display unit 81 is a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays the display image created by the display image creation unit 75 and various processing results. The audio output unit 82 is an audio output device such as a speaker, and outputs various sounds, music, sound effects, and the like as audio.

[0073] The storage unit 54 is composed of a hard disk drive (HDD), a solid state drive (SSD), a semiconductor memory, or the like, and writes or reads various data and programs under the control of the control unit 51. The storage unit 54 includes a prediction method storage unit 91 and a position information accumulation unit 92.

[0074] The communication unit 55 is controlled by the control unit 51 and realizes communication via wired or wireless means, such as LAN (Local Area Network) or Bluetooth (registered trademark), and transmits and receives various data and programs to and from the vehicle 34 via the network 33 as necessary.

[0075] The drive 56 reads and writes data from and to a removable storage medium 57 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0076] <Example of Vehicle Configuration> Next, an example of the configuration of the vehicle 34 will be described with reference to the block diagram of FIG.

[0077] The vehicle 34 includes a vehicle control ECU (Electronic Control Unit) 111, a communication unit 112, a map information storage unit 113, a location information acquisition unit 114, an external recognition sensor 115, an in-vehicle sensor 116, a vehicle sensor 117, a memory unit 118, a driving control unit 119, a DMS (Driver Monitoring System) 120, an HMI (Human Machine Interface) 121, and a vehicle control unit 122.

[0078] The vehicle control ECU 111, communication unit 112, map information storage unit 113, position information acquisition unit 114, external recognition sensor 115, in-vehicle sensor 116, vehicle sensor 117, memory unit 118, driving control unit 119, DMS 120, HMI 121, and vehicle control unit 122 are connected to each other so as to be able to communicate with each other via a communication network 123. The communication network 123 is configured by an in-vehicle communication network or bus that complies with a digital two-way communication standard such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark), for example.

[0079] The vehicle control ECU 111 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 111 controls the entire or part of the functions of the vehicle 34.

[0080] The communication unit 112 communicates with the operator terminal 32, various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 112 can communicate using multiple communication methods.

[0081] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 112. The communication unit 112 communicates with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 112 communicates is, for example, the Internet, a cloud network, or a network specific to an operator. The communication method used by the communication unit 112 with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0082] Furthermore, for example, the communication unit 112 can communicate with a terminal located near the vehicle using P2P (Peer to Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 112 can also perform V2X (Vehicle to Extensible Communication) communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle (V2V) communication with another vehicle, vehicle-to-infrastructure (V2I) communication with a roadside unit, vehicle-to-home (V2H) communication with a home, and vehicle-to-pedestrian (V2P) communication with a terminal carried by a pedestrian.

[0083] The communication unit 112 can receive, for example, a program for updating software that controls the operation of the remote control system 11 from the outside (over the air). The communication unit 112 can also receive map information, traffic information, information about the surroundings of the vehicle 34, and the like from the outside. For example, the communication unit 112 can also transmit information about the vehicle 34 and the surroundings of the vehicle 34 to the outside. Information about the vehicle 34 that the communication unit 112 transmits to the outside includes, for example, data indicating the status of the vehicle 34. Furthermore, for example, the communication unit 112 can perform communication compatible with a vehicle emergency notification system such as e-call.

[0084] For example, the communication unit 112 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0085] The map information storage unit 113 stores one or both of a map acquired from an external source and a map created in the vehicle 34. For example, the map information storage unit 113 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0086] The position information acquisition unit 114 receives GNSS (Global Navigation Satellite System) signals from GNSS satellites and acquires position information of the vehicle 34. The acquired position information is supplied to the driving control unit 119. Note that the method of the position information acquisition unit 114 is not limited to a method using GNSS signals, and it may acquire position information using a beacon, for example.

[0087] The external recognition sensor 115 includes various sensors used to recognize the situation outside the vehicle 34, and supplies sensor data from each sensor to each part of the vehicle 34. The types and number of sensors included in the external recognition sensor 115 are arbitrary.

[0088] For example, the external recognition sensor 115 includes a camera 131, a radar 132, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 133, and an ultrasonic sensor 134. Without being limited to this, the external recognition sensor 115 may be configured to include one or more types of sensors selected from the camera 131, the radar 132, the LiDAR 133, and the ultrasonic sensor 134. Note that a sensor other than the LiDAR 133 may be used as long as it can measure the distance to surrounding vehicles and pedestrians, and may be, for example, a depth sensor or a stereo camera.

[0089] The interior sensor 116 includes various sensors for detecting information about the interior of the vehicle and supplies sensor data from each sensor. The types and number of the various sensors included in the interior sensor 116 are not particularly limited as long as they are of the types and number that can be realistically installed in the vehicle 34.

[0090] The vehicle sensor 117 includes various sensors for detecting the state of the vehicle 34 and supplies sensor data from each sensor. The types and number of the various sensors included in the vehicle sensor 117 are not particularly limited as long as they are of the types and number that can be realistically installed on the vehicle 34.

[0091] For example, the vehicle sensor 117 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these sensors. For example, the vehicle sensor 117 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 117 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 117 includes a battery sensor that detects the remaining battery charge and temperature, and an impact sensor that detects external impacts.

[0092] The storage unit 118 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The storage unit 118 is used, for example, as an electrically erasable programmable read-only memory (EEPROM) and a random access memory (RAM). The storage medium may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage unit 118 stores various programs and data used by each part of the vehicle 34.

[0093] The driving control unit 119 controls the driving functions of the vehicle 34. The driving control unit 119 includes a steering control unit 141, a sensor information acquisition unit 142, and a vehicle information communication unit 143.

[0094] The steering control unit 141 controls the communication unit 112 to acquire steering information transmitted from the operator terminal 32, and based on the acquired steering information, controls the vehicle control unit 122 to control operations such as steering, acceleration, deceleration, and stopping related to the running of the vehicle 34.

[0095] The sensor information acquisition unit 142 acquires, as vehicle surroundings information, detection results acquired by the external recognition sensor 115 that enable the positions and distances of other vehicles and pedestrians around the vehicle 34 to be recognized.

[0096] The vehicle information communication unit 143 compiles vehicle information by compiling the position information of the vehicle 34 acquired by the position information acquisition unit 114, the surrounding object information acquired by the sensor information acquisition unit 142, which allows the positions and distances of other vehicles and pedestrians to be recognized from the vehicle 34, and the steering angle and speed information of the vehicle 34 detected by the vehicle sensor 117, and controls the communication unit 112 to transmit the vehicle information to the operator terminal 32. In addition, the vehicle information may include information related to the body and specifications of the vehicle 34, such as the wheelbase length and tire diameter of the vehicle 34, which are required for predicting the vehicle position, as will be described later.

[0097] The DMS 120 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensors 116 and input data input to the HMI 121 (described later). Examples of the driver's state to be recognized include physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.

[0098] The DMS 120 may be configured to perform authentication processing for users other than the driver and recognition processing for the status of the users. Furthermore, for example, the DMS 120 may be configured to perform recognition processing for the status inside the vehicle based on sensor data from the in-vehicle sensor 116. Examples of the status inside the vehicle that may be recognized include temperature, humidity, brightness, and odor.

[0099] The HMI 121 inputs various data and instructions and presents various data to the user.

[0100] The vehicle control unit 122 controls each unit of the vehicle 34. The vehicle control unit 122 includes a steering control unit 151, a brake control unit 152, a drive control unit 153, a body control unit 154, a light control unit 155, and a horn control unit 156.

[0101] The steering control unit 151 detects and controls the state of the steering system of the vehicle 34. The brake control unit 152 detects and controls the state of the brake system of the vehicle 34. The drive control unit 153 detects and controls the state of the drive system of the vehicle 34. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The body system control unit 154 detects and controls the state of the body system of the vehicle 34. The light control unit 155 detects and controls the states of various lights of the vehicle 34. The horn control unit 156 detects and controls the state of the car horn of the vehicle 34.

[0102] <Method of Displaying Own Vehicle> Next, a method of displaying the own vehicle in the display image created by the display image creation unit 75 will be described with reference to FIG.

[0103] The display method of the vehicle has two elements: a display viewpoint and display information. The display image creation unit 75 switches between the display viewpoint and the display information depending on the delay time to create a display image showing the vehicle.

[0104] FIG. 6 shows an example of a display image generated by switching the display viewpoint and the display information to change the display method.

[0105] As shown in FIG. 6, there are three types of display viewpoints: bird's-eye view, third-person view, and first-person view.

[0106] A bird's-eye view is a display method that looks down on the vehicle and surrounding vehicles and pedestrians. It is a display image that depicts the predicted positions of the vehicle and surrounding vehicles and pedestrians on a two-dimensional map that is also used in general car navigation systems, and is a display viewpoint that requires the lightest processing load for generating the display image.

[0107] A third-person perspective view is a display image that depicts the predicted positions of the vehicle, surrounding vehicles, and pedestrians from a third-person perspective within an image captured by camera 131 included in the vehicle information, and is a display perspective that requires a heavier processing load for generating the display image than a bird's-eye view.

[0108] The first-person perspective view is a display image that depicts the predicted positions of surrounding vehicles and pedestrians on a CG (Computer Graphics) image based on 3D map information, which corresponds to an image captured by camera 131 from the viewpoint of a passenger in vehicle 34, and is a display perspective that imposes a heavier processing load on generating the display image than a third-person perspective view.

[0109] The first-person perspective view is a CG image that reproduces an image captured by the camera 131 installed in the vehicle when no delay occurs. Therefore, generating a display image made up of a first-person perspective view has a low cognitive load and learning cost, but requires 3D map information of the location to be traveled in advance. Furthermore, since the first-person perspective view is a CG image that reproduces an image captured by the camera 131 installed in the vehicle when no delay occurs, the image is expected to be viewed when riding in the vehicle, and therefore the vehicle itself is not displayed.

[0110] There are three types of display information: position only, vehicle body angle, and route.

[0111] When the display information is only the position, the display image displays only the predicted position together with the recognized own position as a two-dimensional figure such as a circle or a square.

[0112] When the displayed information is the vehicle angle, the angle of the vehicle body and the angle of the tires can be shown by adding a tail to the circle attached to the predicted position or by displaying it as a cube, etc. This makes it easier to perform steering operations such as changing lanes and turning right or left.

[0113] If the display information is a route, the route that is expected to be traveled is drawn in addition to the predicted position.

[0114] In Figure 6, example display images of a bird's-eye view, a third-person perspective view, and a first-person perspective view are shown horizontally from left to right in the figure, and example display images of position only, vehicle angle, and route are shown vertically from top to bottom.

[0115] When the display viewpoint is a bird's-eye view and the display image only shows the position, a nearby vehicle, a motorcycle, is represented by a square mark in the lane on the left side of the figure. The vehicle's current position based on the vehicle information is represented by a light gray circle mark at the bottom of the right side of the lane. Furthermore, the current predicted position based on the vehicle's position and delay time from the vehicle information is represented by a dark gray circle mark at the top.

[0116] In the case of a display image in which the display viewpoint is a bird's-eye view and the display information is the vehicle body angle, a nearby vehicle, a motorcycle, is represented by a square mark in the lane on the left side of the figure. Furthermore, the vehicle's own position based on the vehicle information is represented by a light gray circle mark at the bottom of the right side of the lane. Furthermore, the vehicle's own position predicted based on the vehicle information's own position and delay time is represented by a dark gray circle mark at the top, and the vehicle's body angle is represented by the direction of the convex part.

[0117] In the case of a display image in which the display viewpoint is a bird's-eye view and the display information is a route, a nearby vehicle, a motorcycle, is represented by a square mark in the lane on the left side of the figure. Furthermore, the position of the vehicle based on its own position information is represented by a light gray circle mark at the bottom of the right lane. Furthermore, the predicted position based on the own position and delay time from the vehicle information is represented by a dark gray circle mark at the top, and the route is represented by a curved line on the right lane.

[0118] In the case of a display image in which the display viewpoint is a third-person viewpoint and the display information is only the position, a motorcycle is displayed as a nearby vehicle in the left lane in the image captured by camera 131. In addition, a gray circle mark in the right lane represents the vehicle's own position predicted based on the vehicle's own position and delay time in the vehicle information.

[0119] In the case of a display image in which the display viewpoint is a third-person viewpoint and the display information is the vehicle body angle, a motorcycle is displayed as a nearby vehicle in the left lane in the image captured by camera 131. In addition, a box-shaped mark in the right lane represents the vehicle's own position predicted based on the vehicle's own position and delay time in the vehicle information.

[0120] In the case of a display image in which the display viewpoint is a third-person viewpoint and the display information is a route, a motorcycle is displayed as a nearby vehicle in the left lane in the image captured by camera 131. In addition, a circle mark in the right lane represents the vehicle's own position predicted based on the vehicle's own position and delay time in the vehicle information, and the curved line on the right lane represents the route.

[0121] In the case of a display image in which the display viewpoint is a first-person viewpoint and the display information is only the position, a motorcycle is displayed as a nearby vehicle in the left lane in the figure on a CG image generated based on 3D map information, which corresponds to the image captured by camera 131.

[0122] In the case of a display image in which the display viewpoint is a first-person viewpoint and the display information is the vehicle body angle, a motorcycle is displayed in the left lane in the figure on a CG image generated based on 3D map information, which corresponds to the image captured by camera 131, and the vehicle body angle is represented by a circle mark and an arrow in the right lane.

[0123] In the case of a display image in which the display viewpoint is a first-person viewpoint and the display information is a route, a motorcycle is displayed in the lane on the left side of the figure on a CG image generated based on 3D map information, which corresponds to an image captured by camera 131, and the route is represented by the curve of the lane on the right side.

[0124] The display viewpoints that have the heaviest processing load are the first-person view, the third-person view, and the bird's-eye view. Therefore, basically, the longer the delay time, the lighter the processing load of the display viewpoint that is selected.

[0125] Furthermore, the display information is selected according to visibility. For example, when there are few surrounding vehicles, there are few superimposed displays, so the route is selected as the display information. When there are a certain number of surrounding vehicles, if there are long straight stretches, it is considered that there will be few changes in the vehicle body, so only the position is selected as the display information, and in other cases, the vehicle body angle is selected as the display method.

[0126] <Display Method of Peripheral Objects> Next, a display method of peripheral objects will be described with reference to FIG.

[0127] The positions of surrounding objects are predicted based on accumulated position information, and are not absolute position information.

[0128] Therefore, the positions of surrounding objects such as nearby vehicles and pedestrians are predicted results, and are not absolute positions, but the range in which they may exist is explicitly displayed.

[0129] There are three methods for displaying surrounding objects: superimposing the predicted positions, displaying an area, and displaying only the predicted positions.

[0130] The superimposed display of the predicted position is, for example, a display that indicates the predicted position by pinpointing it on the display image, such as the display images (1-1) and (1-2) in FIG.

[0131] In the display image (1-1), the predicted position is represented by a circle mark in the camera image of the third-person viewpoint.

[0132] In addition, in the display image (1-2), the predicted position is drawn as a box-shaped mark on the CG image of the first-person perspective view, but if it overlaps with the target object on the camera image of the third-person perspective view, the box-shaped mark is displayed so that it does not overlap with the target object.

[0133] The area display of the predicted position does not pinpoint the prediction result, but depicts the range in which the target object may exist as an area. However, while area display increases safety by allowing a wide range of prediction results to be seen, when there are many target objects, the displays may overlap, making it difficult to identify them individually.

[0134] In the display image (2-1), the target object is displayed in a box or the like to surround the entire area where it may exist. The target object image may be clipped and enlarged to depict the area.

[0135] In the display image (2-2), the target object is drawn with a gradation effect according to the probability of its existence. In addition to a method of applying the effect to the entire target object, it may also be drawn so that it is superimposed on the road surface.

[0136] In the display image (2-3), the target object is not subject to any effects, but is displayed in a lower quality image. This makes it difficult to see the target object, making it easier to recognize the need to slow down.

[0137] In the display image (2-4), only the effect is drawn at the predicted position of the target object.

[0138] Displaying only the predicted position erases the target object from the generated CG image and only the predicted position is drawn. Therefore, displaying only the predicted position is simple and easy to understand, but it may lead to misunderstandings if the delay is large and the prediction accuracy is low.

[0139] In the display image (3-1), the image of the target object is clipped and drawn at the predicted position, and is superimposed. Note that the dotted ellipse and arrow in the display image (3-1) are not displayed on the display image; the dotted ellipse represents the original position of the image of the target object, the motorcycle, before it was clipped, and the arrow represents the distance and direction that the clipped image has moved.

[0140] In the display image (3-2), a figure such as a box is drawn at the predicted position of the target object and is superimposed.

[0141] In the peripheral object display method, the display methods that have the heaviest load for generating the display image are display of only the predicted position, area display, and superimposed display of the predicted position, so basically, the longer the delay time, the lower the processing load of the display method that is selected.

[0142] Although the first-person perspective view has been described in FIG. 7, a bird's-eye view or a third-person perspective view may also be used.

[0143] That is, as shown in the display image on the left side of Figure 8, in the case of a bird's-eye view, a rectangular gray mark representing a wider predicted position is superimposed on the motorcycle, which is a nearby vehicle represented by a rectangular mark.

[0144] Furthermore, as shown in the display image in the center of Figure 8, in the third-person perspective view, as in the first-person perspective view, the predicted range is superimposed and displayed as a semi-transparent box at the position where the surrounding vehicle, the motorcycle, is drawn.

[0145] The display image on the right side of FIG. 8 shows an example in which a prediction range is displayed as a semi-transparent box superimposed on the position where a motorcycle, which is a nearby vehicle, is drawn in the first-person viewpoint view.

[0146] <Variations of the method for displaying peripheral objects> The above has described examples of methods for displaying peripheral objects based on predicted positions, but if position information of peripheral objects for multiple frames can be obtained along with the predicted positions, the speed of the peripheral objects can also be estimated.

[0147] Here, if there is a nearby object ahead of the vehicle that can be recognized as decelerating from a change in speed, and the vehicle is traveling behind the nearby object, it is necessary to decelerate early.

[0148] Therefore, if there is a surrounding object ahead of the vehicle that can be recognized as decelerating based on a change in speed, the surrounding object may be displayed so that the deceleration can be recognized.

[0149] That is, for example, as shown in the display image on the left side of FIG. 9, a box-shaped mark indicating that the predicted position is behind the peripheral object may be superimposed and displayed.

[0150] Also, for example, as shown in the display image in the center of FIG. 9, a box-shaped mark extending backward including the surrounding objects may be superimposed and displayed.

[0151] Furthermore, for example, the image of the surrounding object may be clipped and shifted backward as shown in the display image on the right side of Fig. 9. In the display image on the right side of Fig. 9, the image of the motorcycle, which is the surrounding object, is clipped from the position of the dotted line portion of the circle mark and is displayed shifted toward the vehicle as represented by the arrow, but the dotted line portion and arrow are not actually displayed.

[0152] <Remote Control Processing> Next, the remote control processing by the remote control system 11 of FIG. 2 will be described with reference to the flowchart of FIG.

[0153] In step S31, the sensor information acquisition unit 142 acquires detection results that are acquired by the external recognition sensor 115 and that enable recognition of the positions and distances of other vehicles and pedestrians around the vehicle 34. That is, the sensor information acquisition unit 142 acquires, as surrounding object information, camera images that capture the surroundings and distance images that enable recognition of the distances to surrounding objects.

[0154] In step S32 , the sensor information acquisition unit 142 acquires the position information of the vehicle from the position information acquisition unit 114 .

[0155] In step S33 , the sensor information acquisition unit 142 acquires information on the steering angle (steering angle) and speed of the vehicle itself detected by the vehicle sensor 117 .

[0156] In step S34, the vehicle information communication unit 143 compiles the vehicle's position information acquired by the position information acquisition unit 114, surrounding object information consisting of sensing results acquired by the sensor information acquisition unit 142 that enable the position and distance of other vehicles and pedestrians to be recognized from the vehicle 34, and the steering angle (steering angle) and speed information of the vehicle detected by the vehicle sensor 117, to form vehicle information, and controls the communication unit 112 to transmit the vehicle information to the operator terminal 32.

[0157] At this time, the vehicle information communication unit 143 controls the communication unit 112 to transmit the vehicle information together with information for identifying the vehicle 34 and time information indicating the transmission timing.

[0158] In step S51 , the vehicle information acquisition unit 73 controls the communication unit 55 to acquire vehicle information transmitted from the vehicle 34 .

[0159] At this time, the vehicle information acquisition unit 73 associates the vehicle information with time information at the timing when the vehicle information is acquired.

[0160] In step S52, the delay time acquisition unit 72 measures the delay time related to communication between the vehicle 34 and the operator terminal 32 from the difference between the time information indicating the transmission timing attached to the vehicle information acquired by the vehicle information acquisition unit 73 and the time information at which the vehicle information was received, which is stored in correspondence with the vehicle information, and acquires the delay time that is the measurement result.

[0161] In step S53, the current position prediction unit 74 executes a vehicle position prediction process based on the acquired vehicle information to predict the vehicle position of the vehicle 34 that has transmitted the vehicle information.

[0162] The vehicle position prediction process will be described in detail later with reference to FIGS.

[0163] In step S54, the display image creating unit 75 executes a vehicle position display method determination process based on the delay time to determine a method for displaying the vehicle position.

[0164] The vehicle position display method determination process will be described in detail later with reference to FIGS.

[0165] In step S55, the current position prediction unit 74 executes a peripheral object prediction process based on the vehicle information, and predicts the positions of vehicles and pedestrians around the vehicle 34 that has transmitted the vehicle information.

[0166] The peripheral object prediction process will be described in detail later with reference to FIG.

[0167] In step S56, the display image creation unit 75 executes a peripheral object display method determination process based on the delay time, and determines a method for displaying peripheral objects.

[0168] The peripheral object display method determination process will be described in detail later with reference to FIG.

[0169] In step S57, the display image creation unit 75 displays the vehicle position using the vehicle position display method determined by the vehicle position display method determination process, generates a display image displaying surrounding objects using the surrounding object display method determined by the surrounding object display method determination process, and outputs it to the display unit 81 for display.

[0170] This processing results in a display image of the vehicle and its surrounding objects being displayed on the display unit 81 in a manner that takes communication delays into account, and at a predicted position that takes communication delays into account.

[0171] This allows the operator 31 to view the display image displayed on the display unit 81 in a state in which the effects of communication delays have been reduced, allowing him or her to operate the vehicle 34 while properly visually checking the positional relationship between the vehicle's own position and surrounding objects.

[0172] In step S58, the steering information acquisition unit 71 acquires steering information input by the operator 31 operating the input unit 52.

[0173] In step S59, the steering information supply unit 76 controls the communication unit 55 to transmit the steering information acquired by the steering information acquisition unit 71 and input by the operator operating the input unit 52 to the vehicle 34.

[0174] In step S35, the steering control unit 141 controls the communication unit 112 to acquire steering information transmitted from the operator terminal 32.

[0175] In step S36, the steering control unit 141 controls the vehicle control unit 122 based on the steering information to control the running of the vehicle 34.

[0176] In steps S37 and S60, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to steps S31 and S51, respectively, and the subsequent steps are repeated.

[0177] Then, in steps S37 and S60, when an instruction to end the process is given, the process ends.

[0178] Through the above series of processes, based on the vehicle information from the vehicle 34, the display unit 81 of the operator terminal 32 can display an image of the vehicle and its surrounding objects in a display method that corresponds to the delay time related to communication, and at a predicted position that corresponds to the delay time related to communication.

[0179] This allows the operator 31 to operate the vehicle 34 while properly understanding the positional relationship between the vehicle's own position and surrounding objects, while viewing the display image displayed on the display unit 81 in a state in which the effects of communication delays have been reduced.

[0180] As a result, it is possible to improve the operability and safety of remote control using a remote control system.

[0181] <Vehicle Position Prediction Processing> Next, the vehicle position prediction processing will be described with reference to the flowchart of FIG.

[0182] In step S71, the current position prediction unit 74 acquires the steering angle S of the vehicle 34 in the vehicle information.

[0183] In step S72, the current position prediction unit 74 acquires the speed V of the vehicle 34 from the vehicle information.

[0184] In step S73 , the current position prediction unit 74 acquires the delay time T acquired by the delay time acquisition unit 72 .

[0185] In step S74, the current position prediction unit 74 acquires the position information Pc in the vehicle information.

[0186] In step S75, the current position prediction unit 74 reads out, for example, an arithmetic formula (Pp = Pc + f(S, V, T)) as a prediction method required for predicting the current position of the vehicle stored in the prediction method memory unit 91, and predicts the current position of the vehicle based on the steering angle S, speed V, delay time T, and position information Pc.

[0187] For example, when the steering angle is other than 0 degrees and the steering angle is set in any direction, the current position of the vehicle 34 is predicted as shown in Fig. 12. Here, the position of the vehicle 34 at the time of transmitting the vehicle information is defined by position information Pc(X, Z), and the steering angle S is set in the X direction, which is the right direction in the figure, relative to the Z direction, which is the upward direction in the figure, and when the vehicle moves at a speed V for a delay time T, X' and Z' in the predicted position Pp(X', Z') of the vehicle 34 at the current time are expressed by the following equations (1) and (2), respectively.

[0188] X'=R-Rcos(ωT+θ0)[m]...(1)

[0189] Z'=Rsin(ωT+θ0)[m]...(2)

[0190] Here, R is the turning radius of the vehicle 34, and is expressed by the following equation (3).

[0191] R=W / sinS[m]...(3)

[0192] Here, W is the wheelbase length of the vehicle 34, and S is the steering angle.

[0193] Furthermore, ω is the yaw angular velocity of the vehicle 34, and is expressed by the following equation (4).

[0194] ω=(1000V / 3600) / R[rad / s]...(4)

[0195] Furthermore, when the steering angle is 0, the vehicle 34 moves linearly in the Z direction, which is the upward direction in FIG. 12, and therefore X′ and Z′ at the predicted position Pp(X′, Z′) are expressed by the following equations (5) and (6), respectively.

[0196] X'=0[m]...(5)

[0197] Z'=(1000V / 3600)×T[m]...(6)

[0198] Through the above calculations, the operator terminal 32 can estimate the distance traveled by the vehicle 34 during the communication delay time based on the position information of the vehicle information, and predict the current position.

[0199] <Vehicle Position Display Method Determination Process> Next, the vehicle position display method determination process will be described with reference to the flowchart in Fig. 13. As described above, the vehicle position display method requires the determination of two elements: the display viewpoint and the display information.

[0200] That is, in step S91, the display image generating unit 75 executes a display viewpoint determination process to determine a display viewpoint.

[0201] The display viewpoint determination process will be described in detail later with reference to the flowchart of FIG.

[0202] In step S92, the display image creating unit 75 executes a display information determination process to determine the display information.

[0203] The display information determination process will be described in detail later with reference to the flowchart of FIG.

[0204] The display viewpoint and display information are determined by the above processing, and as shown in Figure 6, the display viewpoint is determined to be either a bird's-eye view, a third-person view, or a first-person view, and further, the display information is determined to be either the position only, the vehicle body angle, or the route, and the display method for the display image is specified.

[0205] <Display Viewpoint Determination Processing> Next, the display viewpoint determination processing will be described with reference to the flowchart of FIG.

[0206] In step S111, the display image creation unit 75 determines whether the delay time T is equal to or greater than a predetermined time N1.

[0207] If it is determined in step S111 that the delay time T is equal to or greater than the predetermined time N1, the process proceeds to step S112.

[0208] In step S112, the display image creating unit 75 determines the display viewpoint to be a bird's-eye view.

[0209] If it is determined in step S111 that the delay time T is not equal to or greater than the predetermined time N1, the process proceeds to step S113.

[0210] In step S113, the display image creation unit 75 determines whether the delay time T is equal to or greater than a predetermined time N2 (<N1).

[0211] If it is determined in step S113 that the delay time T is equal to or greater than the predetermined time N2, the process proceeds to step S114.

[0212] In step S114, the display image creation unit 75 determines the display viewpoint to be the third-person viewpoint.

[0213] If it is determined in step S113 that the delay time T is not equal to or greater than the predetermined time N2, the process proceeds to step S115.

[0214] In step S115, the display image creation unit 75 determines whether or not there is a 3D map required to create a first-person perspective view.

[0215] If it is determined in step S115 that there are 3D maps required to generate a first-person view, the process proceeds to step S116.

[0216] In step S116, the display image creation unit 75 determines the display viewpoint to be the first-person viewpoint.

[0217] If it is determined in step S115 that there is no 3D map required to generate a first-person view, the process proceeds to step S114, where the display viewpoint is determined to be a third-person view.

[0218] That is, depending on the length of the delay time T, a display viewpoint that can generate a display image with a small processing load for generating the display image is selected.

[0219] In this case, if the delay time T is longer than the longest predetermined time N1, the bird's-eye view, which has the smallest processing load and is least likely to cause a delay in generating the display image, is selected.

[0220] Furthermore, if the delay time T is shorter than the predetermined time N1 but longer than the predetermined time N2 (<N1), that is, if there is enough time to generate a bird's-eye view but not enough time to generate a first-person perspective view, a third-person perspective view is selected, which imposes a greater processing load than a bird's-eye view but is less than displaying a first-person perspective view.

[0221] Furthermore, if the delay time T is shorter than the predetermined time N2 and there is sufficient time to generate a first-person perspective view, and the 3D map information required for the first-person perspective view exists, a first-person perspective view that places a small visual burden on the operator 31 is selected.

[0222] However, even if the delay time T is shorter than the predetermined time N2 and there is sufficient time to generate a first-person viewpoint, if the 3D map information required for the first-person viewpoint does not exist, the third-person viewpoint is selected.

[0223] By performing such processing, it is possible to determine an appropriate display viewpoint depending on the delay time T, the margin of the processing load required to generate the display image, and the presence or absence of 3D map information.

[0224] <Display Information Determination Process> Next, the display information determination process will be described with reference to the flowchart of FIG.

[0225] In step S131, the display image creation unit 75 determines whether the number of vehicles surrounding the host vehicle is equal to or less than a predetermined number based on, for example, an image captured by a camera, which is included in the surrounding object information.

[0226] If it is determined in step S131 that the number of nearby vehicles is equal to or less than the predetermined number, the process proceeds to step S132.

[0227] In step S132, the display image creation unit 75 determines the route to the destination as the display information, since there are few surrounding vehicles, multiple displays are displayed, visibility is not greatly reduced, and there is little need to be aware of the surroundings.

[0228] If it is determined in step S131 that the number of nearby vehicles is not equal to or less than the predetermined number, the process proceeds to step S133.

[0229] In step S133, the display image generating unit 75 determines whether or not the vehicle is traveling on a road with a continuous straight line, based on an image captured by a camera, for example, which is included in the surrounding object information.

[0230] If it is determined in step S133 that the vehicle is traveling on a road consisting of a series of straight lines, the process proceeds to step S134.

[0231] In step S134, the display image creation unit 75 determines only the position as display information, since even if there are a number of surrounding vehicles, as long as the vehicle is traveling on a straight road, the traveling routes of the surrounding vehicles and the like are unlikely to change significantly.

[0232] If it is determined in step S133 that the vehicle is not traveling on a road consisting of continuous straight lines, the process proceeds to step S135.

[0233] In step S135, the display image creation unit 75 determines the vehicle body angle as display information because, even if there are many surrounding vehicles, the vehicle is traveling on a road that is not a continuous straight line, and it is necessary to take into consideration that the driving paths of surrounding vehicles, etc. may change significantly.

[0234] That is, if there are few vehicles in the vicinity, the route is displayed because it is considered that the route will not be superimposed on the surrounding vehicles even if it is displayed.

[0235] In addition, if there are many surrounding vehicles, the system determines whether the road is straight or not. If the road is a series of straight lines, steering operation is likely to be minimal, so only the position is displayed. In other cases, the system displays the vehicle body angle to provide information that makes it easier to make decisions regarding steering operation.

[0236] <Peripheral Object Prediction Processing> Next, the peripheral object prediction processing will be described with reference to the flowchart of FIG.

[0237] In step S151 , the current position prediction unit 74 acquires the delay time T acquired by the delay time acquisition unit 72 .

[0238] In step S152, the current position prediction unit 74 acquires the position information of the vehicle itself based on the vehicle information.

[0239] In step S153, the current position prediction unit 74 acquires sensing results acquired by the sensor information acquisition unit 142 included in the vehicle information, which constitute surrounding object information that enables the positions and distances of other vehicles and pedestrians to be recognized from the vehicle 34.

[0240] In step S154, the current position prediction unit 74 individually recognizes the objects around the vehicle based on the sensing results that constitute the surrounding object information, and calculates the position information of each of the surrounding objects with the vehicle as the origin.

[0241] In step S155, the current position prediction unit 74 sets an unprocessed object among the objects in the vicinity of the host vehicle as an object to be processed.

[0242] In step S156, the current position prediction unit 74 accesses the position information storage unit 92 and determines whether or not N frames of position information relating to the object to be processed have been stored.

[0243] If it is determined in step S156 that position information relating to the object to be processed has been accumulated for N frames, the process proceeds to step S157.

[0244] In step S157, the current position prediction unit 74 acquires N frames of position information relating to the object to be processed, which is stored in the position information storage unit 92.

[0245] In step S158, the current position prediction unit 74 calculates the amount of movement corresponding to the delay time from changes in the time-series position information for N frames based on the position information for the object to be processed and the position information for the object to be processed calculated based on the peripheral object information of the acquired vehicle information, and predicts the current position of the object to be processed by adding the amount of movement to the position information for the object to be processed calculated based on the peripheral object information of the acquired vehicle information.

[0246] In step S159, the current position prediction unit 74 stores information on the current position, which is the prediction result of the object to be processed, in the position information storage unit 92 in association with information identifying the object to be processed.

[0247] If it is determined in step S156 that position information relating to the object to be processed has not been accumulated for N frames, steps S157 and S158 are skipped.

[0248] In other words, in this case, since the current position of the object to be processed cannot be predicted (estimated), the position information recognized from the surrounding object information is stored in the position information storage unit 92 as the current position information of the object to be processed.

[0249] In step S160, the current position prediction unit 74 determines whether or not there is an unprocessed peripheral object, and if there is an unprocessed peripheral object, the process returns to step S155.

[0250] That is, the processing of steps S155 to S160 is repeated for all surrounding objects until the current positions are predicted, updated, and registered in the position information storage unit 92, or until the position information of the surrounding object information is registered in the position information storage unit 92.

[0251] Then, in step S160, if it is determined that the current positions of all surrounding objects have been predicted and updated and registered in the position information storage unit 92, or if it is determined that the position information of the surrounding object information has been registered in the position information storage unit 92, the processing ends.

[0252] Through the above process, the current positions of the surrounding objects are predicted and stored sequentially, and the current positions are predicted based on the stored information. Therefore, it is possible to predict the current positions of the surrounding objects while improving the prediction accuracy each time the prediction is repeated.

[0253] As a result, it becomes possible to appropriately predict the current positions of surrounding objects.

[0254] <Peripheral Object Display Method Decision Processing> Next, the peripheral object display method decision processing will be described with reference to the flowchart of FIG.

[0255] In step S171, the display image creating unit 75 determines whether the delay time T is equal to or greater than a predetermined time M1.

[0256] If it is determined in step S171 that the delay time T is equal to or greater than the predetermined time M1, the process proceeds to step S172.

[0257] In step S172, the display image creation unit 75 sets the display method of only the predicted position as the peripheral object display method.

[0258] If it is determined in step S171 that the delay time T is not equal to or greater than the predetermined time M1, the process proceeds to step S173.

[0259] In step S173, the display image creation unit 75 determines whether the delay time T is equal to or greater than a predetermined time M2 (<M1).

[0260] If it is determined in step S173 that the delay time T is equal to or greater than the predetermined time M2, the process proceeds to step S174.

[0261] In step S174, the display image creation unit 75 sets the display method for displaying in the area to the peripheral object display method.

[0262] If it is determined in step S174 that the delay time T is not equal to or greater than the predetermined time M2, the process proceeds to step S175.

[0263] In step S175, the display image creation unit 75 sets the display method for superimposing and displaying the predicted position as the peripheral object display method.

[0264] With the above processing, if the delay time T is longer than the longest predetermined time M1, sufficient time cannot be secured to generate the display image, so only the predicted position that has the lightest processing load and allows high-speed processing is used.

[0265] Furthermore, when the delay time T is shorter than the predetermined time M1 but longer than the predetermined time M2 (<M1), there is enough time to generate a display image of only the predicted position, but when there is not enough time to generate a display image that superimposes the predicted position, an area display is used.

[0266] When the delay time T is shorter than the predetermined time M2 and a certain amount of time for generating the display image can be secured, the predicted position is superimposed and displayed.

[0267] This makes it possible to appropriately switch and display the display method of the peripheral objects depending on the processing time and load involved in generating the display image based on the delay time.

[0268] <Modification of Bird's-Eye View> In the above, as described with reference to FIG. 6, an example in which a bird's-eye view is displayed as 2D data has been described.

[0269] However, as described with reference to FIG. 6, the bird's-eye view may not only be displayed as 2D data, but may also be an illustrated bird's-eye view or a camera-image bird's-eye view.

[0270] Fig. 18 shows examples of display images when an illustrated bird's-eye view and a camera image bird's-eye view are used as display viewpoints. The left part of Fig. 18 is a display image BV1 showing an example of the illustrated bird's-eye view, and the right part of Fig. 18 is a display image BV2 showing an example of the camera image bird's-eye view.

[0271] The display image BV1, which is an illustrated bird's-eye view of Figure 18, is an example in which a vehicle straddling a lane is illustrated, one moving from the left toward the bottom in the figure, and the other moving toward the top in the figure. By using illustrations to draw the display image BV1, the required data can be made smaller, which also reduces the processing load related to generating and drawing the display image, and shortens the processing time.

[0272] The display image BV2 of FIG. 18, which is a bird's-eye view of a camera image, is a bird's-eye view of a camera image that shows the state of the vehicle as seen from directly above, that is, a so-called around-view monitor image.

[0273] Compared to display image BV1, which is an illustrated bird's-eye view, display image BV2, which is a camera image bird's-eye view, requires communication of camera images of the surroundings of vehicle 34 for vehicle information, which places a greater communication load and may result in communication delays.

[0274] However, since the display image BV2, which is a bird's-eye view of a camera image, is an image in which the vehicle is always present in the center position, the amount of data involved in communication can be reduced compared to a third-person perspective view, so the processing load involved in generating the display image is smaller than that of a third-person perspective view, and the processing time can also be shortened.

[0275] <Display Viewpoint Determination Process Including Illustration Bird's-Eye View and Camera Image Bird's-Eye View> Next, the display viewpoint determination process including the illustration bird's-eye view and camera image bird's-eye view of FIG. 18 will be described with reference to the flowchart of FIG.

[0276] The processes in steps S195 to S198 in the flowchart of FIG. 19 are the same as the processes in steps S113 to S116 described with reference to the flowchart of FIG. 14, and therefore will not be described again.

[0277] That is, in step S191, the display image generating unit 75 determines whether the delay time T is equal to or greater than a predetermined time N0 (>T1).

[0278] If it is determined in step S191 that the delay time T is equal to or greater than the predetermined time N0 (>T1), the process proceeds to step S192.

[0279] In step S192, the display image creation unit 75 determines the display viewpoint to be the illustration bird's-eye view.

[0280] If it is determined in step S191 that the delay time T is not equal to or greater than the predetermined time N0 (>T1), the process proceeds to step S193.

[0281] In step S193, the display image creating unit 75 determines whether the delay time T is equal to or greater than a predetermined time N1 (<N0).

[0282] If it is determined in step S193 that the delay time T is equal to or greater than the predetermined time N1 (<N0), the process proceeds to step S194.

[0283] In step S194, the display image creating unit 75 determines the display viewpoint to be the bird's-eye view camera image.

[0284] If it is determined in step S193 that the delay time T is not equal to or greater than the predetermined time N1 (<N0), the process proceeds to step S195, where the view is set to either a third-person perspective view or a first-person perspective view.

[0285] That is, depending on the length of the delay time T, when it is longer than a predetermined time N0 (> N1), an illustrated bird's-eye view is selected from among the bird's-eye views, and when it is shorter than the predetermined time N0 (> N1) and longer than the predetermined time N1, a camera image bird's-eye view is selected from among the bird's-eye views.

[0286] By this processing, a display viewpoint is appropriately set in accordance with the length of the delay time T, taking into consideration visibility and processing time.

[0287] <Example of including warning information in surrounding object display method> Next, for example, in the case where the delay time T is extremely long and the accuracy of position estimation is reduced, making remote control dangerous, the display information may include a specific indication of the delay time or warning information to stop nearby and end remote control, which may be used as one of the surrounding information display methods.

[0288] The warning information may be displayed in a warning message display field WT on the display image, as shown in the display image WV1 in Fig. 20. The warning message display field WT of the display image WV1 displays "A communication delay has occurred. Please stop the vehicle." Since the delay time T is long and the accuracy of the displayed information cannot be guaranteed, information is presented urging the user to stop the vehicle 34 and end remote control.

[0289] Furthermore, the warning information may be displayed as a warning mark WC, as shown in the display image WV2 of Fig. 20. The color and size of the warning mark WC change depending on the length of the delay time.

[0290] That is, the warning mark WC may be displayed in blue and in a smaller size as the delay time T becomes shorter, for example. The warning mark WC may change from blue to yellow and gradually increase in size as the delay time T becomes longer. Furthermore, if the delay time T is longer than a predetermined time and it is necessary to stop the vehicle and stop remote control, the warning mark WC may change from yellow to red and increase in size to the extent that remote control is difficult while viewing the display image WV2.

[0291] Furthermore, the warning information may be displayed in a manner that reduces visibility by applying semi-transparent hatching to the entire image, as shown in display image WV3 in Fig. 20. By displaying such a warning, the operator 31 may be made to feel that it is difficult to control the vehicle, thereby encouraging the operator 31 to reduce speed voluntarily.

[0292] As described above, depending on the delay time, if the delay time is longer than a predetermined time, the prediction accuracy of the vehicle position and surrounding objects decreases, and remote monitoring and operation becomes dangerous, safety can be ensured by displaying a warning as shown in display images WV1 to WV3 in Figure 20.

[0293] 20 is a display image generated by displaying only the predicted position, etc., in which information for making the driver aware of the warning is superimposed on the display image. Therefore, when the warning display is selected as the method for displaying nearby vehicles, for example, nearby objects are represented by a method for displaying only the predicted position, and then a warning message display field WT, a warning mark WC, and hatching are applied.

[0294] <Peripheral Object Display Method Decision Processing Including Warning Information> Next, peripheral object display method decision processing including warning information will be described with reference to the flowchart of FIG.

[0295] In the flowchart of FIG. 21, the processes of steps S213 to S217 are the same as the processes of steps S171 to S175 described with reference to the flowchart of FIG. 17, and therefore a description thereof will be omitted.

[0296] That is, in step S211, the display image generating unit 75 determines whether the delay time T is equal to or greater than a predetermined time M0 (>M1).

[0297] If it is determined in step S211 that the delay time T is equal to or greater than the predetermined time M0, the process proceeds to step S212.

[0298] In step S2122, the display image creation unit 75 sets the warning display as shown in the display images WV1 to WV3 described with reference to FIG. 20 as the peripheral object display method.

[0299] If it is determined in step S211 that the delay time T is not equal to or greater than the predetermined time M0, the process proceeds to step S213, where the method for displaying surrounding objects is determined to be either displaying the predicted position only, displaying an area, or displaying the predicted position superimposed.

[0300] By the above processing, the predetermined time M0 is set to the time at which the delay time T is expected to cause problems with remote control, and if this time is exceeded, a warning message will be displayed as shown in the display images WV1 to WV3 in Figure 20, thereby ensuring safety.

[0301] <Process for determining the peripheral object display method based on conditions other than delay time> In the above, an example has been described in which the peripheral object display method is determined based on the delay time T, but the peripheral object display method may also be switched based on conditions other than the delay time.

[0302] For example, when the delay time T is not longer than the predetermined time M1, the delay time is relatively short, so rather than making a judgment based on the delay time T, it is also possible to make a judgment based on, for example, whether the number of surrounding vehicles is less than a predetermined number.

[0303] In this case, when there are more than a predetermined number of surrounding vehicles or when the road width is narrow, information about many surrounding vehicles will be presented on the display image, which may result in a cluttered display image and reduced visibility, so for example, it may be possible to set only the position as a method for displaying surrounding objects.

[0304] Conversely, when the number of surrounding vehicles is less than a predetermined number, the amount of information about the surrounding vehicles presented on the display image will be reduced, resulting in a clear, easy-to-read display image and ensuring visibility, so the area display method may be set as the surrounding object display method. In other words, the determination may be based on visibility rather than on delay time.

[0305] FIG. 22 is a flowchart illustrating a peripheral object display method determination process in which it is determined whether the number of peripheral vehicles is less than a predetermined number when the delay time T is not equal to or greater than a predetermined time M1.

[0306] In the flowchart of FIG. 22, the processes of steps S231, S232, S234, and S235 are the same as the processes of steps S171, S172, S174, and S175 described with reference to the flowchart of FIG. 17, and therefore description thereof will be omitted.

[0307] That is, in the flowchart of FIG. 22, if it is determined in step S231 that the delay time T is not equal to or greater than the predetermined time M1, the process proceeds to step S233.

[0308] In step S233, the display image creation unit 75 determines whether the number of surrounding vehicles is less than a predetermined number based on the surrounding object information.

[0309] If it is determined in step S233 that the number of surrounding vehicles is less than a predetermined number, the process may proceed to step S234, whereby area display may be set as the method for displaying surrounding objects.

[0310] In step S233, if the number of surrounding vehicles is greater than the predetermined number, the process proceeds to step S235, where only the positions are set as the surrounding object display method.

[0311] By the above processing, it becomes possible to select a method for displaying surrounding objects taking into consideration not only delay time but also visibility according to the number of surrounding vehicles, road width, etc.

[0312] <Application Example (1)> In the above, we have described an example in which the display method for the vehicle itself and the surrounding vehicles is switched depending on the delay time, but it is also possible to present information related to driving.

[0313] Bird's-eye views and third-person perspective views clearly indicate the user's predicted position, making it easier to focus on the user's position. Therefore, by presenting information that does not require the user to predict changes in position, such as traffic lights and traffic information, in the vicinity of the user's easily noticeable position information, the user's line of sight may be reduced, making it easier to grasp the information.

[0314] For example, as shown in the display image WV11 on the left side of FIG. 23, the speed of the vehicle and information about traffic lights ahead may be displayed in characters or symbols.

[0315] In the display image WV11, the vehicle's traveling speed is displayed on the vehicle position mark WL in the lower right corner, which indicates the vehicle's position, and is displayed as "52 km / h" in Fig. 23. In addition, a traffic light mark SG is displayed to the upper right of the mark WL, and in Fig. 23, it is displayed that the rightmost red light is lit.

[0316] As shown in the display image WV11, the speed is displayed on the vehicle position mark WL, which is likely to attract the operator's attention, and the lighting information of the nearest traffic light is displayed nearby, thereby minimizing the operator's 31 eye movement and making it easier for the operator to grasp the information.

[0317] Furthermore, as shown in the display image WV12 on the right side of Figure 23, the color, color intensity, and size of the vehicle position mark WS indicating the position of the vehicle may be displayed so that the speed is perceived by changing depending on the speed.

[0318] That is, in the display image WV12 of Figure 23, the vehicle position mark WS is displayed with its color, intensity, and size changing according to the speed of the vehicle, and although the exact speed is unknown, it is possible to intuitively perceive how fast the vehicle is traveling.

[0319] The vehicle position mark WS may change in shape in addition to color, intensity, and size depending on the speed of the vehicle.

[0320] More specifically, when the vehicle speed is close to the legal speed limit, the vehicle position mark WS may be displayed with high transparency and small size. In other words, since the current speed is within the legal speed limit, there is no need to be aware of the current speed, and the display can be made to make the surrounding information easier to see, allowing the driver to concentrate more on driving.

[0321] Furthermore, if the vehicle speed exceeds the legal speed limit, the vehicle position mark WS may be displayed with low transparency and large size. In other words, since the current speed exceeds the legal speed limit, the surrounding information may be made somewhat less visible to make the driver aware of the current speed, thereby making it possible to display the mark in a way that encourages the driver to slow down.

[0322] In either case, it is possible to intuitively recognize the speed. In the case of remote control, if the speed is exceeded, the visibility of surrounding information will be reduced more than usual, which is expected to naturally encourage deceleration.

[0323] <Application Example (Part 2)> In the above, an example has been described in which the predicted positions of nearby vehicles are presented taking into consideration the delay time associated with communication. However, it is also possible to set a risk level according to the type of nearby object or the type of behavior, and display the predicted positions by applying an offset according to the risk level.

[0324] The type of surrounding object may be determined using information from the position information accumulation unit 92. For example, as shown in the left part of Fig. 24, "children and elderly people" may be set to a danger level of 10, "pedestrians" to a danger level of 9, "bicycles" to a danger level of 8, "motorcycles" to a danger level of 7, "light cars" to a danger level of 6, "passenger cars" to a danger level of 5, and "trucks" to a danger level of 4. The higher the danger level number, the higher the danger level.

[0325] In addition, as a risk level according to the type of behavior, for example, as shown in the right part of Figure 24, the risk level for behaviors such as "close distance to other vehicles" and "changing lanes without using turn signals" is set to "high," and the risk level for behaviors such as "frequent lane changes" and "sudden acceleration / deceleration, high speed" is set to "medium."

[0326] For example, consider a case where, as shown in Figure 24, the risk level is set according to the type of vehicle and the type of behavior, and the range in which the target object may exist is depicted with a box-shaped mark, as shown in the display example (2-1) in Figure 7.

[0327] Here, for example, in the case of a normal display state in which the offset according to the risk level is not taken into consideration, the surrounding vehicles are displayed as marks MB and the own vehicle is displayed as mark MC, as shown in the display image OFSOFF on the left side of Fig. 25. Here, the distance between the marks MB and MC is expressed as the distance between the ends Pb and Pc.

[0328] In contrast to this, for example, in a display state in which an offset according to the risk level is taken into consideration, the surrounding vehicles are displayed as mark MB' and the own vehicle is displayed as mark MC, as shown in the display image OFSON on the right side of Fig. 25. As shown in the display image OFSON, the mark MB' is set at a position where an offset is applied so that the end Pb' is closer to the mark MC of the own vehicle with respect to the end Pb of the mark MB by a width Da according to the value of the risk level of the type.

[0329] Such a display will cause the operator 31 to perceive the surrounding object for which a danger level has been set as being closer than the predicted positional relationship, thereby encouraging the operator to have a stronger sense of danger.

[0330] Furthermore, by taking into consideration the magnitude of the offset width Da according to the degree of danger set according to the type as well as the degree of danger set according to the behavior, the operator 31 can recognize the sense of crisis according to the degree of danger simply by looking at the distance between the marks MB' and MC.

[0331] Furthermore, the display may be made more visible depending on the degree of danger. For example, the higher the degree of danger, the darker the color may be displayed.

[0332] In the display image WV31 on the left side of Figure 26, for example, a mark MB1 of a nearby object consisting of a motorcycle is displayed on the left side of the figure, and a mark MC1 of a nearby object consisting of a passenger car is displayed on the right side of the figure.

[0333] 24, the danger level for motorcycles is 7 and the danger level for passenger cars is 5, so the marks MB1 and MC1 on the display image WV31 are expressed with densities corresponding to the respective set danger levels. Furthermore, when setting the density corresponding to the danger level, the danger level classified by behavior may also be taken into account in addition to the danger level set based on the type.

[0334] Also, for example, the lower the degree of danger, the lower the resolution at which the object is displayed, so that surrounding objects with a high degree of danger can be recognized in detail.

[0335] For example, in the display image WV32 second from the left in Fig. 26, a nearby object mark MB2 consisting of a motorcycle is displayed on the left side of the figure, and a nearby object mark MC2 consisting of a passenger car is displayed on the right side of the figure. In Fig. 26, the marks MB2 and MC2 are displayed at resolutions corresponding to the motorcycle danger level of 7 and the passenger car danger level of 5, respectively, which are set in Fig. 24. Therefore, since the danger level of motorcycles is higher than that of passenger cars, the passenger car mark MC2 has a lower resolution than the motorcycle mark MB2.

[0336] Furthermore, for example, items with a high degree of risk may be displayed surrounded by a frame.

[0337] For example, in the third display image WV33 from the left in Fig. 26, a mark MB3 representing a nearby object consisting of a motorcycle is displayed on the left side of the figure, and a mark MC3 representing a nearby object consisting of a passenger car is displayed on the right side of the figure. In Fig. 24, the danger level for motorcycles is 7 and the danger level for passenger cars is 5, so for example, motorcycles with a danger level higher than 6 are surrounded by a frame as shown by mark MB3, and the mark MC3 for passenger cars with a danger level lower than 6 is not surrounded by a frame.

[0338] Furthermore, for example, an object with a high degree of risk may be displayed with its outline emphasized.

[0339] For example, in the fourth display image WV34 from the left in Fig. 26, a mark MB4 representing a nearby object consisting of a motorcycle is displayed on the left side of the figure, and a mark MC4 representing a nearby object consisting of a passenger car is displayed on the right side of the figure. In Fig. 24, the danger level for motorcycles is 7 and the danger level for passenger cars is 5, so for example, motorcycles with a danger level higher than 6 are highlighted by having their outlines surrounded by a thick line, as shown by mark MB4, while mark MC3 for passenger cars with a danger level lower than 6 is not highlighted.

[0340] Furthermore, for example, the higher the risk, the wider the mark displayed.

[0341] For example, in the display image WV35 on the right side of Fig. 26, a mark MB5 representing a nearby object consisting of a motorcycle is displayed on the left side of the drawing, and a mark MC5 representing a nearby object consisting of a passenger car is displayed on the right side of the drawing. In Fig. 24, the danger level for motorcycles is 7 and the danger level for passenger cars is 5, so the marks MB5 and MC5 are displayed with widths corresponding to their respective danger levels.

[0342] Note that the width of the mark is not simply set according to the level of danger, but is set wider depending on the level of danger than the default width set for each mark. Therefore, in Figure 26, mark MB5 is narrower than mark MC5, but this does not mean that the motorcycle represented by mark MB5 is less dangerous than the passenger car represented by mark MC5. This is because marks MB5 and MC5 are respectively modeled after a motorcycle and a passenger car, and the default width of both marks MB5 is narrower than the width of mark MC5.

[0343] 26 shows that the width of mark MB5 is wider than the default size by the amount of risk. Conversely, the width of mark MC5 remains the default size because the risk is smaller than threshold 6.

[0344] <<3. Second Embodiment>> In the above, the remote control system 11 has been described, which is an example of a configuration in which the operator terminal 32 and the vehicle 34 transmit and receive vehicle information and steering information via the network 33 .

[0345] However, by using V2V communication, a specific vehicle 34 may collect vehicle information of surrounding vehicles through vehicle-to-vehicle communication, and transmit the vehicle information of multiple vehicles 34 to the operator terminal 32 on behalf of the surrounding vehicles.

[0346] Fig. 27 shows an example of the configuration of a remote control system 11' that uses V2V communication. In Fig. 27, components that have the same functions as those in the remote control system 11 of Fig. 2 are given the same reference numerals, and their explanations will be omitted as appropriate.

[0347] The above-described vehicle 34 has a function for realizing V2V communication. Therefore, for example, in Fig. 27, when vehicle 34-1 acquires vehicle information from vehicles 34-2 and 34-m through vehicle-to-vehicle communication, the vehicle information may be transmitted to operator terminals 32-1 to 32-n via network 33 together with the vehicle information of the vehicle itself.

[0348] When any of the operator terminals 32-1 to 32-n receives vehicle information from multiple vehicles 34, the information may be distributed to other operator terminals 32 for processing, or a single operator terminal 32 may use the vehicle information from multiple vehicles 34 to remotely control multiple vehicles 34.

[0349] <Remote Control Processing (V2V)> Next, with reference to the flowchart of FIG. 28, the remote control processing by the remote control system 11′ using the V2V communication of FIG. 27 will be described.

[0350] The processing of steps S292 to S300 by the operator terminal 32 is similar to the processing of steps S52 to S60 in the flowchart of Fig. 10, and therefore a description thereof will be omitted. Also, the processing of steps S255 to S257 and the processing of steps S277 to S279 are similar to the processing of steps S35 to S37 in the flowchart of Fig. 10, and therefore a description thereof will be omitted.

[0351] Furthermore, since the processes other than the process of transmitting and receiving vehicle information are basically the same as those described with reference to the flowchart of FIG. 10, descriptions of the processes other than the process of transmitting and receiving vehicle information will be omitted as appropriate.

[0352] Furthermore, in the flowchart of Figure 28, the flowchart on the left is the processing of vehicles 34-2 and 34-m, which are surrounding vehicles as seen from vehicle 34-1 in Figure 27, the flowchart in the center is the processing of vehicle 34-1, and the flowchart on the right is the processing of operator terminal 32.

[0353] That is, in steps S251 to S254, the sensor information acquisition unit 142 of the surrounding vehicles 34-2, 34-m acquires surrounding object information such as camera images capturing the surroundings and distance images that allow the distance to surrounding objects to be recognized, as well as the position information of the vehicle itself, and the steering angle (steering angle) and speed information of the vehicle itself detected by the vehicle sensor 117.

[0354] Then, in step S254, the vehicle information communication unit 143 of the surrounding vehicles 34-2 and 34-m compiles the vehicle position information acquired by the position information acquisition unit 114, surrounding object information consisting of sensing results acquired by the sensor information acquisition unit 142 that enable the position and distance of other vehicles and pedestrians to be recognized from vehicle 34, and the steering angle (steering angle) and speed information of the surrounding vehicle detected by the vehicle sensor 117 to form vehicle information, and controls the communication unit 112 to transmit the vehicle information to vehicle 34-1 via vehicle-to-vehicle communication.

[0355] In steps S271 to S273, the sensor information acquisition unit 142 of vehicle 34-1 acquires surrounding object information such as camera images capturing the surroundings and distance images that enable the distance to surrounding objects to be recognized, position information of the vehicle itself, and steering angle (steering angle) and speed information of the vehicle itself detected by the vehicle sensor 117.

[0356] Then, in step S274, the vehicle information communication unit 143 of vehicle 34-1 compiles the vehicle information, including the position information of the vehicle itself acquired by the position information acquisition unit 114, surrounding object information consisting of sensing results acquired by the sensor information acquisition unit 142 that enable the position and distance of other vehicles and pedestrians to be recognized from vehicle 34, and the steering angle (steering angle) and speed information of the vehicle itself detected by the vehicle sensor 117.

[0357] In step S275, the vehicle information communication unit 143 of the vehicle 34-1 controls the communication unit 112 to receive the vehicle information transmitted from the vehicles 34-2 and 34-m via inter-vehicle communication.

[0358] In step S276, the vehicle information communication unit 143 of the vehicle 34-1 controls the communication unit 112 to transmit to the operator terminal 32 the vehicle information of the vehicle itself and the vehicle information of the surrounding vehicles 34-2 and 34-m.

[0359] In step S291, the vehicle information acquisition unit 73 controls the communication unit 55 to acquire the vehicle information of the vehicles 34-1, 34-2, and 34-m transmitted from the vehicle 34-1.

[0360] At this time, the vehicle information acquisition unit 73 associates the time information at the timing when the vehicle information of the vehicles 34-1, 34-2, and 34-m is acquired with the vehicle information.

[0361] Through the subsequent processing, steering information is generated individually based on the vehicle information of the vehicles 34-1, 34-2, and 34-m, and is supplied individually.

[0362] The above processing reduces the communication load between the operator terminal 32 and the vehicle 34, making it possible to reduce communication delays overall.

[0363] <<4. Modification of the Second Embodiment>> The above has described the remote control system 11′ having an example configuration in which vehicle information and steering information are transmitted and received between the operator terminal 32 and the vehicle 34 using V2V communication.

[0364] However, vehicle information may be transmitted to the operator terminal 32 by communicating with equipment other than the vehicle 34 using V2X communication.

[0365] FIG. 29 shows an example configuration of a remote control system 11″ that uses V2X communication. Note that in FIG. 29, components having the same functions as those in the remote control system 11 in FIG. 2 and the remote control system 11′ in FIG. 27 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0366] The remote control system 11'' in Figure 29 differs from the remote control system 11 in Figure 2 and the remote control system 11' in Figure 27 in that it further includes a traffic information cloud server 201 and on-road communication equipment 202.

[0367] The traffic information cloud server 201 is a cloud server that collects information on traveling vehicles 34 and infrastructure on roads, acquires vehicle information from the vehicles 34 and supplies it to the operator terminal 32 .

[0368] The road communication equipment 202 is a device that acquires information on road facilities such as traffic lights, and transmits the road facility information to the traffic information cloud server 201 via the network 33 .

[0369] The traffic information cloud server 201 provides vehicle information and road facility information to the operator terminal 32 .

[0370] The operator terminal 32 acquires vehicle information of the vehicle 34 supplied via the traffic information cloud server 201 and supplies steering information. At this time, by also acquiring corresponding road facility information, it is possible to recognize, for example, the control status of traffic lights and the operating status of railroad crossings, and use this information for remote control.

[0371] <Remote Control Processing (V2X)> Next, with reference to the flowchart of FIG. 30 , the remote control processing by the remote control system 11″ using V2X communication in FIG. 29 will be described.

[0372] The processing of steps S352 to S360 by the operator terminal 32 is similar to the processing of steps S52 to S60 in the flowchart of Fig. 10, and therefore a description thereof will be omitted. Also, the processing of steps S315 to S317 is similar to the processing of steps S35 to S37 in the flowchart of Fig. 10, and therefore a description thereof will be omitted.

[0373] Furthermore, since the processes other than the process of transmitting and receiving vehicle information are basically the same as those described with reference to the flowchart of FIG. 10, descriptions of the processes other than the process of transmitting and receiving vehicle information will be omitted as appropriate.

[0374] Furthermore, in the flowchart of FIG. 30, the flowchart on the left is the processing of the vehicle 34 in FIG. 27, the flowchart in the center is the processing of the traffic information cloud server 201, and the flowchart on the right is the processing of the operator terminal 32.

[0375] That is, in steps S311 to S313, the sensor information acquisition unit 142 of the vehicle 34 acquires surrounding object information such as camera images capturing the surroundings and distance images that enable the distance to surrounding objects to be recognized, position information of the vehicle itself, and information on the steering angle (steering angle) and speed of the vehicle itself detected by the vehicle sensor 117.

[0376] Then, in step S314, the vehicle information communication unit 143 of the vehicle 34 compiles the position information of the vehicle itself acquired by the position information acquisition unit 114, surrounding object information consisting of sensing results acquired by the sensor information acquisition unit 142 that enable the positions and distances of other vehicles and pedestrians of the vehicle 34 to be recognized, and the steering angle (steering angle) and speed information of the vehicle itself detected by the vehicle sensor 117, to form vehicle information, and controls the communication unit 112 to transmit the vehicle information to the traffic information cloud server 201 via V2X communication.

[0377] In step S331, the traffic information cloud server 201 acquires the vehicle information transmitted from the vehicle 34, and stores the information in association with each vehicle 34.

[0378] In step S332, the traffic information cloud server 201 acquires the road facility information transmitted from the road communication facility 202.

[0379] In step S333 , the traffic information cloud server 201 transmits the vehicle information and the road facility information to the operator terminal 32 .

[0380] In step S351, the vehicle information acquisition unit 73 controls the communication unit 55 to receive vehicle information and road facility information.

[0381] Through the subsequent processing, steering information is generated and supplied individually based on the vehicle information of the vehicle 34 .

[0382] The above processing reduces the communication load between the operator terminal 32 and the vehicle 34, making it possible to reduce communication delays overall.

[0383] <Example of application using V2X communication> Furthermore, by using V2X communication, it is possible to obtain vehicle information between vehicles 34 where vehicle-to-vehicle communication is not possible, and to obtain road facility information from road communication equipment 202.

[0384] This makes it possible to obtain information about surrounding vehicles in blind spots at traffic lights, intersections, etc.

[0385] Therefore, by using this information, the system can warn the driver to apply the brakes if the light is predicted to turn red just before passing or if a collision with a nearby vehicle is predicted, thereby avoiding sudden braking and collisions.

[0386] That is, for example, as shown in the display image WV51 on the left side of Figure 31, a warning column can be provided at the top to display a message such as "The light will turn red in 4 seconds. Please begin slowing down."

[0387] Also, for example, as shown in the display image WV52 on the right side of FIG. 31, a warning column can be provided at the top to display a message such as "There is a vehicle in the intersection. Please begin slowing down."

[0388] In either case, sudden braking can be avoided.

[0389] In addition, it is possible to add information about surrounding vehicles obtained through V2X communication to the display using the prediction results of surrounding objects, making it possible to properly recognize the direction of travel of surrounding vehicles.

[0390] For example, as shown in Figure 32, when a mark WB11 of a nearby vehicle consisting of a motorcycle is displayed, triangular marks WC11 to WC13 are placed along the direction of travel based on the predicted position of the nearby vehicle, and as shown in display images WV61 to WV63, an animation is displayed in which the color of the triangular marks WC11 to WC13 changes in order in chronological order from left to right in the figure, making it possible to properly recognize the direction of travel of the nearby vehicle.

[0391] Furthermore, by utilizing V2X communication, the traffic information cloud server 201 can acquire blinking information of turn signals from surrounding vehicles indicated by the mark WB11. By supplying this information to the operator terminal 32 via the traffic information cloud server 201, the blinking direction of the turn signals may be reflected on the display image displayed on the operator terminal 32.

[0392] <<5. Example of Execution by Software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0393] 33 shows an example of the configuration of a general-purpose computer. This computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0394] The input / output interface 1005 is connected to an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected is a drive 1010 that reads and writes data from / to a removable storage medium 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0395] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0396] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0397] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0398] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

[0399] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0400] 33 realizes the functions of the control unit 51 of the operator terminal 32 in FIG.

[0401] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0402] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0403] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0404] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0405] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0406] The present disclosure may also be configured as follows. <1> A signal processing device comprising: a prediction unit that predicts a current position of a peripheral object, which is an object present around a mobile object and is received through communication with the mobile object, taking into account a delay time associated with the communication. <2> The signal processing device according to <1>, further comprising: a storage unit that stores prediction results of the current positions of the peripheral objects in time series, wherein the prediction unit predicts the current position of the peripheral object based on the prediction results stored in the storage unit. <3> The signal processing device according to <2>, wherein the prediction unit calculates the position of the peripheral object at a time when the information about the peripheral object is generated based on the information about the peripheral object, calculates a movement amount over the delay time from a change in the prediction results in the time series stored in the storage unit, and predicts the current position of the peripheral object by adding the movement amount to the position of the peripheral object at the time when the information about the peripheral object is generated. <4> The signal processing device according to <1>, further including a display image creation unit that creates a display image displaying the peripheral object based on the predicted current position. <5> The signal processing device according to <4>, wherein the display image creation unit creates the display image by switching a peripheral object display method that is a method of displaying the peripheral objects based on the delay time. <6> The signal processing device according to <5>, wherein the display image creation unit creates the display image by switching, based on the delay time, between: a first peripheral object display method that displays only the current positions of the peripheral objects based on information about the peripheral objects; a second peripheral object display method that displays an area including the current positions of the peripheral objects based on the information about the peripheral objects; and a third peripheral object display method that displays the peripheral objects based on the information about the peripheral objects and the current positions in a superimposed manner. <7> The signal processing device according to <6>, wherein if the delay time is longer than a first predetermined time, the display image creation unit generates the display image using the first peripheral object display method; and if the delay time is shorter than the first predetermined time, the display image creation unit creates the display image by switching between the second peripheral object display method and the third peripheral object display method.<8> The signal processing device according to <7>, wherein, when the delay time is shorter than the first predetermined time, When the delay time is shorter than the first predetermined time and longer than a second predetermined time, the display image creation unit creates the display image by the second peripheral object display method, and When the delay time is shorter than the first predetermined time and shorter than the second predetermined time, the display image creation unit creates the display image by the third peripheral object display method. <9> The signal processing device according to <7>, wherein, when the delay time is shorter than the first predetermined time, When the number of peripheral objects is less than a predetermined number, the display image creation unit creates the display image by the second peripheral object display method, and When the number of peripheral objects is more than the predetermined number, the display image creation unit creates the display image by the third peripheral object display method. <10> The signal processing device according to <6>, wherein the display image creation unit creates the display image by switching, based on the delay time, between the first peripheral object display method, the second peripheral object display method, the third peripheral object display method, and a fourth peripheral object display method that displays warning information that warns that prediction accuracy of the current position is lower than a predetermined value. <11> The signal processing device according to <10>, wherein, if the delay time is longer than a first predetermined time, the display image creation unit creates the display image using the first peripheral object display method, if the delay time is shorter than the first predetermined time, the display image creation unit creates the display image by switching between the second peripheral object display method and the third peripheral object display method, and if the delay time is longer than the first predetermined time or longer than a fourth predetermined time, the display image creation unit creates the display image using the fourth peripheral object display method. <12> The signal processing device described in <6>, wherein the prediction unit predicts the current position of the moving body taking the delay time into consideration based on position information and steering information of the moving body received through the communication with the moving body, and the display image creation unit, when creating the display image using the second surrounding object display method, offsets the area to be closer to the current position of the moving body depending on the degree of danger of the surrounding object.<13> The signal processing device according to <12>, wherein the risk level of the peripheral object is set according to the type and behavior of the peripheral object. <14> The signal processing device according to <4>, wherein the prediction unit predicts the current position of the moving object taking into account the delay time associated with the communication based on position information and steering information of the moving object received through the communication with the moving object, and the display image creation unit generates the display image by switching a moving object display method, which is a method of displaying the moving object, to display the moving object and the peripheral objects according to the delay time, using the current position of the moving object and the current positions of the peripheral objects. <15> The signal processing device according to <14>, wherein the moving object display method includes a display viewpoint and display information, and the display image creation unit switches the display viewpoint based on the delay time to generate the display image. <16> The signal processing device according to <15>, wherein the display viewpoint includes a bird's-eye view, a third-person viewpoint, and a first-person viewpoint, and the display image creation unit: when the delay time is longer than a first predetermined time, switches the display viewpoint to the bird's-eye view and generates the display image; when the delay time is shorter than the first predetermined time or longer than a second predetermined time, switches the display viewpoint to the third-person viewpoint and generates the display image; and when the delay time is shorter than the second predetermined time, switches the display viewpoint to the first-person viewpoint and generates the display image. <17> The signal processing device described in <16>, wherein the first-person perspective view is a CG (Computer Graphics) image generated based on 3D map information, and the display image creation unit switches the display viewpoint to the first-person perspective view and generates the display image when the delay time is shorter than the second predetermined time and the 3D map information has been acquired, and switches the display viewpoint to the third-person perspective view and generates the display image when the 3D map information has not been acquired.<18> The signal processing device according to <16>, wherein the bird's-eye view includes an illustrated bird's-eye view and a camera image bird's-eye view, and the display image creation unit generates the display image by switching the display viewpoint to the illustrated bird's-eye view when the delay time is longer than the first predetermined time and longer than a third predetermined time, and generates the display image by switching the display viewpoint to the camera image bird's-eye view when the delay time is shorter than the third predetermined time and longer than the first predetermined time. <19> A signal processing method including: predicting a current position of a peripheral object, which is an object present around a moving object and is received through communication with the moving object, taking into account a delay time associated with the communication, based on information about the peripheral object. <20> A signal processing system including: a signal processing device including: a prediction unit that predicts a current position of a peripheral object, which is an object present around a mobile body, based on information about the peripheral object received through communication with the mobile body, taking into account a delay time related to the communication; and a display image creation unit that creates a display image showing information about the peripheral object based on the predicted current position; and a display device that displays the display image.

[0407] 11, 11', 11'' Remote control system, 31, 31-1 to 31-n Operator, 32, 32-1 to 32-n Operator terminal, 33 Network, 34-1 to 34-m Vehicle, 55 Communication unit, 71 Steering information acquisition unit, 72 Delay time acquisition unit, 73 Vehicle information acquisition unit, 74 Current position prediction unit, 75 Display image creation unit, 76 Steering information supply unit, 81 Display unit, 91 Prediction method storage unit, 92 Position information accumulation unit, 112 Communication unit, 115 External recognition sensor, 131 Camera, 132 Radar, 133 LiDAR, 134 Ultrasonic sensor, 141 Steering control unit, 142 Sensor information acquisition unit, 143 Vehicle information communication unit, 201 Traffic information cloud server, 202 Roadside communication facilities

Claims

1. A signal processing device having a prediction unit that predicts the current position of a surrounding object, which is an object that exists around a moving body, based on information about the surrounding object received through communication with the moving body, taking into account the delay time associated with the communication.

2. The signal processing device according to claim 1, further comprising a storage unit that stores prediction results of the current positions of the surrounding objects in a time series, the prediction unit predicting the current positions of the surrounding objects based on the prediction results stored in the storage unit.

3. The signal processing device according to claim 2, wherein the prediction unit calculates the position of the peripheral object at the time the information of the peripheral object is generated based on the information of the peripheral object, calculates the amount of movement of the delay time from changes in the prediction results of the time series stored in the storage unit, and predicts the current position of the peripheral object by adding the amount of movement to the position of the peripheral object at the time the information of the peripheral object is generated.

4. The signal processing device according to claim 1, further comprising a display image generating unit that generates a display image showing the surrounding objects based on the predicted current position.

5. The signal processing device according to claim 4, wherein the display image creation section creates the display image by switching a peripheral object display method, which is a method for displaying the peripheral objects, based on the delay time.

6. The signal processing device according to claim 5, wherein the display image creation unit creates the display image by switching between a first peripheral object display method of displaying only the current positions of the peripheral objects based on information about the peripheral objects, a second peripheral object display method of displaying an area including the current positions of the peripheral objects based on information about the peripheral objects, and a third peripheral object display method of superimposing the peripheral objects displayed based on information about the peripheral objects and their current positions.

7. The signal processing device according to claim 6, wherein, when the delay time is longer than a first predetermined time, the display image creation unit generates the display image using the first peripheral object display method, and, when the delay time is shorter than the first predetermined time, the display image creation unit creates the display image by switching between the second peripheral object display method and the third peripheral object display method.

8. The signal processing device according to claim 7, wherein: when the delay time is shorter than the first predetermined time; when the delay time is shorter than the first predetermined time and longer than the second predetermined time, the display image creation unit creates the display image using the second peripheral object display method; and when the delay time is shorter than the first predetermined time and shorter than the second predetermined time, the display image creation unit creates the display image using the third peripheral object display method.

9. A signal processing device as described in claim 7, wherein: when the delay time is shorter than the first predetermined time, and when the number of surrounding objects is less than a predetermined number, the display image creation unit creates the display image using the second surrounding object display method; and when the number of surrounding objects is more than the predetermined number, the display image creation unit creates the display image using the third surrounding object display method.

10. The signal processing device according to claim 6, wherein the display image creation unit creates the display image by switching between the first peripheral object display method, the second peripheral object display method, the third peripheral object display method, and a fourth peripheral object display method that displays warning information that the prediction accuracy of the current position is lower than a predetermined value, in addition to the first peripheral object display method, the second peripheral object display method, and the third peripheral object display method, based on the delay time.

11. The signal processing device of claim 10, wherein, when the delay time is longer than a first predetermined time, the display image creation unit generates the display image using the first peripheral object display method; when the delay time is shorter than the first predetermined time, the display image creation unit creates the display image by switching between the second peripheral object display method and the third peripheral object display method; and when the delay time is longer than the first predetermined time and longer than a fourth predetermined time, the display image creation unit generates the display image using the fourth peripheral object display method.

12. The signal processing device described in claim 6, wherein the prediction unit predicts the current position of the moving body taking into account the delay time based on position information and steering information of the moving body received through the communication with the moving body, and the display image creation unit, when creating the display image using the second surrounding object display method, offsets the area so as to be closer to the current position of the moving body depending on the degree of danger of the surrounding object.

13. The signal processing device according to claim 12, wherein the degree of danger of the surrounding object is set according to the type and behavior of the surrounding object.

14. The signal processing device of claim 4, wherein the prediction unit predicts the current position of the moving body taking into account the delay time related to the communication based on position information and steering information of the moving body received through the communication with the moving body, and the display image creation unit uses the current position of the moving body and the current positions of the surrounding objects to generate the display image of the moving body and the surrounding objects by switching a moving body display method, which is a display method of the moving body, in accordance with the delay time.

15. The signal processing device according to claim 14, wherein the moving object display method includes a display viewpoint and display information, and the display image creation unit switches the display viewpoint based on the delay time to generate the display image.

16. A signal processing device as described in claim 15, wherein the display viewpoint includes a bird's-eye view, a third person viewpoint, and a first person viewpoint, and the display image creation unit: when the delay time is longer than a first predetermined time, switches the display viewpoint to the bird's-eye view and generates the display image; when the delay time is shorter than the first predetermined time and longer than a second predetermined time, switches the display viewpoint to the third person viewpoint and generates the display image; and when the delay time is shorter than the second predetermined time, switches the display viewpoint to the first person viewpoint and generates the display image.

17. The signal processing device according to claim 16, wherein the first-person viewpoint is a CG (Computer Graphics) image generated based on 3D map information, and the display image creation unit switches the display viewpoint to the first-person viewpoint and generates the display image when the delay time is shorter than the second predetermined time and the 3D map information has been acquired, and switches the display viewpoint to the third-person viewpoint and generates the display image when the 3D map information has not been acquired.

18. The signal processing device according to claim 16, wherein the bird's-eye view includes an illustrated bird's-eye view and a camera image bird's-eye view, and the display image creation unit: when the delay time is longer than the first specified time and longer than a third specified time, switches the display viewpoint to the illustrated bird's-eye view to generate the display image; and when the delay time is shorter than the third specified time and longer than the first specified time, switches the display viewpoint to the camera image bird's-eye view to generate the display image.

19. A signal processing method including a step of predicting the current position of a surrounding object, which is an object present around a moving body, based on information about the surrounding object received through communication with the moving body, taking into account a delay time associated with the communication.

20. A signal processing system having a signal processing device including: a prediction unit that predicts a current position of a peripheral object, which is an object existing around a moving body, based on information of the peripheral object received through communication with the moving body, taking into account a delay time related to the communication; and a display image creation unit that creates a display image showing information of the peripheral object based on the predicted current position; and a display device that includes a display unit that displays the display image.

Images