In-vehicle device, in-vehicle system, control method, and computer program

Abstract

An in-vehicle device according to the present invention is an in-vehicle device installed in a vehicle that has an automated driving function. The in-vehicle device includes: an allowable latency estimating unit that estimates, as an allowable latency, time until the vehicle reaches a dynamic object; a transfer latency estimating unit that estimates, on the basis of a load state of information processing and information transmission of the vehicle, a time from the in-vehicle device receiving data from outside of the vehicle till the in-vehicle device transfers this data to an executing unit of the automated driving function, as transfer latency; a determining unit that, on the basis of difference between the allowable latency and the transfer latency, selects a particular piece of analysis processing from among a plurality of pieces of analysis processing for analyzing data that is externally received; and a driver-assistance information generating unit that executes the particular analysis processing selected by the determining unit, and generates driver-assistance information. The externally-received data includes information relating to the dynamic object, and the driver-assistance information is transferred to the executing unit of the of the automated driving function.

Description

[Technical Field]

[0001] This disclosure relates to an in-vehicle device, an in-vehicle system, a control method, and a computer program. This application claims priority to Japanese Patent Application No. 2021-164872 filed on October 6, 2021, and incorporates by reference all of the contents of that application. [Background technology]

[0002] Various systems have been proposed to assist drivers of automobiles, motorcycles, etc. (hereinafter referred to as vehicles). For example, it has been proposed to collect sensor information from roadside devices equipped with various sensor devices (e.g., cameras, radars, etc.) installed on roads and their surrounding areas, analyze the collected information, and provide traffic-related information (e.g., accidents, congestion, etc.) to vehicles as dynamic driving assistance information. Furthermore, with the increasing speed of mobile communication lines, it has also been proposed to collect information not only from sensor devices installed in roadside devices but also from sensor devices installed in vehicles, communicate it via a server computer, or communicate it directly between vehicles, and effectively use it for driving assistance.

[0003] The introduction of plug-in hybrid electric vehicles (PHEVs) and electric vehicles (EVs) is progressing, and recent vehicles, including these, are equipped with various electronic devices and ECUs (Electric Control Units) that control them. For example, vehicles capable of autonomous driving are equipped with autonomous driving ECUs. The autonomous driving ECUs communicate with external devices as appropriate to obtain necessary information (e.g., road traffic information, dynamic driving assistance information). Other ECUs include engine control ECUs, stop-start control ECUs, transmission control ECUs, airbag control ECUs, power steering control ECUs, and hybrid control ECUs.

[0004] Although not related to driving assistance information, Patent Document 1 below discloses a technology for controlling the transmission of hierarchical information to a user terminal based on a determination result according to the positional relationship (e.g., distance) and moving state (e.g., acceleration) between two user terminals. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2014 / 038323 Summary of the Invention

[0006] An on-board device according to one aspect of the present disclosure is an on-board device mounted on a vehicle having an autonomous driving function, and includes: an allowable delay estimation unit that estimates the time it takes for the vehicle to reach a dynamic object as an allowable delay; a transfer delay estimation unit that estimates the time from when the on-board device receives data from outside the vehicle until when the on-board device transfers the data to an execution unit of the autonomous driving function as a transfer delay based on the load state of information processing and information transmission in the vehicle; a determination unit that selects a specific analysis process from among a plurality of analysis processes for analyzing data received from outside based on the difference between the allowable delay and the transfer delay; and a driving assistance information generation unit that executes the specific analysis process selected by the determination unit and generates driving assistance information, wherein the data received from outside includes information regarding the dynamic object, and the driving assistance information is transferred to the execution unit of the autonomous driving function.

[0007] An in-vehicle system according to another aspect of the present disclosure is an in-vehicle system mounted on a vehicle having an autonomous driving function, and includes an execution unit for the autonomous driving function, a communication unit for acquiring data including information regarding a dynamic object, and the above-mentioned in-vehicle device.

[0008] A control method according to yet another aspect of the present disclosure is a control method for assisting an autonomous driving function of a vehicle, and includes an allowable delay estimation step of estimating, as an allowable delay, the time it takes for the vehicle to reach a dynamic object; a transfer delay estimation step of estimating, based on the load state of information processing and information transmission in the vehicle, the time from when an on-board device mounted on the vehicle receives data from outside the vehicle until when the on-board device transfers the data to an execution unit of the autonomous driving function; a determination step of selecting, based on the difference between the allowable delay and the transfer delay, a specific analysis process from among a plurality of analysis processes for analyzing data received from outside; and a driving assistance information generation step of executing the specific analysis process selected by the determination step and generating driving assistance information, wherein the data received from outside includes information regarding the dynamic object, and the driving assistance information is transferred to the execution unit of the autonomous driving function.

[0009] A computer program relating to yet another aspect of the present disclosure is a computer program for causing a computer mounted on a vehicle to realize the following: an allowable delay estimation function that estimates the time it takes for the vehicle to reach a dynamic object as an allowable delay; a transfer delay estimation function that estimates the time from when the computer receives data from outside the vehicle until when the computer transfers the data to an execution unit of an autonomous driving function as a transfer delay based on the load state of information processing and information transmission in the vehicle; a determination function that selects a specific analysis process from among multiple analysis processes for analyzing data received from outside based on the difference between the allowable delay and the transfer delay; and a driving assistance information generation function that executes the specific analysis process selected by the determination function and generates driving assistance information, wherein the data received from outside includes information regarding the dynamic object, and the driving assistance information is transferred to the execution unit of the autonomous driving function. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a usage form of an in-vehicle system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of the in-vehicle system shown in FIG. [Figure 3]FIG. 3 is a block diagram showing a hardware configuration of the vehicle gateway shown in FIG. [Figure 4] FIG. 4 is a block diagram illustrating a functional configuration of the vehicle gateway illustrated in FIG. [Figure 5] FIG. 5 is a schematic diagram showing a processing state by the function shown in FIG. [Figure 6] FIG. 6 is a flowchart showing the process of generating and transferring hierarchical information executed by the vehicle gateway. [Figure 7] FIG. 7 is a flowchart showing the process when each additional process is executed in order. [Figure 8] FIG. 8 is a schematic diagram showing a situation in which data provided to an autonomous driving ECU changes depending on the distance from the vehicle to a dynamic object in an in-vehicle system of the same vehicle. [Figure 9] FIG. 9 is a plan view showing vehicles approaching an intersection and traffic conditions at the intersection over time. [Figure 10] FIG. 10 is a diagram showing an example of information presented inside a vehicle. [Figure 11] FIG. 11 is a diagram showing an example of information presented in the vehicle following FIG. [Figure 12] FIG. 12 is a diagram showing an example of information presented in the vehicle following FIG. [Figure 13] FIG. 13 is a diagram showing an example of information presented in the vehicle following FIG. [Figure 14] FIG. 14 is a block diagram showing the configuration of an in-vehicle system according to a modified example. [Figure 15] FIG. 15 is a block diagram showing the hardware configuration of the extension device shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] [Problem to be solved by this disclosure] Acquiring sensor data, analyzing it, and generating and integrating dynamic information about detected objects (i.e., dynamic objects such as people and vehicles) can improve the quality of driving assistance information, such as the level of detail and accuracy. However, transmitting, receiving, and analyzing sensor data, etc., requires time, resulting in latency. There is a trade-off between the quality of driving assistance information and latency, and the time range over which the driving assistance information can be applied to vehicle control varies depending on traffic conditions. For example, when providing driving assistance information from a server computer to a vehicle, if the distance between the vehicle and a dynamic object is relatively large, detailed information can be generated and provided to the vehicle through time-consuming analytical processing. However, if the distance between the vehicle and a dynamic object is relatively small, the vehicle does not have time to effectively utilize the detailed information, resulting in waste. Furthermore, depending on the vehicle's condition, it may take some time between receiving the driving assistance information and actually using it (e.g., due to the presence of latency within the vehicle). Therefore, when providing dynamic information to a vehicle as driving assistance information, it is desirable to provide appropriate driving assistance information at a time that allows for effective use, taking latency within the vehicle into consideration.

[0012] This demand cannot be met by Patent Document 1. In Patent Document 1, the output of hierarchical information cannot be controlled according to delay, making it difficult to apply the technology to highly real-time services such as driving assistance and autonomous driving of vehicles.

[0013] Therefore, the present disclosure aims to provide an in-vehicle device, an in-vehicle system, a control method, and a computer program that can generate appropriately hierarchical driving assistance information within the vehicle according to the predicted time until the vehicle reaches a dynamic object, and that can use the driving assistance information to control the driving of the vehicle.

[0014] [Effects of this disclosure] According to the present disclosure, it is possible to provide an in-vehicle device, an in-vehicle system, a control method, and a computer program that can generate appropriately hierarchical driving assistance information within the vehicle according to the predicted time until the vehicle reaches a dynamic object, and that can use the driving assistance information to control the driving of the vehicle.

[0015] [Description of the embodiments of the present disclosure] The contents of the embodiments of the present disclosure will be listed and described below. At least some of the embodiments described below may be combined in any combination.

[0016] (1) An on-board device according to a first aspect of the present disclosure is an on-board device mounted on a vehicle having an autonomous driving function, the on-board device including: an allowable delay estimator that estimates, as an allowable delay, a time until the vehicle reaches a dynamic object; a transfer delay estimator that estimates, based on the load state of information processing and information transmission in the vehicle, a transfer delay that is a time from when the on-board device receives data from outside the vehicle until when the on-board device transfers the data to an execution unit of the autonomous driving function; a determination unit that selects, based on a difference between the allowable delay and the transfer delay, a specific analysis process from among multiple analysis processes for analyzing the data received from outside the vehicle; and a driving assistance information generator that executes the specific analysis process selected by the determination unit and generates driving assistance information, the data received from outside including information about the dynamic object, and the driving assistance information is transferred to the execution unit of the autonomous driving function. As a result, driving assistance information hierarchically organized into appropriate hierarchies can be generated within the vehicle depending on the time until the vehicle reaches the dynamic object, i.e., the distance between the vehicle and the dynamic object, and the driving assistance information can be used for driving control of the vehicle. The on-board device is not limited to a device that is installed as a standard device in a vehicle with an autonomous driving function, but also includes a device that can be installed later as an extension device. The autonomous driving preferably includes all levels from Level 1 (i.e., driving assistance) described below.

[0017] (2) In the above (1), the data received from the outside may further include sensor data, and the information about the dynamic object may include position information and simplified attribute information of the dynamic object. The driving assistance information generation unit may generate hierarchical driving assistance information including the results of executing a specific analysis process and the position information and simplified attribute information as layers. This allows the position information and simplified attribute information of the dynamic object provided from outside the vehicle to be effectively used as driving assistance information. Furthermore, by analyzing the sensor data, it is possible to generate driving assistance information including detailed attributes of the dynamic object. By analyzing the position information and simplified attribute information of the dynamic object, it is possible to generate driving assistance information including a movement prediction of the dynamic object.

[0018] (3) In the above (2), the driving assistance information may include a first layer including analysis results from a specific analysis process that processes sensor data, and a second layer including analysis results from a specific analysis process that does not process sensor data. This allows for the generation of driving assistance information that includes detailed attributes of dynamic objects and predicted movements of dynamic objects as separate layers, and when the driving assistance information is provided to an autonomous driving execution unit (i.e., an autonomous driving ECU), it can be used efficiently.

[0019] (4) In the above (2) or (3), the specific analysis process that does not process sensor data may process at least one of the analysis results of the specific analysis process that processes sensor data and information about dynamic objects, thereby improving the accuracy of the specific analysis process that does not process sensor data.

[0020] (5) In any one of (1) to (4) above, the determination unit may calculate a difference by subtracting the transfer delay from the allowable delay and determine whether the difference is greater than a predetermined value equal to or greater than 0. If the difference is greater than the predetermined value, a specific analysis process may be selected, and if the difference is equal to or less than the predetermined value, no specific analysis process may be selected. This allows an appropriate specific analysis process to be selected, and unnecessary processing to be reduced.

[0021] (6) In the above (5), if the difference is equal to or less than a predetermined value, the information about the dynamic object may be transferred to the execution unit together with information indicating that the transfer delay is equal to or greater than the allowable delay. This allows the execution unit for autonomous driving to determine whether or not to use the information about the dynamic object, and there is a possibility that the information about the dynamic object will be used.

[0022] (7) In the above (5) or (6), the on-board device may further include a storage unit that stores a processing time table that records processing times corresponding to the amount of data to be processed for each of the multiple analysis processes, and if the difference is greater than a predetermined value, the determination unit may use the amount of data to refer to the processing time table to identify the data processing time, and then determine whether the processing time is equal to or less than the difference, thereby selecting a specific analysis process. This allows an appropriate specific analysis process to be selected, and the analysis results can be effectively used for vehicle driving control.

[0023] (8) In (7) above, the processing time table may further include, for an analysis process that processes sensor data among the plurality of analysis processes, an acquisition time for newly acquiring the sensor data to be processed, and if the difference is greater than a predetermined value, the determination unit may select a specific analysis process by determining whether the total value of the processing time and acquisition time identified by referring to the processing time table is less than or equal to the difference. This makes it possible to select an appropriate specific analysis process even when newly acquiring and analyzing sensor data, and the analysis results can be effectively used for vehicle driving control.

[0024] (9) An in-vehicle system according to a second aspect of the present disclosure is an in-vehicle system mounted on a vehicle having an autonomous driving function, and includes an execution unit for the autonomous driving function, a communication unit that acquires data including information about a dynamic object, and any one of the in-vehicle devices described above in (1) to (8). This allows driving assistance information hierarchically organized into appropriate hierarchies within the vehicle to be generated according to the time it takes for the vehicle to reach the dynamic object, i.e., the distance between the vehicle and the dynamic object, and makes the driving assistance information available for driving control of the vehicle.

[0025] (10) In the above (9), the communication unit may further transmit the driving assistance information generated by the in-vehicle device to the other vehicle together with information on the vehicle's position and driving direction. This allows the other vehicle to control the vehicle's driving using the driving assistance information without performing analysis processing.

[0026] (11) In the above (10), the determination unit of the in-vehicle device may estimate the communication time of the driving assistance information transmitted from the communication unit, and may select a specific analysis process from among a plurality of analysis processes based on the difference between the allowable delay and the total value of the transfer delay and the communication time. This allows the selection of an appropriate specific analysis process and prevents unnecessary analysis.

[0027] (12) A control method according to a third aspect of the present disclosure is a control method for supporting an autonomous driving function of a vehicle, the control method including: an allowable delay estimation step of estimating, as an allowable delay, a time until the vehicle reaches a dynamic object; a transfer delay estimation step of estimating, based on the load state of information processing and information transmission in the vehicle, a time from when an on-board device mounted on the vehicle receives data from outside the vehicle until when the on-board device transfers the data to an execution unit of the autonomous driving function; a determination step of selecting, based on a difference between the allowable delay and the transfer delay, a specific analysis process from among multiple analysis processes for analyzing the data received from outside the vehicle; and a driving assistance information generation step of executing the specific analysis process selected in the determination step to generate driving assistance information, wherein the data received from outside includes information about the dynamic object, and the driving assistance information is transferred to the execution unit of the autonomous driving function. As a result, driving assistance information hierarchically organized into appropriate hierarchies can be generated within the host vehicle depending on the time until the host vehicle reaches the dynamic object, i.e., the distance between the host vehicle and the dynamic object, and the driving assistance information can be used for driving control of the host vehicle.

[0028] (13) A computer program according to a fourth aspect of the present disclosure is a computer program for causing a computer mounted on a vehicle to realize the following: an allowable delay estimation function that estimates, as an allowable delay, a time until the vehicle reaches a dynamic object; a transfer delay estimation function that estimates, based on the load state of information processing and information transmission in the vehicle, a transfer delay from when the computer receives data from outside the vehicle until when the computer transfers the data to an execution unit of an autonomous driving function; a determination function that selects a specific analysis process from among multiple analysis processes for analyzing data received from outside based on a difference between the allowable delay and the transfer delay; and a driving assistance information generation function that executes the specific analysis process selected by the determination function to generate driving assistance information, wherein the data received from outside includes information about the dynamic object, and the driving assistance information is transferred to the execution unit of the autonomous driving function. As a result, driving assistance information hierarchically organized into appropriate layers can be generated within the vehicle depending on the time until the vehicle reaches the dynamic object, i.e., the distance between the vehicle and the dynamic object, and the driving assistance information can be used for driving control of the vehicle.

[0029] [Details of the embodiments of the present disclosure] In the following embodiments, the same components are denoted by the same reference numerals, and their names and functions are also the same, so detailed descriptions thereof will not be repeated.

[0030] [Overall configuration] Referring to FIG. 1, an in-vehicle system 100 according to an embodiment of the present disclosure is mounted on a vehicle 102. The in-vehicle system 100 communicates with infrastructure sensors 104 fixedly installed on roads (including intersections) and their surrounding areas (hereinafter also referred to as roadside), and receives sensor data and dynamic information that is an analysis result of the sensor data. The in-vehicle system 100 communicates with an in-vehicle system 110 mounted on another vehicle 112, and receives sensor data and dynamic information from the in-vehicle sensors. The in-vehicle system 100 also communicates with a traffic light 106, and obtains information regarding the traffic light status (hereinafter referred to as traffic light information). These communications may be via a base station 108, or may be direct communications without the base station 108.

[0031] The base station 108 provides mobile communication services using, for example, 4G (i.e., fourth generation mobile communication system) lines and 5G (i.e., fifth generation mobile communication system) lines. The base station 108 is connected to a network 114. The infrastructure sensors 104 and traffic lights 106 may also be connected to the network 114.

[0032] The in-vehicle system 100 and the in-vehicle system 110 mounted on the vehicle 102 and the vehicle 112, respectively, have a communication function according to the communication specifications (e.g., 4G line, 5G line, etc.) provided by the base station 108. As described above, the in-vehicle system 100 and the in-vehicle system 110 also have a function to communicate directly with each other without going through the base station 108 (i.e., V2V (Vehicle to Vehicle)). For example, Wi-Fi communication is used for the mutual communication not going through the base station 108.

[0033] 1 are detection targets of the infrastructure sensor 104. The pedestrian 900 is also a detection target of the sensors mounted on the vehicles 102 and 112.

[0034] The infrastructure sensor 104 is a device installed on the roadside and has a function of acquiring information on the roadside, and has a function of communicating with the base station 108. The infrastructure sensor 104 is, for example, an image sensor (such as a digital surveillance camera), a radar (such as a millimeter-wave radar), or a laser sensor (such as a LiDAR (Light Detection and Ranging)). Note that the infrastructure sensor 104 may be equipped in or connected to a roadside device having a calculation function.

[0035] Sensor data acquired by sensors mounted on each of the vehicles 102 and 112 is analyzed in the in-vehicle system 100 and the in-vehicle system 110, and the analysis results are stored as dynamic information. The dynamic information is used in the autonomous driving function of the host vehicle. Autonomous driving is classified into levels 1 to 5 according to the driver (i.e., human or system) and the driving area (i.e., limited or unlimited). Autonomous driving in which dynamic information can be used is not limited to fully autonomous driving at level 4 or higher (i.e., the system is the driving subject, not a human driver), but preferably also includes levels 1 and 2 in which a human is the driving subject, such as driving assistance, and conditional autonomous driving (i.e., level 3). In other words, the autonomous driving in which dynamic information can be used may be any of levels 1 to 5, or any of levels 1 to 5. Furthermore, the sensor data and dynamic information may be mutually communicated between the in-vehicle system 100 and the in-vehicle system 110 as described above. The in-vehicle system 100 and the in-vehicle system 110 also mutually communicate information about the vehicles in which they are installed (e.g., position information, speed information, driving direction information, etc.). Hereinafter, the position information, speed information, driving direction information, etc. will also be simply referred to as position, speed, and driving direction, respectively. The vehicle information can be used to identify the position and direction at which sensor data transmitted from the vehicle was acquired.

[0036] Dynamic information is information about dynamic objects detected by sensors (i.e., infrastructure sensors and on-board sensors). Dynamic objects are not limited to moving objects (e.g., people, vehicles, etc.), but also include objects that have the ability to move but are stationary. Dynamic information may include information about the dynamic object itself (hereinafter referred to as attributes) and information about the displacement of the dynamic object (e.g., position, movement speed, movement direction, time, etc.). Dynamic information is used to generate driving assistance information, which will be described later. Note that driving assistance information to be used for autonomous driving of the vehicle may be information about a predetermined area including the driving route of the vehicle (i.e., the road on which the vehicle is scheduled to travel).

[0037] Attributes are divided into, for example, simple attributes (hereinafter referred to as simple attributes) and detailed attributes (hereinafter referred to as detailed attributes). Simple attributes are used to roughly classify dynamic objects and include, for example, people, bicycles, motorcycles, and automobiles. Detailed attributes are used to classify dynamic objects in detail and include the state of the dynamic object. For example, if the simple attribute is "person," its detailed attributes may include children, adults, elderly people, etc., and may further include so-called "walking while using a smartphone" (i.e., a state of looking at a smartphone while walking), ignoring traffic lights, etc. For example, if the simple attribute is "automobile," its detailed attributes may include, for example, ordinary cars, large vehicles, etc., and may further include buses, taxis, emergency vehicles (e.g., ambulances and fire engines), distracted driving, etc. Note that the simple attributes and detailed attributes are not limited to these and may include any attribute.

[0038] Among the information regarding the displacement of a dynamic object, time information is, for example, the generation time of position information, movement speed information, movement direction information, etc. The dynamic information may also include prediction information. For example, if the in-vehicle system 100 and the in-vehicle system 110 have a prediction function, they can predict the movement trajectory, movement speed, and movement direction in the future (for example, within a predetermined time from the present) using the movement trajectory, movement speed, and movement direction up to the present obtained from the change in the position of the dynamic object. These may also be included in the dynamic information.

[0039] 1 exemplarily illustrates one base station 108, one infrastructure sensor 104, one traffic light 106, and two vehicles 102 and 112 equipped with on-board systems. However, this is merely an example. Typically, multiple base stations are provided, and three or more vehicles are equipped with on-board systems. There may be vehicles that do not have on-board systems. Vehicles that do not have on-board systems are detected as dynamic objects.

[0040] [Hardware configuration of in-vehicle system] 2, an example of the hardware configuration of an in-vehicle system 100 mounted on a vehicle 102 is shown. An in-vehicle system 110 mounted on a vehicle 112 is similarly configured. The in-vehicle system 100 includes a communication unit 120, an in-vehicle gateway 122, a sensor 124, an autonomous driving ECU 126, an ECU 128, and a bus 130. Note that the in-vehicle system 100 includes multiple ECUs in addition to the autonomous driving ECU 126, and FIG. 2 shows ECU 128 as a representative of these.

[0041] The communication unit 120 performs wireless communication with external devices of the vehicle 102 (for example, communication with the in-vehicle system 110 via the base station 108). The communication unit 120 includes an IC for performing modulation and multiplexing employed in wireless communication, an antenna for transmitting and receiving radio waves at a predetermined frequency, an RF circuit, and the like. The communication unit 120 also has a function for communicating with a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System). The communication unit 120 may also have a communication function such as Wi-Fi.

[0042] The in-vehicle gateway 122, which is an in-vehicle device, plays a role in connecting communication functions with the outside of the vehicle (i.e., communication specifications) with communication functions within the vehicle (i.e., communication specifications) (i.e., communication protocol conversion, etc.). The autonomous driving ECU 126 can communicate with external devices via the in-vehicle gateway 122 and the communication unit 120. The in-vehicle gateway 122 acquires dynamic information and sensor data used to generate the dynamic information from the information received from the outside via the communication unit 120, and generates and updates driving assistance information as described below. The driving assistance information is transmitted to the autonomous driving ECU 126. The bus 130 handles communication functions within the vehicle, and communication (data exchange) between the in-vehicle gateway 122, the sensor 124, the autonomous driving ECU 126, and the ECU 128 is performed via the bus 130. For example, a CAN (Controller Area Network) is used for the bus 130.

[0043] The sensor 124 is mounted on the vehicle 102 and includes sensors for acquiring information outside the vehicle 102 (for example, a video imaging device (for example, a digital camera (for example, a CCD camera or a CMOS camera)), a laser sensor (for example, a LiDAR), etc.), and sensors for acquiring information about the vehicle itself (for example, an acceleration sensor, a load sensor, etc.). The sensor 124 acquires information within a detection range (for example, an imaging range in the case of a camera) and outputs it as sensor data. In the case of a digital camera, it outputs digital image data. The detection signal (i.e., an analog or digital signal) of the sensor 124 is output as digital data to the bus 130 via an I / F unit (not shown), and is transmitted to the in-vehicle gateway 122, the autonomous driving ECU 126, etc.

[0044] The autonomous driving ECU 126 controls the driving of the vehicle 102. For example, the autonomous driving ECU 126 acquires sensor data, analyzes it to understand the situation around the vehicle, and controls mechanisms related to autonomous driving (for example, mechanisms such as the engine, transmission, steering, and brakes). The autonomous driving ECU 126 uses driving assistance information acquired from the in-vehicle gateway 122 for autonomous driving.

[0045] [Hardware configuration of the in-vehicle gateway] 3, the in-vehicle gateway 122 includes a control unit 140 and a memory 142. The control unit 140 is configured to include a CPU (Central Processing Unit) and controls the memory 142. The memory 142 is, for example, a rewritable non-volatile semiconductor memory, and stores a computer program (hereinafter simply referred to as a program) executed by the control unit 140. The memory 142 provides a work area for the program executed by the control unit 140. The control unit 140 obtains data to be processed directly from the communication unit 120, and obtains the data from sources other than the communication unit 120 via the bus 130. The control unit 140 stores the processing results in the memory 142 and outputs the results to the bus 130.

[0046] [Functional configuration] The function of the in-vehicle gateway 122 will be described with reference to FIG. 4. In the following description, the vehicle 102 shown in FIG. 1 is referred to as the subject vehicle, and the vehicle 112 traveling ahead of the vehicle 102 is referred to as the other vehicle. The in-vehicle system 100 acquires the position and simple attributes of a dynamic object as dynamic information from an external device (e.g., the infrastructure sensor 104 and the in-vehicle system 110, etc.). The position and simple attributes may be transmitted with the sensor data that was the subject of analysis when they were generated added. That is, the vehicle 102 receives only the dynamic information (i.e., the position and simple attributes), receives only the sensor data, or receives the dynamic information and the corresponding sensor data.

[0047] The in-vehicle gateway 122 includes a storage unit 200, an allowable delay estimation unit 202, a determination unit 204, a transfer delay estimation unit 206, an additional analysis processing unit 208, and an output unit 210. The storage unit 200 stores data received by the communication unit 120 and sensor data from the sensor 124 input via the bus 130. The data input from the communication unit 120 includes dynamic information (i.e., position and simple attributes), sensor data, traffic light information, position information of the vehicle, etc. The storage unit 200 is realized by the memory 142 in FIG. 3. Other functions, which will be described later, are realized by the control unit 140. As will be described later, the additional analysis processing unit 208 and the output unit 210 constitute a driving assistance information generation unit.

[0048] The allowable delay estimation unit 202 estimates the allowable delay from the distance between the dynamic object and the host vehicle contained in the dynamic information acquired from the communication unit 120. Specifically, the allowable delay estimation unit 202 calculates the distance L between the dynamic object and the host vehicle from the positions of the dynamic object and the host vehicle at the same time (including a predetermined error), and divides the distance L by the speed V of the host vehicle to calculate the allowable delay Tp (=L / V). The allowable delay Tp is the predicted time until the host vehicle reaches the dynamic object. The time of the dynamic object can be the reception time of the dynamic information (i.e., the position and simple attributes) (e.g., the time when packet data is received and configured as dynamic information). If the position of the host vehicle (e.g., GPS data) acquired from the communication unit 120 is associated with the reception time and stored, the position of the host vehicle at the same time when the position of the dynamic object is obtained can be identified. The speed of the host vehicle is acquired from a drive unit (i.e., a control target of the autonomous driving ECU 126) that drives the host vehicle. The speed V of the vehicle can be, for example, the current speed or the average speed within a recent predetermined time. The distance L may be a straight-line distance, but is preferably a distance along the road on which the vehicle is scheduled to travel. The allowable delay estimation unit 202 outputs the estimated allowable delay Tp to the determination unit 204.

[0049] The transfer delay estimation unit 206 observes the load state of information processing and information transmission within the vehicle and estimates the delay time (hereinafter referred to as transfer delay) required to transfer data (i.e., driving assistance information, described later) to the autonomous driving ECU 126. The transfer delay is, for example, the time from when the in-vehicle gateway 122 starts transferring data received by the communication unit 120 to the autonomous driving ECU 126 until the autonomous driving ECU 126 completes receiving the data. The transfer delay Tt depends on the idle state of the bus 130 (i.e., a state in which data transfer via the bus 130 is not being performed, corresponding to the load state of information transmission), the load state of information processing in the control unit 140 itself, and the like. For example, when the bus 130 is a CAN, a multi-master system and an event-driven system are adopted. That is, when the bus 130 is idle, the node (e.g., an ECU) that first starts transmission acquires the transmission right, and to avoid collisions during data transmission via the bus 130, a node with a higher priority acquires the transmission right. Therefore, the transfer delay estimation unit 206 (i.e., the control unit 140) can observe the load state of the bus 130 (i.e., whether it is free or not) in addition to the load state of the control unit 140 itself, and estimate the load state of information processing and information transmission. For example, if a transmission right has been acquired, the load state of information transmission can be estimated by observing the priority of that node. The transfer delay estimation unit 206 outputs the estimated transfer delay Tt to the determination unit 204.

[0050] The determination unit 204 determines whether further analysis (hereinafter referred to as additional analysis) of the dynamic information (i.e., location and simple attributes) and sensor data stored in the storage unit 200 is possible using the allowable delay Tp input from the allowable delay estimation unit 202 and the transfer delay Tt input from the transfer delay estimation unit 206. Specifically, the determination unit 204 determines whether the allowable delay Tp is greater than the transfer delay Tt (i.e., Tp > Tt). If the allowable delay Tp is equal to or less than the transfer delay Tt (i.e., Tp ≦ Tt), there is no time for additional analysis. However, if the allowable delay Tp is greater than the transfer delay Tt (i.e., Tp > Tt), there is enough time for additional analysis. If the allowable delay Tp is greater than the transfer delay Tt, the determination unit 204 selects a process to be executed from multiple predetermined additional analysis processes based on the difference (i.e., Tp − Tt). The determination unit 204 outputs information for identifying the selected additional analysis process (hereinafter referred to as analysis process identification information) to the additional analysis processing unit 208.

[0051] If the allowable delay Tp is greater than the transfer delay Tt, the determination unit 204 selects an additional analysis process to be executed by determining whether each additional analysis process can be completed within Tp-Tt. As described above, additional analysis processes include analysis processes targeting sensor data and analysis processes targeting data other than sensor data (e.g., dynamic information, hereinafter also referred to as non-sensor data). Even for the same analysis process, the larger the amount of data to be processed, the longer the processing time. Therefore, for example, a processing time table 212 that associates the amount of data to be processed with the processing time for each additional analysis process is stored in advance in the storage unit 200. This allows for the selection of an appropriate additional analysis process, as described below, and the analysis results can be effectively used for vehicle driving control. Furthermore, driving assistance information can be generated that includes a layer (i.e., a first layer) including detailed attributes of dynamic objects and a layer (i.e., a second layer) including predicted movement of dynamic objects as different layers. Such driving assistance information can be provided to an autonomous driving ECU for efficient use.

[0052] The determination unit 204 reads the processing time τi corresponding to the combination of the additional analysis process i and the data volume from the processing time table 212, and determines whether τi < Tp - Tt. If τi < Tp - Tt, since it can be completed within Tp - Tt, the additional analysis process i is selected. The order of the additional analysis processes for determining whether τi < Tp - Tt is arbitrary. For example, the processes targeting sensor data may be preferentially determined, or the processes targeting non-sensor data may be preferentially determined. Also, the processes with shorter processing times may be preferentially determined, or the processes with longer processing times may be preferentially determined. When one additional analysis process j is selected, for an additional analysis process i different from the additional analysis process j, it is only necessary to determine whether τi < Tp - Tt - τj. Each time a new additional analysis process is selected, the value of τj may be replaced with Στj, and the determination may be made in the same manner. Note that Σ represents an operator for calculating the sum with respect to the processing time τj of the already selected additional analysis processes.

[0053] Note that the processing time also varies depending on the computing resources. Therefore, the processing time table 212 may correspond the combination of the data volume to be processed and the computing resources with the processing time for each additional analysis process. In that case, the processing time τi corresponding to the combination of the additional analysis process i, the data volume, and the computing resources is read from the processing time table 212, and the magnitude relationship with Tp - Tt may be determined in the same manner as above.

[0054] The additional analysis processing unit 208 includes a plurality of functions (i.e., additional analysis processes) for analyzing the dynamic information and sensor data. The plurality of analysis functions are realized by the first processing unit to the Nth processing unit. The analysis processes executed by the first processing unit to the Nth processing unit are hierarchically arranged according to their types, and the analysis results are also hierarchically arranged. For example, the first processing unit to the Nth processing unit are classified (e.g., hierarchically) into processes for analyzing sensor data and processes for analyzing non-sensor data.

[0055] For example, the first processing unit and the second processing unit analyze sensor data. For example, the first processing unit and the second processing unit read and analyze original sensor data in which dynamic objects included in the dynamic information (i.e., position and simple attributes) are detected from the storage unit 200, and generate detailed information about each dynamic object. For example, if the simple attribute of a dynamic object is a person, the first processing unit detects (i.e., identifies) whether the object is a child, an adult, an elderly person, etc., and the second processing unit detects whether the object is walking while using a smartphone, running a red light, etc. For example, if the simple attribute of a dynamic object is a car, the first processing unit detects whether the object is a regular car, a large vehicle, etc., and the second processing unit detects whether the object is a bus, a taxi, an emergency vehicle, a distracted driver, etc. When reading traffic light information from the storage unit 200, the second processing unit may detect whether a person or a car is running a red light.

[0056] For example, processing units other than the first and second processing units analyze non-sensor data. For example, the third to fifth processing units (not shown) read dynamic information (i.e., location and simple attributes) and traffic light information from the storage unit 200 and predict the future location of a dynamic object (e.g., a person, a vehicle, etc.) contained therein. This analysis result is referred to as "movement prediction." For example, the third processing unit analyzes changes in the location of the same dynamic object stored in the storage unit 200 over time and predicts the movement area of ​​the dynamic object t seconds from now. For example, the fourth processing unit uses traffic light information to detect the current behavior of the dynamic object (e.g., ignoring a traffic light, etc.). For example, the fifth processing unit uses traffic light information to predict the behavior of the dynamic object t seconds from now (e.g., the possibility of a collision, etc.). The analysis result of the fourth and fifth processing units is referred to as "traffic condition prediction." The traffic condition prediction may include the result of detecting the current traffic condition (e.g., congestion, accidents, etc.) and predicting the traffic condition t seconds from now.

[0057] As described above, the additional analysis processing unit 208 executes the additional analysis processing specified by the analysis processing specification information input from the determination unit 204. That is, the additional analysis processing unit 208 executes the processing unit specified by the analysis processing specification information from among the first to Nth processing units. The additional analysis processing unit 208 outputs the processing results obtained by the executed processing unit to the output unit 210.

[0058] The output unit 210 reads out dynamic information (i.e., position and simple attributes) from the storage unit 200, combines it with the analysis results input from the additional analysis processing unit 208 to generate hierarchical driving assistance information, and outputs it to the autonomous driving ECU 126. In other words, the additional analysis processing unit 208 and the output unit 210 constitute a driving assistance information generation unit. This generates hierarchical driving assistance information according to the predicted time (i.e., the allowable delay) until the host vehicle reaches the dynamic object, and transfers it to the autonomous driving ECU 126. Therefore, the autonomous driving ECU 126 can appropriately control the driving of the host vehicle using the driving assistance information.

[0059] As described above, the data received from the outside includes dynamic information and sensor data, and the information about the dynamic object includes the position and simple attributes of the dynamic object. The driving assistance information generation unit generates hierarchical driving assistance information including the result of executing the additional analysis process and the position and simple attributes as layers. This allows the position and simple attributes of the dynamic object provided from outside the vehicle to be effectively used as driving assistance information. Furthermore, by analyzing the sensor data, it is possible to generate driving assistance information including detailed attributes of the dynamic object. By analyzing the position and simple attributes of the dynamic object, it is possible to generate driving assistance information including movement predictions of the dynamic object.

[0060] 5, the in-vehicle system 100 mounted on the vehicle 102 acquires dynamic information (i.e., position, simple attributes) and sensor data of a dynamic object (i.e., a pedestrian 900) from the infrastructure sensor 104 and the vehicle 112 (specifically, the in-vehicle system 110). The dynamic information is generated by analyzing sensor data (e.g., image data), and the sensor data includes the dynamic object. The dynamic information and sensor data are stored in the memory 142 of the in-vehicle gateway 122 in the in-vehicle system 100. Although not shown in FIG. 5, the in-vehicle system 100 also receives signal information from traffic lights. As described above, the allowable delay estimator 202 calculates the allowable delay Tp according to the distance L from the host vehicle (i.e., the vehicle 102) to the dynamic object (i.e., the pedestrian 900) and the speed V of the host vehicle, and the transfer delay estimator 206 calculates the transfer delay Tt by taking into account the load state of the host vehicle. Based on the difference between the allowable delay Tp and the transfer delay Tt, the determination unit 204 selects an executable additional analysis process, and the additional analysis processing unit 208 executes the selected additional analysis process on the data read from the storage unit 200 (i.e., the memory 142). As a result of analyzing the sensor data, for example, detailed attributes are generated. Also, as a result of analyzing the non-sensor data (for example, location information, simple attributes, and traffic signal information), for example, movement predictions and traffic situation predictions are generated.

[0061] The processing results (e.g., detailed attributes, movement predictions, and traffic situation predictions) and the position and simple attributes (see dashed lines) read from the storage unit 200 (i.e., memory 142) are layered to generate driving assistance information, which is then transferred to the autonomous driving ECU 126. The driving assistance information transferred to the autonomous driving ECU 126 is layered information generated in consideration of delay times (e.g., allowable delay, transfer delay, and analysis processing time) as described above. Therefore, the autonomous driving ECU 126 can effectively use the driving assistance information to control the driving of the host vehicle.

[0062] [Operation of the in-vehicle gateway] With reference to Fig. 6, the operation of the in-vehicle gateway 122 will be described with reference to the functions shown in Fig. 4. The processing shown in Fig. 6 is realized by the control unit 140 shown in Fig. 3 reading out a predetermined program from the memory 142 and executing it.

[0063] 6, in step 300, control unit 140 determines whether data has been received by communication unit 120. If it is determined that data has been received, control proceeds to step 302. Otherwise, step 300 is repeated.

[0064] In step 302, the control unit 140 stores the received data in the memory 142. The received data includes sensor data transmitted from the infrastructure sensors 104 and other vehicles 112, dynamic information, and traffic light information transmitted from the traffic lights 106.

[0065] In step 304, the control unit 140 determines whether or not dynamic information has been received. If it is determined that dynamic information has been received, control proceeds to step 306. If not, control proceeds to step 320.

[0066] In step 306, the control unit 140 estimates the allowable delay Tp. Specifically, the control unit 140 calculates the predicted time (i.e., L / V) required to reach the dynamic object from the distance L from the host vehicle to the dynamic object and the host vehicle's speed V, and sets this time as the allowable delay Tp. This corresponds to the function of the allowable delay estimation unit 202 (see FIG. 4) described above. Thereafter, control proceeds to step 308.

[0067] In step 308, the control unit 140 estimates the transfer delay Tt. Specifically, the load state inside the vehicle is observed, and the time required to transfer the driving assistance information to the autonomous driving ECU 126 is calculated and set as the transfer delay Tt. This corresponds to the function of the transfer delay estimation unit 206 (see FIG. 4) described above. Thereafter, control proceeds to step 310.

[0068] In step 310, the control unit 140 determines whether the allowable delay Tp estimated in step 306 is greater than the transfer delay Tt estimated in step 308 (i.e., Tp>Tt). This corresponds to the function of the determination unit 204 (see FIG. 4) described above. If it is determined that the allowable delay Tp is greater, control proceeds to step 312. If not (i.e., Tp≦Tt), control proceeds to step 320.

[0069] In step 312, the control unit 140 refers to the processing time table 212 (see FIG. 4) and acquires the processing time τ corresponding to the amount of data to be processed for each additional analysis process. This corresponds to the function of the determination unit 204 (see FIG. 4) described above.

[0070] In step 314, the control unit 140 identifies one or more additional analysis processes that can be completed within the time represented by the value obtained by subtracting the transfer delay Tt from the allowable delay Tp (i.e., Tp-Tt). Specifically, the control unit 140 determines whether the sum of the processing times of the one or more additional analysis processes is equal to or less than Tp-Tt. This corresponds to the function of the determination unit 204 (see FIG. 4) described above.

[0071] In step 316, the control unit 140 executes the additional analysis process selected in step 314. This corresponds to the function of the additional analysis processing unit 208 (see FIG. 4) described above. If multiple additional analysis processes are selected in step 314, the control unit 140 executes these additional analysis processes by multitasking. The analysis results are stored in the memory 142 as appropriate. If multitasking is not possible, the processes may be executed, for example, according to the flowchart shown in FIG. 7, which will be described later.

[0072] In step 318, the control unit 140 transfers the analysis results obtained in step 316 to the autonomous driving ECU 126 as driving assistance information. Specifically, the control unit 140 reads out the dynamic information (i.e., the position and simple attributes) stored in the memory 142, and combines it with the processing results of step 316 to generate hierarchical driving assistance information, which is then transferred to the autonomous driving ECU 126. This corresponds to the function of the output unit 210 (FIG. 4) described above. As a result, the autonomous driving ECU 126 uses the transferred analysis results for driving control of the host vehicle.

[0073] In step 320, the control unit 140 determines whether an end instruction has been received. If it is determined that an end instruction has been received, the program ends. If not, control returns to step 300, and the above processing is repeated. The end instruction is issued, for example, by turning off the power source installed in the vehicle 102.

[0074] As a result, when the in-vehicle gateway 122 receives dynamic information (i.e., position and simple attributes), it can execute additional analysis processing selected based on the allowable delay Tp and provide the analysis results to the autonomous driving ECU 126. The additional analysis processing executed varies depending on the distance from the host vehicle to the dynamic object. That is, if the distance from the host vehicle to the dynamic object is relatively large, the autonomous driving ECU 126 can acquire detailed attributes and prediction information, etc., and can perform driving control that predicts the state ahead of the host vehicle. It also becomes possible to provide information such as warnings to the driver in advance. If the host vehicle approaches a dynamic object, the autonomous driving ECU 126 cannot acquire detailed attributes and prediction information, etc., but can perform driving control using the position and simple attributes.

[0075] The flowchart shown in FIG. 6 can be executed with various modifications. In FIG. 6, if step 310 returns NO, the process proceeds to step 320, but this is not limited to this. For example, if the additional analysis process does not include prediction processes such as movement prediction, the received dynamic information (i.e., the location and simplified attributes) may be discarded. This prevents unused data from remaining in memory 142 and reducing available capacity. Furthermore, the received dynamic information (i.e., the location and simplified attributes) may be transferred to autonomous driving ECU 126 along with information indicating that the transfer delay Tt is equal to or greater than the allowable delay Tp. This allows autonomous driving ECU 126 to determine whether or not to use the dynamic information, and the dynamic information can be used.

[0076] Furthermore, in step 310, it is determined whether the allowable delay Tp is greater than the transfer delay Tt, but this is not limiting. Even if Tp > Tt, if the difference is small, there is no room to execute additional analysis processing. Therefore, it is preferable to determine whether the difference between Tp and Tt is equal to or greater than a predetermined value greater than or equal to 0. The predetermined value can be, for example, the minimum value of the processing time of multiple planned additional analysis processing. This allows an appropriate additional analysis processing to be selected, and prevents unnecessary execution of steps 312 and 314.

[0077] In the above, the case where the sensor data stored in the memory 142 is analyzed as the additional analysis process of the sensor data has been described, but the present invention is not limited to this. The dynamic information (i.e., the position and simple attributes) received by the in-vehicle system 100 may not have sensor data attached. In such a case, the sensor data cannot be analyzed and detailed attributes cannot be detected for the dynamic object included in the dynamic information. Therefore, it is preferable that the in-vehicle gateway 122 newly acquires sensor data and analyzes it to detect detailed attributes. Such a process will be described with reference to FIG. 7.

[0078] When acquiring new sensor data, a request is made to the infrastructure sensor 104, the in-vehicle system 110, etc. to transmit the sensor data, and time is required to receive the sensor data, which becomes the delay time. Therefore, it is preferable to select an executable additional analysis process taking into consideration the data reception time. For example, for an additional analysis process that processes sensor data, the data reception time is also stored in the processing time table 212. For example, for an additional analysis process that processes sensor data, whether the process is executable can be determined by determining whether the sum of the processing time and the data reception time is equal to or less than the difference between the allowable delay and the transfer delay. This allows an appropriate additional analysis process to be selected even when new sensor data is received.

[0079] The process shown in Fig. 7 is a specific example of step 316 shown in Fig. 6. In step 400, the control unit 140 specifies one of the additional analysis processes identified in step 314. Thereafter, control proceeds to step 402.

[0080] In step 402, the control unit 140 determines whether the additional analysis process specified in step 400 is a process that processes sensor data (i.e., sensor data processing). If it is sensor data processing, control proceeds to step 404. If not (i.e., non-sensor data processing), control proceeds to step 408.

[0081] In step 404, the control unit 140 determines whether sensor data including a dynamic object included in the dynamic information is stored in the memory 142. As described above, the infrastructure sensors 104, the in-vehicle system 110, etc. may transmit dynamic information and corresponding sensor data. When the in-vehicle system 110 receives such data, the sensor data is stored in the memory 142. If it is determined that the sensor data is stored, control proceeds to step 408. Otherwise, control proceeds to step 406.

[0082] In step 406, the control unit 140 transmits a request to an external device to transmit sensor data including the dynamic object, and receives the sensor data transmitted in response. For example, the control unit 140 requests an infrastructure sensor located near the position of the dynamic object (e.g., stored as dynamic information in the memory 142) to transmit sensor data. At this time, if the infrastructure sensor stores sensor data from a certain period of time in the past, the control unit 140 may issue a transmission request specifying the time when the sensor data was stored. For example, the control unit 140 requests sensor data from a time period that includes the acquisition time of the dynamic information stored in the memory 142. This increases the likelihood of acquiring sensor data including the target dynamic object. The control unit 140 may also request the transmission of sensor data from an in-vehicle system of a vehicle traveling near the position of the dynamic object.

[0083] In step 408, the control unit 140 executes the additional analysis process specified in step 400. At this time, the additional analysis process is a process of analyzing sensor data, and if the sensor data has been acquired in step 406, the sensor data is analyzed. Note that if the sensor data cannot be received within the predetermined time in step 406, the additional analysis process is not executed.

[0084] In step 410, the control unit 140 determines whether or not additional analysis processes remain to be executed. If it is determined that there are, the control returns to step 400. If not, the control returns to the flowchart of FIG. 6 and proceeds to step 318. If the control returns to step 400, a new additional analysis process is designated in step 400 so as not to overlap with any additional analysis processes that have already been executed, and the above process is repeated.

[0085] As a result, for additional analysis processing that processes sensor data, if the sensor data is not stored in memory 142, the sensor data can be received from an external device and analyzed to detect detailed attributes. Even when newly acquired sensor data is analyzed, an appropriate specific analysis processing can be selected, and the analysis results can be effectively used for vehicle driving control.

[0086] In the above, in the flowchart of FIG. 6, a case has been described in which all executable additional analysis processes are identified in step 314, and then each identified additional analysis process is executed in step 316. However, this is not limiting. Each time an executable additional analysis process is identified, it may be executed. For example, instead of steps 314 and 316, for any one additional analysis process, it is determined whether the process can be completed within Tp-Tt, and if so, that process is executed. It is determined whether another additional analysis process can be completed within a value (i.e., time) obtained by subtracting the processing time required for the executed additional analysis process from Tp-Tt, and if so, the process is executed. By repeating this process, multiple additional analysis processes can be executed.

[0087] 6, the case where the received dynamic information is transferred to the autonomous driving ECU 126 along with the results of the additional analysis process has been described, but this is not limiting. The process of transferring the received dynamic information to the autonomous driving ECU 126 and the selected additional analysis process may be executed in parallel. This allows new data (e.g., dynamic information) to be quickly transferred to the autonomous driving ECU 126, and the autonomous driving ECU 126 can quickly reflect the transferred data in the driving control of the host vehicle.

[0088] Furthermore, the results of the additional analysis processing of sensor data may be used as the target of the additional analysis processing of non-sensor data. That is, the additional analysis processing of non-sensor data may be performed using at least one of the results of the additional analysis processing of sensor data and dynamic information as the processing target. For example, detailed attributes may be obtained as the result of the additional analysis processing of sensor data. The detailed attributes may be added to the dynamic information (i.e., location and simple attributes) to generate data to be processed, and additional analysis processing may be performed to obtain movement predictions and traffic situation predictions. This can improve the accuracy of the additional analysis processing of non-sensor data.

[0089] [Changes in driving assistance information] Referring to FIG. 8, we will explain how the driving assistance information generated by the in-vehicle system 100 of a vehicle 102 changes as the vehicle 102 approaches a dynamic object. In FIG. 8, vehicles 102A to 102D represent the same vehicle 102 whose position changes over time. Similarly, pedestrians 900A to 900D represent the same pedestrian 900 whose position changes over time. Pedestrians 900A to 900D represent, for example, a state where the vehicle is walking while using a smartphone. Vehicles and people with the same alphabet at the end of their reference symbols represent states at the same time. As described above, the driving assistance information is generated taking into account the allowable delay Tp, which is the predicted time it takes for the vehicle to reach the dynamic object. Therefore, FIG. 8 shows allowable delays T1 to T4 calculated according to the distance between the traveling vehicle and the dynamic object. As the vehicle 102 approaches the pedestrian 900, the allowable delays decrease in the order of T1 to T4. Note that in FIG. 8, the position and simple attributes are collectively referred to as "position and simple attributes."

[0090] Here, the selectable additional analysis processes are analysis processes that obtain detailed attributes, movement predictions, and traffic situation predictions, respectively, and the processing times are assumed to be longer for the analysis processes that generate detailed attributes, movement predictions, and traffic situation predictions, in that order. Assume that vehicle 102A is traveling on a road where the distance between the vehicle and pedestrian is relatively large and the allowable delay Tp is T1≧Tp>T2. The vehicle gateway 122 of vehicle 102A executes additional analysis processes to generate detailed attributes, movement predictions, and traffic situation predictions. These analysis results, along with the location and simple attributes received from external devices such as infrastructure sensors 104, are used to generate hierarchical driving assistance information. The generated driving assistance information is transferred to the autonomous driving ECU 126 and stored in memory 142.

[0091] When the distance between the vehicle and the pedestrian becomes shorter and the vehicle 102B begins traveling on a road where the allowable delay Tp is T2≧Tp>T3, the in-vehicle gateway 122 performs additional analysis processing to generate detailed attributes and movement predictions. The in-vehicle gateway 122 of the vehicle 102B does not perform additional analysis processing to generate traffic situation predictions. Hierarchical driving assistance information is generated using the analysis results (i.e., detailed attributes and movement predictions) and the location and simple attributes. The generated driving assistance information is transferred to the autonomous driving ECU 126. In FIG. 8, a solid right-facing arrow indicates that the corresponding information is generated and updated during that time, and a dashed right-facing arrow indicates that the corresponding information is not generated or updated during that time. Information indicated by dashed lines indicates information that is not updated.

[0092] The distance between the vehicle and the pedestrian becomes smaller, and the allowable delay Tp becomes T 3 ≧Tp>T 4 When the vehicle 102C starts traveling on a road, the vehicle gateway 122 executes additional analysis processing to generate detailed attributes. The vehicle gateway 122 of the vehicle 102C does not execute additional analysis processing to generate movement predictions and traffic situation predictions. Hierarchical driving assistance information is generated using the analysis results (i.e., detailed attributes) and the position and simple attributes. The generated driving assistance information is transferred to the autonomous driving ECU 126.

[0093] The distance between the vehicle and the pedestrian becomes smaller and the allowable delay Tp becomes T 4 When the vehicle 102D starts traveling on a road where Tp≧0, the in-vehicle gateway 122 does not perform any additional analysis processing. The location and simple attributes received from the outside are transferred to the autonomous driving ECU 126 as driving assistance information.

[0094] In this way, the driving assistance information used to control the driving of a vehicle changes within a single vehicle. By providing the driving assistance information that changes depending on the driving conditions of the vehicle to the autonomous driving ECU 126, the driving of the vehicle is appropriately controlled.

[0095] [Present to the driver] The in-vehicle system 100 can provide appropriate information to the driver using the driving assistance information. Examples of changes in the information provided by the in-vehicle system 100 will be described with reference to FIGS. 9 to 13. FIG. 9 shows vehicles 102A to 102D and pedestrians 900A to 900D shown in FIG. 8 in two dimensions, i.e., on a road map. Traffic lights and infrastructure sensors 104 are located at an intersection 910. FIG. 9 shows a state in which a traffic light 106a for vehicles is green and a traffic light 106b for pedestrians is red. Pedestrians 900 (i.e., pedestrians 900A to 900D) cross the crosswalk while walking and using their smartphones (i.e., ignoring the traffic light) even though the traffic light 106b for pedestrians is red. In this situation, the in-vehicle system 100 of the vehicle 102 provides information to the driver over time, for example, as shown in FIGS. 10 to 13.

[0096] As described above, the driving assistance information generated by the in-vehicle gateway 122 of the vehicle 102A traveling at a position where the distance to the dynamic object (i.e., the pedestrian 900A) is large (i.e., the allowable delay Tp is T1≧Tp>T2) includes the analysis results, i.e., detailed attributes, movement prediction, and traffic situation prediction, as well as the received dynamic information (i.e., location and simplified attributes). Based on the dynamic information (i.e., location and simplified attributes), the in-vehicle system 100 displays a graphic 920A representing the current pedestrian (i.e., the pedestrian 900A) on a map near the intersection 910, for example, as shown in FIG. 10 , on a part of the display screen of the car navigation system. Because the graphic 920A is located on a pedestrian crossing even though the traffic light 106b is red, the in-vehicle system 100 can identify the occurrence of a dangerous situation (i.e., a pedestrian has begun to ignore the traffic light at the intersection 910 in the direction of travel of the vehicle). Therefore, the in-vehicle system 100 displays a warning message 230. Furthermore, the in-vehicle system 100 displays a graphic 922 showing a dynamic object included in the movement prediction after t1 seconds. In Fig. 10, the graphic showing the current dynamic object is displayed by a solid line, and the graphic showing the future dynamic object identified from the movement prediction is displayed by a dashed line (the same applies to Figs. 11 to 13).

[0097] This allows the driver of vehicle 102 to know that there is a pedestrian who has started to cross the crosswalk ignoring the traffic light at intersection 910 ahead. The driver can also know that there is a high possibility that the pedestrian will still be on the crosswalk in the future (for example, t1 seconds later), and the driver of the vehicle determines that he or she needs to drive carefully.

[0098] Subsequently, when the distance to the dynamic object (i.e., pedestrian 900B) becomes smaller (i.e., the allowable delay Tp is T2≧Tp>T3), the driving assistance information generated by the in-vehicle gateway 122 of the vehicle 102B includes the detailed attributes and movement predictions that are the analysis results, and the received dynamic information (i.e., location and simplified attributes). Based on the dynamic information (i.e., location and simplified attributes), the in-vehicle system 100 displays a graphic 920B representing the current pedestrian (i.e., pedestrian 900B) on the map, as shown in FIG. 11 . Because the graphic 920B is on a pedestrian crossing, the in-vehicle system 100 can identify that a dangerous situation continues and maintains the displayed message 230. Furthermore, the in-vehicle system 100 uses the movement prediction to display a graphic 924 representing a future pedestrian (e.g., a dynamic object t2 seconds later).

[0099] Subsequently, when the distance to the dynamic object (i.e., pedestrian 900C) becomes even shorter (i.e., the allowable delay Tp is T3≧Tp>T4), the driving assistance information generated by the in-vehicle gateway 122 of the vehicle 102C includes the detailed attributes that are the analysis results and the received dynamic information (i.e., the location and simple attributes). Based on the dynamic information (i.e., the location and simple attributes), the in-vehicle system 100 displays a graphic 920C representing the current pedestrian (i.e., pedestrian 900C) on the map, as shown in FIG. 12. Because the graphic 920C is on a pedestrian crossing, the in-vehicle system 100 can identify that a dangerous situation continues and maintains the displayed message 230. Because no movement prediction is generated, the same graphic 924 as in FIG. 11 is maintained in FIG. 12.

[0100] By displaying the information as shown in Figures 11 and 12, the driver of the vehicle can know that pedestrians are still crossing the crosswalk at the intersection 910 ahead, ignoring the traffic lights, and understands that he needs to drive carefully.

[0101] Subsequently, when the distance to the dynamic object (i.e., pedestrian 900D) becomes even smaller (i.e., the allowable delay Tp is T4≧Tp>0), the in-vehicle gateway 122 of the vehicle 102D does not perform additional analysis processing. Therefore, the real-time information included in the driving assistance information is only the received dynamic information (i.e., location and simple attributes). Based on the dynamic information (i.e., location and simple attributes), the in-vehicle system 100 displays a graphic 920D representing the current pedestrian (i.e., pedestrian 900D) on the map, as shown in FIG. 13 . Because the graphic 920D is on the sidewalk, the in-vehicle system 100 can determine that the danger has passed and that the pedestrian has finished crossing the crosswalk at the intersection 910 ahead, and erases the displayed message 230. This informs the driver of the vehicle that the danger has passed and that the pedestrian has finished crossing the crosswalk at the intersection 910 ahead.

[0102] In this way, the in-vehicle gateway 122 generates hierarchical driving assistance information according to the predicted time (i.e., allowable delay) for the vehicle 102 to reach the dynamic object. As a result, the in-vehicle system 100 can notify and warn the driver of the vehicle that a dangerous situation has occurred. The type of information (i.e., the hierarchy) to be included in the driving assistance information changes according to the allowable delay. Therefore, the in-vehicle system 100 can provide appropriate driving assistance without generating information that is useless to the vehicle.

[0103] Although the above description has been given of a case where a pedestrian is a dynamic object, this is not limiting. The detection target can also be a moving object that may be damaged by a collision with a vehicle, such as a person riding a bicycle or an animal.

[0104] [Variations] In the above, as shown in Fig. 3, the case where the in-vehicle gateway 122, which is an in-vehicle device that is standard equipment in the in-vehicle system 100, generates the driving assistance information has been described, but this is not limiting. In a modified example, the driving assistance information is generated by a device that is not standard equipment in the in-vehicle system and can be installed (i.e., mounted in the vehicle) later.

[0105] 14, an in-vehicle system 150 mounted on a vehicle includes a communication unit 120, an in-vehicle gateway 154, a sensor 124, an autonomous driving ECU 126, an ECU 128, and a bus 130 and a bus 132. The in-vehicle system 150 is not equipped with a standard feature but is equipped with an expansion device 152 that was installed later. In FIG. 14, components with the same reference numerals as those in FIG. 3 have the same functions as those in FIG. 3. The following description will mainly focus on the different components.

[0106] The in-vehicle system 150 includes a bus 132 similar to the bus 130. The communication unit 120 exchanges data with the in-vehicle gateway 154 via the bus 132. That is, the communication unit 120 transfers data received from the outside to the in-vehicle gateway 154 via the bus 132, and transmits data transferred from the in-vehicle gateway 154 to the outside via the bus 132.

[0107] 15, the extension device 152 includes a control unit 160 and a memory 162. The control unit 160 is configured to include a CPU and controls the memory 162. The memory 162 is, for example, a rewritable nonvolatile semiconductor memory, and stores programs executed by the control unit 160. The memory 162 provides a work area for the programs executed by the control unit 160. The control unit 160 acquires data to be processed via the bus 132, stores the processing results in the memory 162, and also outputs them to the bus 132 as appropriate. The extension device 152 has the same functions as the in-vehicle gateway 122 shown in FIG. 3, i.e., the functions shown in FIG. 4.

[0108] The extension device 152 can acquire data received by the communication unit 120 (e.g., traffic light information, vehicle information (e.g., position, speed, driving direction), dynamic information, sensor data, etc.) via the bus 132. Unlike the in-vehicle gateway 122, the in-vehicle gateway 154 does not have the functions shown in FIG. 4 . The in-vehicle gateway 154 transmits sensor data output from the sensor 124 to the bus 130 to the extension device 152 via the bus 132. The in-vehicle gateway 154 acquires the speed of the host vehicle from a drive unit that drives the host vehicle, and transmits the speed to the extension device 152 via the bus 132. The in-vehicle gateway 154 also transmits data output from the extension device 152 to the bus 132 to the autonomous driving ECU 126 via the bus 130. This allows the extension device 152 to perform additional analysis processing and generate hierarchical driving assistance information depending on the time it takes for the host vehicle to reach a dynamic object (i.e., the allowable delay). The driving assistance information is transferred to the automatic driving ECU 126 and used for driving control of the vehicle.

[0109] The driving assistance information generated in the vehicle 102 may be transmitted to an in-vehicle system of another vehicle, such as the in-vehicle system 110 of the vehicle 112. For example, the in-vehicle gateway 122 generates packet data including the driving assistance information and transmits it from the communication unit 120 to the in-vehicle system 110 of the vehicle 112 via the base station 108. The driving assistance information is transmitted from the communication unit 120 by broadcasting, for example. This allows the driving assistance information to be used for the autonomous driving of the other vehicle. For example, if the vehicle 112 is traveling near the vehicle 102 at a speed similar to that of the vehicle 102, it is considered that the time it takes to reach the same dynamic object is approximately the same. Therefore, the in-vehicle system 110 may be able to use the received driving assistance information for the autonomous driving of the vehicle 112.

[0110] When the in-vehicle gateway 122 of the vehicle 102 determines which additional analysis process to perform to generate driving assistance information usable by other vehicles, it is preferable to consider the communication time between vehicles (i.e., the time it takes to transmit data to other vehicles) as a delay time in addition to the above-mentioned allowable delay, transfer delay, processing time, etc. This increases the possibility that the driving assistance information generated in the vehicle 102 will be effectively used for driving control of other vehicles.

[0111] Each process (each function) in the above-described embodiments may be realized by a processing circuit including one or more processors. The processing circuit may be configured by an integrated circuit or the like that combines one or more memories, various analog circuits, and various digital circuits in addition to the one or more processors. The one or more memories store programs (instructions) that cause the one or more processors to execute each process. The one or more processors may execute each process according to the program read from the one or more memories, or may execute each process according to a logic circuit designed in advance to execute each process. The processor may be any of various processors suitable for computer control, such as a CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0112] Furthermore, the processing of the in-vehicle device 100 (specifically, the processing executed by the in-vehicle gateway 122 (for example, the processing shown in FIGS. 6 and 7) )A recording medium having recorded thereon a program for causing a computer to execute the above can be provided. The recording medium is, for example, an optical disc (such as a DVD (Digital Versatile Disc)) or a removable semiconductor memory (such as a USB (Universal Serial Bus) memory). Although the computer program can be transmitted via a communication line, the recording medium is a non-transitory recording medium. By having a computer mounted on a vehicle load the program stored on the recording medium, the computer can generate appropriately hierarchical driving assistance information within the vehicle according to the predicted time until the vehicle reaches the dynamic object, as described above, and use the driving assistance information for driving control of the vehicle.

[0113] (Addendum) That is, the computer-readable non-transitory recording medium is The vehicle's on-board computer an allowable delay estimation function that estimates a time required for the vehicle to reach a dynamic object as an allowable delay; a transfer delay estimation function that estimates, as a transfer delay, the time from when the computer receives data from outside the vehicle until when the computer transfers the data to an execution unit of an automatic driving function, based on the load state of information processing and information transmission in the vehicle; and a determination function that selects a specific analysis process from among a plurality of analysis processes for analyzing the data received from an external device based on a difference between the allowable delay and the transfer delay; a driving assistance information generation function that executes the specific analysis process selected by the determination function and generates driving assistance information, the data received from the outside includes information about the dynamic object; The driving assistance information is stored in a computer program that is transferred to the execution unit of the automatic driving function.

[0114] Although the present disclosure has been described above by explaining the embodiments, the above-described embodiments are merely examples, and the present disclosure is not limited to only the above-described embodiments. The scope of the present disclosure is defined by the claims in the scope of the claims, taking into consideration the description of the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wordings described therein. [Explanation of symbols]

[0115] 100, 110, 150 In-vehicle systems 102, 112, 102A, 102B, 102C, 102D vehicles 104 Infrastructure Sensors 106, 106a, 106b traffic lights 108 Base Station 114 Network 120 Communications Department 122, 154 Vehicle Gateway 124 sensors 126 Autonomous Driving ECU 128 ECU Buses 130 and 132 140, 160 control unit 142, 162 memory 152 Expansion Unit 200 Storage section 202 Allowable Delay Estimation Unit 204 Judgment section 206 Transfer Delay Estimation Unit 208 Additional analysis processing section 210 Output section 212 Processing Time Table 230 Messages 300, 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 400, 402, 404, 406, 408, 410 steps 900, 900A, 900B, 900C, 900D Pedestrian 910 Intersection 920A, 920B, 920C, 920D, 922, 924 shapes T1, T2, T3, T4, Tp Allowable delay

Claims

1. An in-vehicle device mounted on a vehicle having an automatic driving function, an allowable delay estimation unit that estimates a time required for the vehicle to reach a dynamic object as an allowable delay; a transfer delay estimation unit that estimates, based on a load state of information processing and information transmission in the vehicle, a time from when the in-vehicle device receives data from outside the vehicle until when the in-vehicle device transfers the data to the execution unit of the automatic driving function, as a transfer delay; a determination unit that selects a specific analysis process from among a plurality of analysis processes for analyzing the data received from the outside based on a difference between the allowable delay and the transfer delay; a driving assistance information generation unit that executes the specific analysis process selected by the determination unit and generates driving assistance information; the data received from the outside includes information about the dynamic object; The driving assistance information is transferred to the execution unit of the automatic driving function.

2. The data received from the outside further includes sensor data; the information about the dynamic object includes position information and simplified attribute information about the dynamic object; The in-vehicle device according to claim 1 , wherein the driving assistance information generation unit generates the hierarchical driving assistance information including, as layers, a result of executing the specific analysis process, the location information, and the simplified attribute information.

3. The driving assistance information is a first layer including an analysis result of the specific analysis process targeting the sensor data; The in-vehicle device according to claim 2 , further comprising: a second layer including an analysis result of the specific analysis process that does not process the sensor data.

4. 4. The in-vehicle device according to claim 2, wherein the specific analysis process that does not process the sensor data processes at least one of an analysis result of the specific analysis process that processes the sensor data and the information related to the dynamic object.

5. The determination unit calculating the difference by subtracting the transfer delay from the allowable delay; determining whether the difference is greater than a predetermined value that is equal to or greater than 0; If the difference is greater than the predetermined value, the specific analysis process is selected; The in-vehicle device according to claim 1 , wherein the specific analysis process is not selected if the difference is equal to or smaller than the predetermined value.

6. The in-vehicle device according to claim 5 , wherein if the difference is equal to or smaller than the predetermined value, the information regarding the dynamic object is transferred to the execution unit together with information indicating that the transfer delay is equal to or larger than the allowable delay.

7. a storage unit that stores a processing time table that records processing times according to the amount of data to be processed for each of the plurality of analysis processes; 6. The in-vehicle device according to claim 5, wherein if the difference is greater than the predetermined value, the determination unit uses the data amount of the data to refer to the processing time table to identify a processing time for the data, and then selects the specific analysis process by determining whether the processing time is less than or equal to the difference.

8. the processing time table further includes, for an analysis process that processes sensor data among the plurality of analysis processes, an acquisition time for newly acquiring the sensor data to be processed; 8. The in-vehicle device according to claim 7, wherein if the difference is greater than the predetermined value, the determination unit selects the specific analysis process by determining whether a sum of the processing time and the acquisition time specified by referring to the processing time table is less than or equal to the difference.

9. An in-vehicle system installed in a vehicle having an automatic driving function, an execution unit for the automatic driving function; a communication unit for acquiring data including information about a dynamic object; An in-vehicle system comprising: an in-vehicle device according to any one of claims 1 to 3.

10. The in-vehicle system according to claim 9 , wherein the communication unit further transmits the driving assistance information generated by the in-vehicle device to another vehicle together with information on the position and driving direction of the vehicle.

11. The determination unit of the in-vehicle device Estimating a communication time of the driving assistance information transmitted from the communication unit; The in-vehicle system according to claim 10 , wherein the specific analysis process is selected from the plurality of analysis processes based on a difference between the allowable delay and the sum of the transfer delay and the communication time.

12. A control method for supporting an automatic driving function of a vehicle, comprising: an allowable delay estimation step of estimating a time required for the vehicle to reach the dynamic object as an allowable delay; a transfer delay estimation step of estimating, based on the load state of information processing and information transmission in the vehicle, a time from when an in-vehicle device mounted on the vehicle receives data from outside the vehicle until when the in-vehicle device transfers the data to the execution unit of the automatic driving function, as a transfer delay; a determining step of selecting a specific analysis process from among a plurality of analysis processes for analyzing the data received from an external device based on a difference between the allowable delay and the transfer delay; a driving assistance information generating step of executing the specific analysis process selected by the determining step and generating driving assistance information; the data received from the outside includes information about the dynamic object; A control method in which the driving assistance information is transferred to an execution unit of the automatic driving function.

13. The vehicle's on-board computer an allowable delay estimation function that estimates a time required for the vehicle to reach a dynamic object as an allowable delay; a transfer delay estimation function that estimates, as a transfer delay, the time from when the computer receives data from outside the vehicle until when the computer transfers the data to an execution unit of an automatic driving function, based on the load state of information processing and information transmission in the vehicle; and a determination function that selects a specific analysis process from among a plurality of analysis processes for analyzing the data received from an external device based on a difference between the allowable delay and the transfer delay; a driving assistance information generation function that executes the specific analysis process selected by the determination function and generates driving assistance information, the data received from the outside includes information about the dynamic object; A computer program in which the driving assistance information is transferred to an execution unit of the automatic driving function.

Images