In-vehicle device, roadside device, control method, and computer program

Abstract

This in-vehicle device is mounted in a vehicle, and comprises: a transmission unit for transmitting transmission data to a roadside device that is located outside the vehicle; an estimation unit for estimating an end delay time and a usable band that can be used by the in-vehicle device for wireless communication with the roadside device; a determination unit for determining a parameter to be used for generating, from to-be-transmitted data, transmission data to be transmitted from the transmission unit so as to satisfy the usable band and the end delay time; and a generation unit for generating, by using the parameter, transmission data from the to-be-transmitted data. The end delay time represents a time period from when the to-be-transmitted data has been generated to when the transmission data is handed over to a computer program for realizing a service to be provided by the roadside device.

Description

[Technical Field]

[0001] This disclosure relates to an in-vehicle device, a roadside device, a control method, and a computer program. This application claims priority to Japanese Patent Application No. 2021-177405 filed on October 29, 2021, and incorporates by reference all of the contents of that Japanese application. [Background technology]

[0002] A system for linking an on-board device mounted on an automobile, motorcycle, or the like (hereinafter referred to as a vehicle) with an external device such as a server has been proposed. For example, data is uploaded from the on-board device to the external device via wireless communication, and the external device uses the received data in various services that it provides. One of the services provided by the external device is a service that provides information to assist the driver of the vehicle.

[0003] Modern vehicles are equipped with various electronic devices and ECUs (Electronic Control Units) that control them. For example, vehicles capable of autonomous driving are equipped with an autonomous driving ECU. The autonomous driving ECU communicates with external devices as needed to obtain necessary information (including, for example, road traffic information and dynamic driving assistance information) and controls the driving of the vehicle using the obtained information. Other in-vehicle ECUs include an engine control ECU, a stop-start control ECU, a transmission control ECU, an airbag control ECU, a power steering control ECU, and a hybrid control ECU. For autonomous vehicles, external devices provide services such as remote monitoring and remote control.

[0004] Patent Document 1 listed below discloses a communication device that is mounted on a moving body such as a vehicle and that can transmit sensor information acquired by the moving body to a remote device, etc., based on the importance of the sensor (for example, a camera, etc.). This communication device determines the priority of each sensor based on the moving state and surrounding state of the vehicle, and controls communication so that information detected by a sensor with a higher priority can be transmitted with higher quality than information detected by a sensor with a lower priority.

[0005] The following Patent Document 2 discloses a transmission method capable of transmitting high-quality video suitable for remote driving while suppressing noise and delays in remote vehicle monitoring and control via a mobile phone network. This transmission method transmits images taken by multiple cameras mounted on a vehicle via a network, estimating the available bandwidth that can be used on the network, and allocating bandwidth to each camera according to the available bandwidth and the importance of each camera. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2020 / 090285 [Patent Document 2] International Publication No. 2021 / 070214 Summary of the Invention

[0007] An on-board device according to one aspect of the present disclosure is an on-board device mounted on a vehicle, and includes a transmitter that transmits transmission data to a roadside device that is a device located outside the vehicle, an estimation unit that estimates an available bandwidth and an end delay time that the on-board device can use in wireless communication with the roadside device, a determination unit that determines parameters used when generating transmission data to be transmitted from the transmitter from data to be transmitted so as to satisfy the available bandwidth and the end delay time, and a generation unit that generates transmission data from the data to be transmitted using the parameters, wherein the end delay time is the time from when the data to be transmitted is generated to when the transmission data is passed to a computer program that realizes a service provided by the roadside device. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a linkage system including an in-vehicle device and a server. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the in-vehicle device shown in FIG. [Figure 3] FIG. 3 is a block diagram illustrating a hardware configuration of the in-vehicle / out-vehicle linking unit illustrated in FIG. [Figure 4] FIG. 4 is a block diagram showing a hardware configuration of the server shown in FIG. [Figure 5] FIG. 5 is a block diagram showing a schematic hierarchical structure of software in a vehicle (specifically, an on-board device) and a server. [Figure 6] FIG. 6 is a block diagram showing a functional configuration of the in-vehicle / out-vehicle linking unit shown in FIG. [Figure 7] FIG. 7 is a schematic diagram illustrating various delays that occur in data transmission from an in-vehicle device to a server. [Figure 8] FIG. 8 is a diagram showing parameters relating to the acquisition of moving image data in a table format. [Figure 9] FIG. 9 is a flowchart showing the process of uploading video data executed by the vehicle interior / exterior linking unit. [Figure 10]FIG. 10 is a flowchart showing a process executed by the server to transmit information used for uploading video data executed by the in-vehicle / out-of-vehicle linking unit. [Figure 11] FIG. 11 is a diagram showing a state in which parameters are changed by time division. [Figure 12] FIG. 12 is a flowchart showing a process for uploading video data executed by the in-vehicle / out-of-vehicle linking unit according to the first modification. [Figure 13] FIG. 13 is a block diagram showing functions of the in-vehicle / out-vehicle linking unit according to the second modification. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Problem to be solved by this disclosure] In order for data uploaded from an in-vehicle device to an external device (e.g., a roadside device) to be effectively used by an application for implementing a service provided by the external device, it is important to consider latency. This is particularly important for services such as remote control and remote monitoring, which require real-time performance. Latency includes not only communication latency but also latency within the vehicle as an end system. However, Patent Documents 1 and 2 do not adequately consider latency. That is, while Patent Documents 1 and 2 both consider the available communication bandwidth in the communication line (i.e., the wireless section), they do not consider latency in the in-vehicle device (e.g., latency due to the load on the internal network, etc.). In particular, problems may arise when multiple services and back traffic executed in the external device coexist.

[0010] Therefore, the present disclosure aims to provide an in-vehicle device, a roadside device, a control method, and a computer program that, when uploading data to an external device such as a roadside device, can transmit data that can be effectively used by services provided by the external device, taking into account delay time and communication bandwidth.

[0011] [Effects of the invention] According to the present disclosure, it is possible to provide an in-vehicle device, a roadside device, a control method, and a computer program that can transmit data that can be effectively used by services provided by an external device, taking into account delay time and communication bandwidth when uploading data to an external device such as a roadside device.

[0012] [Description of the embodiments of the present disclosure] The contents of the embodiments of the present disclosure will be listed and described below. At least some of the embodiments described below may be combined in any combination.

[0013] (1) An on-board device according to a first aspect of the present disclosure is an on-board device mounted on a vehicle, the on-board device including: a transmitter that transmits transmission data to a roadside device that is a device located outside the vehicle; an estimation unit that estimates an available bandwidth and an end delay time that the on-board device can use in wireless communication with the roadside device; a determination unit that determines parameters used when generating transmission data to be transmitted from the transmitter from data to be transmitted so as to satisfy the available bandwidth and the end delay time; and a generation unit that generates transmission data from the data to be transmitted using the parameters, the end delay time being the time from when the data to be transmitted is generated to when the transmission data is passed to a program that realizes a service provided by the roadside device. This allows data that can be effectively used by a service provided by the external device to be transmitted, taking into account the delay time and the communication bandwidth when uploading data to an external device such as the roadside device.

[0014] (2) In the above (1), the data to be transmitted can include video data acquired from a sensor mounted on the vehicle, and the parameters can include at least one of a compression ratio, a resolution, a frame rate, and a bit rate related to the transmission data. This allows the video data acquired from the sensor to be transmitted so as to satisfy the available bandwidth and end delay time.

[0015] (3) In the above (1) or (2), the transmission target data may include video data acquired from each of a plurality of sensors mounted on the vehicle, the determination unit may include a priority determination unit that determines the priority of each of the plurality of sensors according to the service provided by the roadside device, and the generation unit may generate transmission data from the video data acquired from each of the plurality of sensors according to the priority. This allows the video data acquired from the plurality of sensors to be appropriately used by the service provided by the roadside device.

[0016] (4) In the above (3), the priority may be a ratio of the communication bandwidth used to transmit video data acquired from each of the multiple sensors to the roadside device, and the determination unit may determine parameters for each video data item so as to satisfy the communication bandwidth and end delay time determined from the available bandwidth and the ratio. This makes it possible to efficiently determine an appropriate bit rate for transmitting video data acquired from each sensor to the roadside device.

[0017] (5) In any one of (1) to (4) above, the available bandwidth may be the lower of the communication speed estimated from the communication line quality of the wireless communication and the upper limit communication speed allowed for the in-vehicle device in the wireless communication service provided. This allows appropriate determination of parameters used when uploading data to an external device such as a roadside device.

[0018] (6) In any one of (1) to (5) above, the generator may repeatedly determine the parameters at a predetermined cycle, and the predetermined cycle may be shorter than at least one of the fluctuation cycle of the wireless communication channel in wireless communication and the fluctuation cycle of the communication speed inside the vehicle. This allows appropriate determination of the parameters used when uploading data to an external device such as a roadside device.

[0019] (7) In any one of the above (1) to (6), the transmission target data can also be used by a function control device installed in a vehicle, thereby avoiding a situation where the function control device cannot use the sensor data because the sensor data is transmitted to the roadside device.

[0020] (8) In any one of (1) to (7) above, the transmitter may transmit predetermined data before transmitting the transmission data, and the estimation unit may estimate the available bandwidth based on the communication speed at which the predetermined data is transmitted, and the predetermined data may include an analysis result of sensor data acquired from a sensor mounted on the vehicle or data related to the vehicle. This makes it possible to suppress data transmission for estimating the available bandwidth.

[0021] (9) In any one of (1) to (7) above, the estimation unit may estimate the available bandwidth to be used next by the determination unit based on the communication speed at which the transmission data is transmitted by the transmission unit. This makes it possible to suppress transmission of unnecessary data. This makes it possible to suppress data transmission for estimating the available bandwidth.

[0022] (10) In any one of (1) to (9) above, when a roadside device provides multiple services in parallel that have different requirements that can be used to estimate end delay times, the determination unit may determine parameters corresponding to each of the requirements, and the generation unit may generate transmission data and the transmission unit may transmit the transmission data to the roadside device for a predetermined period for each of the requirements. This allows data that can be effectively used by each service to be transmitted.

[0023] (11) In any one of (1) to (9) above, when a roadside device provides multiple services in parallel, each of which has different requirements that can be used to estimate the end delay time, the estimation unit may determine parameters using the result of an AND operation of the multiple requirements. This allows data transmitted to the roadside device to be effectively used by any of the services.

[0024] (12) In any one of (1) to (5) above, the transmitter may transmit the transmission data to the roadside device in response to an occurrence of an event, and the event may be related to at least one of the running state of the vehicle and the traffic state around the vehicle. This allows the amount of data transmitted within a certain period to be reduced compared to the case of periodic transmission.

[0025] (13) In the above (4), the transmitter may transmit the transmission data to the roadside device in response to the occurrence of an event, and the event may be related to at least one of the vehicle's running state and the traffic state around the vehicle, and the ratio may be changed in response to the occurrence of the event. This reduces the amount of data transmitted within a certain period compared to the case of periodic transmission, and enables appropriate sensor data to be transmitted depending on the situation.

[0026] (14) In any one of (1) to (13) above, a roadside device according to a second aspect of the present disclosure includes a communication unit that communicates with the in-vehicle device and an infrastructure linkage unit that observes the state of the service provided and identifies an allowable delay time that can be used to estimate the end delay time, and the communication unit transmits the allowable delay time to the in-vehicle device. As a result, the in-vehicle device that receives the allowable delay time can determine appropriate parameters to be used when generating transmission data from sensor data acquired from a sensor of the vehicle, and the data uploaded to the roadside device can be effectively used for the service provided by the roadside device.

[0027] (15) A control method according to a third aspect of the present disclosure is a control method for an on-board device mounted on a vehicle, the control method including: a transmitting step of transmitting transmission data to a roadside device located outside the vehicle; an estimating step of estimating an available bandwidth and an end delay time that the on-board device can use in wireless communication with the roadside device; a determining step of determining parameters used when generating the transmission data to be transmitted from data to be transmitted so as to satisfy the available bandwidth and the end delay time; and a generating step of generating the transmission data from the data to be transmitted using the parameters, the end delay time being the time from when the data to be transmitted is generated to when the transmission data is passed to a computer program that realizes a service provided by the roadside device. Thus, when uploading data to an external device such as a roadside device, the on-board device can transmit data that can be effectively used for a service provided by the external device, taking into account the delay time and the communication bandwidth.

[0028] (16) A computer program according to a fourth aspect of the present disclosure causes a computer mounted on a vehicle to implement a transmitting function for transmitting transmission data to a roadside device located outside the vehicle, an estimating function for estimating an available bandwidth and an end delay time that the computer can use in wireless communication with the roadside device, a determining function for determining parameters used when generating transmission data to be transmitted from data to be transmitted so as to satisfy the available bandwidth and the end delay time, and a generating function for generating transmission data from the data to be transmitted using the parameters, wherein the end delay time is the time from when the data to be transmitted is generated to when the transmission data is passed to a computer program that realizes a service provided by the roadside device. As a result, when uploading data to an external device such as a roadside device, the computer (on-board device) can transmit data that can be effectively used for a service provided by the external device, taking into account the delay time and communication bandwidth.

[0029] [Details of the embodiments of the present disclosure] In the following embodiments, the same components are denoted by the same reference numerals, and their names and functions are also the same, so detailed descriptions thereof will not be repeated.

[0030] [Overall configuration] Referring to FIG. 1 , the cooperation system according to the embodiment of the present disclosure includes an in-vehicle device 100 and an in-vehicle device 110 mounted on a vehicle 102 and a vehicle 112, respectively, a base station 104, and a server 106. The base station 104 is connected to a network 108 such as the Internet. The base station 104 is a base station for wide-area communication such as cellular communication. The base station 104 provides mobile communication services using, for example, LTE (Long Term Evolution), 4G (i.e., a fourth-generation mobile communication system) lines, and 5G (i.e., a fifth-generation mobile communication system) lines. The in-vehicle device 100 and the in-vehicle device 110 can communicate with the server 106 via the base station 104 and the network 108. The base station 104 may provide wireless communication functions such as Wi-Fi and C-V2X (Cellular-Vehicle to Everything).

[0031] Each of the vehicles 102 and 112 is equipped with a sensor such as an image sensor, and the sensor data output from the sensor is acquired by the in-vehicle device 100 and the in-vehicle device 110 and transmitted (hereinafter also referred to as "uploaded") to the server 106. The server 106 performs services such as providing driving assistance information, remote monitoring, and remote control, and uses the sensor data received from the in-vehicle device 100 and the in-vehicle device 110 for the services it provides. The server 106 may be a device installed outside the vehicle 102, the vehicle 112, etc., or may be a roadside device fixedly installed on a road or its surroundings.

[0032] FIG. 1 exemplarily shows one base station 104 and two vehicles 102 and 112 equipped with on-board devices. However, this is merely an example. Typically, multiple base stations are provided, and three or more vehicles are equipped with on-board devices. There may be vehicles that do not have on-board devices. Vehicles that do not have on-board devices are the detection targets of sensors installed in vehicles equipped with on-board devices.

[0033] [Hardware configuration of on-board device] Referring to FIG. 2, an example of the hardware configuration of an in-vehicle device 100 mounted on a vehicle 102 is shown. The in-vehicle device 110 mounted on a vehicle 112 is similarly configured. The in-vehicle device 100 includes a communication unit 120 and an in-vehicle / out-of-vehicle linking unit 122. FIG. 2 shows the communication unit 120, sensors 124, an autonomous driving ECU 126, an ECU 128, and a bus 130 mounted on the vehicle 102. The vehicle 102 is equipped with a plurality of ECUs as devices for controlling various functions of the vehicle (i.e., function control devices). The autonomous driving ECU 126 is one of the function control devices. The vehicle 102 is equipped with a plurality of ECUs in addition to the autonomous driving ECU 126, and FIG. 2 shows ECU 128 as a representative of these.

[0034] The communication unit 120 performs wireless communication with devices outside the vehicle 102 via the base station 104. The communication unit 120 includes an integrated circuit (IC) for performing modulation and multiplexing adopted in the wireless communication service provided by the base station 104, an antenna for transmitting and receiving radio waves of a predetermined frequency, an RF circuit, etc. The communication unit 120 also has a function of communicating with a global navigation satellite system (GNSS) such as a global positioning system (GPS).

[0035] The in-vehicle / out-of-vehicle linking unit 122 is responsible for connecting communication functions (i.e., communication specifications) with communication functions within the vehicle (i.e., communication specifications) (for example, communication protocol conversion, etc.). The autonomous driving ECU 126 can communicate with external devices via the in-vehicle / out-of-vehicle linking unit 122 and the communication unit 120. The in-vehicle / out-of-vehicle linking unit 122 transfers driving assistance information received from the outside via the communication unit 120 to the autonomous driving ECU 126. The bus 130 is responsible for communication functions within the vehicle. Communication (i.e., data exchange) between the in-vehicle / out-of-vehicle linking unit 122, the sensor 124, the autonomous driving ECU 126, and the ECU 128 is performed via the bus 130. For example, Ethernet (registered trademark), CAN (Controller Area Network), etc. are used for the bus 130.

[0036] The sensor 124 is mounted on the vehicle 102 and includes a sensor for acquiring information about the inside and outside of the vehicle 102. The sensor for acquiring information about the outside of the vehicle includes an imaging device (for example, a digital camera (for example, a CCD camera or a CMOS camera)), a radar (for example, a millimeter-wave radar), and a laser sensor (for example, a LiDAR (Light Detection and Ranging)). The sensor for acquiring information about the inside of the vehicle includes an imaging device. The sensor 124 acquires information within a detection range (for example, an imaging range in the case of a camera) and outputs it as sensor data. In the case of a digital camera, it outputs digital moving image data. The detection signal (i.e., an analog or digital signal) of the sensor 124 is output as digital data to the bus 130 via an I / F unit (not shown) and transmitted to the vehicle interior / exterior linking unit 122, the autonomous driving ECU 126, etc.

[0037] The autonomous driving ECU 126 controls the driving of the vehicle 102. For example, the autonomous driving ECU 126 acquires sensor data, analyzes it to understand the situation around the vehicle, and controls mechanisms related to autonomous driving (for example, mechanisms such as the engine, transmission, steering, or brakes). The autonomous driving ECU 126 uses driving assistance information (for example, dynamic information) acquired from the in-vehicle / out-of-vehicle cooperation unit 122 for autonomous driving.

[0038] 3, the vehicle interior / exterior linking unit 122 includes a control unit 140, a memory 142, an I / F unit 144, and a transmission data generating unit 146. The control unit 140 includes a CPU (Central Processing Unit) and controls the memory 142, the I / F unit 144, and the communication unit 120. Control signals for this purpose are indicated by dotted arrows in FIG. 3. The memory 142 is, for example, a rewritable nonvolatile semiconductor memory, and stores a computer program (hereinafter simply referred to as a program) executed by the control unit 140. The memory 142 provides a work area for the program executed by the control unit 140.

[0039] The I / F unit 144 interfaces with the sensor 124 (see FIG. 2 ) under the control of the control unit 140. If the sensor 124 is a camera that outputs video data and multiple imaging conditions (e.g., resolution and frame rate of the output data) can be set, the control unit 140 sets the imaging conditions of the camera via the I / F unit 144. Under the control of the control unit 140, the transmission data generation unit 146 generates transmission data from the sensor data acquired from the sensor 124 and stored in the memory 142. This sensor data is data to be transmitted to the server 106 (hereinafter also referred to as transmission target data). For example, if the sensor data is video data, the data volume is large, so the transmission data generation unit 146 converts the resolution and frame rate and compresses the data to avoid congesting the communication bandwidth. If the transmission data generation unit 146 can generate transmission data from the sensor data based on multiple conditions, the control unit 140 specifies the conditions (e.g., resolution, frame rate, compression rate, etc.) to the transmission data generation unit 146. The transmission data generation unit 146 generates transmission data based on specified conditions. The generated transmission data is stored in memory 142. The communication unit 120 is controlled by the control unit 140 to generate packet data from the transmission data stored in memory 142 and transmit it. If the sensor data is video data, a predetermined amount of data sequentially acquired from the sensor 124 is buffered in memory 142, and transmission data is generated from the buffered data and sequentially transmitted.

[0040] [Server hardware configuration] Referring to FIG. 4, the server 106 includes a control unit 160, a memory 162, a communication unit 164, and a bus 166. Data transfer between the units is performed via the bus 166. The control unit 160 includes a CPU, controls the units, and provides various services. The communication unit 164 receives information (including sensor data, vehicle information, etc.) uploaded from the in-vehicle device 100 and the in-vehicle device 110 via the base station 104 and the network 108. The memory 162 includes a rewritable nonvolatile semiconductor memory and a large-capacity storage device such as an HDD (Hard Disk Drive). The data received by the communication unit 164 is transferred to and stored in the memory 162. The uploaded data stored in the memory 162 is used by an application program (hereinafter simply referred to as an application) for providing the service. The communication unit 164 has a function of wirelessly or wiredly accessing the network 108. Under the control of the control unit 160, the communication unit 164 transmits request data, etc. to the vehicles 102 and 112 from the memory 162. The server 106 includes an operation unit (not shown) such as a computer keyboard and mouse, which allows an administrator or the like to input instructions to the control unit 160.

[0041] [Software configuration] Referring to FIG. 5, the software of the collaboration system configured by the vehicle 102 and the server 106 is configured hierarchically. The hierarchical structure shown in FIG. 5 corresponds, for example, to the hierarchical structure of the OSI (Open Systems Interconnection) reference model. The vehicle 102 includes multiple ECUs as described above, and the upper layers include application programs (1-ECU application to M-ECU application) for realizing the functions of each ECU. The upper layer applications are executed in parallel by microcomputers or the like mounted on each ECU. The upper layer corresponds, for example, to the application layer of the OSI reference model. The vehicle 102 includes, in its lower layers, a communication stack (for example, the session layer and below of the OSI reference model) that handles communication with the outside world, and in its middle layers, a sublayer program that mediates between the upper layer programs and the lower layer programs. The middle layer corresponds, for example, to the presentation layer of the OSI reference model.

[0042] The functions (or corresponding programs) of the in-vehicle-external linking unit 122 of the in-vehicle device 100 are mainly positioned in the sub-layer programs. The in-vehicle-external linking unit 122 controls the communication stack of the lower layer and transmits sensor data such as video image data to the server 106 as described above. The in-vehicle-external linking unit 122 controls the communication stack of the lower layer and receives request data from the server 106 regarding the transmission of sensor data such as video image data as described below. The in-vehicle-external linking unit 122 also controls the communication stack of the lower layer and receives service data (e.g., driving assistance information) transmitted from the server 106 as described above, and passes the service data to, for example, the autonomous driving ECU 126 (see FIG. 2 ). The functions (or corresponding programs) of the in-vehicle-external linking unit 122 may include some of multiple applications in the upper layer. That is, the in-vehicle-external linking unit 122 may include the function of an ECU.

[0043] As described above, the server 106 executes a plurality of services (including driving assistance, remote monitoring, remote control, etc.), and includes application programs (first service application to Nth service application) for realizing each service in its upper layer. The server 106 includes a communication stack in its lower layer that handles communication with the outside (for example, the in-vehicle device 100 of the vehicle 102), and includes a sub-layer program in its middle layer that mediates between the programs in the upper layer and the programs in the lower layer. The upper layer, middle layer, and lower layer correspond to, for example, the application layer, presentation layer, and session layer and lower layers of the OSI reference model, respectively.

[0044] The programs for implementing the infrastructure interworking unit that communicates with the in-vehicle device 100 of the vehicle 102 to configure the interworking system are mainly positioned as sub-layer programs. The upper-layer and middle-layer programs are executed as multitasks by the control unit 160 (see FIG. 4). The infrastructure interworking unit controls the lower-layer communication stack and, as described above, receives sensor data such as video data transmitted from the vehicle 102 and passes it to the upper-layer program. The infrastructure interworking unit observes the operation of the upper-layer programs and acquires their required conditions (e.g., request data such as an allowable delay time). The server 106 controls the lower-layer communication stack and transmits the request data. The infrastructure interworking unit also controls the lower-layer communication stack and transmits service data (e.g., driving assistance information) provided by the upper-layer program to the in-vehicle device 100 of the vehicle 102.

[0045] [Functional configuration of the vehicle interior / exterior connection unit] The function of the in-vehicle-external linking unit 122 will be described with reference to Fig. 6. As shown in Fig. 4, the in-vehicle-external linking unit 122 includes a control unit 140, a memory 142, an I / F unit 144, and a transmission data generation unit 146. The control unit 140 includes an estimation unit 220, a determination unit 222, and a communication control unit 224, and the determination unit 222 includes a parameter determination unit 226 and a priority determination unit 228. The transmission data generation unit 146 includes a frame rate conversion unit 210, a resolution conversion unit 212, and a compression unit 214.

[0046] In FIG. 6 , sensor 124 includes camera 200, camera 202, and camera 204. These cameras are arranged on vehicle 102 such that camera 200 captures an image in front of vehicle 102, camera 202 captures an image in rear of vehicle 102, and camera 204 captures an image in the interior of vehicle 102. Cameras 200, 202, and 204 may be cameras with the same specifications or different specifications. For example, camera 200 capturing an image in front of vehicle 102 may have a higher resolution than cameras 202 and 204. Camera 204 capturing an image in the interior of vehicle 102 may include a microphone that detects sound. As described above, the image capturing conditions (including resolution, frame rate, etc.) of cameras 200, 202, and 204 can be set by control unit 140 via I / F unit 144.

[0047] The frame rate conversion unit 210 reads the video data stored in the memory 142 and converts the video data to the frame rate specified by the parameter determination unit 226. For example, if the frame rate of the video data stored in the memory 142 (i.e., the frame rate of the output data of the sensor 124) is an integer multiple of the frame rate specified by the parameter determination unit 226, the converted video data can be generated by thinning out the original video data. If there is no corresponding frame, frame data of the converted video data can be generated by interpolation using multiple adjacent frame data in the original video data. The converted video data is stored in the memory 142.

[0048] The resolution conversion unit 212 reads out the video data stored in the memory 142 and converts (i.e., resizes) the video data so that it has the resolution specified by the parameter determination unit 226. Specifically, the resolution conversion unit 212 generates pixel data for the converted frame from the pixel data of each frame by interpolation or the like. The video data after the resolution change is stored in the memory 142. When the frame rate conversion unit 210 and the resolution conversion unit 212 perform their processing, the resolution conversion unit 212 may be executed on the result of execution of the frame rate conversion unit 210, or the frame rate conversion unit 210 may be executed on the result of execution of the resolution conversion unit 212.

[0049] The compression unit 214 reads video data stored in memory and compresses it to reduce the data size. The compression unit 214 compresses the video data so that the ratio of the compressed data size (e.g., per unit time) to the data size (e.g., per unit time) of the original image is equal to the compression rate specified by the parameter determination unit 226. The compression method may be, for example, MPEG (Moving Picture Experts Group) (including MPEG2 and MPEG4). Since compressed video data typically does not have a frame-by-frame data structure, when changing either the frame rate or the resolution, the compression unit 214 compresses the video data after the corresponding processing by the frame rate conversion unit 210 or the resolution conversion unit 212 has been performed. The compressed video data is stored in the memory 142.

[0050] The estimation unit 220 estimates the UL communication speed and end delay time when uploading data from the in-vehicle device 100 to the server 106. The UL communication speed refers to the available bandwidth for wireless communication, i.e., the wireless communication speed. The end delay time refers to the time from when sensor data is output from a sensor mounted on the vehicle to when the data is passed to a service application executed by the server. The estimation unit 220 also calculates a transmission delay from the estimated end delay time. The transmission delay is the time required from when data is transmitted from the vehicle's communication unit to when the data is received by the server and passed to the application executing the service. In other words, the transmission delay is the time obtained by subtracting the delay time within the vehicle (e.g., the time required for sensor data to be transmitted and processed) from the end delay time. The estimation unit 220 outputs the estimated UL communication speed and the calculated delay time to the parameter determination unit 226.

[0051] (UL communication speed estimate) The estimation unit 220 determines the maximum communication speed X (unit: bps) at which stable communication is possible through observation. The maximum communication speed X (unit: bps) represents the quality of the wireless communication line (i.e., the communication line quality). For example, before uploading video data to the server 106, the estimation unit 220 transmits a pilot signal (hereinafter referred to as a pilot packet) of a predetermined data amount (specifically, the data size of the payload). The estimation unit 220 can calculate the communication speed by dividing the data amount transmitted as the pilot packet by the transmission time of the pilot packet. If the in-vehicle device 100 and the server 106 are synchronized (i.e., the timers of both indicate the same time), the transmission time can be calculated by subtracting the timestamp at which the transmission of the pilot packet started from the timestamp included in the packet notifying normal reception transmitted from the server 106. If the in-vehicle device 100 and the server 106 are asynchronous (i.e., their timers indicate different times), the estimation unit 220 can calculate the communication speed by, for example, executing a PING command and using the RTT (Round-Trip Time) included in the response. That is, since the RTT represents the round-trip time, half of that can be used as the transmission time.

[0052] Typically, when providing a wireless communication service, an upper limit Y (unit: bps) of the upload communication speed available to a single communication terminal is set. The upper limit Y (unit: bps) of the communication speed may be stored in advance in the memory 142, for example. Furthermore, data other than video data (e.g., data related to the vehicle) is also uploaded from the in-vehicle device 100 to the server 106. The communication speed allocated to this is assumed to be a (unit: bps). The estimation unit 220 can determine the communication speed a by observing actual data transmission. The communication speed a may be stored in the memory 142 in advance. Taking these factors into consideration, the estimation unit 220 calculates (i.e., estimates) the UL communication speed Z (unit: bps) by subtracting a (unit: bps) from the smaller of the values ​​X (unit: bps) and Y (unit: bps) obtained as described above. That is, Z = min(X, Y) - a. min is a function representing the minimum value of multiple variables in parentheses.

[0053] (Estimation of end delay time and calculation of transmission delay) The real-time performance required varies depending on the service (i.e., service application) provided by the server 106, and accordingly the delay time allowed for video data received by the server 106 (hereinafter referred to as the allowable delay time) also varies. In order for uploaded video data to be effectively used by the service application of the server 106, the upper limit of the end delay time is limited.

[0054] Referring to FIG. 7, the end delay time is comprised of times T1 to T5. Time T1 is the time (hereinafter referred to as the transmission period or capture period) required for data (e.g., video data) periodically transmitted (i.e., output) from camera 200 in frame units to reach memory 142 after being output from camera 200. The transmission period (i.e., capture period) is expressed as the reciprocal of the frame rate (fps) of the video data. Since video data output from camera 200 is composed of frames and subsequent processing targets frames, the time until the data becomes processable (i.e., the transmission period) is the delay time. Times T2 and T3 are the time required for image capture and processing by processing unit 250. Image capture and processing unit 250 corresponds to frame rate conversion unit 210, resolution conversion unit 212, and compression unit 214 shown in FIG. 6. Time T2 is the time required for video data to be stored in memory 142 and for compression processing to begin (i.e., image capture delay). Time T2 is the time required to convert the video data, including the time required to convert the frame rate and resolution. Time T3 is the time required to compress the converted video data (i.e., compression time). Time T4 is the time required for the transmission data (i.e., compressed video data) to be read from memory 142, transmitted within in-vehicle device 100 to communication unit 120, and made available for transmission by communication unit 120 (i.e., other processing time). Time T5 is the transmission delay described above.

[0055] As mentioned above, the end delay time = T1 + T2 + T3 + T4 + T5. Therefore, the transmission delay can be calculated by the following formula: Transmission delay (i.e., time T5) = End delay time - (T1 + T2 + T3 + T4) (Equation 1) Regarding the term on the right side of Equation 1, since the end delay time depends on the service provided by the server 106 as described above, information for estimating the end delay time can be obtained from the server 106. The estimation unit 220 estimates the end delay time used in Equation 1 from the allowable delay time transmitted from the server 106. Since the allowable delay time is the upper limit of the end delay time, for example, the estimation unit 220 can set the allowable delay time as the end delay time in Equation 1. Furthermore, to be safe, the end delay time in Equation 1 may be set to a value slightly smaller than the allowable delay time. Time T1 (i.e., the transmission period) is determined by the frame rate of the camera. Since times T2 to T4 depend on the target data size (e.g., resolution, etc.), if values ​​corresponding to the data size (e.g., actual measured values) are stored in advance in the memory 142 as a table, the estimation unit 220 can determine times T2 to T4 by referring to the table. Note that time T3 (i.e., the compression time) also depends on the compression rate, so it is preferable that the table also takes the compression rate into consideration. It should be noted that, among the proportions of the end delay time that each of times T1 to T5 occupies, time T5 (i.e., transmission delay) is the largest (usually several hundred milliseconds to several seconds). When video data compression is realized using hardware such as a semiconductor integrated circuit, times T1 to T4 other than time T5 are considered to be less than one hundred milliseconds. Therefore, times T1 to T4 may be set to fixed values.

[0056] The priority determination unit 228 determines the priorities of multiple sensors (i.e., cameras 200, 202, and 204). The priorities indicate which of the output data from the multiple sensors is to be uploaded with a higher priority. For example, the priority determination unit 228 determines the ratio of communication bandwidths (i.e., communication speeds) to be allocated to uploading the output data (i.e., video data) from each sensor, based on the driving state of the vehicle 102 and a request from the server 106. For example, the ratio is set as r1:r2:r3 for cameras 200, 202, and 204 (r1, r2, and r3 are real numbers equal to or greater than 0, such that r1+r2+r3=100(%)). The request from the server 106 is made from the perspective of the server 106 using the video data uploaded from the in-vehicle device 100 (i.e., output data from cameras 200, 202, and 204). The request from the server 106 depends, for example, on the program (i.e., service application) running on the server 106. For example, in the case of a service such as driving assistance, there is a higher demand for video image data acquired from cameras 200 and 202 that capture images outside the vehicle than from camera 204 that captures images inside the vehicle. Therefore, for example, r1:r2:r3=100:0:0, 50:50:0, or 0:100:0 is set. In the case of a service that monitors the driver's driving state (e.g., drowsy driving or inattentive driving), a communication bandwidth is also allocated to camera 204 that captures images inside the vehicle. The priority may be specified from outside (e.g., an operation unit) of in-vehicle device 100. The priority determination unit 228 outputs the determined priority (i.e., ratio) to the parameter determination unit 226.

[0057] The parameter determination unit 226 determines the bit rate for uploading sensor data (i.e., video data) for each of the multiple sensors (i.e., camera 200, camera 202, and camera 204) and determines parameters related to the video data. The parameters include the frame rate, resolution, and compression rate. The parameter determination unit 226 outputs the determined parameters to the frame rate conversion unit 210, the resolution conversion unit 212, and the compression unit 214, and operates the frame rate conversion unit 210, the resolution conversion unit 212, and the compression unit 214 as described above. This generates transmission data from the video data, which is data to be transmitted. Note that the parameters (e.g., frame rate, resolution, and compression rate) are parameters related to the transmission data, not parameters related to the video data output from the cameras 200, 202, and 204. In other words, the compression rate is the data size (e.g., per unit time) of the transmission data relative to the data size (e.g., per unit time) of the video data output from the cameras 200, 202, and 204. After the transmitted data is received and decompressed by the server 106, the video data has the frame rate and resolution specified by the parameters.

[0058] (Determining the bit rate) The parameter determination unit 226 calculates a communication speed (hereinafter referred to as an allocated UL communication speed) to be allocated to uploading video data from each of the cameras 200, 202, and 204 in accordance with the priority (i.e., ratio) input from the priority determination unit 228. That is, the parameter determination unit 226 proportionally distributes the UL communication speed input from the estimation unit in accordance with the ratio input from the priority determination unit 228 to calculate each allocated UL communication speed. Next, the parameter determination unit 226 multiplies each allocated UL communication speed by the transmission delay input from the estimation unit 220 to calculate (i.e., determine) a transmission bit rate for each video data. The bit rate is an upper limit value for the amount of data that can be transmitted from the in-vehicle device 100 to the server 106 during the transmission delay. If the amount of data transmitted per unit time is equal to or less than the determined bit rate, data can be transmitted during the transmission delay, and the end delay time is within the allowable delay range for the service provided by the server 106. Therefore, the transmitted data (i.e., video data) is effectively used by the service provided by the server 106. On the other hand, if the amount of data transmitted per unit time is greater than the determined bit rate, the data cannot be transmitted during the transmission delay, and the end delay time exceeds the allowable delay range of the service provided by the server 106. Therefore, the transmitted data (i.e., video data) is not effectively used by the service provided by the server 106.

[0059] (Parameter determination) Next, the parameter determination unit 226 determines video data parameters (i.e., frame rate, resolution, and compression rate) for each camera so that the bit rate is equal to or less than the determined bit rate. The data volume of video data per unit time is calculated by multiplying the number of pixels in one frame (i.e., the product of the number of vertical and horizontal pixels), the data volume per pixel (e.g., 24 bits), the compression rate, and the frame rate (i.e., the number of frames per unit time). Typically, the data volume per pixel of a camera is fixed. The number of pixels in one frame is determined by the resolution, i.e., the number of vertical and horizontal pixels. Therefore, a table such as that shown in FIG. 8 can be stored in advance in the memory 142. The parameter determination unit 226 can refer to the table to determine a combination of parameters that results in a data volume per unit time (i.e., the video data volume shown in FIG. 8) equal to or less than the bit rate and closest to the bit rate. In FIG. 8, a unique ID is assigned to each parameter set. If there are multiple parameter sets that result in the same video data volume, one parameter set can be determined based on a predetermined criterion. For example, if real-time performance is required, a parameter set with a higher frame rate can be selected, whereas if accuracy in detecting an object is required, a parameter set with a higher resolution or a lower compression rate can be selected.

[0060] The resolution may be any number of pixels, or may be a commonly used number of pixels. Resolution classifications such as SD (Standard Definition), HD (High Definition), FHD (Full High Definition), and 4K may be used. For example, SD refers to an image with 720 horizontal pixels and 480 vertical pixels, HD refers to an image with 1280 horizontal pixels and 720 vertical pixels, Full-HD refers to an image with 1920 horizontal pixels and 1080 vertical pixels, and 4K refers to an image with 3840 horizontal pixels and 2160 vertical pixels. Some parameters may also be fixed. For example, at least one of the compression rate and the frame rate may be fixed. In this case, by changing at least the resolution, the amount of data per unit time (i.e., the amount of video data shown in FIG. 8) can be kept below the required bit rate.

[0061] The communication control unit 224 controls the receiving unit 242 and the transmitting unit 240 of the communication unit 120 to communicate with external devices. Specifically, the communication control unit 224 controls the receiving unit 242 and the transmitting unit 240 to transmit pilot packets and receive packets indicating successful reception from the server 106, as described above. The communication control unit 224 may also control the receiving unit 242 and the transmitting unit 240 to transmit PING packets and receive responses thereto, as described above. The communication control unit 224 controls the receiving unit 242 to receive requests (e.g., allowable delay time and camera priority) transmitted from the server 106 and store them in the memory 142. The communication control unit 224 controls the transmitting unit 240 of the communication unit 120 to transmit transmission data generated by the frame rate conversion unit 210, the resolution conversion unit 212, and the compression unit 214 to the server 106. The communication control unit 224 may transmit data periodically at regular time intervals or in response to a trigger generated when some event occurs.

[0062] As described above, the in-vehicle / external link unit 122 can determine parameters for generating transmission data from video image data according to the priority of each of the multiple sensors (camera 200, camera 202, and camera 204) while taking into account the available bandwidth for wireless communication (i.e., wireless communication speed) and end delay time. That is, when uploading data to an external device (e.g., server 106) such as a roadside device, data that can be effectively used by a service provided by the external device can be transmitted while taking into account the delay time and communication bandwidth. Note that, although the above description has been given of a case in which the sensor is a camera, this is not a limitation. Furthermore, the transmission data is not limited to data generated from sensor data. Analysis results of sensor data may also be used. When transmitting a relatively large amount of data that can be used by a service provided by the roadside device to an external device, the data can be transmitted while taking into account the available bandwidth (i.e., wireless communication speed) and end delay time.

[0063] As described above, the in-vehicle / out-of-vehicle linking unit 122 (specifically, the parameter determining unit 226) uses the compression ratio, resolution, and frame rate of the transmission data as parameters when generating transmission data from video data acquired from a sensor mounted on the vehicle. Therefore, the video data acquired from the sensor can be transmitted so as to satisfy the available bandwidth and end delay time.

[0064] In the above description, the transmission bit rate for each video data is calculated (i.e., determined) taking into account the priority and transmission delay, and parameters (i.e., compression rate, resolution, and frame rate) are determined so as not to exceed the calculated bit rate. However, this is not limiting. The bit rate itself may be used as a parameter. For example, the bit rate may be limited to a predetermined value or less to prevent the transmission of unnecessary large amounts of data (e.g., high-quality data). That is, an upper limit may be set for the bit rate, and if the bit rate determined as described above exceeds the predetermined upper limit, the parameter determination unit 226 may replace the determined bit rate with the upper limit and determine the parameters of the video data as described above. Therefore, the parameter may be at least one of the compression rate, resolution, frame rate, and bit rate for the transmission data.

[0065] As described above, the in-vehicle / out-of-vehicle cooperation unit 122 (specifically, the priority determination unit 228) determines the priority of a sensor from among the multiple sensors that acquires transmission target data to be used to generate transmission data, depending on the service provided by the roadside device. Therefore, video data acquired from the multiple sensors can be appropriately used by the service provided by the roadside device. Furthermore, the in-vehicle / out-of-vehicle cooperation unit 122 (specifically, the priority determination unit 228) uses, as the priority, the ratio of the communication bandwidth used to transmit video data acquired from each of the multiple sensors to the roadside device. Therefore, it is possible to efficiently determine an appropriate bit rate when transmitting video data acquired from each sensor to the roadside device.

[0066] As described above, the in-vehicle-external link unit 122 (specifically, the estimation unit 220) determines the available bandwidth (i.e., the UL communication speed) from the lower of the communication speed estimated from the communication line quality of the wireless communication and the upper limit communication speed allowed for the in-vehicle device in the wireless communication service provided. Therefore, it is possible to appropriately determine the parameters used when uploading data to an external device such as a roadside device.

[0067] Each functional block shown in Fig. 6 can be realized by hardware, software, or a combination thereof. To realize it using hardware, dedicated hardware (e.g., a circuit board or an ASIC (Application Specific Integrated Circuit)) may be used that executes some or all of the processes executed by the frame rate conversion unit 210, resolution conversion unit 212, compression unit 214, estimation unit 220, parameter determination unit 226, and priority determination unit 228. To realize it using software, the control unit 140 may execute a predetermined program (see Fig. 9), as described below.

[0068] [Operation of the vehicle interior / exterior linkage unit] With reference to FIG. 9, the operation of the in-vehicle / out-vehicle linking unit 122 will be described with reference to the functions shown in FIG. 6. The process shown in FIG. 9 is realized by the control unit 140 shown in FIG. 3 reading and executing a predetermined program from the memory 142. The process shown in FIG. 9 is started in response to an instruction to upload sensor data (i.e., video data) being issued to the in-vehicle device 100. The instruction to upload may be issued by turning on the power of the in-vehicle device 100, or by an explicit instruction to the in-vehicle device 100 via an operation unit. The above-mentioned upper limit Y (unit: bps) of the upload communication speed available to one communication terminal is stored in the memory 142. Furthermore, it is assumed that predetermined values ​​are stored in the memory 142 as fixed values ​​for the periods T1 to T4. It is also assumed that a table for determining parameters (i.e., frame rate, resolution, and compression rate) such as that shown in FIG. 8 is also stored in the memory 142.

[0069] In step 300, the control unit 140 controls the communication unit 120 to determine whether a service request has been received from the server 106. If it is determined that a service request has been received, control proceeds to step 302. Otherwise, control proceeds to step 314. The service request is sent from the server 106 and is a request (e.g., an allowable delay time and a priority) of an application program being executed by the server 106.

[0070] In step 302, the control unit 140 stores the service request received in step 302 in the memory 142. Thereafter, the control proceeds to step 304.

[0071] In step 304, control unit 140 estimates UL communication speed Z. This corresponds to the function of estimation unit 220 described above. Control unit 140 measures maximum communication speed X (units of bps), reads upper limit Y (units of bps) of the upload communication speed from memory 142, measures or reads from memory 142 communication speed a allocated to transmitting data other than video data, and calculates UL communication speed Z by Z = min(X, Y) - a. Then, control proceeds to step 306.

[0072] In step 306, the control unit 140 estimates the end delay time and calculates the transmission delay as described above. This corresponds to the function of the estimation unit 220 described above. The control unit 140 reads the service request (i.e., the allowable delay time) and times T1 to T4 from the memory 142, estimates the end delay time from the allowable delay time, and calculates the transmission delay by subtracting the sum of times T1 to T4 from the end delay time. Then, control proceeds to step 308.

[0073] In step 308, control unit 140 determines the UL communication speed for each sensor (i.e., the allocated UL communication speed). This corresponds to the function of parameter determination unit 226 described above. Control unit 140 reads the service request (i.e., priority) from memory 142, and determines the allocated UL communication speed by proportionally allocating the UL communication speed determined in step 304 according to the priority. The priority is, for example, the ratio of the communication bandwidth to be allocated, and as described above, the ratio for the front camera, rear camera, and in-car camera is transmitted from server 106. Thereafter, control proceeds to step 310.

[0074] In step 310, control unit 140 determines a bit rate for each sensor using the assigned UL communication speed determined in step 310 and the transmission delay estimated in step 306. This corresponds to the function of parameter determination unit 226 described above. Control unit 140 multiplies the assigned UL communication speed by the transmission delay to determine the bit rate for transmitting video data acquired by each sensor. Thereafter, control proceeds to step 312.

[0075] In step 312, the control unit 140 uses each bit rate determined in step 310 to refer to a table stored in memory 142 and determine parameters for each sensor. This corresponds to the function of the parameter determination unit 226 described above. Thereafter, control proceeds to step 314. The determined parameters are stored in memory 142. For example, in the case of a table such as that shown in FIG. 8, the control unit 140 stores information specifying the ID of the determined parameter set in memory 142.

[0076] In step 314, the control unit 140 determines whether or not to transmit (i.e., upload) the sensor data (i.e., video image data) acquired from each sensor to the server 106. If it is determined that the data should be transmitted, control proceeds to step 316. Otherwise, control proceeds to step 318. For example, if the data is set to be uploaded periodically at a fixed time interval ΔT, the control unit 140 acquires the current time, and if the time ΔT has elapsed since the previous upload, it determines that the data should be transmitted, and if not, it determines that the data should not be transmitted. The control unit 140 acquires the current time from a timer inside the in-vehicle device 100.

[0077] In step 316, the control unit 140 reads the parameters determined in step 312 from the memory 142 and generates transmission data from the sensor data (i.e., video image data) in accordance with the parameters. The control unit 140 controls the communication unit 120 to transmit the generated transmission data to the server 106. Thereafter, control proceeds to step 318. Note that if the control unit 140 has not received a service request, it generates the transmission data using default parameters. The default parameters may be stored in the memory 142 in advance.

[0078] In step 318, the control unit 140 determines whether an end command has been received. If it is determined that an end command has been received, the program ends. If not, control returns to step 300, and the above-described processing is repeated. The end command is issued, for example, by turning off the power source installed in the vehicle 102.

[0079] This allows the in-vehicle / out-of-vehicle link unit 122 to determine parameters for generating transmission data from video data in accordance with the priority of each of the multiple sensors, taking into account the available bandwidth for wireless communication (i.e., wireless communication speed) and end delay time. The transmission data is generated using the determined parameters and uploaded to the server 106, allowing the server 106 to more effectively use the received sensor data for the services it provides.

[0080] [Server Operation] The operation of server 106 will be described with reference to Fig. 10. The process shown in Fig. 10 is realized by control unit 160 shown in Fig. 4 reading and executing a predetermined program from memory 162. It is assumed that a program that monitors the operation of each service application runs in parallel with the program shown in Fig. 10 as part of the function of the infrastructure linking unit (see Fig. 5) described above. By monitoring the operation of the service application, it is possible to obtain service requests (for example, the above-mentioned allowable delay time and priority).

[0081] In step 400, the control unit 160 determines whether or not to start a service that uses sensor data (i.e., video data) uploaded from the in-vehicle device. The service is started, for example, according to a predetermined schedule or by an administrator operating the operation unit of the server 106 to give an instruction. If it is determined that the service should be started, control proceeds to step 402. Otherwise, control proceeds to step 404. The service is, for example, the provision of driving assistance information, remote monitoring, remote control, etc.

[0082] In step 402, the control unit 160 starts the corresponding application. Then, the control proceeds to step 404.

[0083] In step 404, the control unit 160 determines whether to send a service request. The service request includes the delay time (i.e., the allowable delay time) required for the running service to effectively use the uploaded data, the priority of the on-board sensor (e.g., camera), etc. If it is determined to send the service request, the control proceeds to step 406. Otherwise, the control proceeds to step 408. For example, if the service application can use any uploaded data, it is determined not to send a service request.

[0084] In step 406, the control unit 160 transmits a service request. The transmission is performed, for example, by broadcasting. Then, control proceeds to step 408. The transmitted request is used by the in-vehicle device that receives it to determine parameters as described above.

[0085] In step 408, the control unit 160 determines whether sensor data (e.g., video image data) has been received. If it is determined that sensor data has been received, control proceeds to step 410. Otherwise, control proceeds to step 412. The received data is stored in the memory 162.

[0086] In step 410, the control unit 160 passes the sensor data received in step 408 to the running service application. Thereafter, control proceeds to step 412. For example, the control unit 160 notifies the service application of access information (e.g., memory address) for the sensor data stored in the memory 162. In response to this, the service application accesses the memory 162 to acquire the sensor data. If the received sensor data satisfies the service request sent from the server 106, the service application can effectively use the sensor data.

[0087] In step 412, control unit 160 determines whether or not to terminate the running service. The service may be terminated, for example, according to a predetermined schedule or by an administrator operating the operation unit of server 106 to issue an instruction. If it is determined that the service should be terminated, control proceeds to step 414. Otherwise, control proceeds to step 416.

[0088] In step 414, the control unit 160 terminates the corresponding service application. Then, the control proceeds to step 416.

[0089] In step 416, the control unit 160 determines whether an instruction to terminate has been received. If it is determined that an instruction to terminate has been received, the program terminates. If not, control returns to step 400, and the above processing is repeated. The instruction to terminate is given, for example, by an administrator operating the operation unit of the server 106.

[0090] This allows the server 106 to send a service request (e.g., allowable delay time, priority, etc.) to the sender of the sensor data (i.e., the in-vehicle device) so that the service it provides (i.e., the service application) can effectively use the received sensor data. The in-vehicle device that receives the service request can determine appropriate parameters to use when generating transmission data from the sensor data acquired from the sensor of its own vehicle, and the data uploaded to the server 106 can be effectively used by the service provided by the server 106.

[0091] As described above, the in-vehicle / out-of-vehicle linking unit 122 and the server 106 work in cooperation with each other, so that the in-vehicle / out-of-vehicle linking unit 122 can determine parameters for generating transmission data from video data in accordance with the priority of each of the multiple sensors, taking into consideration the available bandwidth for wireless communication (i.e., wireless communication speed) and the end delay time. That is, when uploading data to an external device such as a roadside device (e.g., the server 106), data that can be effectively used by a service provided by the external device can be transmitted, taking into consideration the delay time and communication bandwidth.

[0092] Although the above describes the case where video data is transmitted periodically, it may also be transmitted when an event occurs (hereinafter referred to as event transmission). In this case, in step 314, the control unit 140 determines whether a trigger requesting upload has occurred. For example, the autonomous driving ECU 126 or ECU 128 (see FIG. 2) may output a trigger when a specific traffic situation occurs (e.g., an accident occurs) or when the vehicle 102 enters a specific driving state (e.g., entering an intersection). For example, the autonomous driving ECU 126 or ECU 128 may detect objects requiring caution on road traffic (e.g., traffic signs, pedestrians, red lights, etc.) by analyzing video data acquired from the cameras 200 and 202 that capture images outside the vehicle. Furthermore, it may detect drowsy driving, inattentive driving, etc. by analyzing video data acquired from the camera 204 that captures images inside the vehicle. A trigger may also be output when any one or a combination of these is detected. Event transmission allows for a smaller amount of data to be transmitted within a certain period of time than periodic transmission.

[0093] Furthermore, in the event transmission described above, the ratio of communication bandwidths allocated to the transmitting in-vehicle cameras 200, 202, and 204 may be changed. For example, when entering an intersection, priority may be given to camera 200 capturing images of the front (i.e., a higher ratio may be allocated), and when exiting the intersection, priority may be given to camera 204 capturing images of the rear (i.e., a higher ratio may be allocated). This allows appropriate sensor data to be transmitted depending on the situation.

[0094] There are cases where multiple services are provided simultaneously by the server 106, that is, multiple service applications are executed in parallel. In such cases, if the requirements (e.g., allowable delay time, etc.) of the service applications for the data to be uploaded are the same (including cases where there are slight differences), the in-vehicle / out-of-vehicle link unit 122 can appropriately determine parameters for generating video data to be uploaded. Therefore, the uploaded data can be effectively used by multiple applications. However, if the requirements (e.g., allowable delay time, etc.) are different, some service applications cannot effectively use the received data if the same parameters are used.

[0095] When the requests (e.g., allowable delay times) differ, the in-vehicle-external linkage unit 122 classifies application programs according to the similarity of the requests, determines different parameters for each request, and uploads the application programs to the service applications corresponding to each request in a time-division manner, as shown in FIG. 11 . That is, the in-vehicle-external linkage unit 122 generates and uploads transmission data from video data using a first parameter set during a period ΔT1. The uploaded data is passed to the corresponding service application. The in-vehicle-external linkage unit 122 generates and uploads transmission data from video data using a second parameter set during a period ΔT2 following the period ΔT1. The uploaded data is passed to the corresponding service application. Thereafter, the period ΔT1 in which the first parameter set is used and the period T2 in which the second parameter set is used are repeated. The lengths of the periods ΔT1 and ΔT2 may be the same or different. Even if there are three or more allowable delay times (i.e., three or more parameter sets), the in-vehicle-external linkage unit 122 similarly increases the number of divisions and sequentially uses different parameter sets in a time-division manner. This allows data to be transmitted that can be effectively used by each service application.

[0096] Alternatively, the in-vehicle / out-of-vehicle linking unit 122 may perform an AND operation on multiple requests, determine parameters using the results, and generate transmission data from video data using the parameters. If the request is for an allowable delay time, the smallest allowable delay time is determined by the AND operation on the multiple requests, and the in-vehicle / out-of-vehicle linking unit 122 uses this to determine parameters corresponding to each camera, as described above. The uploaded data is passed to all service applications running on the server 106. Since the uploaded data satisfies the most stringent requirements, it can be effectively used by any service application.

[0097] A port number is used to identify a service application from among multiple service applications running on the server 106 that will deliver uploaded sensor data that satisfies a specific request. That is, when a service request is transmitted from the server 106, the port number of the corresponding service application is added, and when the in-vehicle device that received the service request uploads sensor data, the port number is added and uploaded. This allows the server 106 to identify the service application corresponding to the port number added to the received sensor data. In the in-vehicle device 100, the control unit 140 simply stores the port number added to the received service request in memory 142 in association with the service request (see step 302 in FIG. 9). When transmitting transmission data generated using parameters determined in accordance with the service request, the control unit 140 can identify the corresponding port number and add the identified port number to the transmission data (see step 316 in FIG. 9).

[0098] In the above description, the video data acquired from the cameras is temporarily stored in memory 142, and then transmission data is generated by frame rate conversion unit 210 and resolution conversion unit 212 using determined parameters (i.e., frame rate and resolution). However, this is not limiting. Cameras 200, 202, and 204 may have a function for outputting video data by changing the frame rate and resolution. Control unit 140 may instruct cameras 200, 202, and 204 via I / F unit 144 to output video data based on determined parameters (i.e., frame rate and resolution). In this case, in-vehicle / out-vehicle linking unit 122 may not include frame rate conversion unit 210 and resolution conversion unit 212. Furthermore, the frame rate used by frame rate conversion unit 210 and the resolution used by resolution conversion unit 212 may be determined so that the frame rate and resolution of the transmission data are the determined parameters, taking into account the adjustable ranges of cameras 200, 202, and 204.

[0099] In the above description, a case has been described in which a service request (e.g., allowable delay time, priority, etc.) is transmitted from server 106 and used to determine parameters used in generating transmission data to be uploaded by the in-vehicle device, but this is not limiting. For example, if the in-vehicle device knows the content of the service provided by server 106, the in-vehicle device may independently determine (i.e., estimate) the service request (e.g., allowable delay time, priority, etc.) and determine the parameters as described above.

[0100] [First Modification] In the flowchart shown in Fig. 9, when a service request is received from server 106, steps 302 to 312 are executed to update the parameters. The wireless communication situation changes depending on the location and driving conditions of vehicle 102, etc. In order to determine appropriate parameters in response to this, for example, it is conceivable that server 106 would periodically send a service request, but this would be cumbersome for server 106. Therefore, in the first modified example, as shown in Fig. 12, after the parameters are determined, the parameters are updated as appropriate even if a service request is not received.

[0101] FIG. 12 shows a flowchart in which step 340 has been added to the flowchart in FIG. 9. In the flowchart in FIG. 12, the processing contents of steps denoted by the same reference numerals as in the flowchart in FIG. 9 are the same as those in FIG. 9. Furthermore, the in-vehicle device that executes the flowchart shown in FIG. 12 has the same configuration as in FIGS. 2, 3, and 6. The processing shown in FIG. 12 is realized by control unit 140 shown in FIG. 3 reading and executing a predetermined program from memory 142. Therefore, redundant explanations will not be repeated, and the following mainly describes the differences.

[0102] In step 300, the control unit 140 determines whether a service request has been received from the server 106. If it is determined that a service request has been received, control proceeds to step 302. As a result, the parameters (i.e., frame rate, resolution, and compression ratio) are determined as described above. On the other hand, if it is determined that a service request has not been received, control proceeds to step 340.

[0103] In step 340, the control unit 140 determines whether or not to update the parameters. If it is determined that the parameters should be updated, control proceeds to step 304, where the parameters (i.e., frame rate, resolution, and compression ratio) are determined as described above. Otherwise, control proceeds to step 314.

[0104] Parameter updates may be performed at regular intervals or according to a predetermined schedule. Because the communication speed may change due to fluctuations in the wireless communication channel, when updating parameters at regular intervals, it is preferable to set the update interval taking into account the fluctuation interval of the wireless communication channel. The fluctuation interval of the wireless communication channel is the time (e.g., the average value) over which the channel is maintained. In other words, the parameter update interval is set to a value shorter than the fluctuation interval of the wireless communication channel. If the parameter update interval is longer than the fluctuation interval of the wireless communication channel, video data with a data volume exceeding the available bandwidth may be transmitted during upload, possibly exceeding the allowable delay time. Furthermore, packet errors and packet loss may occur. This allows the parameters used when uploading data to be appropriately determined.

[0105] Typically, the wireless section, which is the external environment of the vehicle, often becomes a bottleneck in the available bandwidth. However, some abnormality may occur inside the vehicle, causing the delay time within the vehicle (i.e., time T1 to time T4 shown in FIG. 7) to fluctuate. Therefore, it is also preferable to set the parameter update period to a time that is sufficiently short compared to the fluctuation period of the vehicle state. The fluctuation period of the vehicle state is, for example, the fluctuation period of the in-vehicle communication speed due to fluctuations in the load rate and processing capacity of the in-vehicle network and ECU. The fluctuation period of the in-vehicle communication speed is the time during which the in-vehicle communication speed is maintained within a certain range (e.g., average value). "Sufficiently short" means, for example, 1 / 10 or less. This allows the parameters used when uploading data to be appropriately determined.

[0106] [Second Modification] 3 and 6, a case has been described in which the in-vehicle device 100 uploads video data acquired from the sensor 124 (i.e., camera 200, camera 202, and camera 204) to the server 106, but the autonomous driving ECU 126 and ECU 128 also use the video data acquired from the sensor 124. In the second modified example, the ECU (i.e., the function control device) and the in-vehicle / out-of-vehicle linking unit 122 use the video data acquired from the sensor 124 simultaneously.

[0107] 13, the in-vehicle device according to the second modification includes a communication unit 120 and an in-vehicle / outside linking unit 122, similar to the in-vehicle device 100 shown in FIG. 2. The communication unit 120 and the in-vehicle / outside linking unit 122 are configured similarly to those in FIG. 3 and have similar functions to those in FIG. 6. The bus 130 (see FIG. 3) is not shown in FIG. 13. For convenience, one camera 200 is shown as a sensor in FIG. 13, but the vehicle is equipped with multiple cameras, similar to FIG. 6. FIG. 13 also shows an autonomous driving ECU 126 as an ECU that uses video data output from the camera 200. The ECU that uses the video data output from the camera 200 may be an ECU other than the autonomous driving ECU 126.

[0108] The autonomous driving ECU 126 controls the camera 200 to acquire video data as indicated by the dotted arrow, temporarily stores the data in a memory within the autonomous driving ECU 126, and uses the analysis results for autonomous driving. The frame rate and resolution of the video data output from the camera 200 to the autonomous driving ECU 126 are suitable for autonomous driving. The in-vehicle / out-of-vehicle linking unit 122 copies the data output from the camera 200 and stores it in the memory 142 (see FIG. 3 ). As a result, the video data output from the camera 200 is stored in the memory 142. Therefore, as described above, the in-vehicle / out-of-vehicle linking unit 122 can use the determined parameters to convert (i.e., convert the frame rate and resolution) the video data stored in the memory 142 using the frame rate conversion unit 210 and the resolution conversion unit 212, and compress the data using the compression unit 214 to generate transmission data. This prevents a situation in which the autonomous driving ECU 126 cannot use the sensor data when transmitting the sensor data to the server 106.

[0109] Furthermore, the autonomous driving ECU 126 may temporarily store video data used in autonomous driving in memory 142 of the vehicle interior / exterior linkage unit 122, and then read the data from memory 142 (see the dashed arrow in FIG. 13), store the data in a memory within the autonomous driving ECU 126, and analyze the data. For example, the autonomous driving ECU 126 may instruct the control unit 140 (see FIG. 3) of the vehicle interior / exterior linkage unit 122 to capture output data from the camera 200, and upon receiving the instruction, the control unit 140 may store the output data from the camera 200 in memory 142 via the I / F unit 144 (see FIG. 3). This allows the autonomous driving ECU 126 to use the analysis results for autonomous driving, and the vehicle interior / exterior linkage unit 122 to generate transmission data using the determined parameters.

[0110] In the above, a case where a pilot packet is transmitted to estimate the UL communication speed has been described, but this is not limiting. For example, before transmitting video data, data other than video data (e.g., analysis results of sensor data or vehicle data) may be scheduled to be transmitted. The data other than video data transmitted at this time can replace the pilot packet. As the data other than video data, it is preferable to use data that has a data size similar to that of a pilot packet and has a high degree of freedom in transmission timing. This makes it possible to suppress the transmission of pilot packets for estimating the available bandwidth.

[0111] Alternatively, the UL communication speed may be estimated by observing the amount of data transmitted and the transmission time of the previous transmission of data to be transmitted (e.g., video data). Alternatively, the UL communication speed may be calculated using an estimation model (e.g., based on deep learning or mathematical techniques) that estimates future communication quality from current and past communication quality data. This allows the transmission of pilot packets for estimating the available bandwidth to be suppressed.

[0112] Each process (each function) in the above-described embodiments may be realized by a processing circuit including one or more processors. The processing circuit may be configured by an integrated circuit or the like that combines one or more memories, various analog circuits, and various digital circuits in addition to the one or more processors. The one or more memories store programs (instructions) that cause the one or more processors to execute the respective processes. The one or more processors may execute the respective processes according to the programs read from the one or more memories, or according to logic circuits pre-designed to execute the respective processes. The processor may be a CPU, a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an ASIC, or any of various processors suitable for computer control.

[0113] In addition, a recording medium can be provided that stores a program that causes a computer to execute the processing of the in-vehicle device 100 (specifically, the processing executed by the in-vehicle / out-vehicle linking unit 122 (for example, the processing shown in FIGS. 9 and 12 ). The recording medium is, for example, an optical disc (such as a DVD (Digital Versatile Disc)) or a removable semiconductor memory (such as a USB (Universal Serial Bus) memory). Although a computer program can be transmitted over a communication line, the recording medium refers to a non-transitory recording medium. By having the computer mounted on the vehicle load the program stored in the recording medium, the computer can transmit data that can be effectively used by a service provided by the external device, taking into account delay time and communication bandwidth when uploading data to an external device such as a roadside device, as described above.

[0114] (Addendum) That is, the computer-readable non-transitory recording medium is The vehicle's on-board computer a transmission function for transmitting transmission data to a roadside device that is a device located outside the vehicle; an estimation function for estimating an available bandwidth and an end delay time that can be used by the computer in wireless communication with the roadside device; a determining function that determines parameters used when generating the transmission data to be transmitted by the transmitting function from transmission target data so as to satisfy the available bandwidth and the end delay time; a generating function for generating the transmission data from the transmission target data using the parameters; The end delay time is the time from when the data to be transmitted is generated until when the transmission data is passed to a computer program that realizes the service provided by the roadside device, and a computer program is stored.

[0115] Although the present disclosure has been described above by explaining the embodiments, the above-described embodiments are merely examples, and the present disclosure is not limited to only the above-described embodiments. The scope of the present disclosure is defined by the claims in the scope of the claims, taking into consideration the description of the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wordings described therein. [Explanation of symbols]

[0116] 100, 110 In-vehicle equipment 102, 112 cars 104 Base station 106 servers 108 Network 120, 164 Communications Department 122 In-vehicle and out-vehicle coordination department 124 sensors 126 Autonomous Driving ECU 128 ECU Buses 130 and 166 140, 160 control unit 142, 162 memory 144 I / F section 146 Transmission data generation unit 200, 202, 204 cameras 210 Frame rate conversion unit 212 Resolution conversion unit 214 Compression section 220 Estimation section 222 Decision Section 224 Communication Control Unit 226 Parameter Determination Unit 228 Priority determination section 240 Transmitter 242 Receiving unit 250 Image Acquisition and Processing 300, 302, 304, 306, 308, 310, 312, 314, 316, 318, 340, 400, 402, 404, 406, 408, 410, 412, 414, 416 steps

Claims

1. An in-vehicle device mounted on a vehicle, a transmitter that transmits transmission data to a roadside device that is an external device of the vehicle; an estimation unit that estimates an available bandwidth and an end delay time that the in-vehicle device can use in wireless communication with the roadside device; a determination unit that determines parameters used when generating the transmission data to be transmitted from the transmission unit from transmission target data so as to satisfy the available bandwidth and the end delay time; a generating unit that generates the transmission data from the transmission target data using the parameters, The end delay time is a time from when the transmission target data is generated until when the transmission data is passed to a computer program that realizes a service provided by the roadside device.

2. the transmission target data includes video data acquired from a sensor mounted on the vehicle; The in-vehicle device according to claim 1 , wherein the parameters include at least one of a compression ratio, a resolution, a frame rate, and a bit rate for the transmission data.

3. the transmission target data includes video image data acquired from each of a plurality of sensors mounted on the vehicle; the determination unit includes a priority determination unit that determines a priority of each of the plurality of sensors in accordance with the service provided by the roadside device; The in-vehicle device according to claim 1 , wherein the generating unit generates the transmission data from the video image data acquired from each of the plurality of sensors in accordance with the priority.

4. the priority is a ratio of a communication bandwidth used for transmitting the video data acquired from each of the plurality of sensors to the roadside device; The in-vehicle device according to claim 3 , wherein the determination unit determines the parameters for each of the video data so as to satisfy the communication band and the end delay time determined from the available band and the ratio.

5. 5. The in-vehicle device according to claim 1, wherein the available bandwidth is a lower communication speed between a communication speed estimated from a communication line quality of the wireless communication and an upper limit communication speed allowed for the in-vehicle device in the wireless communication service provided.

6. the generator repeatedly determines the parameters at a predetermined cycle; 5. The in-vehicle device according to claim 1, wherein the predetermined period is shorter than at least one of a fluctuation period of a wireless communication channel in the wireless communication and a fluctuation period of a communication speed inside the vehicle.

7. The in-vehicle device according to claim 1 , wherein the transmission target data is also used by a function control device mounted on the vehicle.

8. the transmitting unit transmits predetermined data before transmitting the transmission data; the estimation unit estimates the available bandwidth based on a communication speed at which the predetermined data is transmitted; The in-vehicle device according to claim 1 , wherein the predetermined data includes an analysis result of sensor data acquired from a sensor mounted on the vehicle or data related to the vehicle.

9. The in-vehicle device according to claim 1 , wherein the estimation unit estimates the available bandwidth to be used next by the determination unit based on a communication speed at which the transmission data is transmitted by the transmission unit.

10. In a state in which a plurality of services having mutually different requirements that can be used for estimating the end delay time are provided in parallel by the roadside device, the determination unit determines the parameter corresponding to each of the requests; The in-vehicle device according to claim 1 , wherein the generation unit generates the transmission data and the transmission unit transmits the transmission data to the roadside device for a predetermined period for each request.

11. 5. The in-vehicle device according to claim 1, wherein, in a state in which the roadside device provides a plurality of services in parallel, each of which has a different request that can be used to estimate the end delay time, the estimation unit determines the parameter using a result of an AND operation of the plurality of requests.

12. the transmitting unit transmits the transmission data to the roadside device in response to the occurrence of the event; The in-vehicle device according to claim 1 , wherein the event is related to at least one of a driving state of the vehicle and a traffic state around the vehicle.

13. the transmitting unit transmits the transmission data to the roadside device in response to the occurrence of the event; The event is related to at least one of a driving state of the vehicle and a traffic state around the vehicle; The in-vehicle device according to claim 4 , wherein the ratio is changed in response to the occurrence of the event.

14. a communication unit that communicates with the in-vehicle device according to any one of claims 1 to 4; an infrastructure linking unit that observes a state of a service to be provided and specifies an allowable delay time that can be used to estimate the end delay time; The communication unit transmits the allowable delay time to the vehicle-mounted device.

15. A method for controlling an in-vehicle device mounted on a vehicle, comprising: a transmitting step of transmitting the transmission data to a roadside device that is an device located outside the vehicle; an estimation step of estimating an available bandwidth and an end delay time that can be used by the in-vehicle device in wireless communication with the roadside device; a determining step of determining parameters used when generating the transmission data to be transmitted from the transmission target data so as to satisfy the available bandwidth and the end delay time; generating the transmission data from the transmission target data using the parameters; The control method, wherein the end delay time is a time from when the transmission target data is generated until when the transmission data is passed to a computer program that realizes a service provided by the roadside device.

16. The computer installed in the vehicle a transmission function for transmitting transmission data to a roadside device that is a device located outside the vehicle; an estimation function for estimating an available bandwidth and an end delay time that can be used by the computer in wireless communication with the roadside device; a determining function that determines parameters used when generating the transmission data to be transmitted by the transmitting function from transmission target data so as to satisfy the available bandwidth and the end delay time; a generating function for generating the transmission data from the transmission target data using the parameters; The computer program, wherein the end delay time is a time from when the transmission target data is generated until when the transmission data is passed to a computer program that realizes a service provided by the roadside device.

Images