Information processing apparatus, control method, program, and storage medium

Abstract

An on-vehicle machine 1 or a server device 5 is an information processing apparatus used in a system that intermediates communication between a driver of a target vehicle which is a movable body and a user of an out-of-vehicle terminal 2 which is present outside the target vehicle, and comprises a calculation means, a detection means, and a display control means. The calculation means calculates a workload value which is an index value relating to the degree of driving load (workload) of the driver of the target vehicle. The detection means detects a cause of increasing the driving load when the workload value indicates that the driving load is higher than a prescribed reference. The display control means displays, on the out-of-vehicle terminal 2, information relating to the cause of increasing the driving load.

Description

Information processing device, control method, program, and storage medium

[0001] The present invention relates to a system that mediates communication between a driver of a mobile body and a user outside the mobile body.

[0002] Conventionally, there have been known technologies for enabling a conversation between a vehicle driver and a person outside the vehicle. For example, Patent Literature 1 discloses a conversation providing system that enables a conversation between the vehicle driver and a conversation partner by communicating with an in-vehicle system via a communication network.

[0003] JP 2017-138277 A

[0004] When the driver is talking to someone outside the vehicle, it is difficult for the person outside the vehicle to understand the driver's psychological state, so even when the driver is under heavy driving stress, they are unable to interrupt the conversation, which increases the driver's driving stress.

[0005] In view of the above-mentioned problems, one of the objects of the present invention is to provide an information processing device that can suitably mediate communication between a driver of a mobile body and a user outside the mobile body.

[0006] The invention described in claim 1 is an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body, characterized in that the information processing device has: a calculation means for calculating an index value related to the degree of driving load on the driver of the mobile body; a detection means for detecting a cause that is increasing the driving load when the index value indicates that the driving load is higher than a predetermined standard; and a display control means for displaying information related to the cause on the user's terminal device.

[0007] The invention described in claim 10 is a control method executed by an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body, characterized in that the control method has the following steps: a calculation step of calculating an index value related to the degree of driving load on the driver of the mobile body; a detection step of detecting a cause of increasing the driving load when the index value indicates that the driving load is higher than a predetermined standard; and a display control step of displaying information related to the cause on the user's terminal device.

[0008] The invention described in claim 11 is a program executed by a computer of an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body, the program causing the computer to function as: a calculation means for calculating an index value related to the degree of driving load on the driver of the mobile body; a detection means for detecting the cause of increasing the driving load when the index value indicates that the driving load is higher than a predetermined standard; and a display control means for displaying information related to the cause on the user's terminal device.

[0009] 1 shows an example of the configuration of a driving call system according to an embodiment. FIG. 2 shows an example of the schematic configuration of an in-vehicle device. FIG. 3 shows an example of the schematic configuration of an exterior terminal. FIG. 4 shows a first example of display on the exterior terminal when the target vehicle is traveling. FIG. 5 shows a second example of display on the exterior terminal when the target vehicle is traveling. FIG. 6 shows a third example of display on the exterior terminal when the target vehicle is traveling. FIG. 7 shows an example of a flowchart illustrating the procedure of processing executed by an in-vehicle device. FIG. 8 shows a fourth example of display on the exterior terminal according to a modified example. FIG. 9 shows an example of the configuration of a driving call system according to a modified example. FIG. 10 shows an example of the schematic configuration of a server device.

[0010] In one preferred embodiment of the present invention, an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body includes a calculation means for calculating an index value related to the degree of driving load on the driver of the mobile body, a detection means for detecting a cause that is increasing the driving load when the index value indicates that the driving load is higher than a predetermined standard, and a display control means for displaying information related to the cause on the user's terminal device.

[0011] The information processing device can conveniently allow a user outside the vehicle to recognize the cause of the driver's increased driving load, thereby enabling the user to communicate with the driver according to the cause of the increased driving load.

[0012] In one aspect of the information processing device, the calculation means calculates the index value based on a current state of at least one of the driving behavior of the mobile object, the driver, and the driving environment. In a preferred example, the detection means calculates a score by evaluating the current state of at least one of the driving behavior of the mobile object, the driver, and the driving environment for each element related to the driving load, and calculates the index value based on the score. This allows the information processing device to accurately calculate the index value of the driving load.

[0013] In another aspect of the information processing device, the detection means determines the factor that is causing the increase in driving load based on the score. This aspect enables the information processing device to accurately detect the factor that is causing the increase in driving load.

[0014] In another aspect of the information processing device, the display control means causes the terminal device to display information about the cause and information representing the magnitude of the index value. With this aspect, the information processing device can preferably present the degree of driving load together with the cause of the increase in driving load to a user outside the mobile body.

[0015] In another aspect of the information processing device, the display control means causes the terminal device to display a line representing the trajectory of the driver's line of sight superimposed on a captured image of an area outside the mobile body captured from the mobile body. This aspect allows the user of the terminal device to conveniently grasp the direction in which the driver is paying less attention, and the user of the terminal device can communicate with the driver to assist in confirming safety in the direction in which the driver is paying less attention. In a preferred example, the display control means may change the display mode of the line depending on the index value. In another preferred example, the display control means may display the line when the index value indicates that the driving load is higher than a predetermined standard, and may hide the line when the index value indicates that the driving load is equal to or lower than the predetermined standard.

[0016] In another aspect of the information processing device, the display control means highlights an object that the driver should look at in the captured image. This aspect enables a user of the terminal device to recognize an object that the driver is not looking at and to communicate with the driver to encourage him or her to look at the object.

[0017] In another preferred embodiment of the present invention, a control method is executed by an information processing device used in a system that mediates communication between a driver of a mobile body and a user located outside the mobile body, the control method comprising: a calculation step of calculating an index value relating to a degree of driving load on the driver of the mobile body; a detection step of detecting a cause of increasing the driving load when the index value indicates that the driving load is higher than a predetermined standard; and a display control step of displaying information relating to the cause on a terminal device of the user. By executing this control method, the information processing device can make a user located outside the mobile body preferably aware of the cause of increasing the driver's driving load.

[0018] In yet another embodiment of the present invention, a program executed by a computer of an information processing device used in a system that mediates communication between a driver of a mobile body and a user located outside the mobile body causes the computer to function as a calculation means for calculating an index value related to the degree of driving load of the driver of the mobile body, a detection means for detecting a cause of the increased driving load when the index value indicates that the driving load is higher than a predetermined standard, and a display control means for displaying information related to the cause on a terminal device of the user. By executing this program, the computer of the information processing device can conveniently allow a user located outside the mobile body to recognize the cause of the increased driving load of the driver. Preferably, the program is stored in a storage medium.

[0019] Preferred embodiments of the present invention will now be described with reference to the drawings.

[0020] (1) System Configuration Fig. 1 shows an example of the configuration of a driving call system according to a first embodiment. The driving call system includes an in-vehicle device 1 installed in a vehicle and an external terminal 2 used by a person outside the vehicle. In the driving call system, the in-vehicle device 1 and the external terminal 2 communicate with each other to mediate a voice call between the driver of the vehicle and the user of the external terminal 2, and the user of the external terminal 2 can communicate with the driver of the vehicle while checking various information obtained from the vehicle on the external terminal 2.

[0021] The on-board device 1 travels with the vehicle and performs processing to enable a call between the driver of the vehicle and the user of the external terminal 2. Hereinafter, the vehicle equipped with the on-board device 1 will also be referred to as the "target vehicle." The on-board device 1 performs data communication with the external terminal 2 via a communication network 3 such as the Internet or a dedicated communication network. Data exchanged between the on-board device 1 and the external terminal 2 includes voice data generated during the call between the driver of the target vehicle and the user of the external terminal 2, and display instruction data for displaying information about the target vehicle or the driver. In this embodiment, the on-board device 1 transmits display instruction data including information about the driver's driving load (workload) (also referred to as "workload-related information") to the external terminal 2.

[0022] The vehicle-mounted device 1 may be a navigation device installed in a target vehicle and providing route guidance to a set destination, or may be a mobile terminal such as a smartphone. The vehicle-mounted device 1 may also be incorporated into the target vehicle. The vehicle-mounted device 1 is an example of an "information processing device." The target vehicle is an example of a "mobile body."

[0023] The external terminal 2 is a terminal operated by a person outside the target vehicle, and performs data communication with the onboard device 1 via the communication network 3. The external terminal 2 is, for example, a mobile terminal such as a smartphone. For example, the external terminal 2 establishes communication with the onboard device 1 while the target vehicle is being driven, and exchanges voice data with the onboard device 1 for a call between the driver and the user of the external terminal 2. The external terminal 2 also receives display instruction data including workload-related information from the onboard device 1 during the call, and displays the driver's workload status, etc., based on the display instruction data. In this way, the external terminal 2 allows the user of the external terminal 2 to recognize the driver's workload status and preferably provides information that can be used to determine a convenient time to talk to the driver. The external terminal 2 is an example of a "terminal device."

[0024] Note that communication between the in-vehicle device 1 and the external terminal 2 may be realized by being relayed by a server device (not shown). Even when the in-vehicle device 1 and the external terminal 2 establish direct communication, the in-vehicle device 1 and / or the external terminal 2 may exchange information (e.g., communication address information) necessary for establishing communication between the in-vehicle device 1 and the external terminal 2 with the server device. In these cases, the server device performs processing necessary for data communication between the in-vehicle device 1 and the external terminal 2 (including authentication processing for the in-vehicle device 1 and the external terminal 2).

[0025] (2) Device Configuration Fig. 2 shows an example of a schematic configuration of the on-board device 1. The on-board device 1 mainly includes a communication unit 11, a memory unit 12, an input unit 13, a control unit 14, a sensor group 15, a display unit 16, and a sound output unit 17. The elements within the on-board device 1 are interconnected via a bus line 10.

[0026] The communication unit 11 performs data communication with other terminals under the control of the control unit 14. For example, the communication unit 11 may receive map data for updating a map DB (Database) 4 from a map management server (not shown).

[0027] The storage unit 12 is configured with various types of memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory (including a hard disk drive, flash memory, etc.). The storage unit 12 stores programs for the in-vehicle device 1 to execute predetermined processes. The above-mentioned programs may include an application program for making calls, an application program for causing the external terminal 2 to display a map or an image captured in the target vehicle, etc. The storage unit 12 is also used as a working memory for the control unit 14. Note that the programs executed by the in-vehicle device 1 may be stored in a storage medium other than the storage unit 12.

[0028] The storage unit 12 also stores a map DB (Data Base) 4. The map DB 4 stores various data necessary for route guidance. The map DB 4 contains data necessary for displaying a map based on a predetermined position, such as the current position of the target vehicle. The map DB 4 is a database that includes, for example, road data that represents a road network using a combination of nodes and links, and facility data that indicates facilities that are candidates for destinations, stop-off points, or landmarks. The map DB 4 may be updated based on information received by the communication unit 11 from a map management server under the control of the control unit 14.

[0029] The input unit 13 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 16 is a display, etc. that displays information under the control of the control unit 14. The sound output unit 17 is a speaker, etc. that outputs sound under the control of the control unit 14.

[0030] The sensor group 15 includes various sensors that sense the state of the target vehicle or the environment outside the vehicle. The sensor group 15 includes an exterior camera 51, a driver camera 52, a vehicle behavior detector 53, and a biometric sensor 54.

[0031] The exterior camera 51 is one or more cameras that capture images of the area outside the target vehicle, such as the area in front of the target vehicle, and generates images captured at predetermined time intervals (also referred to as "exterior image"). The driver camera 52 is a camera installed so as to include the driver's face in its capture range, and generates images captured at predetermined time intervals (also referred to as "driver image").

[0032] The vehicle behavior detector 53 generates detection signals indicating the behavior of the target vehicle, such as the current position, vehicle speed, acceleration, steering angle, etc. The vehicle behavior detector 53 includes, for example, a Global Navigation Satellite System (GNSS) receiver, a gyro sensor, an Inertial Measurement Unit (IMU), a vehicle speed sensor, an acceleration sensor, and a steering angle sensor.

[0033] The biological sensor 54 is one or more sensors that generate biological signals that indicate biological phenomena such as the driver's heart rate and the amount of sweat.

[0034] In addition, the sensor group 15 may include various external sensors (including cameras, lidars, radars, ultrasonic sensors, infrared sensors, sonars, etc.) and internal sensors in addition to the exterior camera 51, the driver camera 52, the vehicle behavior detector 53, and the biometric sensor 54.

[0035] The control unit 14 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire in-vehicle device 1. The control unit 14 functions as a "calculation means," a "detection means," a "display control means," a computer that executes programs, etc.

[0036] The processing performed by the control unit 14 is not limited to being realized by software programs, but may be realized by any combination of hardware, firmware, and software. The processing performed by the control unit 14 may also be realized by using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the control unit 14 in this embodiment may be realized by using this integrated circuit.

[0037] The configuration of the vehicle-mounted device 1 shown in FIG. 2 is an example, and various modifications may be made to the configuration shown in FIG. 2 . For example, instead of storing the map DB 4 in the memory unit 12, the control unit 14 may receive map information from a map management server (not shown) via the communication unit 11. In another example, at least one of the input unit 13, the display unit 16, and the sound output unit 17 may be provided in the target vehicle as an external device of the vehicle-mounted device 1, and may supply the generated signals to the vehicle-mounted device 1. Furthermore, at least some of the sensors in the sensor group 15 may be sensors mounted on the target vehicle. In this case, the vehicle-mounted device 1 may acquire information output by sensors mounted on the target vehicle from the target vehicle based on a communication protocol such as CAN (Controller Area Network).

[0038] 3 shows an example of a schematic configuration of the exterior terminal 2. The exterior terminal 2 mainly includes a communication unit 21, a storage unit 22, an input unit 23, a control unit 24, a sensor group 25, a display unit 26, and a sound output unit 27. The elements within the exterior terminal 2 are connected to each other via a bus line 20.

[0039] The communication unit 21 performs data communication with other terminals under the control of the control unit 24. The storage unit 22 is composed of various types of memory such as RAM, ROM, and non-volatile memory. The storage unit 22 stores programs for the external terminal 2 to execute predetermined processes. The above-mentioned programs may include application programs for communicating with the driver, displaying information related to the driving of the target vehicle of the in-vehicle device 1 (including displaying maps and various captured images), and displaying information related to the driver's workload when communication with the in-vehicle device 1 is established. The storage unit 22 is also used as a working memory for the control unit 24. The programs executed by the external terminal 2 may be stored in a storage medium other than the storage unit 22.

[0040] The input unit 23 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 26 is a display, etc. that displays information based on the control of the control unit 24. The sound output unit 27 is a speaker, etc. that outputs sound based on the control of the control unit 24. The sensor group 25 includes an internal sensor that senses the state of the exterior terminal 2 and an external sensor that senses the state of the outside world of the exterior terminal 2.

[0041] The control unit 24 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire exterior terminal 2. Note that the configuration of the exterior terminal 2 shown in Fig. 3 is an example, and various changes may be made to the configuration shown in Fig. 3.

[0042] (3) Display of Workload-Related Information Next, the display of workload-related information regarding the driving load (workload) of the driver of the target vehicle will be described. In summary, when the driver's workload is equal to or greater than a predetermined level, the in-vehicle device 1 identifies the cause of the increased workload (also referred to as a "workload increase cause") and displays the identified workload increase cause on the external terminal 2. In this way, the in-vehicle device 1 makes the user of the external terminal 2 aware of the workload increase cause, and the user of the external terminal 2 can take appropriate action according to the workload increase cause.

[0043] (3-1) Calculation of Workload Value First, a method for calculating the workload value, which is a value (score) indicating the degree of workload, will be described. Hereinafter, for the sake of convenience, the workload value will be assumed to indicate that the higher the workload value, the higher the degree of the driver's workload.

[0044] The vehicle-mounted device 1 calculates a workload value based on scores (also referred to as "element-specific scores") evaluating the current state for each element related to workload (also referred to as "workload-related elements"). The workload-related elements are elements that represent the state of the target vehicle, the driver, or the driving environment. Examples of workload-related elements include elements related to the environment of the road (driving road) on which the target vehicle is traveling (also referred to as "road environment elements"), elements related to driving behavior (driving operations) that affect the workload (also referred to as "driving behavior elements"), and elements related to the driver's physical condition (also referred to as "driver physical condition elements"). In other words, the workload-related elements are elements that are candidates for causing an increase in workload. Note that the workload-related elements may be elements obtained by further subdividing at least one of the road environment elements, driving behavior elements, and driver physical condition elements.

[0045] Therefore, for example, while establishing communication for a call with the external terminal 2, the in-vehicle device 1 calculates a score (also referred to as an "element-specific score") that evaluates the current state of the target vehicle, driver, or driving environment for each workload-related element. The in-vehicle device 1 then determines the total, average, or other representative value of the calculated element-specific scores as the workload value. The workload-related elements for which the element-specific scores are calculated may be all of the road environment elements, driving behavior elements, and driver physical condition elements, or any one of these, or any two of these. Preferably, the element-specific scores may be weighted according to the degree of impact of each workload-related element on the workload. In this case, for example, the weighting coefficient multiplied by the element-specific score of an element with a greater impact on the workload is a larger value, and information indicating these weighting coefficients is pre-stored in the storage unit 12, etc.

[0046] Here, when calculating the element-specific scores of road environment elements, the vehicle-mounted device 1 calculates the element-specific scores by taking into account at least one of the congestion level, width, speed limit, and visibility of the road, for example. In this case, for example, the vehicle-mounted device 1 refers to road information in the map DB 4 or road traffic information acquired by the communication unit 11, identifies the respective degrees of congestion, width, speed limit, visibility, etc. of the road, and converts each identified degree into an element-specific score by referring to a predetermined formula or table. The predetermined formula or table is stored in advance in the storage unit 12, etc. Note that the road environment elements may be subdivided into multiple workload-related elements. In this case, for example, the vehicle-mounted device 1 may consider each of the congestion level, width, speed limit, and visibility of the road as a workload-related element and calculate an element-specific score for each of these.

[0047] Furthermore, when calculating the element-specific scores for driving behavior elements, the in-vehicle device 1 may, for example, determine whether the current driving behavior corresponds to a driving behavior such as a right turn, a left turn, a merge, or a temporary stop, and calculate the element-specific score according to the determined driving behavior. In this case, for example, the in-vehicle device 1 may determine the element-specific scores for the driving behavior elements by referring to a table or the like that associates the corresponding driving behavior with the element-specific scores for the driving behavior elements. Here, the in-vehicle device 1 may determine the current driving behavior based on information obtained by route guidance processing (e.g., the current position of the target vehicle and the route to the set destination), or may determine the score based on a signal indicating the status of the turn signal or the steering state obtained from the target vehicle. The in-vehicle device 1 may also determine the element-specific scores for the driving behavior elements based on whether auto-cruise driving is enabled (or whether other automatic driving functions are enabled). Note that the driving behavior elements may be subdivided into multiple workload-related elements. For example, whether or not to turn right, whether or not to turn left, whether or not to merge, whether or not to stop, and whether or not to drive on auto-cruise are each set as workload-related elements, and the vehicle-mounted device 1 determines the element-specific score for each of these workload-related elements by referring to a specified table, etc., depending on whether or not the corresponding driving action is performed.

[0048] Furthermore, when calculating the element-specific scores for the driver's physical condition elements, the in-vehicle device 1 calculates the element-specific scores by taking into account at least one of the driver's continuous driving time, drowsiness level (alertness level), and other biometric indicators that affect driving. In this case, the in-vehicle device 1, for example, identifies the drowsiness level or other biometric indicators that affect driving based on the signal output by the biometric sensor 54. Then, the in-vehicle device 1 refers to a predetermined formula or table and converts the identified continuous driving time and one or more biometric indicators into element-specific scores for the driver's physical condition elements. Note that the driver's physical condition elements may be subdivided into multiple workload-related elements. In this case, for example, the in-vehicle device 1 may consider each of drowsiness level and continuous driving time as workload-related elements and calculate element-specific scores for these elements.

[0049] In this way, the vehicle-mounted device 1 can determine the workload value taking into consideration various factors that affect the workload.

[0050] (3-2) Detection of Cause of Increase in Workload Next, a method for detecting the cause of an increase in workload will be described.

[0051] The in-vehicle device 1 executes a process for detecting the cause of an increase in workload when the calculated workload value is greater than a predetermined threshold (also referred to as a "workload threshold"). The workload threshold is, for example, determined in advance and stored in the storage unit 12. The workload threshold is an example of a "predetermined standard."

[0052] When the workload value is greater than the threshold, the vehicle-mounted device 1 determines the proportion of the workload value that the element-specific score of each workload-related element accounts for (also referred to as the "workload value occupancy proportion"). Next, the vehicle-mounted device 1 adds up the workload value occupancy proportions in order from the highest to the lowest until the total workload value occupancy proportions reaches a predetermined proportion. When the total workload value occupancy proportions reaches a predetermined proportion, the vehicle-mounted device 1 detects the workload-related element whose workload value occupancy proportions have been added up as the cause of the workload increase. The above-mentioned predetermined proportion is, for example, predetermined and stored in the storage unit 12.

[0053] For example, let us consider a case where the above-mentioned predetermined percentage is 30%. In this case, the on-board device 1 determines whether the workload value occupancy percentage of the workload-related element with the highest workload value occupancy percentage is 30% or more. If the workload value occupancy percentage of the workload-related element with the highest workload value occupancy percentage is 30% or more, the on-board device 1 detects the workload-related element as a cause of workload increase. On the other hand, if the workload value occupancy percentage of the workload-related element with the highest workload value occupancy percentage is not 30% or more, the on-board device 1 determines whether the sum of the workload value occupancy percentages of the workload-related elements with the highest and second highest workload value occupancy percentages is 30% or more. If the sum of the workload value occupancy percentages of the workload-related elements with the highest and second highest workload value occupancy percentages is 30% or more, the on-board device 1 detects these workload-related elements as a cause of workload increase. On the other hand, if the workload value occupancy ratios of the workload-related elements with the first and second highest workload value occupancy ratios are not 30% or more, the on-vehicle device 1 determines whether the total value of the workload value occupancy ratios of the workload-related elements with the first to third highest workload value occupancy ratios is 30% or more. The on-vehicle device 1 repeatedly executes this process until the total value of the workload value occupancy ratios of the workload-related elements is 30% or more, and at the point when the total value is 30% or more, the on-vehicle device 1 detects the workload-related element with the added workload value occupancy ratio as the cause of the workload increase.

[0054] This allows the on-board device 1 to suitably detect one or more workload-related factors that cause an increase in workload. That is, one or more factors among the road environment (congestion level, width, speed limit, visibility), driving behavior, and the driver's physical condition (continuous driving time, drowsiness level based on biological information, etc.) are identified as the cause of the increased workload.

[0055] (3-3) Display Example Fig. 4 shows a first display example of the external terminal 2 while the target vehicle of the in-vehicle device 1 is traveling. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in Fig. 4 on the display unit 26 based on display instruction data received from the in-vehicle device 1.

[0056] The display screen shown in FIG. 4 includes an outside-vehicle image display field 60, a workload-related information display field 61, and a map display field 62.

[0057] The exterior terminal 2 displays the latest exterior image (moving image) generated by the exterior camera 51 in the exterior image display field 60. In this case, the vehicle-mounted device 1 transmits display instruction data including the latest exterior image generated by the exterior camera 51 to the exterior terminal 2.

[0058] Furthermore, the external terminal 2 displays a display window 71 and an indicator 72 on the workload-related information display field 61 based on the workload-related information included in the display instruction data transmitted by the in-vehicle device 1 .

[0059] The display window 71 displays the level of the workload value calculated by the on-board device 1 and the cause of the workload increase detected by the on-board device 1. In this case, since the workload value calculated by the on-board device 1 is at the highest level (for example, the highest level when divided into five levels), "High Workload" indicating the highest level is displayed in the display window 71. Furthermore, since the workload value exceeds the workload threshold, the on-board device 1 detects the cause of the workload increase (here, turning right at an intersection), and the external terminal 2 displays "Turn right at intersection" in the display window 71.

[0060] The indicator 72 also indicates the degree of workload, and the larger the workload value calculated by the vehicle-mounted device 1, the further the gauge extends to the right. Here, since the workload value is near the maximum value, the gauge of the indicator 72 extends to the near right end. The color of the gauge may change depending on the workload value. For example, the higher the workload value, the more noticeable the color of the gauge may be. The "High Workload" display in the display window 71 and the indicator 72 are examples of "information indicating the magnitude of the index value."

[0061] The external terminal 2 also displays a map of the vicinity of the current position of the target vehicle in the map display field 62. The in-vehicle device 1 generates display instruction data for the external terminal 2 to display the above-mentioned map based on the state of the target vehicle estimated using the vehicle behavior detector 53, the map DB 4, and information related to route guidance to the destination, and transmits the display instruction data to the external terminal 2. Note that here, in the map display field 62, a current position mark 73 indicating the current position of the target vehicle and a route line 74 indicating the guidance route along which the in-vehicle device 1 will guide the driver of the target vehicle are superimposed on the map.

[0062] In this way, in the first display example, when the workload is high, the in-vehicle device 1 can appropriately make the user of the external terminal 2 aware of the fact that the workload is high and the cause of this (here, turning right at an intersection). This allows the user of the external terminal 2 to determine that the driver of the in-vehicle device 1 should concentrate on driving, and to take measures such as refraining from talking to the driver about matters other than driving (for example, asking them to do shopping) until the driver has completed turning right at the intersection.

[0063] Fig. 5 shows a second display example of the external terminal 2 while the target vehicle of the on-board device 1 is traveling. The external terminal 2 has established communication with the on-board device 1, and is displaying the display screen shown in Fig. 5 on the display unit 26 based on display instruction data received from the on-board device 1. The external terminal 2 has an external-photographed image display field 60, a workload-related information display field 61, and a map display field 62 on the display screen.

[0064] The external terminal 2 displays the latest externally captured image (moving image) generated by the externally captured camera 51 in the externally captured image display field 60 based on the display instruction data received from the in-vehicle device 1. The external terminal 2 also displays a map of the area around the current position of the target vehicle in the map display field 62 based on the display instruction data received from the in-vehicle device 1.

[0065] Furthermore, the external terminal 2 displays a display window 71A and an indicator 72A indicating the workload level in the workload-related information display area 61 based on the workload-related information included in the display instruction data received from the onboard device 1. Here, because the workload value calculated by the onboard device 1 is equal to or less than the workload threshold, the onboard device 1 does not detect the cause of the workload increase. Therefore, the external terminal 2 displays "Low Workload" in the display window 71, indicating that the workload value level is the lowest, and does not display the cause of the workload increase. Note that, here, the onboard device 1 detects from the vehicle that the current driving mode of the target vehicle is auto-cruise driving, which puts the driver in a low workload state, and therefore includes the fact that the vehicle is currently in auto-cruise driving in the workload-related information display area 61. Based on the workload-related information received from the onboard device 1, the external terminal 2 displays the fact that the vehicle is currently in auto-cruise driving in the workload-related information display area 61. The external terminal 2 also displays an indicator 72A in the workload-related information display area 61, including a gauge whose length corresponds to the level of the workload value.

[0066] In this way, in the second display example, when the workload is low, the in-vehicle device 1 appropriately notifies the user of the external terminal 2 that the workload is low. This allows the user of the external terminal 2 to determine that the driver of the in-vehicle device 1 is relatively relaxed, and to find the right timing to talk to the driver about necessary matters.

[0067] 6 shows a third example of a display on the external terminal 2 when the target vehicle of the onboard device 1 is traveling. The external terminal 2 has established communication with the onboard device 1, and is displaying the display screen shown in FIG. 6 on the display unit 26 based on display instruction data received from the onboard device 1. The external terminal 2 has an external-photographed image display field 60, a workload-related information display field 61, and a map display field 62 on the display screen.

[0068] The external terminal 2 displays the latest externally captured image (moving image) generated by the externally captured camera 51 in the externally captured image display field 60, similar to the first and second display examples, based on the display instruction data received from the in-vehicle device 1. Furthermore, the external terminal 2 displays a map of the vicinity of the current position of the target vehicle in the map display field 62, similar to the first and second display examples, based on the display instruction data received from the in-vehicle device 1.

[0069] Furthermore, based on the workload-related information received from the onboard device 1, the external terminal 2 displays, in the workload-related information display field 61, a display window 71B showing the workload level and the cause of workload increase, and an indicator 72B visually indicating the workload level. Here, since the workload value calculated by the onboard device 1 is greater than the workload threshold, the cause of the workload increase is displayed in the display window 71B. Specifically, the onboard device 1 detects the drowsiness level of the driver's physical condition element and the visibility of the road environment element as causes of the workload increase, respectively. As a result, "drowsiness + low visibility" is displayed in the display window 71B. The display window 71B also displays "moderately high workload," indicating the workload level corresponding to the workload value calculated by the onboard device 1. The external terminal 2 also displays, in the workload-related information display field 61, an indicator 72B including a gauge whose length corresponds to the level of the determined workload value.

[0070] In this way, in the third display example, when the workload is high due to the driver feeling drowsy, the in-vehicle device 1 can appropriately make the user of the external terminal 2 aware of the high workload and the cause of the high workload. This allows the user of the external terminal 2 to take action, such as actively talking to the driver of the in-vehicle device 1 to wake him up.

[0071] The display layouts in the first to third display examples are merely examples, and various modifications may be applied. For example, the external terminal 2 may display an image captured by the driver instead of an image captured outside the vehicle, based on an operation performed by the user of the external terminal 2 on the external terminal 2. In another example, the external terminal 2 may provide only one of the external-vehicle-captured-image display field 60 and the map display field 62 on the display screen. In yet another example, the external terminal 2 may provide only the workload-related information display field 61 on the display screen.

[0072] (4) Processing Flow Fig. 7 is an example of a flowchart showing the procedure of processing executed by the in-vehicle device 1. The in-vehicle device 1 executes the processing of the flowchart shown in Fig. 7 when communication with the external terminal 2 is established and a call is started.

[0073] First, the vehicle-mounted device 1 calculates a score for each workload-related element related to the workload of the driver of the target vehicle (i.e., element-specific score) (step S101). In this case, the vehicle-mounted device 1 calculates an element-specific score for each of the road environment element, the driving behavior element, the driver's physical condition element, or elements obtained by subdividing these elements, based on the map DB 4 and data output by the sensor group 15.

[0074] Next, the vehicle-mounted device 1 calculates a workload value based on the scores of each workload-related element (element-specific scores) (step S102). In this case, the vehicle-mounted device 1 may calculate the workload value by adding up or averaging (including using a weighting factor) the element-specific scores, or by substituting each element-specific score into a formula for calculating the workload value.

[0075] Next, the on-board device 1 determines whether the workload value is greater than the workload threshold (step S103). If the workload value is greater than the workload threshold (step S103; Yes), the on-board device 1 detects the cause of the workload increase (step S104). The on-board device 1 then supplies display instruction data including the workload value and workload-related information related to the detected cause of the workload increase to the external terminal 2 (step S105). Based on the display instruction data, the external terminal 2 then displays the workload value and the cause of the workload increase. If the display instruction data includes at least one of an externally captured image, a driver image, and map display information for displaying a map of the area around the current location, the external terminal 2 may display based on at least one of the externally captured image, the driver image, and the map display information.

[0076] On the other hand, if the workload value is equal to or less than the workload threshold (step S103; No), the in-vehicle device 1 supplies display instruction data including workload-related information related to the workload value to the external terminal 2 (step S106). As a result, the external terminal 2 performs display related to the workload value based on the display instruction data. Note that if the display instruction data includes at least one of an externally captured image, a driver image, and map display information, the external terminal 2 may perform display based on at least one of the externally captured image, the driver image, and the map display information.

[0077] The vehicle-mounted device 1 then determines whether the call has ended (step S107). If the vehicle-mounted device 1 determines that the call has ended (step S107; Yes), the process of the flowchart ends. On the other hand, if the vehicle-mounted device 1 determines that the call has not ended (step S107; No), the process returns to step S101.

[0078] (5) Modifications Next, suitable modifications of the above-described embodiment will be described. The following modifications may be applied in combination to the above-described embodiment.

[0079] (Variation 1) When the vehicle-mounted device 1 displays an outside-vehicle captured image on the external terminal 2, the vehicle-mounted device 1 may superimpose a line (gaze tracking line) representing the recognized path of the driver's gaze on the outside-vehicle captured image.

[0080] 8 shows a fourth display example of the external terminal 2 according to Modification 1. The external terminal 2 has established communication with the in-vehicle device 1, and is displaying the display screen shown in FIG. 8 on the display unit 26 based on the display instruction data received from the in-vehicle device 1. Here, as an example, the external terminal 2 provides an external-captured image display field 60, a workload-related information display field 61, and a map display field 62 on the display screen, similar to the first to third display examples, based on the display instruction data received from the in-vehicle device 1.

[0081] In the exterior-photographed image display field 60, an eye tracking line 66 indicating the trajectory of the driver's gaze direction within a recent predetermined time period is superimposed on the latest exterior-photographed image in correspondence with the gaze direction in the exterior-photographed image. In this case, the in-vehicle device 1 detects the gaze direction in the exterior-photographed image captured at the same time as the driver's image using any gaze detection technology and generates gaze direction data indicating the detected gaze direction. The in-vehicle device 1 then superimposes the gaze tracking line 66, indicating the driver's gaze direction within the most recent predetermined time period indicated by the gaze direction data, on the exterior-photographed image to be included in the display instruction data sent to the exterior terminal 2. The predetermined time period is pre-stored in the storage unit 12, for example. Upon receiving the display instruction data, the exterior terminal 2 displays the exterior-photographed image with the gaze tracking line 66 superimposed on it in the exterior-photographed image display field 60.

[0082] The vehicle-mounted device 1 may store in advance face-gaze correspondence data that associates each position on the image taken outside the vehicle with a face image model when the driver is looking at that position, and generate gaze direction data based on the face-gaze correspondence data. In this case, the vehicle-mounted device 1 identifies a face image model that most closely matches the driver's face image extracted from the driver-photographed image, and generates gaze direction data that indicates the position on the image taken outside the vehicle that corresponds to the identified face image model.

[0083] As described above, according to this modification, the user of the external terminal 2 can easily grasp the direction in which the driver is paying less attention (to the right in the example of FIG. 8 ), and the user of the external terminal 2 can assist the driver in confirming safety in the direction in which the driver is paying less attention by talking to the driver. In addition, the driver can feel a sense of security as if a fellow passenger is watching over him / her, and can feel a sense of unity as if the user of the external terminal 2 were actually riding with him / her.

[0084] In a preferred example, the vehicle-mounted device 1 may change the display mode (display color, whether or not to display, etc.) of the gaze tracking line depending on the workload value. For example, the vehicle-mounted device 1 superimposes the gaze tracking line on the outside-of-vehicle captured image when the workload value is equal to or greater than a predetermined threshold, and hides the gaze tracking line when the workload value is less than the threshold. The predetermined threshold may be the same as or different from the workload threshold. This allows the vehicle-mounted device 1 to display the gaze tracking line only when there is a high probability that the user of the outside-vehicle terminal 2 needs assistance in confirming safety. On the other hand, during normal times when the user of the outside-vehicle terminal 2 does not need assistance in confirming safety, the user of the outside-vehicle terminal 2 can enjoy the scenery using the outside-vehicle captured image. In another example, the vehicle-mounted device 1 may display the gaze tracking line in a color that corresponds to the workload value. In this case, the vehicle-mounted device 1 displays the gaze tracking line in a color that is more noticeable the higher the workload value, for example.

[0085] Furthermore, when displaying gaze tracking lines, the in-vehicle device 1 may highlight an object (also referred to as a "gaze target") in the outside-vehicle captured image to which the driver should direct their gaze, and display it on the external terminal 2. Examples of gaze targets include traffic lights, road signs, obstacles, pedestrians, etc. In this case, the in-vehicle device 1 uses any object recognition technology (including those using deep learning models such as instance segmentation) to extract objects that fit into a predetermined type of gaze target from the outside-vehicle captured image. The in-vehicle device 1 then processes the outside-vehicle captured image so as to emphasize the area of ​​the target in the extracted outside-vehicle captured image by, for example, outlining it and superimpose the gaze tracking line, and transmits the processed outside-vehicle captured image to the external terminal 2 together with display instruction data. Upon receiving the display instruction data, the external terminal 2 displays the outside-vehicle captured image with the gaze tracking line superimposed and the gaze target highlighted. This allows the user of the external terminal 2 to recognize the presence of an object of interest that the driver is not looking at from the image captured outside the vehicle, and to advise the driver to pay attention to the object of interest.

[0086] (Modification 2) At least a part of the processing executed by the vehicle-mounted device 1 may be executed by a server device that performs data communication with the vehicle-mounted device 1 and the external terminal 2 .

[0087] 9 shows a configuration example of a driving call system according to Modification 2. The driving call system includes an in-vehicle unit 1A, an external terminal 2, and a server device 5. Data communication is performed between the in-vehicle unit 1A and the server device 5, and between the external terminal 2 and the server device 5, via a communication network 3.

[0088] The vehicle-mounted device 1A has the same configuration as the vehicle-mounted device 1 described in the first embodiment (see FIG. 2). Note that if the server device 5 performs processing based on the map DB 4, the vehicle-mounted device 1A does not need to have the map DB 4. The vehicle-mounted device 1A transmits to the server device 5 an upload signal including information output by the sensor group 15, input information input by the input unit 13, and the like.

[0089] The server device 5 relays data necessary for a voice call between the in-vehicle device 1A and the external terminal 2. During the voice call, the server device 5 generates display instruction data based on an upload signal or the like received from the in-vehicle device 1A, and transmits the generated display instruction data to the external terminal 2. Specifically, during the voice call, the server device 5 executes the processing of the flowchart shown in Fig. 7 based on an upload signal or the like received from the in-vehicle device 1A.

[0090] 10 shows an example of a schematic configuration of the server device 5. The server device 5 mainly includes a communication unit 41, a storage unit 42, and a control unit 44. The elements within the server device 5 are connected to each other via a bus line 40.

[0091] The communication unit 41 performs data communication with external devices such as the in-vehicle device 1A and the external terminal 2 under the control of the control unit 44. The storage unit 42 is composed of various types of memory such as RAM, ROM, and non-volatile memory (including a hard disk drive, flash memory, etc.). The storage unit 42 stores programs that cause the server device 5 to execute predetermined processes. The storage unit 42 also includes a map DB 4. The control unit 44 includes a CPU, a GPU, etc., and controls the entire server device 5. The control unit 44 also executes the programs stored in the storage unit 42 to perform processes required to display workload-related information on the external terminal 2.

[0092] In this way, even when the server device 5 executes the processing required to display workload-related information on the external terminal 2, the driving call system, as in the embodiment, allows the user of the external terminal 2 to appropriately recognize the state of the driver's workload, and enables the user to determine a convenient time to talk to the driver. In this modification, the server device 5 is an example of an "information processing device."

[0093] As described above, the in-vehicle device 1 or the server device 5 is an information processing device used in a system that mediates communication between a driver of a target vehicle, which is a moving body, and a user of an external terminal 2 located outside the target vehicle, and includes a calculation means, a detection means, and a display control means. The calculation means calculates a workload value, which is an index value related to the degree of driving load (workload) of the driver of the target vehicle. The detection means detects the cause of the increased driving load when the workload value indicates that the driving load is higher than a predetermined standard. The display control means displays information related to the cause of the increased driving load on the external terminal 2. This allows the user of the external terminal 2 to determine a convenient time to communicate with the driver.

[0094] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a control unit, such as a computer. Non-transitory computer-readable media include various types of tangible storage media (tangible storage media). Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs)).

[0095] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference.

[0096] REFERENCE SIGNS LIST 1, 1A In-vehicle device 2 External terminal 3 Communication network 4 Map DB 5 Server device 11, 21, 41 Communication unit 12, 22, 42 Storage unit 13 Input unit 14, 24, 44 Control unit 15, 25 Sensor group 16, 26 Display unit 17, 27 Sound output unit

Claims

1. An information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising: A calculation means for calculating an index value relating to a degree of a driving load of a driver of the moving body; a display control means for displaying the tracking result of the driver's line of sight direction on a captured image captured from the moving body of an area outside the moving body, on a terminal device of the user; having The display control means is an information processing device that changes a display mode of the tracking result in accordance with the index value.

2. The information processing device described in Claim 1, wherein the display control means superimposes the tracking result of the driver's gaze direction within a most recent specified period of time on the captured image and displays it on the user's terminal device.

3. The information processing device described in Claim 1, wherein the display control means changes the display color of the tracking result depending on the index value.

4. 2. The information processing device according to claim 1, wherein the display control means displays the tracking result when the index value indicates that the driving load is higher than a predetermined standard, and does not display the tracking result when the index value indicates that the driving load is equal to or lower than the predetermined standard.

5. The information processing device according to claim 1 , wherein the display control means displays an object to which the driver should direct his or her gaze in the captured image with emphasis.

6. An information processing device as described in any one of claims 1 to 5, wherein the display control means causes information representing the magnitude of the index value to be displayed on the terminal device.

7. A control method executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user present outside the moving body, comprising: A calculation step of calculating an index value related to a degree of a driving load of a driver of the moving body; a display control step of superimposing the tracking result of the driver's line of sight on a captured image captured from the moving body of an area outside the moving body and displaying the result on a terminal device of the user; having The display control step is a control method for changing a display mode of the tracking result in accordance with the index value.

8. A program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, comprising: A calculation means for calculating an index value relating to a degree of a driving load of a driver of the moving body; a display control means for displaying the tracking result of the driver's line of sight direction on a captured image captured from the moving body of an area outside the moving body, on a terminal device of the user; The computer functions as The display control means is a program that changes the display mode of the tracking result depending on the index value.

9. A storage medium storing the program according to claim 8.