Information processing device, control method, program, and storage medium

Abstract

An on-vehicle machine 1 or a server device 5 is an information processing apparatus used in a system that intermediates communication between a driver of a target vehicle which is a movable body and a user of an out-of-vehicle terminal 2 which is present outside the target vehicle, and comprises a calculation means, a detection means, and a display control means. The calculation means calculates a workload value which is an index value relating to the degree of driving load (workload) of the driver of the target vehicle. The detection means detects a cause of increasing the driving load when the workload value indicates that the driving load is higher than a prescribed reference. The display control means displays, on the out-of-vehicle terminal 2, information relating to the cause of increasing the driving load.

Description

[Technical Field]

[0001] The present invention relates to a system that mediates communication between a driver of a mobile body and a user outside the mobile body. [Background technology]

[0002] Conventionally, there have been known technologies that enable a conversation between a vehicle driver and a person outside the vehicle. For example, Patent Document 1 discloses a conversation providing system that enables a conversation between a vehicle driver and a conversation partner by communicating with an in-vehicle system via a communication network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-138277 Summary of the Invention [Problem to be solved by the invention]

[0004] When the driver is talking to someone outside the vehicle, it is difficult for the person outside the vehicle to understand the driver's psychological state, so even when the driver is under heavy driving stress, they are unable to interrupt the conversation, which increases the driver's driving stress.

[0005] In view of the above-mentioned problems, one of the objects of the present invention is to provide an information processing device that can suitably mediate communication between a driver of a mobile body and a user outside the mobile body. [Means for solving the problem]

[0006] The invention described in claim 1 is An information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, a calculation means for calculating an index value relating to a degree of driving load on a driver of the vehicle; Tracking the driver's gaze direction A line representing the locus of a display control means for displaying the image on the user's terminal device by superimposing the image on a captured image of the outside of the moving body taken from the moving body; and The display control means displays the tracking information in accordance with the index value. A line representing the locus of The information processing device is characterized in that it changes the display mode of the information.

[0007] The invention described in claim 7 is A control method executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user located outside the moving body, comprising: a calculation step of calculating an index value relating to a degree of driving load of a driver of the moving body; Tracking the driver's gaze direction A line representing the locus of a display control step of superimposing the image on a captured image of the outside of the moving body and displaying the image on the terminal device of the user; and The display control step includes: A line representing the locus of The present invention is characterized in that the control method changes the display mode of the image.

[0008] The invention described in claim 8 is A program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, a calculation means for calculating an index value relating to a degree of driving load on a driver of the vehicle; Tracking the driver's gaze direction A line representing the locus of a display control means for displaying the image on the user's terminal device by superimposing the image on a captured image of the outside of the moving body taken from the moving body; Make the computer function as The display control means displays the tracking information in accordance with the index value. A line representing the locus of The program is characterized by changing the display mode of the image. [Brief explanation of the drawings]

[0009] [Figure 1] 1 illustrates a configuration example of a driving call system according to an embodiment. [Figure 2] 2 shows an example of a schematic configuration of an in-vehicle device. [Figure 3] 2 shows an example of a schematic configuration of an external terminal. [Figure 4] 10 shows a first example of a display on an external terminal while the target vehicle is traveling. [Figure 5] 10 shows a second example of the display on the external terminal while the target vehicle is traveling. [Figure 6] 10 shows a third example of a display on an external terminal while the target vehicle is traveling. [Figure 7] 10 is a flowchart illustrating an example of a procedure of a process executed by an in-vehicle device. [Figure 8] 10 shows a fourth example of a display on the external terminal according to a modified example. [Figure 9] 10 shows an example of the configuration of a driving call system according to a modified example. [Figure 10] 1 shows an example of a schematic configuration of a server device. DETAILED DESCRIPTION OF THE INVENTION

[0010] In one preferred embodiment of the present invention, an information processing device used in a system that mediates communication between a driver of a mobile body and a user outside the mobile body includes a calculation means for calculating an index value related to the degree of driving load on the driver of the mobile body, a detection means for detecting a cause that is increasing the driving load when the index value indicates that the driving load is higher than a predetermined standard, and a display control means for displaying information related to the cause on the user's terminal device.

[0011] The information processing device can conveniently allow a user outside the vehicle to recognize the cause of the driver's increased driving load, thereby enabling the user to communicate with the driver according to the cause of the increased driving load.

[0012] In one aspect of the information processing device, the calculation means calculates the index value based on a current state of at least one of the driving behavior of the mobile object, the driver, and the driving environment. In a preferred example, the detection means calculates a score by evaluating the current state of at least one of the driving behavior of the mobile object, the driver, and the driving environment for each element related to the driving load, and calculates the index value based on the score. This allows the information processing device to accurately calculate the index value of the driving load.

[0013] In another aspect of the information processing device, the detection means determines the factor that is causing the increase in driving load based on the score. This aspect enables the information processing device to accurately detect the factor that is causing the increase in driving load.

[0014] In another aspect of the information processing device, the display control means causes the terminal device to display information about the cause and information representing the magnitude of the index value. With this aspect, the information processing device can preferably present the degree of driving load together with the cause of the increase in driving load to a user outside the mobile body.

[0015] In another aspect of the information processing device, the display control means causes the terminal device to display a line representing the trajectory of the driver's line of sight superimposed on a captured image of an area outside the mobile body captured from the mobile body. This aspect allows the user of the terminal device to conveniently grasp the direction in which the driver is paying less attention, and the user of the terminal device can communicate with the driver to assist in confirming safety in the direction in which the driver is paying less attention. In a preferred example, the display control means may change the display mode of the line depending on the index value. In another preferred example, the display control means may display the line when the index value indicates that the driving load is higher than a predetermined standard, and may hide the line when the index value indicates that the driving load is equal to or lower than the predetermined standard.

[0016] In another aspect of the information processing device, the display control means highlights an object that the driver should look at in the captured image. This aspect enables a user of the terminal device to recognize an object that the driver is not looking at and to communicate with the driver to encourage him or her to look at the object.

[0017] In another preferred embodiment of the present invention, a control method is executed by an information processing device used in a system that mediates communication between a driver of a mobile body and a user located outside the mobile body, the control method comprising: a calculation step of calculating an index value relating to a degree of driving load on the driver of the mobile body; a detection step of detecting a cause of increasing the driving load when the index value indicates that the driving load is higher than a predetermined standard; and a display control step of displaying information relating to the cause on a terminal device of the user. By executing this control method, the information processing device can make a user located outside the mobile body preferably aware of the cause of increasing the driver's driving load.

[0018] In yet another embodiment of the present invention, a program executed by a computer of an information processing device used in a system that mediates communication between a driver of a mobile body and a user located outside the mobile body causes the computer to function as a calculation means for calculating an index value related to the degree of driving load of the driver of the mobile body, a detection means for detecting a cause of the increased driving load when the index value indicates that the driving load is higher than a predetermined standard, and a display control means for displaying information related to the cause on a terminal device of the user. By executing this program, the computer of the information processing device can conveniently allow a user located outside the mobile body to recognize the cause of the increased driving load of the driver. Preferably, the program is stored in a storage medium. [Example]

[0019] Preferred embodiments of the present invention will now be described with reference to the drawings.

[0020] (1) System Configuration 1 shows an example of the configuration of a driving call system according to a first embodiment. The driving call system has an in-vehicle device 1 installed in a vehicle and an external terminal 2 used by a person outside the vehicle. In the driving call system, the in-vehicle device 1 and the external terminal 2 communicate with each other to mediate a voice call between the driver of the vehicle and the user of the external terminal 2, and the user of the external terminal 2 can communicate with the driver of the vehicle while checking various information obtained from the vehicle on the external terminal 2.

[0021] The in-vehicle device 1 travels with the vehicle and performs processing to enable a call between the driver of the vehicle and the user of the external terminal 2. Hereinafter, the vehicle equipped with the in-vehicle device 1 will also be referred to as the "target vehicle." The in-vehicle device 1 performs data communication with the external terminal 2 via a communication network 3 such as the Internet or a dedicated communication network. Data exchanged between the in-vehicle device 1 and the external terminal 2 includes voice data generated during the call between the driver of the target vehicle and the user of the external terminal 2, and display instruction data for displaying information about the target vehicle or the driver. In this embodiment, the in-vehicle device 1 transmits display instruction data including information about the driver's driving load (workload) (also referred to as "workload-related information") to the external terminal 2.

[0022] The vehicle-mounted device 1 may be a navigation device installed in a target vehicle and providing route guidance to a set destination, or may be a mobile terminal such as a smartphone. The vehicle-mounted device 1 may also be incorporated into the target vehicle. The vehicle-mounted device 1 is an example of an "information processing device." The target vehicle is an example of a "mobile body."

[0023] The external terminal 2 is a terminal operated by a person outside the target vehicle, and performs data communication with the in-vehicle device 1 via the communication network 3. The external terminal 2 is, for example, a mobile terminal such as a smartphone. For example, the external terminal 2 establishes communication with the in-vehicle device 1 while the target vehicle is being driven, and exchanges voice data with the in-vehicle device 1 for a call between the driver and the user of the external terminal 2. The external terminal 2 also receives display instruction data including workload-related information from the in-vehicle device 1 during the call, and displays the driver's workload status, etc., based on the display instruction data. In this way, the external terminal 2 allows the user of the external terminal 2 to recognize the driver's workload status and provides the user with information that can be used to determine a convenient time to talk to the driver. The external terminal 2 is an example of a "terminal device."

[0024] Note that communication between the in-vehicle device 1 and the external terminal 2 may be realized by being relayed by a server device (not shown). Even when the in-vehicle device 1 and the external terminal 2 establish direct communication, the in-vehicle device 1 and / or the external terminal 2 may exchange information (e.g., communication address information) necessary for establishing communication between the in-vehicle device 1 and the external terminal 2 with the server device. In these cases, the server device performs processing necessary for the in-vehicle device 1 and the external terminal 2 to perform data communication (including authentication processing for the in-vehicle device 1 and the external terminal 2).

[0025] (2) Device configuration 2 shows an example of a schematic configuration of the vehicle-mounted device 1. The vehicle-mounted device 1 mainly includes a communication unit 11, a memory unit 12, an input unit 13, a control unit 14, a sensor group 15, a display unit 16, and a sound output unit 17. The elements within the vehicle-mounted device 1 are connected to each other via a bus line 10.

[0026] The communication unit 11 performs data communication with other terminals under the control of the control unit 14. For example, the communication unit 11 may receive map data for updating a map DB (DataBase) 4 from a map management server (not shown).

[0027] The storage unit 12 is configured with various types of memory such as RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory (including a hard disk drive, a flash memory, etc.). The storage unit 12 stores programs for the in-vehicle device 1 to execute predetermined processes. The above-mentioned programs may include an application program for making a call, an application program for causing the external terminal 2 to display a map or an image captured in the target vehicle, etc. The storage unit 12 is also used as a working memory for the control unit 14. The programs executed by the in-vehicle device 1 may be stored in a storage medium other than the storage unit 12.

[0028] The storage unit 12 also stores a map DB (DataBase) 4. The map DB 4 stores various data necessary for route guidance. The map DB 4 contains data necessary for displaying a map based on a predetermined position such as the current position of the target vehicle. The map DB 4 is a database that includes, for example, road data that represents a road network using a combination of nodes and links, and facility data that indicates facilities that are candidates for destinations, stop-off points, or landmarks. The map DB 4 may be updated based on information received by the communication unit 11 from a map management server under the control of the control unit 14.

[0029] The input unit 13 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 16 is a display or the like that displays information under the control of the control unit 14. The sound output unit 17 is a speaker or the like that outputs sound under the control of the control unit 14.

[0030] The sensor group 15 includes various sensors that sense the state of the target vehicle or the environment outside the vehicle. The sensor group 15 includes an exterior camera 51, a driver camera 52, a vehicle behavior detector 53, and a biometric sensor 54.

[0031] The exterior camera 51 is one or more cameras that capture images of the area outside the target vehicle, such as the area in front of the target vehicle, and generates images captured at predetermined time intervals (also called "exterior image"). The driver camera 52 is a camera installed so that the driver's face is included in the capture range, and generates images captured at predetermined time intervals (also called "driver image").

[0032] The vehicle behavior detector 53 generates detection signals indicating the behavior of the target vehicle, such as the current position, vehicle speed, acceleration, steering angle, etc. The vehicle behavior detector 53 includes, for example, a Global Navigation Satellite System (GNSS) receiver, a gyro sensor, an Inertial Measurement Unit (IMU), a vehicle speed sensor, an acceleration sensor, and a steering angle sensor.

[0033] The biological sensor 54 is one or more sensors that generate biological signals that indicate biological phenomena such as the driver's heart rate and the amount of sweat.

[0034] In addition, the sensor group 15 may include various external sensors (including cameras, lidars, radars, ultrasonic sensors, infrared sensors, sonars, etc.) and internal sensors in addition to the exterior camera 51, the driver camera 52, the vehicle behavior detector 53, and the biometric sensor 54.

[0035] The control unit 14 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire vehicle-mounted device 1. The control unit 14 functions as a "calculation means," a "detection means," a "display control means," a computer that executes a program, etc.

[0036] The processes executed by the control unit 14 are not limited to being realized by software programs, but may be realized by any combination of hardware, firmware, and software. The processes executed by the control unit 14 may also be realized by a user-programmable integrated circuit, such as a field-programmable gate array (FPGA) or a microcomputer. In this case, the programs executed by the control unit 14 in this embodiment may be realized by using this integrated circuit.

[0037] The configuration of the vehicle-mounted device 1 shown in FIG. 2 is an example, and various modifications may be made to the configuration shown in FIG. 2. For example, instead of storing the map DB 4 in the memory unit 12, the control unit 14 may receive map information from a map management server (not shown) via the communication unit 11. In another example, at least one of the input unit 13, the display unit 16, and the sound output unit 17 may be provided in the target vehicle as an external device of the vehicle-mounted device 1, and may supply the generated signals to the vehicle-mounted device 1. Furthermore, at least some of the sensors in the sensor group 15 may be sensors mounted on the target vehicle. In this case, the vehicle-mounted device 1 may acquire information output by sensors mounted on the target vehicle from the target vehicle based on a communication protocol such as CAN (Controller Area Network).

[0038] 3 shows an example of a schematic configuration of the exterior terminal 2. The exterior terminal 2 mainly includes a communication unit 21, a storage unit 22, an input unit 23, a control unit 24, a sensor group 25, a display unit 26, and a sound output unit 27. The elements within the exterior terminal 2 are connected to each other via a bus line 20.

[0039] The communication unit 21 performs data communication with other terminals under the control of the control unit 24. The storage unit 22 is composed of various types of memory such as RAM, ROM, and non-volatile memory. The storage unit 22 stores programs for the external terminal 2 to execute predetermined processes. The above-mentioned programs may include application programs for communicating with the driver, displaying information related to the driving of the target vehicle of the in-vehicle device 1 (including displaying maps and various captured images), and displaying information related to the driver's workload when communication with the in-vehicle device 1 is established. The storage unit 22 is also used as a working memory for the control unit 24. The programs executed by the external terminal 2 may be stored in a storage medium other than the storage unit 22.

[0040] The input unit 23 is a button, a touch panel, a remote controller, a voice input device, etc. that the user operates. The display unit 26 is a display or the like that displays information based on the control of the control unit 24. The sound output unit 27 is a speaker or the like that outputs sound based on the control of the control unit 24. The sensor group 25 includes an internal sensor that senses the state of the exterior terminal 2 and an external sensor that senses the state of the external world of the exterior terminal 2.

[0041] The control unit 24 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the entire external terminal 2. Note that the configuration of the external terminal 2 shown in Fig. 3 is an example, and various changes may be made to the configuration shown in Fig. 3.

[0042] (3) Viewing workload-related information Next, the display of workload-related information regarding the driving load (workload) of the driver of the target vehicle will be described. In summary, when the workload of the driver is equal to or greater than a predetermined level, the in-vehicle device 1 identifies the cause of the increased workload (also referred to as a "workload increase cause") and displays the identified workload increase cause on the external terminal 2. This allows the in-vehicle device 1 to make the user of the external terminal 2 aware of the workload increase cause, and the user of the external terminal 2 to take appropriate action according to the workload increase cause.

[0043] (3-1) Calculation of workload value First, a method for calculating a workload value, which is a value (score) indicating the degree of workload, will be described. Hereinafter, for the sake of convenience, it is assumed that a higher workload value indicates a higher degree of workload of the driver.

[0044] The vehicle-mounted device 1 calculates a workload value based on a score (also referred to as an "element-specific score") that evaluates the current state for each element related to workload (also referred to as a "workload-related element"). The workload-related element is an element that represents the state of the target vehicle, the driver, or the driving environment. Examples of workload-related elements include elements related to the environment of the road (driving road) on which the target vehicle is traveling (also referred to as a "road environment element"), elements related to driving behavior (driving operation) that affect the workload (also referred to as a "driving behavior element"), and elements related to the driver's physical condition (also referred to as a "driver physical condition element"). In other words, the workload-related element is an element that is a candidate for causing an increase in workload. Note that the workload-related element may be an element obtained by further subdividing at least one of the road environment element, the driving behavior element, and the driver's physical condition element.

[0045] Therefore, for example, while establishing communication for a call with the external terminal 2, the in-vehicle device 1 calculates a score (also referred to as an "element-specific score") that evaluates the current state of the target vehicle, driver, or driving environment for each workload-related element. The in-vehicle device 1 then determines the total, average, or other representative value of the calculated element-specific scores as the workload value. The workload-related elements for which the element-specific score is calculated may be all of the road environment elements, driving behavior elements, and driver physical condition elements, or any one of these, or any two of them. Preferably, the element-specific score may be weighted according to the degree of impact of each workload-related element on the workload. In this case, for example, the weighting coefficient multiplied by the element-specific score of an element with a greater impact on the workload is a larger value, and information indicating these weighting coefficients is stored in advance in the storage unit 12, etc.

[0046] Here, when calculating the element-specific scores of road environment elements, the vehicle-mounted device 1 calculates the element-specific scores by taking into account, for example, at least one of the congestion level, width, speed limit, and visibility of the road. In this case, for example, the vehicle-mounted device 1 refers to road information in the map DB 4 or road traffic information acquired by the communication unit 11, identifies each degree of congestion, width, speed limit, visibility, etc. of the road, and converts each identified degree into an element-specific score by referring to a predetermined formula or table. The predetermined formula or table is stored in advance in the storage unit 12, etc. Note that the road environment elements may be subdivided into multiple workload-related elements. In this case, for example, the vehicle-mounted device 1 may consider each of the congestion level, width, speed limit, and visibility of the road as workload-related elements and calculate element-specific scores for each of these.

[0047] Furthermore, when calculating the element-specific scores of driving behavior elements, the in-vehicle device 1 determines whether the current driving behavior corresponds to a driving behavior such as a right turn, a left turn, a merge, or a temporary stop, and calculates the element-specific score according to the determined driving behavior. In this case, for example, the in-vehicle device 1 determines the element-specific scores of the driving behavior elements by referring to a table or the like that associates the corresponding driving behavior with the element-specific scores of the driving behavior elements. Here, the in-vehicle device 1 may determine the current driving behavior based on information obtained by route guidance processing (e.g., the current position of the target vehicle and the route to the set destination), or may determine the score based on a signal indicating the state of the turn signal or the steering state obtained from the target vehicle. Furthermore, the in-vehicle device 1 may determine the element-specific scores of the driving behavior elements based on the presence or absence of auto-cruise driving (or the presence or absence of other automatic driving functions). Note that the driving behavior elements may be subdivided into multiple workload-related elements. For example, whether or not to turn right, whether or not to turn left, whether or not to merge, whether or not to stop, and whether or not to drive on auto-cruise are each set as workload-related elements, and the vehicle-mounted device 1 determines the element-specific score for each of these workload-related elements by referring to a predetermined table, etc., depending on whether or not the corresponding driving action is performed.

[0048] Furthermore, when calculating the element-specific scores for the driver's physical condition elements, the in-vehicle device 1 calculates the element-specific scores by taking into account at least one of the driver's continuous driving time, drowsiness level (alertness level), and other biometric indicators that affect driving. In this case, the in-vehicle device 1, for example, identifies the drowsiness level or other biometric indicators that affect driving based on a signal output by the biometric sensor 54. Then, the in-vehicle device 1 refers to a predetermined formula or table and converts the identified continuous driving time and one or more biometric indicators into element-specific scores for the driver's physical condition elements. Note that the driver's physical condition elements may be subdivided into multiple workload-related elements. In this case, for example, the in-vehicle device 1 may consider each of the drowsiness level and the continuous driving time as workload-related elements and calculate element-specific scores for these elements, respectively.

[0049] In this way, the vehicle-mounted device 1 can determine the workload value taking into consideration various factors that affect the workload.

[0050] (3-2) Detecting the cause of increased workload Next, a method for detecting the cause of an increase in workload will be described.

[0051] When the calculated workload value is greater than a predetermined threshold (also referred to as a "workload threshold"), the vehicle-mounted device 1 executes a process for detecting the cause of the workload increase. The workload threshold is, for example, determined in advance and stored in the storage unit 12. The workload threshold is an example of a "predetermined standard."

[0052] When the workload value is greater than the threshold, the vehicle-mounted device 1 determines the proportion of the workload value that the element-specific score of each workload-related element accounts for (also referred to as the "workload value occupancy proportion"). Next, the vehicle-mounted device 1 adds up the workload value occupancy proportions in order from the workload-related element with the highest workload value occupancy proportion until the total value of the workload value occupancy proportions reaches a predetermined proportion. When the total value of the workload value occupancy proportions reaches a predetermined proportion, the vehicle-mounted device 1 detects the workload-related element whose workload value occupancy proportion has been added up as the cause of the workload increase. The above-mentioned predetermined proportion is, for example, predetermined and stored in the storage unit 12.

[0053] For example, a case where the above-mentioned predetermined percentage is 30% will be illustrated. In this case, the on-board device 1 determines whether the workload value occupancy percentage of the workload-related element with the highest workload value occupancy percentage is 30% or more. If the workload value occupancy percentage of the workload-related element with the highest workload value occupancy percentage is 30% or more, the on-board device 1 detects the workload-related element as a cause of workload increase. On the other hand, if the workload value occupancy percentage of the workload-related element with the highest workload value occupancy percentage is not 30% or more, the on-board device 1 determines whether the total value of the workload value occupancy percentages of the workload-related elements with the highest and second highest workload value occupancy percentages is 30% or more. If the total value of the workload value occupancy percentages of the workload-related elements with the highest and second highest workload value occupancy percentages is 30% or more, the on-board device 1 detects these workload-related elements as a cause of workload increase. On the other hand, if the workload value occupancy ratios of the workload-related elements with the first and second highest workload value occupancy ratios are not 30% or more, the on-vehicle device 1 determines whether the total value of the workload value occupancy ratios of the workload-related elements with the first to third highest workload value occupancy ratios is 30% or more. The on-vehicle device 1 repeatedly executes this process until the total value of the workload value occupancy ratios of the workload-related elements is 30% or more, and at the point when the total value is 30% or more, the on-vehicle device 1 detects the workload-related element with the added workload value occupancy ratio as the cause of the workload increase.

[0054] This allows the on-board device 1 to suitably detect one or more workload-related factors that cause an increase in workload. That is, one or more factors among the environment of the road during driving (congestion level, width, speed limit, visibility), driving behavior, and the driver's physical condition (continuous driving time, drowsiness level based on biological information, etc.) are identified as the cause of the increase in workload.

[0055] (3-3) Display example 4 shows a first display example of the external terminal 2 when the target vehicle of the in-vehicle device 1 is traveling. The external terminal 2 has established communication with the in-vehicle device 1, and is displaying the display screen shown in FIG. 4 on the display unit 26 based on display instruction data received from the in-vehicle device 1.

[0056] Here, the display screen shown in FIG. 4 has an outside-vehicle photographed image display field 60, a workload-related information display field 61, and a map display field 62.

[0057] The exterior terminal 2 displays the latest exterior-photographed image (moving image) generated by the exterior-photographed camera 51 on the exterior-photographed image display field 60. In this case, the vehicle-mounted device 1 transmits display instruction data including the latest exterior-photographed image generated by the exterior-photographed camera 51 to the exterior terminal 2.

[0058] Furthermore, the external terminal 2 displays a display window 71 and an indicator 72 on the workload-related information display field 61 based on the workload-related information included in the display instruction data transmitted by the in-vehicle device 1 .

[0059] The display window 71 displays the level of the workload value calculated by the on-vehicle device 1 and the cause of the workload increase detected by the on-vehicle device 1. In this case, since the workload value calculated by the on-vehicle device 1 is at the highest level (for example, the highest level when divided into five levels), the display window 71 displays "High Workload," which indicates the highest level. Furthermore, since the workload value exceeds the workload threshold, the on-vehicle device 1 detects the cause of the workload increase (here, turning right at an intersection), and the external terminal 2 displays "Turn right at intersection" in the display window 71.

[0060] The indicator 72 also indicates the degree of workload, and the larger the workload value calculated by the vehicle-mounted device 1, the further the gauge extends to the right. Here, since the workload value is near the maximum value, the gauge of the indicator 72 extends to the near right end. The color of the gauge may change depending on the workload value. For example, the higher the workload value, the more noticeable the color of the gauge may be. The "High Workload" display in the display window 71 and the indicator 72 are examples of "information indicating the magnitude of the index value."

[0061] The external terminal 2 also displays a map of the vicinity of the current position of the target vehicle on the map display field 62. The in-vehicle device 1 generates display instruction data for the external terminal 2 to display the above-mentioned map based on the state of the target vehicle estimated using the vehicle behavior detector 53, the map DB 4, and information related to route guidance to the destination, and transmits the display instruction data to the external terminal 2. Note that here, in the map display field 62, a current position mark 73 indicating the current position of the target vehicle and a route line 74 indicating the guidance route along which the in-vehicle device 1 will guide the driver of the target vehicle are superimposed on the map.

[0062] As described above, in the first display example, when the workload is high, the in-vehicle device 1 can appropriately make the user of the external terminal 2 aware of the fact that the workload is high and the cause of this (here, turning right at an intersection). This allows the user of the external terminal 2 to determine that the driver of the in-vehicle device 1 should concentrate on driving, and to take measures such as refraining from talking to the driver about matters other than driving (for example, asking them to do some shopping) until the driver has completed turning right at the intersection.

[0063] Fig. 5 shows a second display example of the external terminal 2 when the target vehicle of the in-vehicle device 1 is traveling. The external terminal 2 has established communication with the in-vehicle device 1, and is displaying the display screen shown in Fig. 5 on the display unit 26 based on display instruction data received from the in-vehicle device 1. The external terminal 2 has an external-photographed image display field 60, a workload-related information display field 61, and a map display field 62 on the display screen.

[0064] The external terminal 2 displays the latest externally captured images (moving images) generated by the externally captured camera 51 in the externally captured image display field 60 based on the display instruction data received from the in-vehicle device 1. The external terminal 2 also displays a map of the area around the current position of the target vehicle in the map display field 62 based on the display instruction data received from the in-vehicle device 1.

[0065] Furthermore, the external terminal 2 displays a display window 71A and an indicator 72A indicating the workload level in the workload-related information display field 61 based on the workload-related information included in the display instruction data received from the on-board device 1. Here, because the workload value calculated by the on-board device 1 is equal to or less than the workload threshold, the on-board device 1 does not detect the cause of the workload increase. Therefore, the external terminal 2 displays "Low Workload" in the display window 71, indicating that the workload value level is the lowest level, and does not display the cause of the workload increase. Note that, here, the on-board device 1 detects from the vehicle that the current driving mode of the target vehicle is auto-cruise driving, which puts the driver in a low workload state, and therefore includes in the workload-related information that the vehicle is currently in auto-cruise driving. Based on the workload-related information received from the on-board device 1, the external terminal 2 displays in the workload-related information display field 61 that the vehicle is currently in auto-cruise driving. Furthermore, the external terminal 2 displays in the workload-related information display field 61 an indicator 72A including a gauge whose length corresponds to the level of the workload value.

[0066] In this way, in the second display example, when the workload is low, the in-vehicle device 1 appropriately notifies the user of the external terminal 2 that the workload is low. This allows the user of the external terminal 2 to determine that the driver of the in-vehicle device 1 is relatively relaxed, and to find the right timing to talk to the driver about necessary matters.

[0067] Fig. 6 shows a third display example of the external terminal 2 when the target vehicle of the in-vehicle device 1 is traveling. The external terminal 2 has established communication with the in-vehicle device 1, and is displaying the display screen shown in Fig. 6 on the display unit 26 based on display instruction data received from the in-vehicle device 1. The external terminal 2 has an external-photographed image display field 60, a workload-related information display field 61, and a map display field 62 on the display screen.

[0068] The exterior terminal 2 displays the latest exterior-captured images (moving images) generated by the exterior-captured camera 51 on the exterior-captured image display field 60, similar to the first and second display examples, based on the display instruction data received from the in-vehicle device 1. Also, the exterior terminal 2 displays a map of the area around the current position of the target vehicle on the map display field 62, similar to the first and second display examples, based on the display instruction data received from the in-vehicle device 1.

[0069] Furthermore, based on the workload-related information received from the in-vehicle device 1, the external terminal 2 displays, in the workload-related information display field 61, a display window 71B showing the workload level and the cause of workload increase, and an indicator 72B visually indicating the workload level. In this example, since the workload value calculated by the in-vehicle device 1 is greater than the workload threshold, the cause of the workload increase is displayed in the display window 71B. Specifically, the in-vehicle device 1 detects the drowsiness level of the driver's physical condition element and the visibility of the road environment element as causes of the workload increase, respectively, and as a result, "drowsiness + low visibility" is displayed in the display window 71B. Furthermore, the display window 71B displays "moderately high workload," which indicates the workload level according to the workload value calculated by the in-vehicle device 1. Furthermore, the external terminal 2 displays, in the workload-related information display field 61, an indicator 72B including a gauge whose length corresponds to the level of the determined workload value.

[0070] As described above, in the third display example, when the workload is high due to the driver feeling drowsy, the in-vehicle device 1 can appropriately make the user of the external terminal 2 aware of the high workload and its cause. This allows the user of the external terminal 2 to take action, such as actively talking to the driver of the in-vehicle device 1 to wake him up.

[0071] The display layouts in the first to third display examples are merely examples, and various modifications may be applied. For example, the external terminal 2 may display an image captured by the driver instead of an image captured outside the vehicle, based on an operation on the external terminal 2 by the user of the external terminal 2. In another example, the external terminal 2 may provide only one of the external-vehicle-captured-image display field 60 and the map display field 62 on the display screen. In yet another example, the external terminal 2 may provide only the workload-related information display field 61 on the display screen.

[0072] (4) Processing flow Fig. 7 is an example of a flowchart showing the procedure of processing executed by the vehicle-mounted device 1. The vehicle-mounted device 1 executes the processing of the flowchart shown in Fig. 7 when communication with the external terminal 2 is established and a call is started.

[0073] First, the vehicle-mounted device 1 calculates a score for each workload-related element related to the workload of the driver of the target vehicle (i.e., element-specific score) (step S101). In this case, the vehicle-mounted device 1 calculates an element-specific score for each of road environment elements, driving behavior elements, driver physical condition elements, or elements obtained by subdividing these elements, based on the map DB 4 and data output by the sensor group 15.

[0074] Next, the vehicle-mounted device 1 calculates a workload value based on the scores of each workload-related element (element-specific scores) (step S102). In this case, the vehicle-mounted device 1 may calculate the workload value by adding up or averaging (including using a weighting factor) the element-specific scores, or may calculate the workload value by substituting each element-specific score into a formula for calculating the workload value.

[0075] Next, the on-vehicle device 1 determines whether the workload value is greater than the workload threshold (step S103). If the workload value is greater than the workload threshold (step S103; Yes), the on-vehicle device 1 detects the cause of the workload increase (step S104). The on-vehicle device 1 then supplies display instruction data including the workload value and workload-related information related to the detected cause of the workload increase to the external terminal 2 (step S105). The external terminal 2 then displays the workload value and the cause of the workload increase based on the display instruction data. Note that if the display instruction data includes at least one of an image taken outside the vehicle, an image of the driver, and map display information for displaying a map around the current location, the external terminal 2 may display based on at least one of the image taken outside the vehicle, the image of the driver, and the map display information.

[0076] On the other hand, if the workload value is equal to or less than the workload threshold (step S103; No), the in-vehicle device 1 supplies display instruction data including workload-related information related to the workload value to the external terminal 2 (step S106). As a result, the external terminal 2 performs display related to the workload value based on the display instruction data. Note that if the display instruction data includes at least one of an image taken outside the vehicle, an image of the driver, and map display information, the external terminal 2 may perform display based on at least one of the image taken outside the vehicle, the image of the driver, and the map display information.

[0077] Then, the vehicle-mounted device 1 determines whether the call has ended (step S107). If the vehicle-mounted device 1 determines that the call has ended (step S107; Yes), it ends the processing of the flowchart. On the other hand, if the vehicle-mounted device 1 determines that the call has not ended (step S107; No), it returns the processing to step S101.

[0078] (5) Variations Next, preferred modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in combination.

[0079] (Variation 1) When the vehicle-mounted device 1 displays the outside-of-vehicle captured image on the outside-vehicle terminal 2, the vehicle-mounted device 1 may superimpose a line (gaze tracking line) representing the recognized trajectory of the driver's gaze on the outside-of-vehicle captured image.

[0080] Fig. 8 shows a fourth display example of the external terminal 2 according to the first modification. The external terminal 2 has established communication with the in-vehicle device 1, and displays the display screen shown in Fig. 8 on the display unit 26 based on the display instruction data received from the in-vehicle device 1. Here, as an example, the external terminal 2 provides an external-photographed image display field 60, a workload-related information display field 61, and a map display field 62 on the display screen, similar to the first to third display examples, based on the display instruction data received from the in-vehicle device 1.

[0081] In the outside-vehicle-shot image display field 60, an eye tracking line 66 indicating the trajectory of the driver's gaze direction within a recent predetermined time period is superimposed on the latest outside-vehicle-shot image in association with the gaze direction in the outside-vehicle-shot image. In this case, the in-vehicle device 1 detects the gaze direction in the outside-vehicle-shot image captured at the same time as the driver's shot image using any gaze detection technology, and generates gaze direction data indicating the detected gaze direction. The in-vehicle device 1 then superimposes the gaze tracking line 66, indicating the driver's gaze direction within the most recent predetermined time period indicated by the gaze direction data, on the outside-vehicle-shot image to be included in the display instruction data to be transmitted to the outside-vehicle terminal 2. The predetermined time period is stored in the storage unit 12, for example. Upon receiving the display instruction data, the outside-vehicle-shot image display field 60 displays the outside-vehicle-shot image with the gaze tracking line 66 superimposed on it.

[0082] The vehicle-mounted device 1 may store in advance face-gaze correspondence data that associates each position on the image taken outside the vehicle with a face image model when the driver is looking at that position, and generate gaze direction data based on the face-gaze correspondence data. In this case, the vehicle-mounted device 1 identifies a face image model that most closely matches the driver's face image extracted from the driver-photographed image, and generates gaze direction data that indicates the position on the image taken outside the vehicle that corresponds to the identified face image model.

[0083] As described above, according to this modification, the user of the external terminal 2 can easily grasp the direction in which the driver is paying less attention (the right direction in the example of FIG. 8), and the user of the external terminal 2 can assist the driver in confirming safety in the direction in which the driver is paying less attention by talking to the driver. Also, the driver can feel a sense of security as if a fellow passenger is watching over him / her, and can feel a sense of unity as if the user of the external terminal 2 were actually riding with him / her.

[0084] In a preferred example, the vehicle-mounted device 1 may change the display mode (display color, whether to display, etc.) of the gaze tracking line depending on the workload value. For example, the vehicle-mounted device 1 superimposes the gaze tracking line on the image captured outside the vehicle when the workload value is equal to or greater than a predetermined threshold, and hides the gaze tracking line when the workload value is less than the threshold. The predetermined threshold may be the same as or different from the workload threshold. This allows the vehicle-mounted device 1 to display the gaze tracking line only when there is a high probability that the user of the external terminal 2 needs assistance in confirming safety. On the other hand, during normal times when the user of the external terminal 2 does not need assistance in confirming safety, the user of the external terminal 2 can enjoy the scenery using the image captured outside the vehicle. In another example, the vehicle-mounted device 1 may display the gaze tracking line in a color corresponding to the workload value. In this case, the vehicle-mounted device 1 displays the gaze tracking line in a color that is more noticeable the higher the workload value, for example.

[0085] Furthermore, when displaying gaze tracking lines, the in-vehicle device 1 may highlight an object (also referred to as a "gaze target") in the outside-vehicle captured image to which the driver should direct his or her gaze, and display the highlighted object on the external terminal 2. Examples of gaze targets include traffic lights, road signs, obstacles, pedestrians, etc. In this case, the in-vehicle device 1 uses any object recognition technology (including one using a deep learning model such as instance segmentation) to extract objects that fit into a predetermined type of gaze target from the outside-vehicle captured image. The in-vehicle device 1 then processes the outside-vehicle captured image so as to highlight the area of ​​the target in the extracted outside-vehicle captured image by, for example, outlining it and superimpose the gaze tracking line, and transmits the processed outside-vehicle captured image to the external terminal 2 together with display instruction data. Upon receiving the display instruction data, the external terminal 2 displays the outside-vehicle captured image with the gaze tracking line superimposed and the gaze target highlighted. This allows the user of the external terminal 2 to recognize the presence of an object of interest that the driver is not paying attention to, based on the image captured outside the vehicle, and to advise the driver to pay attention to the object of interest.

[0086] (Variation 2) At least a part of the processing executed by the vehicle-mounted device 1 may be executed by a server device that performs data communication with the vehicle-mounted device 1 and the external terminal 2.

[0087] 9 shows a configuration example of a driving call system according to Modification 2. The driving call system includes an in-vehicle device 1A, an external terminal 2, and a server device 5. The in-vehicle device 1A and the server device 5, and the external terminal 2 and the server device 5, respectively, perform data communication via a communication network 3.

[0088] The vehicle-mounted device 1A has the same configuration (see FIG. 2) as the vehicle-mounted device 1 described in the first embodiment. Note that, if the server device 5 performs processing based on the map DB 4, the vehicle-mounted device 1A does not need to have the map DB 4. The vehicle-mounted device 1A transmits to the server device 5 an upload signal including information output by the sensor group 15, input information input by the input unit 13, and the like.

[0089] The server device 5 relays data necessary for a voice call between the in-vehicle device 1A and the external terminal 2. During the voice call, the server device 5 generates display instruction data based on an upload signal or the like received from the in-vehicle device 1A, and transmits the generated display instruction data to the external terminal 2. Specifically, during the voice call, the server device 5 executes the processing of the flowchart shown in Fig. 7 based on an upload signal or the like received from the in-vehicle device 1A.

[0090] 10 shows an example of a schematic configuration of the server device 5. The server device 5 mainly includes a communication unit 41, a storage unit 42, and a control unit 44. The elements within the server device 5 are connected to each other via a bus line 40.

[0091] The communication unit 41 performs data communication with external devices such as the in-vehicle device 1A and the external terminal 2 under the control of the control unit 44. The storage unit 42 is composed of various types of memory such as RAM, ROM, and non-volatile memory (including a hard disk drive, flash memory, etc.). The storage unit 42 stores programs that cause the server device 5 to execute predetermined processes. The storage unit 42 also includes a map DB 4. The control unit 44 includes a CPU, a GPU, etc., and controls the entire server device 5. The control unit 44 also executes the programs stored in the storage unit 42 to perform processes required to display workload-related information on the external terminal 2.

[0092] In this way, even when the server device 5 executes the processing required to display workload-related information on the external terminal 2, the driving call system, as in the embodiment, allows the user of the external terminal 2 to appropriately recognize the state of the driver's workload, and the user can choose a convenient time to talk to the driver. In this modification, the server device 5 is an example of an "information processing device."

[0093] As described above, the in-vehicle device 1 or the server device 5 is an information processing device used in a system that mediates communication between a driver of a target vehicle, which is a moving body, and a user of an external terminal 2 located outside the target vehicle, and includes a calculation means, a detection means, and a display control means. The calculation means calculates a workload value, which is an index value related to the degree of driving load (workload) of the driver of the target vehicle. The detection means detects the cause of the increased driving load when the workload value indicates that the driving load is higher than a predetermined standard. The display control means displays information related to the cause of the increased driving load on the external terminal 2. This allows the user of the external terminal 2 to determine a convenient time to communicate with the driver.

[0094] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a control unit, such as a computer. Non-transitory computer-readable media include various types of tangible storage media (tangible storage media). Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory).

[0095] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference. [Explanation of symbols]

[0096] 1, 1A in-vehicle unit 2. Outside vehicle terminal 3. Communication Network 4 Map DB 5. Server equipment 11, 21, 41 Communications Department 12, 22, 42 storage section 13 Input section 14, 24, 44 Control unit 15, 25 sensor group 16, 26 Display section 17, 27 Sound output section

Claims

1. An information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, a calculation means for calculating an index value relating to a degree of driving load on a driver of the vehicle; a display control means for displaying a line representing a tracking trajectory of the driver's line of sight on a captured image of an area outside the moving body taken from the moving body, on the user's terminal device; and The display control means is an information processing device that changes a display mode of a line representing the tracking trajectory in accordance with the index value.

2. The information processing device according to claim 1 , wherein the display control means causes the user's terminal device to display a line representing the tracking trajectory of the driver's line of sight within a recent predetermined time period superimposed on the captured image.

3. The information processing apparatus according to claim 1 , wherein the display control means changes a display color of the line representing the tracking trajectory in accordance with the index value.

4. 2. The information processing device according to claim 1, wherein the display control means displays a line representing the tracking trajectory when the index value indicates that the driving load is higher than a predetermined standard, and does not display the line representing the tracking trajectory when the index value indicates that the driving load is equal to or lower than the predetermined standard.

5. The information processing device according to claim 1 , wherein the display control means highlights an object to which the driver should direct his or her gaze in the captured image.

6. 6. The information processing device according to claim 1, wherein the display control means causes the terminal device to display information indicating the magnitude of the index value.

7. A control method executed by an information processing device used in a system that mediates communication between a driver of a moving body and a user located outside the moving body, comprising: a calculation step of calculating an index value relating to a degree of driving load of a driver of the moving body; a display control step of superimposing a line representing a tracking trajectory of the driver's line of sight on a captured image of an area outside the moving body captured from the moving body on the user's terminal device; and The display control step is a control method for changing a display mode of a line representing the tracking trajectory in accordance with the index value.

8. A program executed by a computer of an information processing device used in a system that mediates communication between a driver of a moving body and a user outside the moving body, a calculation means for calculating an index value relating to a degree of driving load on a driver of the vehicle; a display control means for displaying, on the user's terminal device, a line representing a tracking trajectory of the driver's line of sight on a captured image of an area outside the moving body taken from the moving body; Make the computer function as The display control means is a program that changes the display mode of the line representing the tracking trajectory according to the index value.

9. A storage medium storing the program according to claim 8.

Images