User status notification method and user assistance system

Abstract

The present invention relates to a user state notification method and a user assistance system. An assistant can include a time when a user who will be in a content easy to master operation of a mobile body is able to communicate. The user state notification method is executed by a computer. The user state notification method includes: a process of acquiring a traffic condition of a predetermined movement path of the mobile body operated by the user; a process of generating communication possibility information indicating whether or not the user is able to perform communication with an assistant user of an assistant via the mobile body, based on the acquired traffic condition; and a process of causing the generated communication possibility information to be displayed on a display device for the assistant. The communication possibility information includes: present state information indicating whether or not the user is able to perform communication now; and future time information indicating a time when the user becomes able to perform communication in a case where the user is not able to perform communication now.

Description

Technical Field

[0001] This disclosure relates to a technique for notifying assistants of the status of users operating a mobile body. Background Technology

[0002] Patent Document 1 discloses a method in which a recipient notifies the sender in advance whether a call is possible before the sender sends a call via a mobile phone network or the Internet. The method determines whether the recipient can receive the call based on call reception capability information set on the recipient's side. If the recipient can receive the call, a normal transmission is performed. Conversely, if the recipient cannot receive the call, a notification indicating that the call cannot be received is displayed on the sender's mobile phone.

[0003] Furthermore, Patent Document 2 discloses a call management device that displays information about other users who can talk to the user of the terminal device on the display control unit of the terminal device. The display control unit displays the time period during which the second user can talk on the terminal device of the first user. Patent Document 3 discloses a car phone system that overlays an image of the front of the vehicle captured during a hands-free call with the audio call and sends it to the other party in the hands-free call.

[0004] Patent Document 1: Japanese Patent No. 6368887

[0005] Patent Document 2: Japanese Patent No. 7408765

[0006] Patent Document 3: Japanese Patent No. 4453579

[0007] According to the technology described in Patent Document 1, the sender can determine whether a call can be made at the moment the call is sent, but cannot determine whether the receiver (the call recipient) will be able to make a call in the future. Summary of the Invention

[0008] This disclosure was made in view of the aforementioned issues, and its object is to provide a technique that allows an assistant to easily grasp, including in the future, when a user operating a mobile body is able to communicate.

[0009] The user status notification method disclosed herein is performed by a computer. The user status notification method includes: obtaining traffic conditions along a predetermined movement path of a mobile body operated by a user; generating communication capability information based on the obtained traffic conditions, the communication capability information indicating whether the user can perform communication with an assistant assisting the user via the mobile body; and displaying the generated communication capability information on a display device for the assistant. The communication capability information includes: current status information indicating whether the user can perform communication now; and future time information indicating the time when the user will become able to perform communication if the user cannot perform communication now.

[0010] The user assistance system disclosed herein includes a display device and one or more processors. The display device displays information for an assistance person to assist a user operating a mobile body. One or more processors acquire traffic conditions along a predetermined path of the mobile body, generate communication capability information based on the acquired traffic conditions indicating whether the user can communicate with the assistance person via the mobile body, and display the generated communication capability information on the display device. The communication capability information includes: current status information indicating whether the user can perform communication now; and future time information indicating when the user will become able to perform communication if the user cannot perform communication now.

[0011] According to this disclosure, the user's ability to communicate, displayed on the display device to assist personnel, includes current status information and future time information, the future time information indicating the time when the user will become able to communicate if the user cannot currently do so. Thus, the assistive personnel can easily grasp, including the future, when a user operating the mobile device will be able to communicate. Attached Figure Description

[0012] Figure 1 This is a simplified diagram illustrating an example of the configuration of the user assistance system and the vehicle (mobile body) involved in the implementation method.

[0013] Figure 2 This is a flowchart illustrating an example of a process related to user status notifications in an implementation method.

[0014] Figure 3 This is a diagram showing a first example of whether or not communication information is possible according to the implementation method.

[0015] Figure 4 This is a diagram illustrating an example of additional user information involved in the implementation method.

[0016] Figure 5 This is a diagram showing traffic conditions that are included in the communication information provided in the implementation method.

[0017] Figure 6 This is a diagram showing a second example of whether or not communication information is possible in the implementation method.

[0018] Explanation of reference numerals in the attached figures

[0019] 10… User assistance system; 11… Communication device; 12… Display device; 15… Control device; 16… Processor; 17… Storage device; 20… Vehicle; 30… Transportation system; 40… Cloud. Detailed Implementation

[0020] The embodiments of this disclosure will be described with reference to the accompanying drawings.

[0021] 1. System Composition

[0022] Figure 1 This is a simplified diagram illustrating an example of the configuration of the user assistance system 10 and the vehicle 20 according to this embodiment. The user assistance system 10 is a system for remotely assisting a user U operating a mobile body. Examples of mobile bodies include vehicles (e.g., automobiles), airplanes, etc. In the following description, as an example, we consider the case where the mobile body to which the user assistance system 10 is applied is the vehicle 20. In general, "mobile body" will be used to replace "vehicle" in the following description.

[0023] 1-1. User Assistance System

[0024] The user assistance system 10 is operated by an assistant S (operator) who assists the user U. The assistant S is, for example, a receptionist who responds to various requests from the user U (e.g., restaurant recommendations, restaurant reservations) from the operating vehicle 20. The user assistance system 10 includes, for example, a communication device 11, a display device 12, an input device 13, a speaker 14, and a control device 15. The communication device 11, display device 12, input device 13, speaker 14, and control device 15 can be configured as terminals with communication functions, such as smartphones or personal computers.

[0025] Communication device 11 communicates wirelessly with the external environment of user assistance system 10. Specifically, communication device 11 communicates wirelessly with one or more vehicles 20 and traffic systems 30 respectively. For example, communication device 11 communicates wirelessly with one or more vehicles 20 and traffic systems 30 respectively via an external server (e.g., cloud 40). Alternatively, communication device 11 may also communicate wirelessly directly with one or more vehicles 20 and traffic systems 30 respectively. Cloud 40, for example, has a database 41 that stores various information related to the assistance provided by assistant S to user U and used by user assistance system 10 and one or more vehicles 20. Communication between user assistance system 10 and vehicles 20 includes communication (including phone calls) between assistant S and user U via user assistance system 10 and vehicles 20.

[0026] Display device 12 displays various information required by the assistant S to assist the user U. Input device 13 is used by the assistant S when operating the user assistance system 10. Examples of input device 13 include touch panel, keyboard, mouse, button, microphone, etc. Speaker 14, together with the microphone, is used when the assistant S conducts voice communication with the user U of each vehicle 20.

[0027] Control device 15 controls user assistance system 10. Control device 15 includes one or more processors 16 (hereinafter simply referred to as processor 16) and one or more storage devices 17 (hereinafter simply referred to as storage devices 17). Processor 16 performs various processes related to assisting user U via assistant S. Examples of processor 16 include general-purpose processors, application-specific processors, CPUs (Central Processing Units), GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), etc. Processor 16 can also be referred to as processing circuitry. Storage device 17 stores various information required for various processes. Examples of storage devices 17 include volatile memory, non-volatile memory, HDDs (Hard Disk Drives), SSDs (Solid State Drives), etc. The functions of control device 15 can also be implemented through the cooperation of processor 16 executing computer programs and storage device 17. The computer programs are stored in storage device 17. Alternatively, computer programs can be recorded on computer-readable recording media, or they can be provided via a network.

[0028] 1-2. Vehicles

[0029] Vehicle 20 is operated (driven) by user U. Vehicle 20 includes communication device 21, sensor class 22, navigation device 23, display device 24, input device 25, speaker 26 and control device 27.

[0030] Communication device 21 communicates wirelessly with the outside of vehicle 20. Specifically, communication device 21 communicates wirelessly with user assistance system 10 and traffic system 30 respectively. For example, communication device 21 communicates wirelessly with user assistance system 10 and traffic system 30 respectively via an external server (e.g., cloud 40). Alternatively, communication device 21 may also communicate wirelessly directly with user assistance system 10 and traffic system 30 respectively.

[0031] Sensors 22 are mounted on vehicle 20, including, for example, identification sensors, vehicle status sensors, and position sensors. The identification sensors (e.g., one or more cameras) identify (detect) the surrounding conditions of vehicle 20, including at least the area in front of vehicle 20. The vehicle status sensors detect the state of vehicle 20 (e.g., vehicle speed, yaw rate, acceleration, steering angle). The position sensors detect the position and orientation of vehicle 20.

[0032] The navigation device 23 generates a predetermined travel path R for the vehicle 20, for example, based on destination information received from the user U and information from the sensor class 22 and the traffic system 30. The display device 24 displays various information for the user U. This information includes information from the assistant S. The display device 24 may be, for example, a head-up display (HUD) that displays information on the windshield of the vehicle 20, or a display located on the dashboard. The input device 25 is used by the user U when communicating with the assistant S. Examples of input devices 25 include touch panels, buttons, and microphones. The speaker 26, together with the microphone, is used when the user U and the assistant S have a voice conversation.

[0033] Control device 27 controls vehicle 20. Control device 27 includes one or more processors 28 (hereinafter simply referred to as processor 28) and one or more storage devices 29 (hereinafter simply referred to as storage devices 29). Processor 28 performs various processes, including those related to communication (including phone calls) between user U and assistant S. Examples of processor 28 include general-purpose processors, application-specific processors, CPUs, GPUs, ASICs, FPGAs, etc. Processor 28 can also be referred to as a processing circuit. Storage device 29 stores various information required for various processes. Examples of storage device 29 include volatile memory, non-volatile memory, HDDs, SSDs, etc. The functions of control device 27 can also be implemented through the cooperation of processor 28 executing computer programs and storage device 29. The computer program is stored in storage device 29. Alternatively, the computer program can be recorded on a computer-readable recording medium or provided via a network.

[0034] 2. User Status Notification

[0035] The assistant S may sometimes operate the user assistance system 10 (input device 13) to attempt communication (including phone calls) with the user U of the vehicle 20. More specifically, for example, if an expectation from the user U is received through communication from the user U, the assistant S may sometimes temporarily terminate communication with the user U and then attempt to communicate with the user U again when preparation for a response to that expectation is complete. Here, sometimes it is not a suitable situation to respond to a call from the assistant S because the user U is operating the vehicle 20 (e.g., manually driving).

[0036] In view of this, in order for the assistant S, including in the future, to easily know when the user U operating the vehicle 20 can communicate, in this embodiment, the user assistance system 10 notifies the assistant S of the status of the user U via the display device 12.

[0037] Specifically, the control device 15 (processor 16) acquires traffic condition information It for the predetermined movement path R of the vehicle 20 operated by the user U. Based on the acquired traffic condition information It, the control device 15 generates a communication permission information Ic, indicating whether the user U can communicate with the assistant S via the vehicle 20. Furthermore, the control device 15 notifies the assistant S of the communication permission information Ic by displaying it on the display device 12.

[0038] Figure 2This is a flowchart illustrating an example of a process related to user status notifications as described in this embodiment. The processing in this flowchart is repeatedly executed by the processor 16 of the user assistance system 10, for example, at a predetermined period. More specifically, the processing in this flowchart targets user status notifications from multiple users U. Furthermore, as described later... Figure 3 As shown, the user status notification to the assistant S using the display device 12 is displayed as a list L on the screen SC1 of the display device 12. The information displayed in the list L is updated, for example, according to the processing cycle of this flowchart.

[0039] In step S100, the processor 16 determines whether there is communication from a user U (hereinafter referred to as "object user Ux") who needs assistance from the assistant S to make a call. If the result is no communication from the object user Ux (step S100: No), the process proceeds to step S108, which will be described later. On the other hand, if the communication exists (step S100: Yes), the process proceeds to step S102.

[0040] In step S102, the processor 16 appends information related to the target user Ux whose communication was confirmed in step S100 to list L. For example, in step S102, the personal information of the target user Ux (e.g., facial photo, name) is first appended to list L. Then, the process proceeds to step S104.

[0041] In step S104, the processor 16 obtains the traffic condition information It of the vehicle 20 (hereinafter referred to as the object vehicle 20x) ridden by the aforementioned object user Ux. Specifically, as an example, the predetermined travel path R generated by each vehicle 20 is periodically sent to the cloud 40. In addition, the cloud 40 periodically obtains the traffic conditions of the area that is the object of user assistance involved in the user assistance system 10 from the traffic system 30. Moreover, the predetermined travel paths R of each vehicle 20 are stored in the database 41 of the cloud 40 in a state associated with the traffic conditions. The processor 16 obtains the traffic condition (traffic condition information It) of the predetermined travel path R of the object vehicle 20x from the cloud 40. Furthermore, the processor 16 can also obtain (determine) the traffic condition information It of the predetermined travel path R based on the predetermined travel path R directly obtained from the object vehicle 20x and the traffic conditions directly obtained from the traffic system 30.

[0042] In step S106 immediately following step S104, processor 16 displays the communication availability information Ic based on the acquired traffic condition information It in list L.

[0043] Specifically, the processor 16 first generates communication permission information Ic based on traffic condition information It. If the acquired traffic condition information It indicates "no traffic conditions that the target user Ux should be aware of during the operation of the target vehicle 20x," the processor 16 generates communication permission information Ic indicating that the target user Ux can communicate with the assistant S. Conversely, if the traffic condition information It indicates "there are traffic conditions that the target user Ux should be aware of during the operation of the target vehicle 20x," the processor 16 generates communication permission information Ic indicating that the target user Ux cannot communicate with the assistant S. Examples of traffic conditions that the target user Ux should be aware of include "a pedestrian crossing exists on the predetermined travel path R," "a pedestrian is crossing the pedestrian crossing on the predetermined travel path R," and "a traffic light is about to turn red on the predetermined travel path R." The processor 16 displays the generated communication permission information Ic in the list L of the display device 12.

[0044] Figure 3 This is a diagram showing a first example of the communication capability information Ic involved in this embodiment. Figure 3 This indicates the display of screen SC1 on display device 12. In Figure 3 In this example, the status of three target users Ux1 to Ux3 is represented as a list L. The display of each target user Ux includes the display of personal information (e.g., facial photo, name) of the target user Ux. In addition, in the first display example, the display of the status of each target user Ux includes displaying the communicable time period TA and the non-communicable time period TB as communication availability information Ic in time sequence. To further clarify, in the generation of the communication availability information Ic, the processor 16 applies the time period TB to the time period when there are traffic conditions on the predetermined travel path R that the target user Ux should pay attention to.

[0045] More specifically, in Figure 3 In the time series display shown, for example, each time interval TA and TB is displayed as a thin bar, accompanied by a time axis displaying the current time as 0 (e.g., displaying elapsed time at 5-second intervals). The longer the bar is displayed as a time interval, the longer it is. To further clarify, if... Figure 3 As shown, each time period TA and TB can be displayed along with a numerical value indicating the length of the time period. Additionally, as... Figure 3 As shown, the display of the communication information Ic can also include a marker (e.g., pedestrian marker M1, traffic light marker M2) that visually indicates why the target user Ux cannot communicate, and is associated with the time period TB.

[0046] In addition, Figure 3In the list L, the currently communicable target user Ux1 is positioned at the top. Next, the target user Ux2, whose communicable time period TA arrives slightly later than Ux1, is positioned next to Ux1. And at the bottom, the target user Ux3, whose time period TA arrives latest, is positioned. Thus, list L can be configured such that the earlier the communication opportunity (in other words, the higher the priority of the response by the assistant S), the higher the position of the target user Ux. Furthermore, when there are multiple currently communicable target users Ux, they can be displayed with the target user Ux that communicated their expectations to the assistant S earlier positioned higher.

[0047] To further clarify, in the first explicit example, the initial time period TA of object user Ux1 and the time periods TB of object users Ux2 and Ux3 respectively correspond to an example of the "current state information" involved in this disclosure. Furthermore, the time period TA of object users Ux2 and Ux3 who are currently unable to communicate corresponds to an example of the "future time information" involved in this disclosure.

[0048] Figure 4 This is a diagram illustrating an example of the additional user information Iad involved in this embodiment. Figure 4 The user information Iad shown is the highest priority target user Ux (in the response performed by assistant S). Figure 3 The middle section contains additional information related to Ux1. User information Iad can be displayed near the information of the object user Ux in list L, or it can be displayed outside of list L. For example... Figure 4 As illustrated, the user information Iad can contain textual information indicating the time remaining until the target user Ux is able to communicate (e.g., "Safe in 4.3 seconds"). Additionally, the user information Iad can also contain information about the conversation between the assistant S and the target user Ux. This conversation information could, for example, be information representing the expectations conveyed by the target user Ux to the assistant S (e.g., photos of candidate restaurants that the target user Ux wishes to book).

[0049] Figure 5 This is a diagram showing the traffic condition display Dt that is displayed along with the communication availability information Ic according to this embodiment. The display device 12 may include a list L containing the communication availability information Ic (for example, refer to...). Figure 3Simultaneously, screen SC2 displays the traffic condition display Dt. The traffic condition display Dt includes an overhead view showing the current surrounding conditions of the target vehicle 20x operated by the target user Ux. For example, this overhead view is generated by the cloud 40 based on multiple images taken by cameras that capture the surrounding conditions of the target vehicle 20x (e.g., cameras mounted on vehicle 20, infrastructure cameras). The processor 16 displays the overhead view obtained from the cloud 40 on screen SC2. Figure 5 The overhead view shown includes the road 50 on which the target vehicle 20x travels, the pedestrian crossing 51 ahead of the target vehicle 20x, pedestrians 52 crossing the pedestrian crossing 51, and traffic lights 53 ahead of the target vehicle 20x. Additionally, the overhead view includes markers M3 and M4 indicating various locations along the predetermined movement path R of the target vehicle 20x. Marker M3 indicates locations where there is no traffic condition that the target user Ux should be aware of (i.e., locations corresponding to the communication time period TA). On the other hand, marker M4 indicates locations where there is traffic condition that the target user Ux should be aware of (i.e., locations corresponding to the non-communication time period TB). The marker for each location is determined based on the traffic condition information It along the predetermined movement path R. For example, the aforementioned time periods TA and TB are determined based on the information of markers M3 and M4 for each location, and the location and speed information of the target vehicle 20x obtained from the target vehicle 20x via the cloud 40. Figure 5 As shown, the speed information of the target vehicle 20x can be overlaid on the overhead image. Through this traffic condition display Dt, the operator S of the user assistance system 10 can more easily determine, including future timeframes, when the target user Ux operating the target vehicle 20x can communicate.

[0050] Figure 6 This is a diagram illustrating a second example of the communication capability information Ic according to this embodiment. The second example differs from the first example in the following points.

[0051] That is, in the second display example, for the currently communicable target user Ux1, current status information indicating that communication is currently possible (e.g., a text message) is displayed as communication capability information Ic. In other words, the display of current status information in the second display example indicates that the time when communication with target user Ux1 is possible is the current moment. On the other hand, for each of the currently non-communicable target users Ux2 and Ux3, current status information indicating that communication is currently impossible (e.g., a text message) and future time information indicating the time when communication is possible (e.g., the time until communication becomes possible) are displayed as communication capability information Ic. Figure 6In the example, "Communication will be possible in 16.3 seconds" and "Communication will be possible in 17.3 seconds" are shown. In other words, the display of future time information in the second example indicates the future moments when communication with target users Ux2 and Ux3 will be possible, respectively. Another example of future time information could be the moment when target user Ux becomes capable of communication. For example, this moment could be displayed together with the current moment in list L.

[0052] exist Figure 2 In step S108, immediately following step S106, the processor 16 determines whether communication with the target user Ux displayed in list L is complete. If communication has occurred with the target user Ux to whom the assistant S needs a response, then communication with that target user Ux is complete. If communication is complete (step S108: Yes), the processor 16 removes the target user Ux from list L (step S110). On the other hand, if no communication is complete (step S108: No), the process returns.

[0053] 3. Effects

[0054] As explained above, according to this embodiment, the communication capability information Ic of the target user Ux displayed on the display device 12 for the purpose of assisting the personnel S includes "current status information" and "future time information," where the future time information indicates the time when the target user Ux will be able to communicate if it is currently unable to do so. Therefore, the personnel S can easily understand, including the future, when the target user Ux operating the vehicle 20x can communicate. Furthermore, the display of the communication capability information Ic according to this embodiment allows the future safety of one or more target users Ux (drivers) who are the communication targets to be easily visualized for the personnel S.

[0055] Additionally, as in Figure 3 As illustrated in the example, the display of the communication availability information Ic can be a time series display of the communication-enabled time period TA and the communication-incapable time period TB. Thus, the assistant S can visually and easily determine the time when communication with each target user Ux is possible.

[0056] Furthermore, according to this embodiment, in generating the communication availability information Ic, the time period during which the target user Ux should be aware of traffic conditions on the predetermined movement path R can be applied to the time period TB during which the target user Ux cannot perform communication. Thus, the assistant S can attempt to communicate with the target user Ux at an appropriate time without hindering the target user Ux's operation of the target vehicle 20x.

Claims

1. A user status notification method, executed by a computer, wherein, The user status notification method includes: This process involves obtaining traffic conditions for a user-operated mobile vehicle's predetermined movement path. The process of generating communication capability information based on the obtained traffic conditions indicates whether the user can communicate with an assistant via the mobile body; and The process of displaying the generated communication capability information on a display device for the assistive personnel. The communication capability information includes: Current status information, indicating whether the user can perform the communication now; and Future time information indicates the time when the user will become able to perform the communication if the user cannot perform the communication now.

2. The user status notification method according to claim 1, wherein, The process of displaying the communication availability information on the display device includes displaying the time periods during which the user can perform the communication and the time periods during which the user cannot perform the communication in a time sequence.

3. The user status notification method according to claim 2, wherein, The process of generating the communication availability information includes applying the time period during which the user should be aware of traffic conditions on the predetermined travel path to the time period during which the user cannot perform the communication.

4. The user status notification method according to any one of claims 1 to 3, wherein, The user status notification method includes, in response to a situation where the assistant has communicated with a target user who requires a response, performing the process of displaying the communication permission information to the display device, treating the target user as the target.

5. A user assistance system, wherein, The user assistance system includes: A display device for displaying information to assist users in operating a moving object; and One or more processors The one or more processors acquire traffic conditions along the predetermined movement path of the mobile body, generate communication capability information based on the acquired traffic conditions, and display the generated communication capability information on the display device. This communication capability information indicates whether the user can communicate with the assistant via the mobile body. The communication capability information includes: Current status information, indicating whether the user can perform the communication now; and Future time information indicates the time when the user will become able to perform the communication if the user cannot perform the communication now.

Images