Guidance output method and guidance output device

Abstract

To provide a guidance output method and a guidance output device that can output guidance messages using objects around a user in accordance with a line of sight of the user.SOLUTION: A guidance output device, based on surrounding information including a surrounding environment of a user and gaze information of the user, calculates a graph structure which represents surrounding objects located around the user and relations between the surrounding objects and in which at least one surrounding object located in a gaze direction of the user is weighted, generates a guidance message using the surrounding objects based on the graph structure, and outputs the guidance message.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a guidance output method and a guidance output device. [Background technology]

[0002] A technology is known in which attributes are assigned to each of a number of landmarks and stored, the driving environment of the vehicle is recognized, a landmark that has been assigned an attribute suitable for the recognized driving environment is selected from the multiple landmarks, and the selected landmark is used to guide the vehicle along its route (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 7-98800 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 has the problem that the attributes of the landmarks used for route guidance for a specific driving environment are determined in advance, and therefore it is not possible to output guidance messages using objects around the user in accordance with the user's line of sight.

[0005] An object of the present invention is to provide a guidance output method and a guidance output device capable of outputting a guidance message using an object around the user in accordance with the user's line of sight. [Means for solving the problem]

[0006] The present invention solves the above problem by calculating a graph structure that represents surrounding objects located around a user and the relationships between the surrounding objects based on surrounding information including the user's surrounding environment and the user's gaze information, in which a graph structure is weighted for at least one surrounding object located in the user's gaze direction, generating a guidance message using the surrounding objects based on the graph structure, and outputting the guidance message. Effect of the Invention

[0007] According to the present invention, a guidance message using objects around the user can be output in accordance with the user's line of sight. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram of a route guidance system including a guidance output device according to an embodiment of the present invention. [Diagram 2] 4 is a diagram showing an example of calculating a gaze weighting graph structure (scene graph) from a surrounding image in the present embodiment. FIG. [Diagram 3] 4 is a flowchart illustrating an example of a control procedure for executing a guidance output method according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a guidance output method and a guidance output device according to the present invention will be described using an example in which the method and device are applied to a route guidance system.

[0010] FIG. 1 is a block diagram of a route guidance system 100 including a guidance output device 10 according to the present embodiment. As shown in FIG. 1, the route guidance system 100 includes a vehicle 1, a position acquisition device 2, an external sensor 3, a line of sight estimation device 4, a database 5, and a guidance output device 10. The position acquisition device 2, the external sensor 3, the line of sight estimation device 4, the database 5, and the guidance output device 10 are mounted on the vehicle 1, and exchange information with each other via an in-vehicle network such as a CAN. In the route guidance system 100, the guidance output device 10 outputs a guidance message for providing route guidance from the user's current position to a destination to a user who is riding in the vehicle 1. The guidance output device 10 is, for example, a navigation device mounted on the vehicle 1.

[0011] In this embodiment, the guidance output device 10 is described as an in-vehicle navigation device, but it is not limited thereto and may be a terminal device carried by a user. That is, route guidance may be provided when the user does not get in the vehicle 1 but travels on foot, bicycle, or the like. In addition, in this embodiment, a guidance output device that outputs a guidance sentence for route guidance is described as an example, but it is not limited thereto and may be a guidance output device that outputs a guidance sentence for providing guidance regarding objects around the user. For example, the guidance output device 10 may output a guidance sentence for providing tourist guidance regarding surrounding objects such as tourist spots.

[0012] In this embodiment, when the vehicle 1 is approaching a guidance point, the guidance output device 10 outputs a guidance sentence including a traveling direction at the guidance point to the user via the output device 13. The guidance point is a point where the user needs route guidance, and is, for example, a point where there are two or more accessible routes, such as an intersection, a branch point, or an interchange on a highway. If necessary, a point other than a point where there are two or more accessible routes can be set as a guidance point for accurately guiding the user's traveling direction. That is, the guidance output device 10 can determine and use a point as a guidance point for accurately guiding the traveling direction of the vehicle in advance. The traveling direction at a guidance point is, for example, a right turn, a left turn, or a straight line.

[0013] The guidance text is, for example, text such as "Follow the ambulance ahead and go straight through the intersection." In this embodiment, the guidance text includes a phrase indicating the guidance point and a phrase indicating the traveling direction at the guidance point. In the above example, the guidance point is "intersection" and the traveling direction at the guidance point is "go straight." In addition, the guidance text includes a phrase using objects around the user as landmarks to more easily convey the guidance point and the traveling direction to the user. The phrase includes, for example, surrounding objects and the user's actions with respect to the surrounding objects. In the above example, the phrase using objects around the user as landmarks is "follow the ambulance ahead." The guidance text is generated using a trained model, as described below.

[0014] The position acquisition device 2 acquires position information of the user. In this embodiment, the position acquisition device 2 acquires the position information of the vehicle 1 as the user's position information. The position acquisition device 2 acquires the position information of the vehicle 1 from, for example, a GPS satellite. An example of the position information of the vehicle 1 is latitude and longitude information. The position acquisition device 2 acquires the position information from the GPS satellite at a predetermined cycle. While the user is traveling in the vehicle 1, the user's position information changes as the vehicle 1 moves.

[0015] The vehicle exterior sensor 3 detects the surrounding environment including surrounding objects located around the user. The vehicle exterior sensor 3 may be, for example, a camera installed in the vehicle 1, and may be a camera equipped with an image sensor such as a CCD or a CMOS. The vehicle exterior sensor 3 acquires a surrounding image capturing the surrounding environment including the surrounding objects. For example, the vehicle exterior sensor 3 detects the surrounding environment ahead of the vehicle 1 in the traveling direction at a predetermined cycle. The surrounding objects are landmarks in the surrounding environment around the user. For example, the surrounding objects include advertising billboards, signs, stop lines, traffic lights, pedestrian crossings, traffic restrictions, road trees, buildings, road structures, automobiles (other vehicles) other than the vehicle itself, motorcycles, bicycles, pedestrians, and the like. That is, the surrounding objects may include static objects and dynamic objects. Road structures include bridges, tunnels, and the like. The surrounding objects also include landmarks. The surrounding objects also include objects stored in the map data and objects not stored in the map data. The objects stored in the map data include, for example, landmarks such as convenience stores, restaurants, and gas stations. Objects that are not stored in the map data are, for example, static objects such as buildings and road structures, and dynamic objects such as automobiles (other vehicles) other than the vehicle itself.

[0016] The external sensor 3 detects the direction in which the surrounding object exists and the distance to the surrounding object. For example, the external sensor 3 is a millimeter wave radar, a laser radar, an ultrasonic radar, a laser range finder, etc. The external sensor 3 detects the relative speed (including the moving direction) of the surrounding object with respect to the vehicle based on the direction in which the surrounding object exists and the change over time in the distance to the surrounding object. Surrounding information including the surrounding environment detected by the external sensor 3 is output to the guidance output device 10 at predetermined intervals.

[0017] The gaze estimation device 4 performs a gaze estimation process for a user. In this embodiment, the gaze estimation device 4 is a trained model for gaze estimation that has been trained to output gaze information of a user. When input data including surrounding information is input, the gaze estimation device 4 performs a gaze estimation process by outputting gaze information including the gaze direction in which the user is directing his / her gaze toward the surrounding environment. The estimated gaze information is output to the guidance output device 10. The trained model for gaze estimation is trained by a gaze estimation network based on teacher data including surrounding information and gaze information including the gaze direction of the user toward the surrounding environment.

[0018] For example, the teacher data is a data set in which surrounding images (surrounding images for learning) in a plurality of driving scenes obtained by simulation and / or experiment are associated with gaze information including the gaze direction in which the driver directs his / her gaze toward the surrounding environment contained in the surrounding images. The gaze information used for learning is not limited to the gaze direction in which the driver directs his / her gaze during normal driving, but may be the gaze direction in which the driver intentionally guides his / her gaze toward an object that is easy to focus on while navigating. This makes it possible to estimate the gaze direction when the user directs his / her gaze toward a surrounding object suitable for navigation.

[0019] For example, a trained model for gaze estimation is the Dilated Inception Network (DINet). DINet is an encoder-decoder network that uses the surrounding image as an input image, estimates the gaze direction from the features of the input image, and learns the gaze direction in the surrounding image by comparing the estimated gaze direction with the gaze direction included in the teacher data. DINet uses the Dilated Residual Network (DRN) as an encoder, captures multi-scale features with the Dilated Inception Module (DIM), and generates a saliency map with the decoder. DRN is a network that introduces dilated convolution processing to ResNet-50, a Residual Network (ResNet) with 50 convolution layers. DRN is a pre-trained network that is often used for gaze estimation, and can acquire feature maps with a wide receptive field compared to VGG-16, which is also often used for gaze estimation. DRN also requires less computational effort than VGG-16. DIM is a network that introduces dilated convolution processing to the Inception module proposed in GoogLeNet. By replacing each convolution process in the Inception module with a dilated convolution process with a different dilation rate, it is possible to achieve the same or higher accuracy with a smaller number of parameters and a shorter training time.

[0020] In this embodiment, the user's gaze direction with respect to the surrounding environment is the position of the gaze area where the user is gazing in the surrounding image captured from the surrounding environment. The gaze area is specified as a rectangular area. For example, the gaze estimation device 4 uses the surrounding image as an input image and outputs a heat map that represents the degree of gaze of the user in the surrounding image with a shade of color. In the heat map, the higher the degree of gaze, the darker the color is displayed. The gaze estimation device 4 specifies, as the gaze area, a circumscribing rectangle that circumscribes an area on the heat map of the surrounding image where the degree of gaze is equal to or greater than a predetermined threshold. The position of the gaze area on the surrounding image is output to the guidance output device 10 as gaze information.

[0021] The database 5 is a database that stores various types of information. The database 5 stores map information. The map information includes at least road information. The road information includes the type of road (e.g., general road, toll road, etc.) and the locations of intersections, stop lines, traffic lights, and road signs. The map information also includes information on landmarks such as convenience stores, restaurants, gas stations, etc. The map information also includes location information of guidance points that require route guidance.

[0022] Further, the database 5 stores the trained model. The trained model to be stored may be a trained model for guidance that has been trained to output a guidance message. When the controller 12 acquires a trained model, the controller 12 stores the acquired trained model in the database 5. In this embodiment, the controller 12 acquires a trained model by generating the trained model using the generation unit 103, but this is not limited thereto. The controller 12 may acquire a trained model from an external server via the communication device 14, or may acquire the input trained model when data of the trained model is input to the input device 11 by a user.

[0023] For example, the external server is a device that manages the trained model. The server generates a trained model based on teacher data and stores the trained model. A specific generation method is, for example, a generation method in the generation unit 103. The external server can exchange information with the guidance output device 10 via a network, and transmits the trained model to the guidance output device 10. The network refers to a telecommunication line network such as the Internet or a LAN, and is not particularly limited as long as information can be exchanged between the guidance output device 10 and the server, and any known network can be used. The trained model stored in the database 5 is used when the controller 12 generates a guidance message.

[0024] The database 5 also stores vehicle information of the vehicle 1. The vehicle information includes travel information of the vehicle 1. The travel information of the vehicle 1 includes position information of the vehicle 1 and vehicle speed information of the vehicle 1. The position information of the vehicle 1 is updated at regular intervals. In this embodiment, the database 5 stores travel information of the vehicle 1 approaching a guidance point. If a destination is set for the vehicle 1, the vehicle information may include information on the destination of the vehicle 1. The vehicle information also includes route information set for the vehicle 1. The vehicle route is the route from the current position of the vehicle to the destination.

[0025] The guidance output device 10 is a device that outputs a guidance message for providing a user with route guidance to a destination. The guidance output device 10 includes an input device 11, a controller 12, an output device 13, and a communication device 14. In the guidance output device 10, the controller 12 acquires information from the position acquisition device 2, the vehicle exterior sensor 3, the line of sight estimation device 4, the database 5, the input device 11, and the communication device 14, generates a guidance message based on the acquired information, and outputs the guidance message to the user via the output device 13, thereby providing the user with route guidance to the destination.

[0026] The input device 11 is an input interface that accepts input from a user. The input device 11 is composed of one or more devices. For example, the input device 11 is composed of a steering switch that allows the user to input, and a microphone that allows the user to input voice. The input device 11 may include a touch panel or a keyboard. The input information that the user inputs to the input device 11 is, for example, the user's destination. The input information may also include a trained model.

[0027] The output device 13 is an output interface that outputs information to the user. The information output to the user is, for example, route information. The output device 13 displays a map, a route to a destination, a current location, etc., on a screen to the user, and also displays and / or outputs a guidance message generated by the guidance output device 10 as audio. The output device 13 is composed of one or more devices. For example, the output device 13 is composed of a display that displays on a screen and a speaker that outputs audio data. Examples of the display include a liquid crystal panel and an organic EL panel. Note that, when a touch panel display is used as the output device 13, it can be used in combination with the input device 11.

[0028] The communication device 14 is a device that exchanges information with the outside of the vehicle 1 via a network. The communication device 14 receives a trained model from, for example, an external server.

[0029] The controller 12 is equipped with a computer having hardware and software, and this computer includes a ROM storing a program, a CPU executing the program stored in the ROM, and a RAM functioning as an accessible storage device. Note that, as the operating circuit, an MPU, a DSP, an ASIC, an FPGA, etc. can be used instead of or together with the CPU. As shown in FIG. 1, the controller 12 is configured to include an acquisition unit 101, a graph structure calculation unit 102, a generation unit 103, and an output unit 104 as functional blocks, and executes each function by cooperation between the software for executing each process and the hardware. Note that, in this embodiment, the functions of the controller 12 are divided into four blocks, and the functions of each functional block are described, but the functions of the controller 12 do not necessarily need to be divided into four blocks.

[0030] The acquisition unit 101 performs an information acquisition process to acquire various kinds of information. In the information acquisition process, the acquisition unit 101 acquires surrounding information including the surrounding environment of the user. The acquisition unit 101 acquires, from the vehicle exterior sensor 3, surrounding information detected by the vehicle exterior sensor 3. The surrounding information is, for example, a surrounding image including surrounding objects.

[0031] The acquisition unit 101 also acquires the user's gaze information. The user's gaze information includes the user's gaze direction with respect to the surrounding environment. The acquisition unit 101 acquires the user's gaze information estimated by the gaze estimation device 4 from the gaze estimation device 4. For example, the acquisition unit 101 inputs input data including the acquired surrounding information to the gaze estimation device 4, and acquires the gaze information by having the gaze estimation device 4 output gaze information including the user's gaze direction with respect to the surrounding environment as output data corresponding to the input data. The user's gaze direction with respect to the surrounding environment is the direction in which the user is looking in the surrounding environment included in the surrounding image.

[0032] In the present embodiment, the acquisition unit 101 acquires surrounding information and gaze information when the vehicle 1 in which the user rides approaches a guide point. The acquisition unit 101 extracts guide points located on the route based on route information and map information. When the vehicle 1 starts traveling along the route, the acquisition unit 101 acquires the current position of the vehicle 1 from the position acquisition device 2 at regular intervals, and determines whether the vehicle 1 approaches the guide point based on the current position of the vehicle 1 and the position of the guide point on the route. For example, the acquisition unit 101 determines that the vehicle 1 approaches the guide point when the vehicle 1 reaches a range within a predetermined distance from the guide point. The predetermined distance is, for example, a distance that corresponds to a distance detectable by the outside-vehicle sensor 3. When it is determined that the vehicle 1 approaches the guide point, the acquisition unit 101 acquires surrounding information and gaze information. That is, the acquisition unit 101 acquires a surrounding image including the surrounding environment of the guide point located ahead in the traveling direction of the vehicle 1, and gaze information including the user's gaze direction with respect to the surrounding environment of the guide point.

[0033] Furthermore, the acquisition unit 101 may acquire surrounding images and gaze information for the surrounding images at a fixed cycle for a predetermined time from the time when it is determined that the vehicle 1 is approaching the guidance point. The predetermined time is, for example, a time from when acquisition of surrounding images starts until a fixed number of surrounding images (fixed number of frames) are acquired. In this embodiment, the acquisition unit 101 acquires time-series information including two or more surrounding images that are consecutive in chronological order and gaze information of the user for the surrounding images at each time point for a predetermined time from the time when it is determined that the vehicle 1 is approaching the guidance point.

[0034] The graph structure calculation unit 102 performs a graph structure calculation process to calculate a graph structure based on the surrounding information and the line of sight information. The graph structure is a graph structure that expresses surrounding objects located around the user and the relationships between the surrounding objects. The graph structure has surrounding objects as nodes and relationships between the surrounding objects as edges between the nodes. The graph structure is a so-called scene graph. In the graph structure calculation process, the graph structure calculation unit 102 performs an image recognition process on the surrounding image to recognize surrounding objects included in the surrounding image. In addition, the graph structure calculation unit 102 connects the nodes of nearby surrounding objects with edges. For example, the graph structure calculation unit 102 calculates the distance between the surrounding objects, and connects the surrounding objects whose distance is equal to or less than a predetermined threshold with edges. The nodes include, for example, information on the object class, position, type, and color of the surrounding objects. The object classes are, for example, the user's vehicle, another vehicle, and a pedestrian. The edges include, for example, information on the relative distance between the surrounding objects.

[0035] In this embodiment, the graph structure is a scene graph, and is a graph structure in which at least one surrounding object located in the user's gaze direction is weighted (hereinafter, also referred to as a gaze-weighted graph structure). The graph structure calculation unit 102 estimates a surrounding object that is a gaze target that the user is gazing at in the surrounding environment, based on the gaze information of the user estimated from the surrounding image using the gaze estimation device 4. For example, the graph structure calculation unit 102 estimates a surrounding object corresponding to the gaze area among the surrounding objects included in the surrounding image as a gaze target, based on a gaze area acquired as the gaze information. The surrounding object corresponding to the gaze area is a surrounding object at least a part of which is included in the gaze area in the surrounding image. The gaze target may be one or more. The graph structure calculation unit 102 calculates a graph structure in which a weight is assigned to a node of the surrounding object that is a gaze target.

[0036] Here, an example of calculating a gaze-weighted graph structure (scene graph) from a surrounding image will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of calculating a gaze-weighted graph structure (scene graph) from a surrounding image in this embodiment. FIG. 2 shows a surrounding image SI acquired by an outside-vehicle sensor 3, an overhead view B1 converted from the surrounding image S1, and a graph structure diagram SG with surrounding objects included in the surrounding image as nodes. For example, the scene in FIG. 2 is a scene in which another vehicle V1 is present ahead of the host vehicle V0 in the traveling direction when the host vehicle V0 approaches an intersection GP that is a guide point, as shown in the overhead view BI. In this scene, the driver of the host vehicle V0 turns his / her gaze to the other vehicle V1. That is, the other vehicle V1 is the gaze target. In the graph structure calculated for the surrounding image SI, the node N1 corresponding to the other vehicle V1 is weighted. As described above, when the graph structure calculation unit 102 acquires the surrounding image SI, it calculates a graph structure SG in which the node N1 is weighted as a graph structure corresponding to the surrounding image S1.

[0037] In addition, in this embodiment, the graph structure calculation unit 102 may calculate the graph structure in chronological order based on the surrounding information and the gaze information. That is, the graph structure calculation unit 102 calculates the gaze-weighted graph structure at each time point based on the time series information including a plurality of surrounding images acquired in chronological order and gaze information for each surrounding image. That is, the gaze-weighted graph structure corresponding to the surrounding image at each time point is calculated in chronological order. The graph structure calculation unit 102 adds an edge connecting the surrounding object included in the graph structure at each time point with the same surrounding object included in the graph structure at another time point adjacent in chronological order to calculate a graph structure (scene temporal graph) based on the time series information. As a result, a graph considering the time series change of the positional relationship of each surrounding object is calculated.

[0038] The generation unit 103 performs a generation process to generate a guidance message using surrounding objects based on the graph structure calculated by the graph structure calculation unit 102. The guidance message is text for providing route guidance so that the vehicle 1 travels along a route from the current position of the vehicle 1 to the destination. For example, the guidance message is text including a traveling direction in which the vehicle 1 should travel at the guidance point. The graph structure used to generate the guidance message is a gaze-weighted graph structure. The graph structure used to generate the guidance message may be a gaze-weighted graph structure based on time-series information.

[0039] In the generation process, first, the generation unit 103 calculates the feature amount of the graph structure. For example, the generation unit 103 inputs the graph structure as input data to a graph convolutional neural network (GCN) and uses the GCN to output the feature amount of the graph structure. The feature amount of the graph structure is the feature amount of each node included in the graph structure.

[0040] Furthermore, when the graph structure used to generate the guidance message is a gaze-weighted graph structure based on time-series information, the generation unit 103 may use a spatio-temporal graph convolutional neural network (ST-GCN) to calculate the feature amount of the graph structure. For example, the generation unit 103 inputs the gaze-weighted graph structure based on time-series information to the ST-GCN, and uses the ST-GCN to output the feature amount of the graph structure. This makes it possible to obtain feature amounts that take into account the movement of each surrounding object.

[0041] Next, the generating unit 103 generates a guidance sentence based on the calculated feature amount of the graph structure. The generating unit 103 inputs input data including the feature amount of the gaze weighting graph structure to a trained model for guidance, and generates a guidance sentence by outputting a guidance sentence from the trained model for guidance as output data corresponding to the input data. The trained model for guidance is a trained model that has been trained to generate a guidance sentence using a gaze target from the feature amount of the gaze weighting graph structure, and is, for example, a Transformer. The generating unit 103 inputs the feature amount of each node included in the graph structure to a Transformer, and generates a guidance sentence by outputting a guidance sentence corresponding to each surrounding object from the Transformer.

[0042] The trained model for guidance may generate the guidance sentence using a neural network that has undergone end-to-end training. The neural network may be a network that learns from the surrounding information in the input layer to the guidance sentence in the output layer all at once. The trained model for guidance is a trained model that has been trained to output the guidance sentence based on training data including the surrounding information and the guidance sentence. The trained model for guidance calculates a graph structure based on the surrounding information included in the training data, and learns based on the feature amount of the calculated graph structure. For example, in the trained model for guidance, GCN is used to calculate the feature amount of the graph structure, and Transformer is used to generate the guidance sentence.

[0043] The teacher data is a data set in which surrounding images (learning surrounding images) in a plurality of driving scenes acquired by simulation and / or experiment are associated with guidance sentences using each surrounding object included in the surrounding image. In addition, for each surrounding image, a gaze target included in the surrounding image is identified by gaze estimation. The guidance sentence included in the teacher data includes a correct guidance sentence using the gaze target identified in the surrounding image. The gaze target may be a surrounding object to which the driver directs his / her gaze during normal driving, or may be a gaze target suitable for navigation. As described above, in this embodiment, the learned model for guidance is a learned model that has been trained to generate guidance sentences using gaze targets included in the surrounding image from the surrounding image, and when a surrounding image is input, outputs guidance sentences using gaze targets estimated from the surrounding image.

[0044] In this embodiment, the learned model for guidance is generated by the generation unit 103 in the controller 12, but this is not limited to the above, and the model may be generated by an external server and obtained via the communication device 14, or may be input by the user to the input device 11 and obtained.

[0045] As described above, in this embodiment, the guidance output device 10 generates a guidance message based on a scene graph using line-of-sight information and time-series information of the surrounding image at the guidance point, so that the guidance message can be generated according to the situation around the guidance point. This makes it easy for the driver to intuitively understand the traveling direction at the guidance point from the output guidance message.

[0046] In particular, in this embodiment, by using gaze information, it is possible to generate a guidance sentence that matches the gaze of the user. That is, since the trained model is trained to generate a guidance sentence using a surrounding object located in the gaze direction of the user, the guidance output device 10 can generate a guidance sentence that matches the gaze of the user using the trained model. Also, in this embodiment, since the trained model is trained using time-series information, the guidance output device 10 can generate a guidance sentence that uses a dynamic object as a landmark using the trained model. As an example, when a vehicle approaches an intersection and another vehicle is traveling ahead of the vehicle, a guidance sentence including following the other vehicle traveling ahead and proceeding straight through the intersection is generated.

[0047] The output unit 104 outputs the guidance message generated by the generation unit 103. For example, the output unit 104 transmits a control instruction to the output device 13 to cause the output device 13 to output the guidance message.

[0048] Next, a control procedure of the guidance output method executed by the guidance output device 10 will be described with reference to Fig. 3. Fig. 3 is a flow chart showing an example of a control procedure for executing the guidance output method according to this embodiment. In this embodiment, when the vehicle 1 starts traveling along a route, the controller 12 starts the control flow from step S101.

[0049] In step S101, the controller 12 acquires the current position of the vehicle 1 from the position acquisition device 2. In step S102, the controller 12 determines whether or not the vehicle 1 is approaching a guide point on the route based on the current position of the vehicle 1 and the map information. If it is determined that the vehicle 1 is approaching the guide point, the controller 12 proceeds to step S103. If it is determined that the vehicle 1 is not approaching the guide point, the controller 12 returns to step S101 and repeats the following flow.

[0050] In step S103, the controller 12 acquires surrounding information including surrounding objects around the vehicle 1 from the outside sensor 3. The surrounding information is, for example, a surrounding image. The surrounding image includes surrounding objects located around a guidance point ahead in the traveling direction of the vehicle 1. In step S104, the controller 12 acquires user's gaze information. For example, the controller 12 acquires the gaze information by estimating the user's gaze direction with respect to the surrounding image.

[0051] In step S105, the controller 12 calculates a graph structure based on the surrounding information acquired in step S103 and the gaze information acquired in step S104. The calculated graph structure is a graph structure that expresses surrounding objects located around the user and relationships between the surrounding objects, and is a graph structure in which at least one surrounding object located in the gaze direction of the user is weighted. In this embodiment, the graph structure may be a gaze-weighted graph structure based on time-series information. In step S106, the controller 12 generates a guidance sentence based on the graph structure. For example, the controller 12 inputs the graph structure to a GCN to calculate a feature amount of the graph structure. When the graph structure is a gaze-weighted graph structure based on time-series information, an ST-GCN is used to calculate the feature amount of the graph structure. The controller 12 generates the guidance sentence by inputting the calculated feature amount of the graph structure to a Transformer and outputting the guidance sentence. In step S107, the controller 12 outputs the guidance sentence generated in step S106 via the output device 13. For example, the controller 12 transmits a control instruction to output a guidance message to the output device 13. Based on the control instruction, the output device 13 displays the guidance message on a screen and / or outputs the guidance message by voice.

[0052] As described above, the guidance output method and guidance output device according to this embodiment acquire surrounding information including the user's surrounding environment and the user's gaze information, calculate a graph structure that represents surrounding objects located around the user and relationships between the surrounding objects based on the surrounding information and the gaze information, weighting at least one of the surrounding objects located in the user's gaze direction, generates a guidance message using the surrounding objects based on the graph structure, and outputs the generated guidance message. This makes it possible to output a guidance message using the objects around the user in accordance with the user's gaze.

[0053] In addition, the guidance output method and the guidance output device according to the present embodiment acquire the surrounding information and the line-of-sight information at a constant cycle, calculate the graph structure in chronological order based on the surrounding information and the line-of-sight information, and generate the guidance sentence based on the graph structure calculated in chronological order, thereby making it possible to generate the guidance sentence using the object that the user is focusing on with higher accuracy.

[0054] Furthermore, the guidance output method and the guidance output device according to the present embodiment calculate features of the graph structure and generate the guidance sentence based on the calculated features, thereby making it possible to generate the guidance sentence using the target that the user is focusing on with higher accuracy.

[0055] Furthermore, the guidance output method and guidance output device according to this embodiment generate the guidance by inputting input data including the calculated feature values ​​into a trained model for guidance and outputting the guidance sentence from the trained model for guidance as output data corresponding to the input data, the trained model for guidance being a trained model that has been trained to output the guidance sentence based on training data including the surrounding information and the guidance sentence, calculating the graph structure based on the surrounding information included in the training data, and learning based on the feature values ​​of the calculated graph structure. This makes it possible to generate guidance sentences using an object that the user is focusing on with greater accuracy.

[0056] Furthermore, the guidance output method and guidance output device according to this embodiment acquire the gaze information by inputting input data including the surrounding information to a trained model for gaze estimation and outputting the gaze information including the gaze direction of the occupant with respect to the surrounding environment from the trained model for gaze estimation as output data corresponding to the input data, and the trained model for gaze estimation is a trained model trained to output the gaze information of the occupant based on teacher data including the surrounding information and the gaze information including the gaze direction of the occupant with respect to the surrounding environment. This makes it possible to acquire the gaze information of the user from the surrounding information.

[0057] The above-described embodiments are described for the purpose of facilitating understanding of the present invention, and are not described for the purpose of limiting the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]

[0058] 100...Route guidance system 10...Guidance output device 11...Input device 12…Controller 101…Acquisition Department 102...Graph structure calculation unit 103...Generation section 104...Output section 13...Output device 14...Communication equipment 2...Position acquisition device 3…External sensor 4... Gaze estimation device 5. Database

Claims

1. A guidance output method executed by a guidance output device, comprising: The guidance output device is Acquire surrounding information including a surrounding environment of a user and line of sight information of the user; Calculating a graph structure that represents surrounding objects located around the user and relationships between the surrounding objects based on the surrounding information and the gaze information, the graph structure being weighted for at least one of the surrounding objects located in the gaze direction of the user; generating a guide message using the surrounding objects based on the graph structure; A guidance output method for outputting the generated guidance sentence.

2. The guidance output method according to claim 1, The guidance output device is The surrounding information and the line of sight information are acquired at a constant period. Calculating the graph structure in chronological order based on the surrounding information and the line of sight information; The guidance output method generates the guidance sentence based on the graph structure calculated in chronological order.

3. The guidance output method according to claim 1 or 2, The guidance output device is Calculating a feature amount of the graph structure; The guidance output method generates the guidance sentence based on the calculated feature amount.

4. The guidance output method according to claim 3, The guidance output device is Inputting input data including the calculated feature amount into a trained model for guidance; generating the guidance sentence by outputting the guidance sentence from the guidance trained model as output data corresponding to the input data; The trained model for guidance is a trained model that has been trained to output the guidance sentence based on training data including the surrounding information and the guidance sentence, and the guidance output method calculates the graph structure based on the surrounding information included in the training data, and learns based on the features of the calculated graph structure.

5. The guidance output method according to claim 1 or 2, The guidance output device is inputting input data including the surrounding information into a trained model for gaze estimation; acquiring the gaze information by outputting, from the trained model for gaze estimation, the gaze information including the gaze direction of the user with respect to the surrounding environment as output data corresponding to the input data; A guidance output method, in which the trained model for gaze estimation is a trained model trained to output the user's gaze information based on teacher data including the surrounding information and the gaze information including the user's gaze direction relative to the surrounding environment.

6. An acquisition unit that acquires surrounding information including a surrounding environment of a user and line of sight information of the user; a graph structure calculation unit that calculates a graph structure that represents surrounding objects located around the user and relationships between the surrounding objects based on the surrounding information and the gaze information, the graph structure being weighted with at least one of the surrounding objects located in the gaze direction of the user; A generation unit that generates a guidance sentence using the surrounding objects based on the graph structure; and an output unit that outputs the generated guidance sentence.

Images