Object guidance control device, object guidance control method, and program

Abstract

To provide a target guiding control device, a target guiding control method, and a program capable of improving the accuracy of a guide about a guide target designated as a target whose guide is requested by a user getting on a vehicle.SOLUTION: A target guiding control device includes: a detection part for detecting candidates for guide target designated by a user on the basis of a voice of the user getting in a vehicle, an operation for designating the guide target being a target whose guide is requested by the user, and a picked up image around the vehicle; a specification part for specifying a position and a direction of the vehicle in space information around the vehicle on the basis of position information of the vehicle, specifying the direction of the guide target in the space information with the specified direction of the vehicle as reference, and specifying the guide target from the candidates on the basis of the specified direction; and an output control part for outputting output information on the guide target on the basis of the space information corresponding to the specified guide target.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an object guidance control device, an object guidance control method, and a program. [Background technology]

[0002] Conventionally, guide systems that provide route guidance and guidance of surrounding facilities in vehicles have become widespread. Car navigation devices equipped with such guide systems are often used for route guidance in vehicles such as automobiles. In addition, such guide systems are also used for providing tourist information to tourists.

[0003] For example, the following Patent Document 1 discloses a guide system that can automatically provide tourist information to tourists even if there is no guide such as a bus guide on board the vehicle. In this technology, when an imaging device that is provided to be able to capture images of the outside of the vehicle captures an image of a target for tourist information, tourist information is provided by audio output based on previously prepared audio data.

[0004] In recent years, with the spread of smartphones, applications that can provide route guidance and guidance to surrounding facilities on smartphones are becoming popular. Therefore, car navigation devices are sometimes substituted with smartphones. Furthermore, with the development of voice recognition technology, speaker-type voice interface devices (e.g., smart speakers) that can be controlled by voice are also becoming popular. Therefore, in the future, car navigation devices may be replaced by smart speakers, and smart speakers may provide route guidance and guidance to facilities around the vehicle based on the user's voice input. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-194898 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, when a conventional smart speaker is installed inside a moving vehicle, the smart speaker can obtain information inside the vehicle, but has difficulty obtaining information outside the vehicle. For example, when a user specifies a landmark outside the vehicle as a guidance target in a manner that does not specify the name of the landmark, such as "What is that building?", the smart speaker can recognize that the user has requested guidance. However, it is difficult for the smart speaker to recognize which landmark outside the vehicle the user specified as the guidance target. Therefore, the smart speaker may provide erroneous guidance because it is difficult for the smart speaker to correctly recognize the landmark specified by the user.

[0007] In view of the above-mentioned problems, an object of the present invention is to provide a target guidance control device, a target guidance control method, and a program that can improve the accuracy of guidance regarding a target specified by a user riding in a vehicle as the target for which guidance is requested. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, a target guidance control device according to one aspect of the present invention includes a detection unit that detects candidates for a guidance target designated by a user based on a voice of a user riding in a vehicle, an in-vehicle captured image in which a pose of the user that designates a guidance target for which the user requests guidance, and an captured image of the surroundings of the vehicle, and a detection unit that identifies a position and orientation of the vehicle in spatial information around the vehicle based on position information of the vehicle, identifies a direction of the guidance target in the spatial information based on the identified orientation of the vehicle, and selects a candidate for the guidance target from the candidates based on the identified direction. It is determined whether the guidance target can be determined univocally, and if it can be determined, Identify the target of the guidance If the guidance target cannot be determined, the user is prompted to speak to narrow down the guidance target, and the speech uttered by the user is supplied to the detection unit. and an output control unit that outputs output information related to the guidance target based on the spatial information corresponding to the specified guidance target.

[0009] A target guidance control method according to one aspect of the present invention includes detecting candidates for a target for guidance designated by a user based on a voice of a user riding in a vehicle, an in-vehicle captured image in which a pose of the user designating a target for guidance for which the user requests guidance, and a captured image of the surroundings of the vehicle; identifying a position and orientation of the vehicle in spatial information surrounding the vehicle based on position information of the vehicle; identifying a direction of the target for guidance in the spatial information based on the identified orientation of the vehicle; and selecting a candidate for the target for guidance from among the candidates based on the identified direction. It is determined whether the guidance target can be determined univocally, and if it can be determined, Identify the target of the guidance If the guidance target cannot be determined, the user is prompted to speak to narrow down the guidance target, and the speech uttered by the user is supplied to the detection unit. and outputting output information related to the identified guidance object based on the spatial information corresponding to the identified guidance object, and the method is executed by a processor.

[0010] A program according to one aspect of the present invention includes a computer, a detection unit that detects candidates for a guidance target designated by a user based on a voice of a user riding in a vehicle, an image captured inside the vehicle in which a pose of the user is captured and an image captured around the vehicle, and a detection unit that identifies a position and orientation of the vehicle in spatial information around the vehicle based on position information of the vehicle, identifies a direction of the guidance target in the spatial information based on the identified orientation of the vehicle, and selects a guidance target from the candidates based on the identified direction. It is determined whether the guidance target can be determined univocally, and if it can be determined, Identify the target of the guidance If the guidance target cannot be determined, the user is prompted to speak to narrow down the guidance target, and the speech uttered by the user is supplied to the detection unit. and an output control unit that outputs output information related to the specified guidance target based on the spatial information corresponding to the specified guidance target. Effect of the Invention

[0011] According to the present invention, it is possible to improve the accuracy of guidance regarding a guidance target designated by a user in a vehicle as a target for which guidance is requested. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram showing an overview of an object guidance control system according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram showing a configuration example of a target guidance control system according to the embodiment. FIG. [Diagram 3] 11 is an explanatory diagram of an example of detection of candidates for guidance targets according to the embodiment; FIG. [Figure 4] 11 is an explanatory diagram of an example of association between vehicle position information and a three-dimensional map according to the embodiment; FIG. [Diagram 5] 10 is a flowchart showing a process flow in the object guidance control system according to the embodiment. [Figure 6] 13 is a flowchart showing a flow of a guidance target specification process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. 1 is a diagram showing an overview of an object guidance control system 1 according to an embodiment of the present invention. The object guidance control system 1 is composed of an object guidance control device 10, a vehicle 20, and a network 30 shown in FIG.

[0014] The object guidance control device 10 is a device that controls guidance (hereinafter also referred to as "guidance control") regarding a guidance target designated as a target for which guidance is requested by a user 40 of a vehicle 20. The object guidance control device 10 is realized, for example, by a computer such as a server device.

[0015] 1, the object guidance control device 10 is connected to the vehicle 20 via a network 30 and is capable of transmitting and receiving information to and from the vehicle 20. The object guidance control device 10 according to this embodiment performs guidance control based on information received from the vehicle 20 via the network 30. In addition, the object guidance control device 10 transmits information related to guidance control to the vehicle 20 via the network 30.

[0016] The vehicle 20 is a moving body that moves with a passenger (e.g., a user 40) on board. The vehicle 20 according to the present embodiment is realized as an autonomous vehicle, as an example. When the vehicle 20 is an autonomous vehicle, the user 40 does not necessarily need to drive it. The vehicle 20 may be a non-autonomous vehicle.

[0017] The vehicle 20 is provided with sensor devices for acquiring various information. For example, as shown in FIG. 1, the vehicle 20 is provided with a microphone 212, an in-vehicle camera 214a, an in-vehicle camera 214b, an in-vehicle camera 214c, an outside camera 216a, an outside camera 216b, and a speaker 242. The microphone 212 is provided so as to be able to acquire sounds generated inside the vehicle 20. The in-vehicle cameras 214a to 214c (hereinafter also referred to as "in-vehicle cameras 214") are provided so as to be able to capture images of the interior of the vehicle 20. For example, as shown in FIG. 1, the in-vehicle cameras 214 are provided at the upper front, upper center, and upper rear of the vehicle 20, respectively, and capture images of the interior of the vehicle 20 from the respective positions to acquire captured images (still images or moving images). The outside camera 216a and the outside camera 216b (hereinafter also referred to as "outside camera 216") are provided so as to be able to capture images of the exterior of the vehicle 20. 1, the exterior cameras 216 are provided at the front and rear of the vehicle 20, respectively, and capture images of the outside of the vehicle 20 from the respective positions. Note that the type, number, and positions of the sensor devices provided on the vehicle 20 are not limited to the example shown in FIG.

[0018] The vehicle 20 is also provided with an output device for outputting output information. For example, as shown in FIG. 1, a speaker 242 is provided in the vehicle 20. The speaker 242 is provided so as to be able to output information related to a guidance target (hereinafter also referred to as "guidance information") as output information toward the interior of the vehicle 20. Note that the type, number, and location of the output device provided in the vehicle 20 are not limited to the example shown in FIG. 1. The output device may be a terminal owned by the user 40. The terminal owned by the user 40 is, for example, a smartphone, a tablet terminal, a hearable terminal, etc.

[0019] The guidance target is a landmark designated by user 40 among the landmarks existing around vehicle 20. As shown in Fig. 1, there are buildings 50 and 51 as landmarks around vehicle 20. As seen from user 40 inside vehicle 20 traveling in the direction of arrow 22, building 50 is in front of user 40 to the right, and building 51 is in front of user 40 to the left. Note that landmarks are not limited to buildings. The guidance information includes introduction information of the guidance target, route information to the guidance target, and the like.

[0020] In FIG. 1, user 40 utters "the building ahead on the right" as shown in speech bubble 42, specifying the guidance target by voice. At the same time, user 40 specifies the guidance target by pointing his / her finger in the direction of arrow 44 (to the front right as seen by user 40). As a result, the guidance target in FIG. 1 is identified as building 50. Note that user 40 may utter a question when specifying the guidance target. For example, user 40 may utter "What building is ahead on the right?"

[0021] The process of identifying such a guidance target (hereinafter also referred to as "guidance target identification process") is performed by the target guidance control device 10 based on the voice acquired by the vehicle 20 and the action of the user 40 designating the guidance target. The action of the user 40 designating the guidance target is, for example, a pointing action with a finger, a pointing action with a rod-shaped object (e.g., a pointer or pen), or a pointing action with the whole arm. The target guidance control device 10 may also perform the guidance target identification process using information indicating the facial orientation and line of sight of the user 40. Details of the guidance target identification process will be described later.

[0022] The object guidance control device 10 may be realized as a device that can include some or all of its components in the vehicle 20. The components of the object guidance control device 10 included in the vehicle 20 can be connected to each device included in the vehicle 20 by wire or wirelessly so as to be able to transmit and receive information to and from each device. On the other hand, the components of the object guidance control device 10 that are not included in the vehicle 20 can be realized by, for example, a server device, and can cooperate with the components of the object guidance control device 10 included in the vehicle 20 and each device included in the vehicle 20 via the network 30.

[0023] The outline of this embodiment has been described above. Next, a configuration example of the object guidance control system 1 according to this embodiment will be described. Fig. 2 is a block diagram showing a configuration example of the object guidance control system 1 according to this embodiment. Note that, below, a configuration example of the object guidance control system 1 in the case where not all of the components of the object guidance control device 10 are provided in the vehicle 20, but are realized by a server device or the like will be described.

[0024] (Example of configuration of vehicle 20) As shown in FIG. 2, the vehicle 20 includes an acquisition unit 210, a communication unit 220, a control unit 230, and a guidance information output unit 240.

[0025] (Acquisition section 210) The acquisition unit 210 has a function of acquiring various information. This function can be realized by various sensor devices. The acquisition unit 210 according to this embodiment includes sensor devices such as the above-mentioned microphone 212, the in-vehicle camera 214, and the outside-vehicle camera 216. The acquisition unit 210 may also include a sensor device that acquires location information of the vehicle 20. As an example, the acquisition unit 210 includes a GPS (Global Positioning System). After acquiring the sensing information by these sensor devices, the acquisition unit 210 outputs the sensing information to the communication unit 220.

[0026] The sensor device included in the acquisition unit 210 is not limited to this example. For example, the acquisition unit 210 may include a distance measurement sensor such as a LiDAR (Laser Imaging Detection and Ranging) as a sensor device that acquires information indicating the state inside and outside the vehicle 20.

[0027] (Communication unit 220) The communication unit 220 has a function of communicating with an external device. The communication unit 220 outputs information received through communication with the external device to the control unit 230. As an example, the communication unit 220 outputs guidance information received from the object guidance control device 10 via the network 30 to the control unit 230. The communication unit 220 also transmits information input from the acquisition unit 210 to the object guidance control device 10. As an example, the communication unit 220 transmits sensing information input from the acquisition unit 210 to the object guidance control device 10 via the network 30.

[0028] (Control unit 230) The control unit 230 has a function of controlling the overall operation of the vehicle 20. The control unit 230 is realized, for example, by causing a CPU (Central Processing Unit) provided as hardware in the vehicle 20 to execute a program. For example, the control unit 230 controls the operation of each sensor device of the acquisition unit 210. Furthermore, the control unit 230 controls the output of guidance information input from the communication unit 220 to the guidance information output unit 240. Furthermore, when the vehicle 20 is an autonomous vehicle, the control unit 230 controls autonomous driving of the vehicle 20.

[0029] (Guide information output unit 240) The guidance information output unit 240 has a function of outputting guidance information. The guidance information may be input from the control unit 230, for example, and output by various output devices. For example, when the guidance information output unit 240 includes an audio output device such as a speaker, the guidance information output unit 240 outputs the guidance information converted into audio from the speaker. When the guidance information output unit 240 includes a display device such as a display, the guidance information output unit 240 displays the guidance information converted into an image on the display.

[0030] (Configuration example of the object guidance control device 10) As shown in FIG. 2, the object guidance control device 10 includes a communication unit 110, a control unit 120, and a storage unit .

[0031] (Communication unit 110) The communication unit 110 has a function of communicating with an external device. The communication unit 110 outputs information received through communication with the external device to the control unit 120. As an example, the communication unit 110 outputs sensing information received from the vehicle 20 via the network 30 to the control unit 120. In addition, the communication unit 110 transmits information input from the control unit 120 to the vehicle 20. As an example, the communication unit 110 transmits guidance information input from the control unit 120 to the vehicle 20 via the network 30.

[0032] (Control unit 120) The control unit 120 has a function of controlling the overall operation of the object guidance control device 10. The control unit 120 is realized, for example, by causing a CPU provided as hardware in the object guidance control device 10 to execute a program. To realize this function, the control unit 120 includes a detection unit 122, an identification unit 124, and an output control unit 126.

[0033] (Detection unit 122) The detection unit 122 has a function of detecting candidates for guidance targets. For example, the detection unit 122 detects candidates for guidance targets based on the voice of the user 40 in the vehicle 20, the action of the user 40 specifying the guidance target, and a captured image of the surroundings of the vehicle 20. Specifically, the detection unit 122 detects a group of candidates for guidance targets by analyzing the captured image, and detects candidates that exist in a direction from the group of candidates based on a natural language related to the guidance target detected by analyzing the voice and a direction detected by analyzing the action of the user 40.

[0034] Here, an example of detection of a candidate for guidance will be described with reference to FIG. 3. FIG. 3 is an explanatory diagram of an example of detection of a candidate for guidance according to the present embodiment. First, the detection unit 122 detects a natural language from the voice by analyzing the voice acquired by the microphone 212 of the vehicle 20. When a voice command indicating that the user 40 is requesting guidance is detected from the detected natural language, the detection unit 122 further analyzes the voice and detects a natural language related to the guidance target. For example, the detection unit 122 detects the voice "building in front of the right" shown in the speech bubble 42 as a natural language related to the guidance target. In the example shown in FIG. 3, the user 40 faces the direction of the arrow 22, which is the traveling direction of the vehicle 20, so the detection unit 122 can detect that the user 40 is pointing to the building 50 in front of the right of the vehicle 20 from the detected natural language.

[0035] Furthermore, the detection unit 122 detects that the user 40 is pointing in the direction of the arrow 44 by analyzing the pose of the user 40 captured in an image of the interior of the vehicle acquired by the in-vehicle camera 214 of the vehicle 20. The detection unit 122 also detects buildings 50 and 51 shown in Fig. 3 as a group of candidates for guidance targets by analyzing the scenery outside the vehicle captured in an image of the exterior of the vehicle acquired by the exterior camera 216 of the vehicle 20. From the above detection results, the detection unit 122 detects the building 50 located to the right of the vehicle 20 in the direction of the arrow 44 as a candidate for guidance targets from the group of candidates.

[0036] As described above, the detection unit 122 can link the group of candidates for guidance targets detected from the scenery outside the vehicle 20 with the guidance target designated by the user 40 based on the voice and the movement of the user 40 inside the moving vehicle 20. With this configuration, the detection unit 122 can detect landmarks outside the moving vehicle 20 as candidates for guidance targets. In addition, the detection unit 122 uses both the voice and the movement of the user 40 inside the vehicle 20 when detecting candidates for guidance targets, and therefore can improve the accuracy of detection of candidates for guidance targets. Note that the detection unit 122 may detect one candidate for guidance targets from the group of candidates, or may detect multiple candidates for guidance targets.

[0037] (Specific Section 124) The identification unit 124 has a function of identifying a guidance target designated by the user 40 from among the candidate guidance targets detected by the detection unit 122. For example, the identification unit 124 identifies the position and orientation of the vehicle 20 in spatial information around the vehicle 20 based on the position information of the vehicle 20, identifies the direction of the guidance target in the spatial information based on the identified orientation of the vehicle 20, and identifies the guidance target from the candidate guidance targets based on the identified direction. The spatial information around the vehicle 20 according to this embodiment is, for example, a 3D map around the vehicle 20, landmark information including metadata of landmarks on the 3D map, and the like. Note that the spatial information is not limited to such an example.

[0038] Here, the correspondence between the position information of the vehicle 20 and the 3D map will be described with reference to Fig. 4. Fig. 4 is an explanatory diagram of an example of the correspondence between the position information of the vehicle 20 and the 3D map according to this embodiment. As shown in Fig. 4, the 3D map 60 has 3D data of landmarks such as a building 50, a building 51, and a mountain 52.

[0039] First, the identification unit 124 identifies the position and orientation of the vehicle 20 on a three-dimensional map from position information acquired by the GPS of the vehicle 20, and maps the vehicle 20 on the three-dimensional map. The orientation of the vehicle 20 may be identified, for example, based on sensing information from a gyro sensor or the movement history of the vehicle 20. Next, the identification unit 124 identifies the direction of the guidance target designated by the user 40 on the three-dimensional map based on the orientation of the vehicle 20 indicated by the arrow 22. Then, the identification unit 124 identifies, from among the candidates for the guidance target, a building 50 that is a candidate located in the identified direction (arrow 44) as the guidance target.

[0040] As described above, the identification unit 124 can uniquely identify the guidance target designated by the user 40 on the 3D map from among the candidate guidance targets detected by the detection unit 122, based on the position information of the vehicle 20 and the 3D map. With this configuration, when the detection unit 122 detects one candidate guidance target, the identification unit 124 can increase the accuracy of identifying the candidate guidance target. Furthermore, when the detection unit 122 detects multiple candidate guidance targets, the identification unit 124 can narrow down the guidance targets. This improves the accuracy of identifying the guidance target, and therefore makes it possible to improve the accuracy of guidance regarding the guidance target designated by the user 40.

[0041] (Output control unit 126) The output control unit 126 has a function of controlling the output in the vehicle 20. For example, the output control unit 126 causes an output device of the vehicle 20 to output output information related to the identified guidance target based on spatial information corresponding to the identified guidance target. Specifically, the output control unit 126 first acquires metadata of the identified guidance target from the landmark information, and generates guidance information based on the metadata. Then, the output control unit 126 causes the communication unit 110 to transmit the generated guidance information to the vehicle 20, and causes an output device of the vehicle 20 to output the guidance information.

[0042] When the guidance target is a building, the metadata is, for example, the name, address, number of floors, tenant information of each floor, etc. of the building. Based on these metadata, the output control unit 126 generates an introductory text such as, for example, "That building is the AAA building" as the guidance information. Then, the output control unit 126 causes the output device to output the guidance information according to the output device of the vehicle 20. For example, when the output device is an audio output device, the output control unit 126 converts the introductory text into audio and outputs it to the audio output device. Also, when the output device is a display device, the output control unit 126 converts the introductory text into an image and outputs it to the display device. With this configuration, flexible guidance can be provided to the user 40 according to the identified guidance target and the acquired landmark information based on the landmark information acquired after the guidance target is identified.

[0043] When generating the guidance information, the output control unit 126 may generate the guidance information according to the status of the vehicle 20 and the user 40, and the profile information of the user 40. The status of the vehicle 20 is, for example, the position, direction, speed, and moving route of the vehicle 20. The status of the user 40 is, for example, the posture of the user 40, and the position of the user 40 in the vehicle. The profile information of the user 40 is, for example, the name, sex, height, weight, hobbies, and the usage history of the target guidance control system of the user 40. The output control unit 126 generates, for example, an introduction text such as "That building is the AAA building. Inside the building, there is the BBB cafe that we introduced before" as the introduction information based on the profile information of the user 40. With this configuration, it is possible to provide flexible guidance to the user 40 according to the status of the vehicle 20 and the user 40, and the profile information of the user 40.

[0044] (Storage unit 130) The storage unit 130 has a function of storing various information. The storage unit 130 is configured by a storage medium, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media. The storage unit 130 can use, for example, a non-volatile memory.

[0045] The storage unit 130 stores spatial information around the vehicle 20, such as a three-dimensional map and landmark information. The storage unit 130 may also store sensing information acquired by the acquisition unit 210 of the vehicle 20, landmarks identified as guidance targets by the identification unit 124 of the target guidance control device 10, output information generated by the output control unit 126, profile information of the user 40, and the like.

[0046] The above describes a configuration example of the object guidance control system 1 in the case where not all of the components of the object guidance control device 10 are provided in the vehicle 20 but are realized by a server device or the like. Note that the configuration example of the object guidance control system 1 is not limited to this example. For example, in the case where all of the components of the object guidance control device 10 are provided in the vehicle 20, at least one of the communication unit 110 and the communication unit 220 may be provided in the vehicle 20 as a component having a function of communicating with an external device.

[0047] (Processing flow in the target guidance control system 1) Next, a process flow in the object guidance control system 1 according to this embodiment will be described. Fig. 5 is a flowchart showing the process flow in the object guidance control system 1 according to this embodiment. Note that, below, an example of the process flow in the object guidance control system 1 in the case where not all of the components of the object guidance control device 10 are provided in the vehicle 20 but are realized by a server device or the like will be described.

[0048] First, the microphone 212 of the vehicle 20 acquires voice within the vehicle 20 (S102). The voice acquired by the microphone 212 is transmitted to the object guidance control device 10 via the network 30. When the microphone 212 acquires the voice, the control unit 230 of the vehicle 20 turns on the in-vehicle camera 214 of the vehicle 20 (S104). The control unit 230 also turns on the exterior camera 216 of the vehicle 20 (S106).

[0049] Next, when the user 40 speaks (S108), the microphone 212 acquires voice including the speech of the user 40, and the voice is transmitted to the object guidance control device 10 via the network 30. Next, the control unit 120 of the object guidance control device 10 determines whether or not the received voice contains a voice command requesting guidance (S110). If the voice does not contain a voice command (S110 / NO), the object guidance control system 1 performs the processes of S108 and S110 until the voice contains a voice command. If the voice contains a voice command (S110 / YES), the control unit 120 performs a guidance target identification process (S112). The guidance target identification process will be described in detail later.

[0050] After the guidance target identification process, the control unit 120 records the landmark identified as the guidance target (S114). Next, the control unit 120 acquires metadata of the landmark from the 3D map (S116). Next, the control unit 120 generates an introduction text according to the situation of the vehicle 20 and the user 40 and the profile information of the user 40 (S118). After generating the introduction text, the control unit 120 converts the introduction text into voice and outputs the converted voice to the speaker 242 of the vehicle 20, thereby reading out the introduction text (S120). After reading out the introduction text, the control unit 120 determines whether or not there is a next dialogue (S122). If there is a next dialogue (S122 / YES), the control unit 120 repeats the process from S110. If there is no next dialogue (S122 / NO), the control unit 120 ends the process until the next voice is acquired.

[0051] (Flow of the guidance target identification process) Next, a detailed description will be given of the guidance target specification process of S112. Fig. 6 is a flowchart showing the flow of the guidance target specification process according to this embodiment.

[0052] First, the control unit 120 analyzes the natural language contained in the voice (S202). Next, the control unit 120 analyzes the pose of the user 40 shown in the image captured by the in-vehicle camera 214 received from the vehicle 20 via the network 30 (S204). Next, the control unit 120 analyzes the scenery outside the vehicle shown in the image captured by the exterior camera 216 received from the vehicle 20 via the network 30 (S206).

[0053] After analyzing the natural language, the pose, and the scenery outside the vehicle, the control unit 120 determines whether or not the guidance target has been detected (S208). If the guidance target has not been detected (S208 / NO), the control unit 120 prompts the user 40 to speak again (S220).

[0054] If the guidance target is detected (S208 / YES), the control unit 120 acquires the position and orientation of the vehicle 20 (S210). Next, the control unit 120 associates the position and orientation of the vehicle 20 with a three-dimensional map (S212). Next, the control unit 120 associates the orientation of the pose of the user 40 with the orientation of the vehicle 20 (S214). Next, the control unit 120 associates a captured image showing the scenery outside the vehicle 20 with the three-dimensional map (S216).

[0055] After each correspondence, the control unit 120 judges whether the guidance target can be uniquely determined on the three-dimensional map (S218). If the guidance target cannot be uniquely determined (S218 / NO), the control unit 120 prompts the user 40 to speak to narrow down the guidance targets (S222). If the guidance target can be uniquely determined (S218 / YES), the control unit 120 ends the guidance target specification process.

[0056] An example of the process flow in the object guidance control system 1 in the case where not all of the components of the object guidance control device 10 are provided in the vehicle 20 but are realized by a server device or the like has been described above. Note that the process flow in the object guidance control system 1 is not limited to this example. For example, in the case where all of the components of the object guidance control device 10 are provided in the vehicle 20, the object guidance control system 1 can perform processing without going through the network 30.

[0057] As described above, the object guidance control device 10 according to this embodiment links a group of candidate guidance targets detected from the scenery outside the vehicle 20 with a guidance target designated by the user 40, based on the voice and actions of the user 40 inside the moving vehicle 20. This allows the object guidance control device 10 to detect landmarks outside the moving vehicle 20 as candidate guidance targets. Moreover, the object guidance control device 10 according to the present embodiment detects candidates for guidance targets using both the voice of the user 40 inside the vehicle 20 and the movement of the user 40. This allows the object guidance control device 10 to improve the accuracy of detection of guidance targets. Moreover, the object guidance control device 10 according to this embodiment uniquely identifies the guidance object on the 3D map based on the position information of the vehicle 20 and the 3D map. This enables the object guidance control device 10 to improve the accuracy of identifying the guidance object, and further improve the accuracy of guidance regarding the guidance object designated by the user 40. Therefore, the object guidance control device 10 according to the present embodiment can improve the accuracy of guidance regarding the object designated by the user 40 in the vehicle 20 as an object for which guidance is requested.

[0058] (Modification) The embodiments of the present invention have been described above. Next, modified examples of the embodiments of the present invention will be described. Each modified example described below may be applied alone to the embodiments of the present invention, or may be applied in combination to the embodiments of the present invention. Furthermore, each modified example may be applied in place of the configuration described in the embodiments of the present invention, or may be applied in addition to the configuration described in each embodiment of the present invention.

[0059] (First Modification) In the above-described embodiment, an example has been described in which the object guidance control device 10 controls guidance regarding a guidance target designated by the user 40, but the present invention is not limited to such an example. If there is a landmark around the vehicle 20 that the user 40 is likely to be interested in, the object guidance control device 10 may control guidance to the landmark as a guidance target even if it is not designated by the user 40. The landmark that the user 40 is likely to be interested in is detected based on, for example, the position information of the vehicle 20 or the profile information of the user 40. With such a configuration, flexible guidance can be provided to the user 40 according to the user 40.

[0060] (Second Modification) In the above-described embodiment, an example in which the object guidance control device 10 causes the output device of the vehicle 20 to output guidance information has been described, but the present invention is not limited to such an example. For example, when a voice command indicating that the user 40 is requesting a route change is detected from the natural language detected by the detection unit 122, the guidance information may be output to the control unit 230 of the vehicle 20. At this time, route information indicating the changed route is input to the control unit 230 as the guidance information. Then, the control unit 230 controls the automatic driving of the vehicle 20 based on the input route information so that the vehicle 20 moves along the changed route.

[0061] (Third Modification) In the above embodiment, an example in which the output control unit 126 generates guidance information based on metadata has been described, but the present invention is not limited to such an example. The output control unit 126 may obtain information on the guidance target from a web service and generate guidance information based on the obtained information.

[0062] The above describes the modified embodiment of the present invention. The object guidance control device 10 in the above embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0063] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0064] 1... object guidance control system, 10... object guidance control device, 20... vehicle, 30... network, 110... communication unit, 120... control unit, 122... detection unit, 124... identification unit, 126... output control unit, 130... memory unit, 210... acquisition unit, 212... microphone, 214... in-vehicle camera, 216... outside-vehicle camera, 220... communication unit, 230... control unit, 240... guidance information output unit

Claims

1. a detection unit that detects candidates for a guidance target designated by a user based on a voice of the user riding in the vehicle, an image captured inside the vehicle in which a pose of the user designating a guidance target for which the user requests guidance, and an image captured around the vehicle; an identification unit that identifies a position and a direction of the vehicle in spatial information around the vehicle based on position information of the vehicle, identifies a direction of the guidance target in the spatial information based on the identified direction of the vehicle, determines whether the guidance target can be uniquely determined from the candidates based on the identified direction, identifies the guidance target if it can be determined, and supplies a voice uttered by the user to the detection unit by prompting the user to speak to narrow down the guidance targets if it cannot be determined; an output control unit that outputs output information related to the identified guidance target based on the spatial information corresponding to the identified guidance target; An object guidance control device comprising:

2. 2. The object guidance control device according to claim 1, wherein the detection unit detects a group of candidates for the guidance object by analyzing the captured image, and detects the candidate that exists in the direction from among the group of candidates based on a natural language related to the guidance object detected by analyzing the voice and the direction detected by analyzing a pose of the user included in the in-vehicle captured image.

3. The object guidance control device according to claim 1 , wherein the output control unit acquires metadata of the specified object to be guided from the spatial information, and generates the output information based on the metadata.

4. Detecting candidates for a guidance target designated by a user based on a voice of the user riding in the vehicle, an in-vehicle captured image in which a pose of the user designating a guidance target for which the user requests guidance is captured, and a captured image of the surroundings of the vehicle; specifying a position and orientation of the vehicle in spatial information around the vehicle based on position information of the vehicle, specifying a direction of the guidance target in the spatial information based on the specified orientation of the vehicle, determining whether or not the guidance target can be uniquely determined from the candidates based on the specified direction, specifying the guidance target if it can be determined, and supplying a voice uttered by the user to the detection unit by prompting the user to speak to narrow down the guidance targets if it cannot be determined; outputting output information related to the identified guidance target based on the spatial information corresponding to the guidance target; 23. An object guidance control method executed by a processor, comprising:

5. Computer, a detection unit that detects candidates for a guidance target designated by a user based on a voice of the user riding in the vehicle, an image captured inside the vehicle in which a pose of the user designating a guidance target for which the user requests guidance, and an image captured around the vehicle; an identification unit that identifies a position and a direction of the vehicle in spatial information around the vehicle based on position information of the vehicle, identifies a direction of the guidance target in the spatial information based on the identified direction of the vehicle, determines whether the guidance target can be uniquely determined from the candidates based on the identified direction, identifies the guidance target if it can be determined, and supplies a voice uttered by the user to the detection unit by prompting the user to speak to narrow down the guidance targets if it cannot be determined; an output control unit that outputs output information related to the identified guidance target based on the spatial information corresponding to the identified guidance target; A program to function as a

Images