Information processing device, mobile object, control method thereof, and program

Abstract

To suitably estimate a user and adjust a junction between the estimated user and a moving object.SOLUTION: An information processor includes: first acquisition means for acquiring at least one of speech information by a user and location information of a communication device from the communication device of the user; identification means for identifying a predetermined area according to a landmark included in the speech information; setting means for acquiring a direction of movement of the user from at least one of the acquired speech information and the location information acquired from the communication device of the user, and setting a probability distribution of presence of the user in the predetermined area based on the acquired direction of movement of the user; and estimation means for estimating the user based on the set probability distribution.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device, a mobile object, a control method thereof, and a program. [Background technology]

[0002] In recent years, electric vehicles (mobiles) with a passenger capacity of around one or two people, known as ultra-compact mobility (also known as micromobility), have become known and are expected to become widespread as a convenient means of transportation.

[0003] A car sharing system has been proposed in which such ultra-compact mobility vehicles are used for sharing (Patent Document 1). In this car sharing system, a vehicle management server receives an application message from a user's communication device, which includes the usage start time and rental location of the vehicle (mobile object) that is the subject of car sharing. Then, based on the content of the application message and the current location of the transportation vehicle, a transport vehicle that can arrive at the rental location by the usage start time is identified, and the identified transport vehicle is made to transport the shared car to the rental location. The user can use the shared car by visiting the rental location at the specified usage start time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-77035 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when a user uses an ultra-compact mobility vehicle, there may be a use case in which the ultra-compact mobility vehicle and the user dynamically adjust the meeting point while moving, rather than the user visiting a rental location where the ultra-compact mobility vehicle is parked. Such a use case is effective in cases where it becomes difficult to meet at a pre-specified location due to congestion, etc., or when the user first specifies a rough area or building, etc., and adjusts the specific meeting point when they arrive nearby each other.

[0006] The present invention has been made in view of the above problems, and has an object to suitably estimate a user. Another object of the present invention is to adjust a meeting position between the estimated user and a moving object. [Means for solving the problem]

[0007] According to the present invention, there is provided an information processing apparatus, comprising: a first acquisition means for acquiring at least one of utterance information by a user and location information of the communication device from the communication device of the user; , indicating the meeting point with the user a specifying means for specifying a predetermined area according to a landmark; a second acquiring means for acquiring an image captured around the predetermined area; a setting means for acquiring a moving direction of the user from at least one of the acquired speech information and position information acquired from the user's communication device, and setting a probability distribution of the user's presence in the predetermined area based on the acquired moving direction of the user; and a setting means for analyzing, from the captured image, the moving directions of one or more people detected in the captured image acquired by the second acquiring means, relative to the landmark, and selecting from the one or more people based on the set probability distribution and the analyzed moving directions of the one or more people. Identifying the location of the user requesting joining and an estimation means for estimating a person corresponding to the user.

[0008] According to the present invention, a mobile object includes a communication means for communicating with a communication device of a user, an imaging means for imaging an image of the surroundings of the mobile object, a first acquisition means for acquiring at least one of utterance information by the user and position information of the communication device from the communication device of the user by the communication means, and , indicating the meeting point with the user a specifying means for specifying a predetermined area in accordance with a landmark; a second acquiring means for acquiring an image captured around the predetermined area; a setting means for acquiring a moving direction of the user from at least one of the acquired speech information and position information acquired from the user's communication device, and setting a probability distribution in which the user is present in a divided area of ​​the predetermined area based on the acquired moving direction of the user; and a setting means for analyzing, from the captured image, the moving directions of one or more people detected in the captured image acquired by the second acquiring means, relative to the landmark, and selecting from the one or more people a probability distribution that has been set and the analyzed moving directions of the one or more people. Identifying the location of the user requesting joining and an estimation means for estimating a person corresponding to the user. Effect of the Invention

[0009] According to the present invention, it is possible to preferably estimate a user, and to adjust a meeting position between the estimated user and a moving object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing an example of the hardware configuration of a moving body according to the embodiment; [Diagram 3] FIG. 2 is a block diagram showing an example of the functional configuration of a moving body according to the embodiment; [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of a server and a communication device according to an embodiment of the present invention; [Diagram 5] FIG. 1 is a diagram for explaining estimation of a meeting position using speech and images according to the present embodiment; [Figure 6]A flowchart showing a series of operations in the process of adjusting the joining position according to the present embodiment. [Figure 7] FIG. 1 is a diagram for explaining estimation of a user based on a probability distribution according to the present embodiment; [Figure 8] A flowchart showing a series of operations in a user estimation process based on a probability distribution according to the present embodiment. [Figure 9] A flowchart showing a series of operations in a user estimation process using an utterance and an image according to the present embodiment. [Figure 10] FIG. 1 is a diagram for explaining user estimation using speech and images according to the present embodiment; [Figure 11] FIG. 13 is a diagram showing an example of a screen displaying an estimated positional relationship between a user and a moving object according to the present embodiment; [Figure 12] FIG. 13 is a diagram illustrating an example of an information processing system according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] <Configuration of information processing system> The configuration of an information processing system 1 according to this embodiment will be described with reference to FIG. 1. The information processing system 1 includes a vehicle (mobile body) 100, a server 110, and a communication device (communication terminal) 120. In this embodiment, the server 110 estimates the user using speech information of the user 130 and a captured image of the surroundings of the vehicle 100, and further estimates a meeting position to merge with the vehicle 100. The user communicates with the server 110 via a predetermined application started on the communication device 120 held by the user, and moves to a meeting position (for example, a nearby red post as a landmark) designated by the user while providing his / her own position and the like by speech. The server 110 controls the vehicle 100 to move to the estimated meeting position while estimating the user and the meeting position. Each configuration will be described in detail below.

[0013] The vehicle 100 is equipped with a battery and is, for example, an ultra-compact mobility that moves mainly by the power of a motor. The ultra-compact mobility is a vehicle that is more compact than a general automobile and has a passenger capacity of about one or two people. In this embodiment, the vehicle 100 is described as an example of an ultra-compact mobility, but this is not intended to limit the present invention, and may be, for example, a four-wheeled vehicle or a saddle-type vehicle. In addition, the vehicle of the present invention is not limited to a vehicle, but may be a vehicle that carries luggage and runs alongside a person walking, or a vehicle that leads a person. Furthermore, the present invention is not limited to vehicles such as four-wheeled or two-wheeled vehicles, and may also be applicable to walking robots that can move independently. In other words, the present invention can be applied to moving bodies such as these vehicles and walking robots, and the vehicle 100 is an example of a moving body.

[0014] The vehicle 100 is connected to the network 140 via wireless communication such as Wi-Fi or fifth generation mobile communication. The vehicle 100 can measure the inside and outside of the vehicle (such as the vehicle's position, driving state, and surrounding object targets) using various sensors and transmit the measured data to the server 110. The data collected and transmitted in this manner is generally called floating data, probe data, traffic information, and the like. Information about the vehicle is transmitted to the server 110 at regular intervals or in response to the occurrence of a specific event. The vehicle 100 can travel by autonomous driving even when the user 130 is not on board. The vehicle 100 receives information such as a control command provided by the server 110, or controls the operation of the vehicle using data measured by the vehicle itself.

[0015] The server 110 is an example of an information processing device, and is configured with one or more server devices, and is capable of acquiring vehicle-related information transmitted from the vehicle 100, and speech information and position information transmitted from the communication device 120, via the network 140, and controlling the traveling of the vehicle 100. The traveling control of the vehicle 100 includes a process of adjusting a junction position between the user 130 and the vehicle 100, which will be described later.

[0016] The communication device 120 is, for example, a smartphone, but is not limited thereto, and may be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, etc. The communication device 120 connects to the network 140 via wireless communication such as Wi-Fi or fifth generation mobile communication.

[0017] The network 140 includes a communication network such as the Internet or a mobile phone network, and transmits information between the server 110 and the vehicle 100 or the communication device 120. In this information processing system 1, when the user 130 and the vehicle 100, who are in distant locations, approach each other to a degree that allows them to visually confirm a target or the like (a visual landmark), the merging position is adjusted using speech information and image information captured by the vehicle 100. In this embodiment, an example in which a camera capturing an image of the surroundings of the vehicle 100 is provided on the vehicle itself will be described, but the vehicle 100 does not necessarily need to be provided with a camera or the like. For example, an image captured using a surveillance camera or the like already installed around the vehicle 100 may be used, or both may be used. This allows an image captured at a more optimal angle to be used when identifying the user's position. For example, when a user utters a speech about a positional relationship between the user and a landmark, the image captured by a camera close to the predicted position of the landmark can be analyzed to more accurately identify the user requesting to merge with the ultra-compact mobility.

[0018] Before the user 130 and the vehicle 100 get close enough to visually confirm a target or the like, the server 110 first moves the vehicle 100 to a rough area including the user's current location or the user's predicted location. Then, when the vehicle 100 reaches the rough area, the server 110 transmits voice information asking about a location related to a visual landmark (e.g., "Are there any stores nearby?" or "What do you see in the direction of travel?") to the communication device 120. The location related to the visual landmark includes, for example, the name of a location included in map information. Here, the visual landmark indicates a physical object that the user can see, and includes various objects such as a building, a traffic light, a river, a mountain, a bronze statue, and a signboard. The server 110 receives speech information by the user including a location related to the visual landmark (e.g., "There is a building with a xx coffee shop") from the communication device 120. Then, the server 110 acquires the position of the corresponding location from the map information and moves the vehicle 100 to the vicinity of the location (i.e., the vehicle and the user approach close enough to visually confirm the landmarks, etc.). If the location cannot be identified from the map information, for example, if there are multiple candidate locations, it is possible to narrow down the candidate locations by asking additional questions.

[0019] <Mobile configuration> Next, the configuration of a vehicle 100 as an example of a moving body according to this embodiment will be described with reference to Fig. 2. Fig. 2(A) shows a side view of the vehicle 100 according to this embodiment, and Fig. 2(B) shows the internal configuration of the vehicle 100. In the figure, an arrow X indicates the front-rear direction of the vehicle 100, F indicates the front, and R indicates the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of the vehicle 100.

[0020] The vehicle 100 is an electric autonomous vehicle equipped with a propulsion unit 12 and using a battery 13 as a main power source. The battery 13 is, for example, a secondary battery such as a lithium ion battery, and the vehicle 100 is self-propelled by the propulsion unit 12 using the power supplied from the battery 13. The propulsion unit 12 is a four-wheeled vehicle equipped with a pair of left and right front wheels 20 and a pair of left and right rear wheels 21. The propulsion unit 12 may be in another form, such as a tricycle. The vehicle 100 is equipped with a seat 14 for one or two people.

[0021] The traveling unit 12 includes a steering mechanism 22. The steering mechanism 22 is a mechanism that changes the steering angle of the pair of front wheels 20 using a motor 22a as a driving source. By changing the steering angle of the pair of front wheels 20, the traveling direction of the vehicle 100 can be changed. The traveling unit 12 also includes a drive mechanism 23. The drive mechanism 23 is a mechanism that uses a motor 23a as a driving source to rotate the pair of rear wheels 21. By rotating the pair of rear wheels 21, the vehicle 100 can be moved forward or backward.

[0022] The vehicle 100 includes detection units 15-17 that detect targets around the vehicle 100. The detection units 15-17 are a group of external sensors that monitor the periphery of the vehicle 100, and in the present embodiment, each is an imaging device that captures an image of the periphery of the vehicle 100, and includes, for example, an optical system such as a lens and an image sensor. However, instead of or in addition to the imaging device, it is also possible to employ a radar or a lidar (Light Detection and Ranging).

[0023] The two detection units 15 are arranged at the front of the vehicle 100, spaced apart in the Y direction, and mainly detect targets in front of the vehicle 100. The detection units 16 are arranged on the left and right sides of the vehicle 100, respectively, and mainly detect targets on the sides of the vehicle 100. The detection unit 17 is arranged at the rear of the vehicle 100, and mainly detects targets behind the vehicle 100.

[0024] <Control configuration of moving object> FIG. 3 is a block diagram of a control system of vehicle 100, which is a moving body. Here, the configuration necessary for implementing the present invention will be mainly described. Therefore, in addition to the configuration described below, other configurations may be included. Vehicle 100 is equipped with a control unit (ECU) 30. Control unit 30 includes a processor represented by a CPU, a storage device such as a semiconductor memory, an interface with an external device, etc. The storage device stores programs executed by the processor, data used by the processor for processing, etc. A plurality of sets of processors, storage devices, and interfaces may be provided according to the functions of vehicle 100, and configured to be able to communicate with each other.

[0025] The control unit 30 receives the detection results of the detection units 15-17, input information from the operation panel 31, voice information input from the voice input device 33, and control commands (such as transmission of captured images and current position) from the server 110, and executes corresponding processes. The control unit 30 controls the motors 22a and 23a (driving control of the driving unit 12), controls the display of the operation panel 31, notifies the occupants of the vehicle 100 by voice, and outputs information.

[0026] The voice input device 33 collects the voice of the passenger of the vehicle 100. The control unit 30 can recognize the input voice and execute a corresponding process. The GNSS (Global Navigation Satellite system) sensor 34 receives a GNSS signal to detect the current position of the vehicle 100. The storage device 35 is a large-capacity storage device that stores map data including information on the routes on which the vehicle 100 can travel, landmarks such as buildings, stores, etc. The storage device 35 may also store programs executed by the processor and data used by the processor for processing. The storage device 35 may store various parameters (e.g., learned parameters and hyperparameters of a deep neural network) of a machine learning model for voice recognition or image recognition executed by the control unit 30. The communication unit 36 ​​is a communication device that can be connected to the network 140 via wireless communication such as Wi-Fi or fifth-generation mobile communication.

[0027] <Server and communication device configuration> Next, with reference to FIG. 4, a configuration example of the server 110 as an example of an information processing device according to the present embodiment and the communication device 120 will be described.

[0028] (Server configuration) First, a configuration example of the server 110 will be described. Here, the configuration necessary for implementing the present invention will be mainly described. Therefore, in addition to the configuration described below, other configurations may be included. The control unit 404 includes a processor represented by a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor and data used by the processor for processing. A plurality of sets of the processor, storage device, and interface may be provided according to the functions of the server 110 and configured to be able to communicate with each other. The control unit 404 executes programs to perform various operations of the server 110 and adjustment processing of the joining position described later. In addition to the CPU, the control unit 404 may further include dedicated hardware suitable for executing processing of a machine learning model such as a GPU or a neural network.

[0029] The user data acquisition unit 413 acquires image and position information transmitted from the vehicle 100. In addition, the user data acquisition unit 413 acquires at least one of utterance information of the user 130 transmitted from the communication device 120 and position information of the communication device 120. The user data acquisition unit 413 may store the acquired image and position information in the storage unit 403. The image and utterance information acquired by the user data acquisition unit 413 is input to a trained model in the inference stage to obtain an inference result, but may also be used as training data for training a machine learning model executed by the server 110.

[0030] The voice information processing unit 414 includes a machine learning model that processes voice information, and executes the learning stage processing and the inference stage processing of the machine learning model. The machine learning model of the voice information processing unit 414 performs, for example, calculations of a deep learning algorithm using a deep neural network (DNN) to recognize the names of places, landmarks such as buildings, store names, and targets included in the speech information. The targets may include passersby, signs, signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like included in the speech information. The DNN becomes in a learned state by performing the learning stage processing, and can perform recognition processing (inference stage processing) for new speech information by inputting new speech information into the trained DNN. Note that in this embodiment, an example will be described in which the server 110 executes the speech recognition processing, but the speech recognition processing may be performed in a vehicle or a communication device, and the recognition result may be transmitted to the server 110.

[0031] The image information processing unit 415 includes a machine learning model that processes image information, and executes a learning stage process and an inference stage process of the machine learning model. The machine learning model of the image information processing unit 415 performs a process of recognizing targets included in the image information, for example, by performing calculations of a deep learning algorithm using a deep neural network (DNN). The targets may include passersby, signs, signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like included in the image.

[0032] The merging position estimation unit 416 executes a process of adjusting the merging position, which will be described later. The merging position adjustment process will be described later. The user estimation unit 417 executes a process of estimating a user, which will be described later. Here, estimating a user means estimating a user who requests merging with the vehicle 100, and estimates the user by identifying the position of the requesting user from one or more people within a predetermined area. The detailed process will be described later.

[0033] Generally, the server 110 can use more abundant computational resources than the vehicle 100. In addition, by receiving and storing image data captured by various vehicles, learning data in a wide variety of situations can be collected, enabling learning to be performed in a wider variety of situations.

[0034] The communication unit 401 is a communication device including, for example, a communication circuit and the like, and communicates with external devices such as the vehicle 100 and the communication device 120. The communication unit 401 receives image information and position information from the vehicle 100, and at least one of speech information and position information from the communication device 120, and transmits control commands to the vehicle 100 and speech information to the communication device 120. The power supply unit 402 supplies power to each component in the server 110. The storage unit 403 is a non-volatile memory such as a hard disk or a semiconductor memory.

[0035] (Configuration of communication device) Next, the configuration of the communication device 120 will be described. The communication device 120 refers to a portable device such as a smartphone carried by the user 130. Here, the configuration necessary for carrying out the present invention will be mainly described. Therefore, in addition to the configuration described below, other configurations may be included. The communication device 120 includes a control unit 501, a storage unit 502, an external communication device 503, a display operation unit 504, a microphone 507, a speaker 508, and a speed sensor 509. The external communication device 503 includes a GPS 505 and a communication unit 506.

[0036] The control unit 501 includes a processor represented by a CPU. The storage unit 502 stores programs executed by the processor, data used by the processor for processing, and the like. The storage unit 502 may be incorporated inside the control unit 501. The control unit 501 is connected to other components 502, 503, 504, 508, and 509 via signal lines such as a bus, can transmit and receive signals, and controls the entire communication device 120.

[0037] The control unit 501 can communicate with the communication unit 401 of the server 110 via the network 140 using the communication unit 506 of the external communication device 503. The control unit 501 also acquires various information via the GPS 505. The GPS 505 acquires the current position of the communication device 120. This makes it possible to provide the server 110 with position information together with user speech information, for example. Note that the GPS 505 is not an essential component in the present invention, and the present invention provides a system that can be used even in facilities such as indoors where the position information of the GPS 505 cannot be acquired. Therefore, the position information from the GPS 505 is treated as supplementary information when estimating the user.

[0038] The display operation unit 504 is, for example, a touch panel type liquid crystal display, and can perform various displays and accept user operations. The display operation unit 504 displays information such as the contents of an inquiry from the server 110 and the merging position with the vehicle 100. When an inquiry is received from the server 110, the user's utterance can be acquired by the microphone 507 of the communication device 120 by operating a microphone button displayed as selectable. The microphone 507 acquires the user's utterance as voice information. The microphone may be switched to an activated state by pressing a microphone button displayed on the operation screen, for example, to acquire the user's utterance. The speaker 508 outputs a voice message when making an inquiry to the user according to an instruction from the server 110 (for example, "What color clothes are you wearing?"). If the inquiry is made by voice, the communication device 120 can communicate with the user even if it has a simple configuration such as a headset without a display screen. Even if the user does not have the communication device 120 in his / her hand, the user can hear the inquiry from the server 110 through, for example, earphones.

[0039] The speed sensor 509 is an acceleration sensor that detects acceleration in the forward / backward, left / right, and up / down directions of the communication device 120. Output values ​​indicating the acceleration output from the speed sensor 509 are stored in a ring buffer of the storage unit 502, and are overwritten starting from the oldest record. The server 110 may obtain these data and use them to detect the moving direction of the user.

[0040] <Outline of merging position estimation using speech and images> An overview of merging position estimation using speech and images, which is executed in the server 110, will be described with reference to Fig. 5. As described above, this process is executed after the user 130 and the vehicle 100, which are in distant locations, approach each other to a degree that allows them to visually confirm a target or the like (a visual landmark). Fig. 5 shows how the relative positional relationship between the user and the vehicle is understood using the user's speech information and image information captured by the vehicle 100.

[0041] First, in S501, the user 130 makes an utterance to the communication device 120 indicating the meeting position (for example, "In front of the postbox!"). The communication device 120 acquires the user's utterance with the microphone 507 and transmits the acquired utterance information to the server 110. In S502, the server 110 performs voice recognition of the user's utterance information, and in S503, extracts information regarding the user's position from the utterance information. Here, the information regarding the user's position is the name of a physical object visible to the user as described above, and information indicating the name of a landmark such as a building is extracted.

[0042] On the other hand, in S511, since the vehicle 100 is approaching the user 130 to a certain extent, the detection units 15-17, which are imaging devices, capture images of the surroundings of the vehicle 100, and transmit one or more pieces of image data to the server 110 as image information. The image data transmitted to the server 110 here is not limited to data captured by the vehicle 100, and data captured by a camera installed in another vehicle or a surveillance camera installed in the vicinity may be transmitted. In S512, the server 110 performs image recognition (image analysis) of the one or more pieces of image data received, and in S513, extracts the landmark name extracted in S503 from the image recognition result. Here, the extraction of the landmark name is taken as an example, but the present invention is not limited to such control, and may be, for example, an object such as a building recognized based on the result of image recognition. After that, in S520, the server 110 uses the results of S503 and S513 to understand the positional relationship between the user and the vehicle, estimates the user, and further estimates the target position that is the junction position.

[0043] Next, a series of operations of the process of adjusting the merging position in the server 110 according to this embodiment will be described with reference to FIG. 6. This process is realized by the control unit 404 executing a program. In the following description, for the sake of simplicity, the control unit 404 is assumed to execute each process, but the corresponding process is executed by each part of the control unit 404 (described above in FIG. 4). Here, a flow in which the user and the vehicle finally merge will be described, but the characteristic configuration of the present invention is a configuration related to the estimation of the user, and the configuration for estimating the merging position is not an essential configuration. That is, in the following, a processing procedure including control related to the estimation of the merging position will be described, but control may be performed so that only the processing procedure related to the estimation of the user is performed.

[0044] In S601, the control unit 404 receives a request (a merging request) to start merging with the vehicle 100 from the communication device 120. In S602, the control unit 404 acquires user position information from the communication device 120. The user position information is position information acquired by the GPS 505 of the communication device 120. In S603, the control unit 404 specifies a rough merging area (also simply referred to as a merging area or a predetermined region) based on the user position acquired in S602. The merging area is, for example, an area with a radius of a predetermined distance (for example, several hundred meters) from the current position of the user 130 (communication device 120).

[0045] In S604, the control unit 404 tracks the movement of the vehicle 100 heading toward the meeting area, for example, based on the position information periodically transmitted from the vehicle 100. Note that the control unit 404 can select, for example, from among a plurality of vehicles located around the current position of the user 130 (or a destination point after a predetermined time), a vehicle closest to the current position as the vehicle 100 that will meet with the user 130. Alternatively, when information designating a specific vehicle 100 is included in the meeting request, the control unit 404 may select the vehicle 100 as the vehicle 100 that will meet with the user 130.

[0046] In S605, the control unit 404 determines whether the vehicle 100 has reached the merging area. For example, when the distance between the vehicle 100 and the communication device 120 is within the radius of the merging area, the control unit 404 determines that the vehicle 100 has reached the merging area and proceeds to S606. If not, the server 110 returns the process to S605 and waits for the vehicle 100 to reach the merging area.

[0047] In S606, the control unit 404 sets a probability distribution for estimating the user using the utterance, and estimates the user in the captured image. Details of the user estimation process using the user's utterance here will be described later. Next, in S607, the control unit 404 further estimates the junction position based on the user estimated in S606. For example, by estimating the user in the captured image, if the user utters "a red postbox nearby" or the like as the junction position, the junction position can be estimated more accurately by searching for a red postbox close to the estimated user. After that, in S608, the control unit 404 transmits position information of the junction position to the vehicle. That is, the control unit 404 moves the vehicle 100 to the junction position by transmitting the junction position estimated in the process of S607 to the vehicle 100. After transmitting the junction position to the vehicle 100, the control unit 404 ends the series of operations.

[0048] <Probability distribution settings> Next, an example of estimating a user by setting a probability distribution of the user's presence in a predetermined area from at least one of the user's speech information and position information will be described with reference to Fig. 7. Here, estimating a user basically means estimating which of the people detected in a captured image capturing the periphery of the predetermined area is the user.

[0049] 7(a) shows a probability distribution when a user utters, "I just passed P." "P" indicates a landmark such as a specific commercial facility, and the server 110 searches for "P" extracted from the utterance information on a map based on rough user location information. The rough user location information is identified from a specific area extracted from the utterance information, location information obtained from the GPS 505 of the communication device 120 carried by the user, etc.

[0050] When identifying the user's approximate location information from the speech information, for example, if the user utters "I just passed P" before uttering another utterance related to a landmark, P may be identified based on the two utterances. For example, if the user utters "I am in front of Q" before uttering "I just passed P", a search is performed on the map for "P" that exists within a predetermined range of "Q". Like "P", "Q" indicates a landmark such as a specific commercial facility. In this way, the landmark P can be identified even when the location information obtained from the GPS 505 cannot be used. When "P" is searched for on the map, the server 110 divides the predetermined area 700 centered on "P" into multiple areas and sets a probability distribution indicating the probability that the user exists in each area.

[0051] Here, the probability that a user exists is set for each divided area, and multiple patterns are prepared in advance according to the user's speech information. Basically, the probability is set by determining the user's moving direction relative to the landmark P. Here, the user's moving direction is a concept that includes various moving directions, for example, the moving direction indicating the direction on the map (north, south, east, west) and the moving direction relative to the landmark such as whether the user is approaching or moving away from the landmark P, and is estimated by information obtained from at least one of the user's speech information and position information. For example, in FIG. 7(a), the user 130 utters "I passed the landmark P," and it can be determined that the user is moving away from the landmark P. Therefore, as shown in FIG. 7(a), the server 110 sets the probability of the area where a person is moving away from the landmark P relatively high (probability "high"), sets the surrounding area next highest (probability "medium"), and sets the other areas relatively low (probability "low"). When determining the area with a "high" probability, the user's moving direction is estimated from at least one of the user's rough position information and speech information relative to the landmark P, and it is possible to determine which area with a high probability is set relative to the landmark P according to the estimated moving direction. In the example of Fig. 7(a), since it is estimated that the user's moving direction is from north to south, the probability of the area corresponding to the south of the landmark P is set high. The user's moving direction is estimated based on at least one of the utterance information and the location information acquired from the GPS 505. When estimating the user's moving direction from the utterance information, the estimation may be performed based on the utterance information before the user utters "I passed the landmark P." For example, if the user uttered an utterance indicating that he / she was in the vicinity of the landmark Q located north of the landmark P before uttering "I passed the landmark P," the user's moving direction can be estimated to be from the north to the south.

[0052] After that, the server 110 performs image recognition of the captured image captured by the detection units 15-17 of the vehicle 100, and detects one or more people existing in the predetermined area. Here, not only the image captured by the vehicle 100 but also image data captured by other imaging devices can be used. The server 110 determines the movement direction of each of the detected people by image analysis, and sets a high probability to a user who is performing an action that matches the movement direction of the user acquired from at least one of the user's speech information and the position information. In FIG. 7(a), the detected people are indicated by "1", "2", and "3", and the movement direction of these people is further indicated by arrows. Therefore, since the user has uttered "I passed the landmark P", the probability of "2" who has passed P is set to the highest, followed by "3" who is currently passing by, and the lowest probability is set to "1" who is approaching the landmark "P". Therefore, the probability is set to each detected person in the relationship of "2">"3">"1". Furthermore, the server 110 obtains a composite probability by combining the probability assigned to the person and the probability set for the area in which the person is located, and estimates the person with the highest probability as the user. In the example of Fig. 7(a), the person "2" is estimated to be the user.

[0053] Fig. 7(b) shows the probability distribution when the user utters "I am approaching P now." The server 110 searches for "P" on the map in the same manner as described in Fig. 7(a). When "P" is searched for on the map, the server 110 divides a predetermined area 710 centered on "P" into multiple areas and sets a probability distribution indicating the probability that the user exists in each area.

[0054] In FIG. 7(b), the user 130 utters "approaching landmark P," and it can be determined that the user is approaching landmark P. Therefore, as shown in FIG. 7(b), the server 110 sets the probability of the area where a person is approaching landmark P relatively high (high probability), sets the surrounding area next high (medium probability), and sets the other areas relatively low (low probability). When determining the area with high probability, it is possible to determine which area with high probability should be set with respect to landmark P based on the user's rough position information with respect to landmark P. In the example of FIG. 7(b), the previous user's position can be roughly recognized as being north of landmark P, so the probability of the corresponding area is set high.

[0055] After that, the server 110 performs image recognition of the captured image captured by the detection units 15-17 of the vehicle 100, and detects one or more people existing in the predetermined area. Here, not only the image captured by the vehicle 100 but also image data captured by other imaging devices can be used. The server 110 determines the movement direction of each of the detected people by image analysis, and sets a high probability to a user who is performing an action that matches the movement direction of the user acquired from at least one of the user's speech information and the position information. In FIG. 7(b), the detected people are indicated by "1", "2", and "3", and the movement direction of these people is further indicated by arrows. Therefore, since the user has uttered "approaching the landmark P", the probability of "1" approaching P is set to the highest, and low probabilities are set to "2" and "3" that are away from the landmark "P". Therefore, the probability is set to each detected person in the relationship of "1">"2"="3". Furthermore, the server 110 obtains a composite probability by combining the probability assigned to the person and the probability set for the area in which the person is located, and estimates the person with the highest probability as the user. In the example of Fig. 7(b), the person "1" is estimated to be the user.

[0056] Next, a series of operations of the process of estimating a user using an utterance (S606) in the server 110 will be described with reference to Fig. 8. Note that this process is realized by the control unit 404 executing a program, similar to the process shown in Fig. 6.

[0057] In S801, the control unit 404 transmits voice information inquiring about a "place associated with a visual landmark" to the communication device 120. The voice information inquiring about a place associated with a visual landmark includes, for example, a voice such as "Is there a store nearby?". This voice information inquiring about a place associated with a visual landmark may be information that is determined in advance and stored in the storage unit 403.

[0058] In S802, the control unit 404 receives the user's speech information from the communication device 120, recognizes the speech content, and identifies a predetermined area centered on a landmark included in the speech content. At this time, the user's speech information includes location information related to a visual landmark, such as "there is a building with a xx coffee shop." Furthermore, in S803, the control unit 404 acquires the user's moving direction according to the speech information and the user's rough position, as described above with reference to FIG. 7, and divides the identified predetermined area to set a probability distribution.

[0059] Next, in S804, the control unit 404 acquires an image of the predetermined area specified in S802 from the vehicle 100 or the like and analyzes it. Specifically, the control unit 404 analyzes the acquired captured image to detect one or more people (candidate users) located within the predetermined area. Furthermore, the control unit 404 estimates the moving direction (moving direction of one or more people) for each detected person from the orientation and posture of the person. Note that the control unit 404 can also acquire time-series image data, and may specify the moving direction from the difference in the time position. Furthermore, in S805, the control unit 404 assigns a probability that each detected person is a user requesting to merge with the vehicle based on the moving direction of the candidate user. In the process here, as described with reference to FIG. 7, the control unit 404 assigns the probability according to the speech information by the user.

[0060] In S806, the control unit 404 estimates the user using the speech information, position information, and image information, and ends this process. Detailed processing will be described later with reference to FIG. 9. If a probability distribution capable of identifying a user has already been set in the processing up to S805, a person having the highest probability or a probability equal to or greater than a predetermined value is identified as the user in S806. On the other hand, if a single user cannot be identified, further conversations with the user are conducted to narrow down the candidate users, as will be described with reference to FIG. 9.

[0061] The detailed process of S806 will be described with reference to Fig. 9. Note that this process is realized by the control unit 404 executing a program, similar to the process shown in Fig. 6.

[0062] In S901, the control unit 404 calculates a composite probability by combining the probability assigned to each divided area in the predetermined area set in the flowchart of FIG. 8 and one or more detected people, and determines whether there are multiple candidate users with a high composite probability. For example, the composite probability is calculated by combining the probability assigned to the candidate user and the probability set for the divided area corresponding to the position of the candidate user. If there are multiple candidate users, proceed to S902, and if not, proceed to S905. In S905, the control unit 404 identifies the candidate user with the highest composite probability as the user, and ends the process.

[0063] On the other hand, if there are multiple candidate users, the user cannot be identified, so in S902, the control unit 404 further analyzes the image captured in the predetermined area to further extract the features of the detected person. The features here are the features of the clothes worn by the user, such as a hat or glasses, and the features of the items carried by the user, such as a bag, and indicate, for example, the color, shape, number, etc. of those items.

[0064] Next, in S903, the control unit 404 transmits additional voice information (e.g., "What color clothes are you wearing?") to the communication device 120 to ask about the user's characteristics according to the characteristics extracted in S902. Here, when there are multiple candidate users, it is desirable for the transmitted voice information to ask about matters that are different characteristics of each candidate user. This makes it possible to identify users more efficiently. For example, when the candidate users are wearing clothes of different colors, it is desirable to ask the user using voice information such as "What color clothes are you wearing?"

[0065] Thereafter, in S904, the control unit 404 receives user utterance information from the communication device 120 and corrects the probability distribution. Note that, here too, location information of the communication device 120 may be received and used to correct the probability distribution. Here, the control unit can select at least one of the probability assigned to a person and the probability set in the divided area as the probability to be corrected in the content of the utterance information. After correcting the probability distribution, the process returns to S901, and the control unit 404 judges again whether or not there are still multiple candidate users. The control unit 404 repeatedly executes the processes of S902 to S904 until the candidate users are narrowed down to one.

[0066] <Example of display on communication device> Fig. 10 shows an example of the display unit of communication device 120 showing the process of estimating a user using speech and images. A display screen 1000 shown in Fig. 10 is displayed on display operation unit 504 of communication device 120 according to screen information provided by server 110, and shows how the user is estimated during adjustment of the merging position between the vehicle and the user. Therefore, display operation unit 504 of communication device 120 may function as a web browser for server 110, which is a web server.

[0067] Display 1001 shows a state where the communication device 120 acquires a user's speech and displays the acquired content as a character string. For example, the user can provide his / her own speech to the communication device 120 by speaking to the communication device 120 while pressing the microphone button 1006. Since the character string to be displayed requires language analysis, it is preferable to receive and display the analysis result of the speech information from the server 110, rather than performing the language analysis in the communication device 120. This can reduce the processing load in the communication device 120, and eliminate the need to implement a language analysis module in the communication device 120. Display 1002 is displayed when there are multiple candidate users in S901 of FIG. 9 as a result of setting the probability distribution for the predetermined area described in FIG. 8, and includes a message indicating that there are multiple candidate users in the corresponding area where the user is likely to be present.

[0068] Display 1003 shows an inquiry from the server 110 to the user, and displays the voice information transmitted from the server 110 to the communication device 120 as a message (e.g., "What color clothes are you wearing?"). At this time, the communication device 120 may output a voice according to the message via the speaker 508. After that, the user speaks a response to the inquiry into the microphone 507 of the communication device 120 while pressing the microphone button 1006. Display 1004 shows the user's response, and displays the speech information analyzed by the server 110 as a message. In display 1004, the user's utterance interpreted by the server 110 (e.g., "You are wearing red clothes") is displayed. After that, when the server 110 narrows down the candidate users to one and identifies the user, the message of display 1005 ("User has been estimated") is displayed.

[0069] A map display button 1007 may be further displayed in an operable manner on the display screen 1000. When the map display button 1007 is operated, a transition to a map display screen 1100, which will be described later, occurs. The map display button 1007 may be displayed in an operable manner at the stage when the user is estimated.

[0070] 11 shows a map display screen 1100 that displays the estimated positional relationship between the user and the vehicle. The map display screen 1100 is displayed on the display operation unit 504 of the communication device 120, and displays a map of the surroundings of a predetermined area.

[0071] Display 1101 on the map shows the user estimated in S606. Display 1002 shows the vehicle 100 merging with user 130. Display 1103 shows the position of the landmark identified from the user's speech information in S802. Display 1004 shows the merging position estimated from the user's speech information in S607. In this way, the map display screen 1100 shows the estimated user, landmark, merging position, etc. on a map of a predetermined area to show their positional relationship. The user can confirm these positional relationships and readjust the merging position. Button 1105 is a button for transitioning to the speech screen, and when operated, transitions to the display screen 1000. The user can operate button 1105 to return to the display screen 1000 and request the server 110 to readjust the merging position, etc., by speech.

[0072] Here, an example of a map display screen displayed at the stage of estimating a user will be described, but this is not intended to limit the present invention. For example, at the stage when a plurality of candidate users are displayed, a map display may be performed, and the plurality of candidate users may be displayed on a map showing a predetermined area in a selectable manner, and the user may select a candidate user that represents the user. Alternatively, a display screen may be provided that allows a plurality of candidate users detected on a captured image to be selected on the captured image. In this case, for example, a detected person may be surrounded by a line, and the user may select the inside of the line to select a user. In this way, by having the user select the user, it is possible to more efficiently and accurately identify the user. In addition, the estimated junction position may not be displayed, and the junction position may not be estimated in the first place. In this case, for example, the vehicle 100 may be controlled so as to approach the estimated user, or the user may be requested to specify the junction position again. In addition, the vehicle 100 may propose a junction position to the user.

[0073] <Modification> A modified example according to the present invention will be described below. In the above embodiment, an example in which the adjustment process of the merging position is executed in the server 110 has been described. However, the above-mentioned adjustment process of the merging position can also be executed on the vehicle side. In this case, as shown in FIG. 12, the information processing system 1200 is composed of a vehicle 1210 and a communication device 120. The user's speech information is transmitted from the communication device 120 to the vehicle 1210. The image information captured by the vehicle 1210 is processed by a control unit in the vehicle instead of being transmitted via a network. The configuration of the vehicle 1210 may be the same as that of the vehicle 100, except that the control unit 30 can execute the adjustment process of the merging position. The control unit 30 of the vehicle 1210 operates as a control device in the vehicle 1210, and executes a stored program to execute the above-mentioned adjustment process of the merging position. The communication between the server and the vehicle in the series of operations shown in FIG. 6, FIG. 8, and FIG. 9 may be performed inside the vehicle (for example, inside the control unit 30, or between the control unit 30 and the detection unit 15). Other processes can be executed in the same manner as the server.

[0074] In this way, in a vehicle control device that adjusts a merging position where a user and a vehicle merge, at least one of speech information and position information related to the merging position, including visual landmarks, is acquired from a communication device. Then, the visual landmarks included in the speech information are identified, the user's movement direction is acquired from at least one of the speech information and position information, and the user in the captured image is estimated based on the acquired user's movement direction. Furthermore, the merging position is estimated based on the estimated user.

[0075] <Summary of the embodiment> 1. The information processing device (e.g., 110) of the above embodiment is A first acquisition means (401, 413) for acquiring at least one of information on a user's speech and location information of the communication device from the communication device of the user; A determination means (417) for determining a predetermined area according to a mark included in the speech information; A setting means (417, S801 to S805) for acquiring a moving direction of the user from at least one of the acquired speech information and position information acquired from the communication device of the user, and for setting a probability distribution of the user's presence in the predetermined area based on the acquired moving direction of the user; and an estimation means (417, S806) for estimating the user based on the set probability distribution.

[0076] According to this embodiment, it is possible to preferably estimate the user.

[0077] 2. In the information processing device of the above embodiment, a second acquisition means (401) is further provided for acquiring an image captured around the identified specific area, and the setting means detects one or more people in the image acquired by the second acquisition means (S804).

[0078] According to this embodiment, the user can be identified from among people detected in a captured image around a specific area identified based on the user's speech information, and the user can be estimated more accurately.

[0079] 3. In the information processing device of the above embodiment, the estimation means analyzes the movement direction of the one or more people relative to the landmark from the captured image, and estimates the user based on the probability distribution set by the setting means and the analyzed movement direction of the one or more people (S805).

[0080] According to this embodiment, when estimating a user, the moving direction of each detected person is analyzed, thereby making it possible to identify the user more accurately.

[0081] 4. In the information processing device of the above embodiment, the estimation means assigns a higher probability to a person whose movement direction matches the user's movement direction among the analyzed movement directions of the one or more people, compared to a person whose movement direction does not match (S805).

[0082] According to this embodiment, when estimating a user, the moving direction of each detected person is analyzed and combined with speech information by the user, thereby enabling a more accurate identification of the user.

[0083] 5. In the information processing device of the above embodiment, the estimation means estimates the user based on a composite probability of the probability distribution set by the setting means and the probability assigned to the one or more people (S901, S905). The estimation means also estimates the person with the highest corresponding composite probability or a person with a predetermined value or more as the user.

[0084] According to this embodiment, when estimating a user, the moving direction of each detected person is analyzed and combined with speech information by the user, thereby enabling a more accurate identification of the user.

[0085] 6. In the information processing device of the above embodiment, when the estimation means cannot identify a single user, it further acquires speech information by the user using the first acquisition means, and updates the composite probability from the acquired speech information and captured image to estimate the user (S902 to S904).

[0086] According to this embodiment, by making additional inquiries to the user, it is possible to narrow down the candidate users and more accurately identify the user.

[0087] 7. In the information processing device of the above embodiment, the first acquisition means makes an inquiry to the user based on an analysis of an image captured by a moving object located in the vicinity of the user, and acquires speech information by the user from the communication device in response to the inquiry (S902, S903).

[0088] According to this embodiment, by conducting additional queries to the user based on image analysis, it is possible to narrow down the candidate users and more accurately identify the user.

[0089] 8. In the information processing device of the above embodiment, the second acquisition means acquires at least one of a captured image captured by a moving object located in the vicinity of a user and a captured image captured by an imaging means located in the vicinity of the moving object.

[0090] According to this embodiment, images captured by not only the imaging means installed on the moving body, but also the imaging means of other moving bodies and surrounding surveillance cameras can be used, making it possible to more accurately estimate the user and the meeting point.

[0091] 9. The information processing device of the above embodiment further includes a providing means for providing the communication device with screen information that displays a result of language analysis of the speech information acquired by the first acquiring means (FIG. 10).

[0092] According to this embodiment, it is possible to notify the user of how the system recognizes the user's speech information, and to prevent inferences based on misinterpreted analysis.

[0093] 10. In the information processing device of the above embodiment, the providing means further provides to the communication device screen information displaying a plurality of selectable candidate users from among one or more people detected in the captured image acquired by the second acquisition means.

[0094] 11. In the information processing device of the above embodiment, the estimation means further estimates a merging position between the user and the vehicle according to the estimated user (S607).

[0095] According to this embodiment, it is possible to suitably estimate the user and adjust the merging position between the user and the vehicle that is about to merge.

[0096] According to this embodiment, it is possible to allow the user to select himself / herself from a plurality of candidate users, and it is possible to identify the user more accurately. [Explanation of symbols]

[0097] 100: vehicle, 110: server, 120: communication device, 404: control unit, 413: user data acquisition unit, 414: voice information processing unit, 415: image information processing unit, 416: junction position estimation unit, 417: user estimation unit

Claims

1. An information processing device, a first acquisition means for acquiring at least one of information on a speech by a user and information on a location of the communication device from the communication device of the user; A specifying means for specifying a predetermined area according to a mark indicating a meeting position with the user, the mark being included in the speech information; A second acquisition means for acquiring an image captured around the predetermined area; a setting means for acquiring a moving direction of the user from at least one of the acquired speech information and position information acquired from a communication device of the user, and setting a probability distribution of the user's presence in the predetermined area based on the acquired moving direction of the user; an estimation means for analyzing, from the captured image, a moving direction of one or more people detected in the captured image acquired by the second acquisition means relative to the landmark, and for identifying a position of the user requesting joining from among the one or more people based on the set probability distribution and the analyzed moving direction of the one or more people, and estimating a person corresponding to the user; An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the estimation means assigns a higher probability to a person whose movement direction matches the user's movement direction among the analyzed movement directions of the one or more people, compared to a person whose movement direction does not match.

3. The information processing device according to claim 2, characterized in that the estimation means identifies the position of the user requesting to meet up among the one or more people based on a composite probability of the probability distribution set by the setting means and the probability assigned to the one or more people, and estimates the person corresponding to the user.

4. The information processing apparatus according to claim 3 , wherein the estimation means estimates the person having the highest corresponding composite probability as the person corresponding to the user among the one or more people.

5. The information processing apparatus according to claim 3 , wherein the estimation means estimates a person whose corresponding composite probability is equal to or greater than a predetermined value as the person corresponding to the user, among the one or more people.

6. The information processing device described in any one of claims 3 to 5, characterized in that when a single user cannot be identified, the estimation means further acquires speech information by the user using the first acquisition means, updates the synthesis probability from the acquired speech information and the captured image, and estimates the person corresponding to the user from among the one or more people.

7. The information processing device according to claim 6, characterized in that, when the estimation means is unable to identify a single user, the first acquisition means analyzes characteristics of candidate users from the captured image, queries the users based on the analyzed characteristics, and acquires speech information from the users from the communication device in response to the query.

8. The information processing device according to any one of claims 3 to 7, characterized in that the second acquisition means acquires at least one of an image captured by a moving object located in the vicinity of the user and an image captured by an imaging means located in the vicinity of the moving object.

9. 9. The information processing apparatus according to claim 3, further comprising a providing unit that provides the communication device with screen information that displays a result of language analysis of the speech information acquired by the first acquiring unit.

10. The information processing device according to claim 9, further characterized in that, when there are multiple candidate users whose combined probability is equal to or greater than a predetermined value, the providing means provides to the communication device screen information displayed on the captured image or map, in which the user requesting to meet up is selectable from the multiple candidate users among one or more people detected in the captured image acquired by the second acquisition means, and the estimation means estimates the user selected via the displayed captured image or map as the user requesting to meet up.

11. The information processing apparatus according to claim 1 , wherein the estimation means further estimates a meeting position between the user and a moving object according to a person corresponding to the estimated user.

12. A method for controlling an information processing device, comprising: a first acquisition step in which a first acquisition means acquires at least one of utterance information by a user and location information of the communication device from the communication device of the user; A specifying step in which a specifying means specifies a predetermined area according to a mark indicating a meeting position with the user, the mark being included in the speech information; A second acquisition step in which a second acquisition means acquires an image captured around the predetermined area; a setting step in which a setting means acquires a moving direction of the user from at least one of the acquired utterance information and position information acquired from a communication device of the user, and sets a probability distribution of the user's presence in the divided areas of the predetermined area based on the acquired moving direction of the user; an estimation step in which an estimation means analyzes, from the captured image acquired in the second acquisition step, a moving direction of one or more people detected in the captured image relative to the landmark, and identifies a position of the user requesting joining from among the one or more people based on the set probability distribution and the analyzed moving direction of the one or more people, thereby estimating a person corresponding to the user; 13. A method for controlling an information processing device, comprising:

13. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

14. A mobile object, A communication means for communicating with a user's communication device; An imaging means for imaging the surroundings of the moving object; a first acquisition means for acquiring at least one of information on a user's speech and information on a location of the communication device from the communication device of the user by the communication means; A specifying means for specifying a predetermined area according to a mark indicating a meeting position with the user, the mark being included in the speech information; A second acquisition means for acquiring an image captured around the predetermined area; a setting means for acquiring a moving direction of the user from at least one of the acquired speech information and position information acquired from a communication device of the user, and setting a probability distribution of the user's presence in the divided regions of the predetermined region based on the acquired moving direction of the user; an estimation means for analyzing, from the captured image, a moving direction of one or more people detected in the captured image acquired by the second acquisition means relative to the landmark, and for identifying a position of the user requesting joining from among the one or more people based on the set probability distribution and the analyzed moving direction of the one or more people, and estimating a person corresponding to the user; A moving object comprising:

15. A method for controlling a moving object including a communication means for communicating with a communication device of a user and an imaging means for imaging an image of a surrounding of the moving object, comprising: a first acquisition step in which a first acquisition means acquires at least one of utterance information by a user and location information of the communication device from the communication device of the user by the communication means; A specifying step in which a specifying means specifies a predetermined area according to a mark indicating a meeting position with the user, the mark being included in the speech information; A second acquisition step in which a second acquisition means acquires an image captured around the predetermined area; a setting step of acquiring a moving direction of the user from at least one of the acquired speech information and position information acquired from a communication device of the user, and setting a probability distribution of the user's presence for the divided regions of the predetermined region based on the acquired moving direction of the user; an estimation step in which an estimation means analyzes, from the captured image acquired in the second acquisition step, a moving direction of one or more people detected in the captured image relative to the landmark, and identifies a position of the user requesting joining from among the one or more people based on the set probability distribution and the analyzed moving direction of the one or more people, thereby estimating a person corresponding to the user; A method for controlling a moving object, comprising:

Images