Information processing device, mobile object control device, information processing device control method, mobile object control method, and program

Abstract

To provide an information processing device which can estimate an indication position to a mobile object from speech information on a user who uses the mobile object.SOLUTION: An information processing device concerning the present invention estimates an indication position to a mobile object used by a user, and comprises: speech acquisition means for acquiring speech information regarding the indication position including a visual marker from a communication device used by the user; image acquisition means for acquiring a photographic image photographed in the mobile object; determination means for determining an object area in the photographic image corresponding to the visual marker included in the speech information; and estimation means for estimating the indication position on the basis of the object area determined by the determination means.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control device for a mobile object, a control method for an information processing device, a control method for a mobile object, and a program. [Background technology]

[0002] In recent years, ultra-compact mobility (also known as micromobility), electric vehicles with a passenger capacity of around one or two people, have become known and are expected to become widespread as a convenient means of transportation.

[0003] A car sharing system has been proposed in which such ultra-compact mobility vehicles are used for sharing (Patent Document 1). In this car sharing system, a vehicle management server receives an application message from a user's communication device, which includes the usage start time and rental location of the vehicle to be car shared. Then, based on the content of the application message and the current location of the transport vehicle, a transport vehicle that can arrive at the rental location by the usage start time is identified, and the identified transport vehicle is made to transport the shared car to the rental location. The user can use the shared car by visiting the rental location at the specified usage start time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-77035 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when a user uses an ultra-compact mobility, there may be a use case in which the ultra-compact mobility and the user dynamically adjust the meeting position while moving, rather than the user visiting a rental location where the ultra-compact mobility stops. Such a use case is effective when it becomes difficult to meet at a pre-specified position due to congestion, or when the user first specifies a rough area or building, etc., and adjusts the specific meeting position when they arrive nearby. Alternatively, when the ultra-compact mobility is already present in the vicinity of the user, there may be a use case in which the ultra-compact mobility is moved according to the user's intention, such as a waiting instruction such as "wait next to that postbox." In these use cases, if the user's intended position is estimated from the user's speech, and the ultra-compact mobility moves to that position, the user can more easily handle the ultra-compact mobility.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to realize a technology capable of estimating a pointed position on a moving object from speech information of a user who uses the moving object. [Means for solving the problem]

[0007] According to the present invention, Autonomous movement of a moving object used by a user Destination An information processing device that estimates a pointing position related to a position, a speech acquisition means for acquiring speech information related to a pointing position, including a visual landmark, from a communication device used by the user; an image acquisition means for acquiring an image captured by the moving object; a determining means for determining an object area in the captured image corresponding to the visual landmark included in the speech information; an estimation means for estimating the pointing position based on the object region determined by the determination means, An information processing device is provided, characterized in that the determination means calculates a probability distribution indicating the probability that one or more object regions recognized in the captured image correspond to the visual landmark included in the speech information, and determines an object region having a probability equal to or greater than a first threshold as the object region corresponding to the visual landmark included in the speech information. Effect of the Invention

[0008] According to the present invention, it is possible to estimate a pointed position on a moving object from speech information of a user who uses the moving object. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a block diagram showing an example of the hardware configuration of a vehicle as an example of a moving body according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram showing an example of a functional configuration of a vehicle according to the present embodiment; [Figure 4] FIG. 2 is a block diagram showing an example of the functional configuration of a server as an example of an information processing device according to the embodiment; [Figure 5A] FIG. 1 is a diagram for explaining estimation of a meeting point using speech and images according to the present embodiment. [Figure 5B] FIG. 2 is a diagram for explaining estimation of a meeting point using speech and images according to the present embodiment. [Figure 6] A flowchart showing a series of operations in the process of adjusting the joining position according to the present embodiment. [Figure 7] A flowchart showing a series of operations in a process for adjusting a meeting position using speech according to the present embodiment. [Figure 8] A flowchart showing a series of operations in a process for adjusting a meeting position using speech and images according to the present embodiment. [Figure 9] FIG. 13 is a diagram illustrating an example of an information processing system according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (Configuration of Information Processing System) The configuration of an information processing system 1 according to this embodiment will be described with reference to Fig. 1. The information processing system 1 includes a vehicle 100, a server 110, and a communication device 120.

[0012] The vehicle 100 is an example of a moving body capable of autonomous movement, and is, for example, an ultra-compact mobility vehicle equipped with a battery and moving mainly by the power of a motor. An ultra-compact mobility vehicle is a very small vehicle that is more compact than a general automobile and has a passenger capacity of about one or two people. In this embodiment, the vehicle 100 is, for example, a four-wheeled vehicle. Note that in the following embodiments, the moving body is not limited to a vehicle, and may include a small mobility vehicle that runs alongside a walking user to carry luggage or lead a person, and may also include other moving bodies capable of autonomous movement (for example, a walking robot, etc.).

[0013] The vehicle 100 is connected to the network 140 via wireless communication such as Wi-Fi or fifth generation mobile communication. The vehicle 100 can measure the inside and outside of the vehicle (such as the vehicle's position, driving state, and surrounding object targets) using various sensors and transmit the measured data to the server 110. The data collected and transmitted in this manner is generally called floating data, probe data, traffic information, and the like. Information about the vehicle is transmitted to the server 110 at regular intervals or in response to the occurrence of a specific event. The vehicle 100 can travel by autonomous driving even when the user 130 is not on board. The vehicle 100 receives information such as a control command provided by the server 110, or controls the operation of the vehicle using data measured by the vehicle itself.

[0014] The server 110 is configured with one or more server devices, and is capable of acquiring vehicle-related information transmitted from the vehicle 100, and speech information and position information transmitted from the communication device 120, via the network 111, and controlling the driving of the vehicle 100. The driving control of the vehicle 100 includes a process of adjusting a junction position between the user 130 and the vehicle 100, which will be described later.

[0015] The communication device 120 is, for example, a smartphone, but is not limited thereto, and may be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, etc. The communication device 120 connects to the network 140 via wireless communication such as Wi-Fi or fifth generation mobile communication.

[0016] The network 111 includes a communication network such as the Internet or a mobile phone network, and transmits information between the server 110 and the vehicle 100 or communication device 120 .

[0017] In this information processing system 1, when a user 130 and a vehicle 100 located at distant locations come close enough to visually confirm a target (visual landmark), the merging position is adjusted using speech information and image information captured by the vehicle 100.

[0018] Before the user 130 and the vehicle 100 approach each other to a degree where they can visually confirm a target or the like, the server 110 first moves the vehicle 100 to a rough area that includes the user's current location or the user's predicted location. Then, when the vehicle 100 reaches the rough area, the server 110 transmits speech information (e.g., "Is there a store nearby?") to the communication device 120 to inquire about a location related to a visual landmark. The location related to the visual landmark includes, for example, a name of a location included in map information. The server 110 receives speech information (e.g., "There is a building with a xx coffee shop") including a location related to a visual landmark from the communication device 120. Then, the server 110 acquires the location of the relevant location from the map information and moves the vehicle 100 to the vicinity of the location (i.e., the vehicle and the user approach each other to a degree where they can visually confirm a target or the like).

[0019] (Vehicle configuration) Next, with reference to FIG. 2, a configuration of a vehicle 100 as an example of a vehicle according to this embodiment will be described.

[0020] Fig. 2(A) shows a side view of vehicle 100 according to this embodiment, and Fig. 2(B) shows the internal configuration of vehicle 100. In the figure, arrow X indicates the front-rear direction of vehicle 100, F indicates the front, and R indicates the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of vehicle 100.

[0021] The vehicle 100 is an electric autonomous vehicle equipped with a propulsion unit 12 and using a battery 13 as a main power source. The battery 13 is, for example, a secondary battery such as a lithium ion battery, and the vehicle 100 is self-propelled by the propulsion unit 12 using the power supplied from the battery 13. The propulsion unit 12 is a four-wheeled vehicle equipped with a pair of left and right front wheels 20 and a pair of left and right rear wheels 21. The propulsion unit 12 may be in another form, such as a tricycle. The vehicle 100 is equipped with a seat 14 for one or two people.

[0022] The traveling unit 12 includes a steering mechanism 22. The steering mechanism 22 is a mechanism that changes the steering angle of the pair of front wheels 20 using a motor 22a as a driving source. By changing the steering angle of the pair of front wheels 20, the traveling direction of the vehicle 100 can be changed. The traveling unit 12 also includes a drive mechanism 23. The drive mechanism 23 is a mechanism that uses a motor 23a as a driving source to rotate the pair of rear wheels 21. By rotating the pair of rear wheels 21, the vehicle 100 can be moved forward or backward.

[0023] The vehicle 100 includes detection units 15-17 that detect targets around the vehicle 100. The detection units 15-17 are a group of external sensors that monitor the periphery of the vehicle 100, and in the present embodiment, each is an imaging device that captures an image of the periphery of the vehicle 100, and includes, for example, an optical system such as a lens and an image sensor. However, instead of or in addition to the imaging device, it is also possible to employ a radar or a lidar (Light Detection and Ranging).

[0024] The two detection units 15 are arranged at the front of the vehicle 100, spaced apart in the Y direction, and mainly detect targets in front of the vehicle 100. The detection units 16 are arranged on the left and right sides of the vehicle 100, respectively, and mainly detect targets on the sides of the vehicle 100. The detection unit 17 is arranged at the rear of the vehicle 100, and mainly detects targets behind the vehicle 100.

[0025] 3 is a block diagram of a control system of the vehicle 100. The vehicle 100 includes a control unit (ECU) 30. The control unit 30 includes a processor such as a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor and data used by the processor for processing, and the like. A plurality of sets of the processor, storage device, and interface may be provided for each function of the vehicle 100, and may be configured to be able to communicate with each other.

[0026] The control unit 30 receives the detection results of the detection units 15-17, input information from the operation panel 31, voice information input from the voice input device 33, and control commands (such as transmission of captured images and current position) from the server 110, and executes corresponding processes. The control unit 30 controls the motors 22a and 23a (driving control of the driving unit 12), controls the display of the operation panel 31, notifies the occupants of the vehicle 100 by voice, and outputs information.

[0027] The voice input device 33 picks up the voice of the passenger of the vehicle 100. The control unit 30 can recognize the input voice and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 34 receives a GNSS signal and detects the current position of the vehicle 100.

[0028] The storage device 35 is a large-capacity storage device that stores map data including information on routes on which the vehicle 100 can travel, landmarks such as buildings, stores, etc. The storage device 35 may also store programs executed by the processor, data used by the processor for processing, etc. The storage device 35 may store various parameters of machine learning models for speech recognition and image recognition executed by the control unit 30 (for example, learned parameters and hyperparameters of a deep neural network, etc.).

[0029] The communication device 36 is a communication device that can be connected to the network 140 via wireless communication such as Wi-Fi or fifth generation mobile communication.

[0030] (Server configuration) Next, with reference to FIG. 4, a configuration of the server 110 as an example of an information processing device according to the present embodiment will be described.

[0031] The control unit 404 includes a processor such as a CPU, a storage device such as a semiconductor memory, an interface with an external device, etc. The storage device stores programs executed by the processor, data used by the processor for processing, etc. A plurality of sets of the processor, storage device, and interface may be provided for each function of the server 110 and configured to be able to communicate with each other. The control unit 404 executes programs to perform various operations of the server 110 and adjustment processing of a joining position, which will be described later. In addition to the CPU, the control unit 404 may further include dedicated hardware suitable for executing processing of a machine learning model such as a GPU or a neural network.

[0032] The user data acquisition unit 413 acquires image and position information transmitted from the vehicle 100. In addition, the user data acquisition unit 413 acquires speech information of the user 130 transmitted from the communication device 120. The user data acquisition unit 413 may store the acquired image and position information in the storage unit 403. The image and speech information acquired by the user data acquisition unit 413 is input to a trained model in the inference stage to obtain an inference result, but may also be used as training data for training a machine learning model executed by the server 110.

[0033] The voice information processing unit 414 includes a machine learning model that processes voice information, and executes the learning stage processing and inference stage processing of the machine learning model. The machine learning model of the voice information processing unit 414 performs calculations of a deep learning algorithm using a deep neural network (DNN), for example, to recognize names of places, landmarks such as buildings, store names, and landmarks included in the speech information. The landmarks may include passersby, signs, signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like included in the speech information.

[0034] The DNN is put into a trained state by performing the learning stage processing, and by inputting new speech information to the trained DNN, it is possible to perform recognition processing (inference stage processing) for the new speech information. Note that, in this embodiment, an example will be described in which the server 110 performs the speech recognition processing, but the speech recognition processing may be performed in a vehicle or a communication device, and the recognition result may be transmitted to the server 110.

[0035] The image information processing unit 415 includes a machine learning model that processes image information, and executes a learning stage process and an inference stage process of the machine learning model. The machine learning model of the image information processing unit 415 performs a process of recognizing targets included in the image information, for example, by performing calculations of a deep learning algorithm using a deep neural network (DNN). The targets may include passersby, signs, signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like included in the image.

[0036] The merging position estimation unit 416 executes a process of adjusting the merging position, which will be described later. The process of adjusting the merging position will be described later.

[0037] Generally, the server 110 can use more abundant computational resources than the vehicle 100. In addition, by receiving and storing image data captured by various vehicles, learning data in a wide variety of situations can be collected, enabling learning to be performed in a wider variety of situations.

[0038] The communication unit 401 is a communication device including, for example, a communication circuit, and communicates with external devices such as the vehicle 100 and the communication device 120. The communication unit 401 receives image information and position information from the vehicle 100 and speech information from the communication device 120, and also transmits control commands to the vehicle 100 and speech information to the communication device 120.

[0039] The power supply unit 402 supplies power to each unit in the server 110. The storage unit 403 is a non-volatile memory such as a hard disk or a semiconductor memory.

[0040] (Outline of merging position estimation using speech and images) 5A and 5B, an overview of estimating a meeting position using speech and images, which is executed in the server 110, will be described. As described above, this process is executed after the user 130 and the vehicle 100, which are in distant locations, approach each other to a degree that allows them to visually confirm a target or the like (a visual landmark).

[0041] FIG. 5A shows a schematic example of a result of the server 110 estimating the user's intended meeting position for an image 510 captured by the vehicle 100. In FIG.

[0042] Image 510 is an image capturing the situation in front of vehicle 100. Image 510 captures vending machine 520 and vending machine 530 installed in front of building 580. For example, vending machine 520 is a red vending machine, and vending machine 530 is a blue vending machine.

[0043] For example, consider a case where user 130 utters "Stop in front of the blue vending machine." Server 110 extracts a visual landmark from the utterance information. The visual landmark is, for example, a vending machine. Server 110 recognizes the vending machine from the image information and identifies the area of ​​the recognized vending machine. The example shown in FIG. 5A indicates that the area of ​​red vending machine 520 is identified as 540, and the area of ​​blue vending machine 530 is identified as 550.

[0044] The server 110 calculates a probability distribution corresponding to the visual landmark for the identified region. In this example, since the visual landmark is a vending machine, a probability distribution that obtains a probability for each "vending machine" is calculated. At this time, since the only blue visual landmark is the vending machine 530, for example, a probability distribution is assigned in which the probability of the blue vending machine 530 is "0.90" and the probability of the red vending machine 520 is "0.10". The setting of the probability distribution described here is an example, and other probability distributions may be set. For example, the "blue vending machine" may be extracted as a visual landmark, and only the region of the blue vending machine 530 may be identified, and the maximum probability may be assigned only to this region.

[0045] FIG. 5B is a schematic diagram showing another example of the result of the server 110 estimating the user's intended meeting position for the image 510 captured by the vehicle 100. In the example shown in FIG. 5B, consider a case where the user 130 utters "Stop in front of the vending machine." The server 110 recognizes the vending machines from the image information and identifies the area of ​​the recognized vending machines. The area of ​​the red vending machine 520 is identified as 540, and the area of ​​the blue vending machine 530 is identified as 550. The server 110 can calculate, for example, a probability distribution in which both the red vending machine 520 and the blue vending machine 530 have the same probability (for example, 0.5). In this case, if the red vending machine 520 is closer to the current position of the user 130 (communication device 120), the server 110 may correct the probability distribution so that the probability of the red vending machine 520 becomes "0.6" and the probability of the blue vending machine 530 becomes "0.4." In this case, although each vending machine is assigned a relatively high probability (e.g., "0.3" or more), the probability does not exceed a predetermined threshold (e.g., 0.9) that allows one to be identified, so server 110 transmits additional speech information (e.g., "Is it a red vending machine?") to communication device 120 to narrow down the visual landmarks. Server 110 repeatedly receives speech information from the user and transmits additional speech information to identify one visual landmark. When server 110 identifies one visual landmark, it estimates that the meeting point is in front of the identified landmark, and moves vehicle 100 to that location to meet the user.

[0046] (A series of operations for adjusting the junction position) Next, a series of operations of the joining position adjustment process in server 110 will be described with reference to Fig. 6. This process is realized by executing a program by control unit 404. In the following description, for simplicity, each process is described as being executed by control unit 404, but the corresponding process is executed by each part of control unit 404 (described above in Fig. 4).

[0047] In S601, the control unit 404 receives a request (a merging request) to start merging with the vehicle 100 from the communication device 120. In S602, the control unit 404 acquires user position information from the communication device 120. The user position information is the position information of the communication device 120. In S603, the control unit 404 specifies a rough merging area (also simply referred to as a merging area) based on the user position acquired in S602. The merging area is, for example, an area with a radius of a predetermined distance (for example, several hundred meters) from the current position of the user 130 (communication device 120).

[0048] In S604, the control unit 404 tracks the movement of the vehicle 100 heading toward the meeting area, for example, based on the position information periodically transmitted from the vehicle 100. Note that the control unit 404 can select, for example, a vehicle closest to the current position of the user 130 (or a destination point after a predetermined time) as the vehicle 100 that will meet with the user 130. Alternatively, when information designating a specific vehicle 100 is included in the meeting request, the control unit 404 may select the vehicle 100 as the vehicle 100 that will meet with the user 130.

[0049] In S605, the control unit 404 determines whether the vehicle 100 has reached the merging area. For example, when the distance between the vehicle 100 and the communication device 120 is within the radius of the merging area, the control unit 404 determines that the vehicle 100 has reached the merging area and proceeds to S606. If not, the server returns the process to S605 and waits for the vehicle 100 to reach the merging area.

[0050] In S606, the control unit 404 executes a process of estimating a meeting position using an utterance. The process of estimating a meeting position using an utterance will be described in detail later.

[0051] In S607, the control unit 404 transmits position information of the junction position to the vehicle. That is, the control unit 404 transmits the junction position estimated in the processing of S606 to the vehicle 100, thereby moving the vehicle 100 to the junction position. After transmitting the junction position to the vehicle 100, the control unit 404 ends the series of operations.

[0052] (A series of operations for estimating meeting points using speech) Furthermore, a series of operations of the process of estimating a meeting point using speech in the server 110 will be described with reference to Fig. 7. Note that this process is realized by the control unit 404 executing a program, similar to the process shown in Fig. 6.

[0053] In S701, the control unit 404 transmits speech information inquiring about a "place associated with a visual landmark" to the communication device 120. The speech information inquiring about a place associated with a visual landmark includes, for example, an utterance such as "Is there a store nearby?". The speech information inquiring about a place associated with a visual landmark may be information that is determined in advance and stored in the storage unit 403.

[0054] In S702, the control unit 404 receives user speech information from the communication device 120, recognizes the speech content, and extracts location information included in the speech content. At this time, the user speech information includes location information related to visual landmarks, such as "there is a building with xx coffee shop."

[0055] In S703, the control unit 404 identifies the position of the place extracted in S702 from the map information. For example, the control unit 404 searches for "xx coffee shop" from the map information in the meeting area, and identifies the position of the place.

[0056] At S704, the control unit 404 transmits the location of the place (associated with the visual landmark) identified from the map information to the vehicle 100 to move the vehicle 100 to the location of the place, and the control unit 404 tracks the movement of the vehicle to the location of the place.

[0057] In S705, the control unit 404 determines whether the vehicle 100 has approached the location of the place. For example, if the control unit 404 determines that the vehicle 100 has approached within a predetermined distance from the location of the place associated with the visual landmark, the process proceeds to S706, otherwise the process of S705 is repeated. Here, the predetermined distance is a distance at which a target or the like serving as a visual landmark can be visually confirmed.

[0058] In S706, the control unit 404 estimates the meeting position using the speech information and image information. Details of this process will be described later. After completing the process of this step, the control unit 404 ends the series of operations and returns to the caller.

[0059] (A series of operations for estimating meeting points using speech and image information) Furthermore, a series of operations of the process of estimating a meeting point using speech and image information in the server 110 will be described with reference to Fig. 8. Note that this process is realized by the control unit 404 executing a program, similar to Figs. 6 and 7.

[0060] In S801, the control unit 404 transmits speech information inquiring about a visual landmark, such as "What is the landmark of the meeting point?" to the communication device 120.

[0061] In S802, the control unit 404 receives the user's speech information, recognizes the speech content, and extracts visual landmark information from the recognized speech content. The visual landmark is, for example, the above-mentioned "vending machine." In S803, the control unit 404 instructs the vehicle 100 to transmit image information of the surroundings (for example, the front), and acquires image information capturing the surroundings of the vehicle 100. The control unit 404 performs object recognition on the acquired image to identify an object region in the image (corresponding to the visual landmark).

[0062] In S804, the control unit 404 calculates a probability distribution indicating the probability that one or more object regions in the image correspond to a visual landmark. For example, if the landmark included in the utterance information is a "vending machine" and there are two or more "vending machine" regions in the image, the control unit 404 may calculate a probability distribution of the object region based on a limited linguistic element of the utterance content (e.g., "blue"). In this case, for example, a probability distribution may be calculated in which the probability of a blue vending machine 530 is "0.90" and the probability of a red vending machine 520 is "0.10".

[0063] When the landmark included in the speech information is a "vending machine" and two or more "vending machine" regions exist in the image, the same probability can be assigned to both object regions. In this case, as illustrated in FIG. 5B, the server 110 may further vary the probability distribution according to the relative positional relationship between the visual landmark and the user 130. If the red vending machine 520 is closer to the current position of the user 130 (communication device 120), the server 110 may correct the probability distribution so that the probability of the red vending machine 520 becomes "0.6" and the probability of the blue vending machine 530 becomes "0.4". A probability distribution can be assigned in which the probability increases in the order of the likelihood of being a candidate when viewed from the direction from which the user is approaching.

[0064] When the spoken information includes a positional relationship with an object, such as "the vending machine on the left side of the building," control unit 404 may calculate a probability distribution that takes into account the relative positional relationship as seen from vehicle 100. For example, the probability of the area of ​​vending machine 520 on the left side of building 580 may be calculated as "0.9," and the probability of the area of ​​vending machine 530 on the right side may be calculated as "0.1."

[0065] Alternatively, when the utterance information is "a vending machine near the building," that is, when the utterance information includes a plurality of object (target) types and each type is recognized by a separate model, a conditional probability may be calculated using the probability distribution of each object. By calculating the conditional probability, even when different models are used for each object type, the probability for each model can be easily combined.

[0066] When the probability distribution of the user position is further taken into consideration, for speech information such as "I am near xx," the probability distribution corresponding to the visual landmark can be calculated as a conditional probability of the probability distribution of the user position and the probability distribution for the area in the image. In this way, even for targets estimated from models with different properties, such as the probability distribution of the user position and the probability distribution for the object area, each probability distribution can be easily combined.

[0067] In S805, the control unit 404 determines whether the probability of the object region calculated in S804 is equal to or greater than a first threshold. If there is an object region having a probability equal to or greater than the first threshold, the process proceeds to S808; otherwise, the process proceeds to S806.

[0068] In S806, the control unit 404 determines whether the probability of two or more object regions corresponding to the visual landmark is equal to or greater than a second threshold. Here, the determination is YES when, as shown in FIG. 5B, all vending machines are given a relatively high probability (e.g., 0.3 or more), but the probability does not exceed the first threshold (e.g., 0.9) that can be identified as one. The second threshold corresponds to, for example, 0.3 in this example. If the determination is YES, the control unit 404 advances the process to S807, and if not, returns the process to S801 to obtain a more effective visual landmark.

[0069] In S807, the control unit 404 transmits additional speech information for narrowing down the visual landmarks, such as, for example, "Is it a red vending machine?" In the case where the visual landmarks cannot be narrowed down to one in relation to the speech information of the user 130 and the image information of the vehicle 100, the ambiguity of the visual landmarks can be reduced by making it possible to obtain additional speech information from the user.

[0070] In S808, the control unit 404 identifies an object region having a probability equal to or greater than the first threshold as an object region corresponding to a visual landmark, and estimates a junction position based on the identified object region. For example, the control unit 404 acquires the self-position of the vehicle 100, and estimates the distance from the self-position to the identified object region in the image. At this time, a known technique can be used for measuring the distance from the vehicle to the specific object in the image. The control unit 404 calculates the position of the object corresponding to the object region based on the self-position and the distance to the object, and estimates the junction position based on the position of the object. At this time, the junction position is estimated in consideration of the relative positional relationship from the vehicle 100. For example, in the example of the image 510 shown in FIG. 5A, in which the user's speech information is "in front of the blue vending machine," the junction position is estimated as the near side as seen from the vehicle 100 with respect to the position of the blue vending machine. The control unit 404 then ends the series of operations and returns to the caller.

[0071] The above-described embodiment is not limited to cases where a user and a vehicle merge, but can be applied to various movement instructions given by the user to the vehicle. That is, the above-described estimation of the merge position for the vehicle is an example of a process of estimating an instruction position for the vehicle. The estimation of the instruction position for the vehicle can also be applied to designation of an instruction position such as a waiting instruction such as "I'm going to buy some coffee, so please wait by the mailbox there" or a destination instruction such as "Please pick up your luggage at the counter over there". That is, the server can estimate the instruction position by receiving speech information related to the instruction position including a visual landmark.

[0072] As described above, in this embodiment, a server that estimates a designated position for a moving body (e.g., a vehicle) used by a user acquires speech information about a junction position including visual landmarks and a captured image captured by the moving body from the communication device 120. Then, an object area in the captured image that corresponds to the visual landmark included in the speech information is determined, and the designated position is estimated based on the determined object area. In this way, it becomes possible to estimate a designated position for a moving body (e.g., a vehicle) from speech information of a user who uses the moving body.

[0073] (Modification) A modified example according to the present invention will be described below. In the above embodiment, an example in which the adjustment process of the junction position is executed in the server 110 has been described. However, the above-mentioned adjustment process of the junction position can also be executed on the vehicle side. In this case, as shown in FIG. 9, the information processing system 900 is composed of a vehicle 910, which is an example of a moving body, and a communication device 120. The user's speech information is transmitted from the communication device 120 to the vehicle 910. The image information captured by the vehicle 910 is processed by a control unit in the vehicle instead of being transmitted via a network. The configuration of the vehicle 910 may be the same as that of the vehicle 100, except that the control unit 30 can execute the adjustment process of the junction position. The control unit 30 of the vehicle 910 operates as a control device in the vehicle 910, and executes a stored program to execute the above-mentioned adjustment process of the junction position. In the series of operations shown in FIG. 6 to FIG. 8, the exchange between the server and the vehicle may be performed inside the vehicle (for example, inside the control unit 30, or between the control unit 30 and the detection unit 15). Other processes can be executed in the same manner as the server.

[0074] In this way, in a mobile object control device that estimates a designated position for a mobile object (e.g., a vehicle) used by a user, speech information regarding the designated position, including visual landmarks, is acquired from a communication device, and a captured image captured by the mobile object is acquired. Then, an object area in the captured image that corresponds to the visual landmarks included in the speech information is determined, and the designated position is estimated based on the determined object area. In this way, it becomes possible to estimate the designated position for the mobile object from the speech information of the user who uses the mobile object.

[0075] <Summary of the embodiment> 1. The information processing device (e.g., 110) of the above embodiment is An information processing device that estimates an indication position of a moving object used by a user, A speech acquisition means (e.g., 413) for acquiring speech information related to a pointing position, including a visual landmark, from a communication device used by a user; An image acquisition means (e.g., 413) for acquiring an image captured in a moving object; A determining means (e.g., 414, 415, 416) for determining object regions in the captured image that correspond to visual landmarks included in the speech information; and an estimation means (eg, 416) for estimating the pointing position based on the object region determined by the determination means.

[0076] According to this embodiment, it becomes possible to estimate the designated position relative to a moving body from the speech information of a user using the moving body (for example, when adjusting the designated position between a user and a moving body (e.g., an ultra-compact mobility) or issuing a standby instruction).

[0077] 2. In the information processing device of the above embodiment, The determination means calculates a probability distribution indicating the probability that one or more object regions recognized in the captured image correspond to a visual landmark, and determines an object region having a probability equal to or greater than a first threshold as the object region corresponding to the visual landmark.

[0078] According to this embodiment, it is possible to accurately calculate the object region that is most likely to correspond to the landmark included in the speech information.

[0079] 3. In the information processing device of the above embodiment, The determination means transmits additional speech information to the user's communication device to narrow down the visual landmarks when the probability that the visual landmarks correspond to two or more object regions is greater than or equal to a second threshold and less than the first threshold.

[0080] According to this embodiment, when the visual landmark cannot be narrowed down to one, it is possible to obtain additional speech information from the user. Also, by obtaining additional speech information, it is possible to reduce the ambiguity of the visual landmark.

[0081] 4. In the information processing device of the above embodiment, The determining means calculates a probability of correspondence to the visual landmark for the two or more object regions as a function of a distance between each object in the two or more object regions and the communication device.

[0082] According to this embodiment, it is possible to assign a higher probability to a candidate in order of likelihood when viewed from the direction from which the user is approaching.

[0083] 5. In the information processing device of the above embodiment, The determining means calculates a probability distribution using a conditional probability when different types of visual markers are included in the speech information.

[0084] According to this embodiment, even if different models are used for different object types, it is possible to easily combine the probabilities of each model. In addition, for objects estimated from models with different properties, the respective probability distributions can be easily combined. 6. In the information processing device of the above embodiment, When the speech information including the pointed position includes a speech indicating a positional relationship between a visual landmark and another object, the determination means calculates a probability distribution for one or more object regions according to the positional relationship with the other object.

[0085] According to this embodiment, it becomes possible to take into account the relative positional relationship from the moving body.

[0086] 7. In the information processing device of the above embodiment, The speech acquisition means further acquires, from the communication device, speech information including a location associated with the visual landmark, before acquiring speech information regarding the pointing position including the visual landmark.

[0087] According to this embodiment, it becomes possible to first approach a target location using location information such as that described in map information, and then adjust the indicated position using visual information.

[0088] 8. In the information processing device of the above embodiment, The speech acquisition means acquires speech information relating to a pointed position including the visual landmark in response to the mobile object approaching within a predetermined distance from a location associated with the visual landmark.

[0089] According to this embodiment, it becomes possible to start estimating the indicated position using visual information when the vehicle approaches within a certain distance such that a visual landmark such as a target can be visually confirmed.

[0090] 9. In the information processing device of the above embodiment, The vehicle is ultra-compact mobility.

[0091] According to this embodiment, it is possible to provide travel instructions using ultra-compact mobility vehicles that can be used more easily than general passenger cars.

[0092] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0093] 10. The control device (e.g., 30) of the moving body (e.g., 910) according to the above embodiment is A control device for a moving object that estimates an indication position of a moving object used by a user, A speech acquisition means (e.g., 30) for acquiring speech information related to a meeting point, including a visual landmark, from a communication device (e.g., 120) used by the user; An image acquisition means (e.g., 30) for acquiring an image captured by the moving body; A determining means (e.g., 30) for determining an object area in the captured image corresponding to the visual landmark included in the speech information; and an estimation means (eg, 30) for estimating the joining position based on the object region determined by the determination means.

[0094] According to this embodiment, it becomes possible to estimate the designated position for a moving body from the speech information of a user using the moving body (for example, when adjusting a meeting position between a user and a moving body (e.g., an ultra-compact mobility vehicle) or issuing a waiting instruction). [Explanation of symbols]

[0095] 100: vehicle, 110: server, 120: communication device, 404: control unit, 413: user data acquisition section, 414: voice information processing section, 415: image information processing section, 416: merging position estimation section

Claims

1. An information processing device that estimates an instruction position regarding a destination position to which a moving object used by a user should autonomously move, comprising: a speech acquisition means for acquiring speech information related to a pointing position, including a visual landmark, from a communication device used by the user; an image acquisition means for acquiring an image captured by the moving object; a determining means for determining an object area in the captured image corresponding to the visual landmark included in the speech information; an estimation means for estimating the pointing position based on the object region determined by the determination means, The information processing device is characterized in that the determination means calculates a probability distribution indicating the probability that one or more object regions recognized in the captured image correspond to the visual landmark included in the speech information, and determines an object region having a probability equal to or greater than a first threshold as the object region corresponding to the visual landmark included in the speech information.

2. 2. The information processing device according to claim 1, wherein the determination means transmits additional speech information to the user's communication device for narrowing down the visual landmarks when the probability that the visual landmarks correspond to two or more object regions is greater than or equal to a second threshold and less than the first threshold.

3. The information processing device according to claim 2 , wherein the determination means calculates a probability of the visual landmark corresponding to the two or more object regions depending on a distance between each object in the two or more object regions and the communication device.

4. 4. The information processing apparatus according to claim 2, wherein the determining means calculates the probability distribution using a conditional probability when the speech information includes different types of the visual landmarks.

5. The information processing device according to any one of claims 1 to 4, characterized in that when the speech information including the indicated position includes an utterance indicating a positional relationship between the visual landmark and another object, the determination means calculates the probability distribution for the one or more object regions according to a relative positional relationship between the visual landmark and the other object as seen from the moving body.

6. The information processing device according to any one of claims 1 to 5, characterized in that the utterance acquisition means further acquires utterance information including a location related to the visual landmark from the communication device before acquiring utterance information regarding the indication position including the visual landmark.

7. The information processing device according to claim 6, characterized in that the speech acquisition means acquires speech information regarding the indicated position including the visual landmark in response to the moving object approaching within a predetermined distance from a location related to the visual landmark.

8. 8. The information processing apparatus according to claim 1, wherein the moving body is a micro mobility vehicle.

9. A method for controlling an information processing device, in which each step is executed by an information processing device that estimates an instruction position related to a destination position to which a moving object used by a user should autonomously move, the method comprising: a speech acquisition step of acquiring speech information related to a pointing position, including a visual landmark, from a communication device used by the user; an image acquisition step of acquiring an image captured by the moving object; determining an object region in the captured image that corresponds to the visual landmark included in the speech information; an estimation step of estimating the pointing position based on the object region determined in the determination step, A control method for an information processing device, characterized in that in the determination process, a probability distribution indicating the probability that one or more object regions recognized in the captured image correspond to the visual landmark included in the speech information is calculated, and an object region having a probability equal to or greater than a first threshold is determined to be the object region corresponding to the visual landmark included in the speech information.

10. A control device for a moving body that estimates an instruction position regarding a destination position to which a moving body used by a user should autonomously move, comprising: a speech acquisition means for acquiring speech information related to a pointing position, including a visual landmark, from a communication device used by the user; an image acquisition means for acquiring an image captured by the moving object; a determining means for determining an object area in the captured image corresponding to the visual landmark included in the speech information; an estimation means for estimating the pointing position based on the object region determined by the determination means, The control device is characterized in that the determination means calculates a probability distribution indicating the probability that one or more object regions recognized in the captured image correspond to the visual landmark included in the speech information, and determines an object region having a probability equal to or greater than a first threshold as the object region corresponding to the visual landmark included in the speech information.

11. A method for controlling a moving object, in which each step is executed by a control device of the moving object that estimates an instruction position regarding a destination position to which the moving object should autonomously move, for the moving object used by a user, the control device comprising: a speech acquisition step of acquiring speech information related to a pointing position, including a visual landmark, from a communication device used by the user; an image acquisition step of acquiring an image captured by the moving object; determining an object region in the captured image that corresponds to the visual landmark included in the speech information; an estimation step of estimating the pointing position based on the object region determined in the determination step, A method for controlling a moving body, characterized in that in the determination process, a probability distribution indicating the probability that one or more object regions recognized in the captured image correspond to the visual landmark included in the speech information is calculated, and an object region having a probability equal to or greater than a first threshold is determined to be the object region corresponding to the visual landmark included in the speech information.

12. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 8.

Images