MOBILE BODY CONTROL DEVICE, MOBILE BODY CONTROL METHOD, MOBILE BODY, INFORMATION PROCESSING METHOD, AND PROGRAM

Abstract

To provide a control device of a mobile object which can reduce occurrence of stop of the mobile object when the mobile object travels to a stop position which is adjusted by a user and the mobile object.SOLUTION: A control device of a mobile object according to the present disclosure adjusts a stop position of the mobile object on the basis of an instruction of a user and comprises: instruction acquisition means of acquiring instruction information for designating a predetermined target from a communication device used by the user: image acquisition means of acquiring a photographed image to be photographed in the mobile object; determination means of determining a position of the predetermined target from the mobile object as the stop position on the basis of an area of the predetermined target identified in the photographed image; and control means of controlling travel of the mobile object so that the mobile object travels toward the determined stop position, wherein the control means controls the travel of the mobile object to the stop position by updating the stop position determined to the predetermined target according to a motion of the mobile object when it becomes impossible to identify the predetermined target in the photographed image while the mobile object travels toward the stop position.SELECTED DRAWING: Figure 8A

Description

[Technical Field]

[0001] The present invention relates to a control device for a mobile body, a control method for a mobile body, a mobile body, an information processing method, and a program. [Background technology]

[0002] In recent years, electric vehicles with a seating capacity of around one to two people, known as ultra-compact mobility (also known as micromobility), have become known and are expected to become widespread as a convenient means of transportation.

[0003] In order to use autonomously driven micro-mobilities as a convenient means of transportation, it is desirable for the micro-mobility to stop at an appropriate location where the user can easily board. Patent Document 1 discloses a technology for minimizing the walking distance to the boarding location when a user leaving their home gets into an autonomously driven vehicle, thereby preventing the user from feeling inconvenienced. Furthermore, Patent Document 2 discloses a technology in which, when an autonomously driven vehicle recognizes that its occupants are in a boarding / disembarking area, it determines a stopping position within that area where the distance between the vehicle and the occupants is within a few meters, and drives to that stopping position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-91216 [Patent Document 2] Japanese Patent Publication No. 2020-142720 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when a user uses an ultra-compact mobility vehicle, there may be a use case in which the stopping position of the mobility vehicle is dynamically adjusted while the vehicle and the user are both moving. Such a use case is effective when it becomes difficult to merge at the planned location due to congestion or restrictions, or when the stopping position needs to be finely adjusted. However, if the mobility vehicle is unable to detect the stopping position when merging, there is a risk that it will be unable to continue traveling and will stop. Note that the above-mentioned conventional technology did not take into consideration a use case in which the vehicle's stopping position is dynamically adjusted while the vehicle and the user are both moving.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can reduce the occurrence of a moving body stopping when the moving body (e.g., a vehicle) travels to a stopping position coordinated between the user and the moving body. [Means for solving the problem]

[0007] According to the present invention, A control device for a moving body that adjusts a stopping position of the moving body based on an instruction from a user, an instruction acquisition means for acquiring instruction information for designating a predetermined target from a communication device used by a user; image acquisition means for acquiring a photographed image taken by the moving body; In the photographed image In front The area of ​​the specified target Identify , from the moving body Measured The specified target relative position is Stop position of a determining means for determining; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, A control device for a moving body is provided, characterized in that, when the specified target becomes unable to be identified in the captured image while the moving body is traveling toward the stopping position, the control means controls the traveling of the moving body to the stopping position by updating the stopping position determined for the specified target in accordance with the movement of the moving body. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the occurrence of stopping of a moving body when the moving body travels to a stopping position coordinated between a user and the moving body. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram showing an example of the hardware configuration of a vehicle according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of a vehicle according to an embodiment of the present invention; [Figure 4] FIG. 1 is a block diagram showing an example of a functional configuration realized by a control unit according to the present embodiment. [Figure 5A] FIG. 1 is a diagram for explaining determination of a vehicle stopping position using speech and images according to the present embodiment. [Figure 5B] FIG. 2 is a diagram for explaining determination of a vehicle stopping position using speech and images according to the present embodiment. [Figure 5C] FIG. 3 is a diagram for explaining how to determine a vehicle stopping position using speech and images according to the present embodiment. [Figure 5D] FIG. 4 is a diagram for explaining how to determine a vehicle stopping position using speech and images according to the present embodiment. [Figure 5E] FIG. 5 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 5F] FIG. 6 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 5G] FIG. 7 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 5H] FIG. 8 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 6] FIG. 10 is a diagram illustrating a state transition of vehicle control in response to a stop position instruction according to the present embodiment. [Figure 7] 1 is a flowchart showing a series of operations in a stop position determination process according to the present embodiment. [Figure 8A] 1 is a flowchart showing a series of operations in a stop position adjustment process using a relative position according to the present embodiment. [Figure 8B] 10 is a flowchart (2) showing a series of operations in a stop position adjustment process using a relative position according to the present embodiment. [Figure 8C] Diagram (1) shows an example of vehicle control when a landmark object can no longer be recognized. [Figure 8D] Figure (2) shows an example of vehicle control when a landmark object can no longer be recognized. [Figure 9] FIG. 10 is a diagram illustrating an example of an information processing system according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0011] (Configuration of information processing system) The configuration of an information processing system 10 according to this embodiment will be described with reference to Fig. 1. The information processing system 10 includes a vehicle 100 and a communication device 120.

[0012] Vehicle 100 is an example of a moving body, and is, for example, an ultra-compact mobility vehicle equipped with a battery and moving mainly by motor power. An ultra-compact mobility vehicle is a very small vehicle that is more compact than a typical automobile and has a passenger capacity of about one or two people. An ultra-compact mobility vehicle may be capable of traveling on roadways or sidewalks. In this embodiment, vehicle 100 is, for example, a four-wheeled vehicle. Note that this embodiment is not limited to vehicles and can also be applied to other moving bodies. Moving bodies are not limited to vehicles, and may include small mobility vehicles that run alongside walking users to carry luggage or lead people, and may also include other moving bodies capable of autonomous movement (for example, walking robots).

[0013] The vehicle 100 connects to the network 140 via, for example, fifth-generation mobile communication, road-to-vehicle communication, or wireless communication such as Wi-Fi. The vehicle 100 measures conditions inside and outside the vehicle (such as the vehicle's position, driving status, and surrounding object landmarks) using various sensors and accumulates the measured data. The data collected and transmitted in this manner is generally referred to as floating data, probe data, traffic information, etc. The vehicle 100 may transmit the accumulated data to a server (not shown). When information about the vehicle is transmitted to the server, it is transmitted at regular intervals or in response to the occurrence of a specific event. The vehicle 100 can travel autonomously even when the user 130 is not aboard the vehicle. The vehicle 100 can start traveling from an initial stopping position toward a position where the user 130 will be picked up, in response to speech information from the communication device 120, etc. As will be described later, the vehicle 100 coordinates a stopping position with the user by acquiring speech information from the communication device 120 or transmitting speech information to the communication device 120 via the network 111. The vehicle 100 stops at the adjusted stop position and lets the user get on.

[0014] The communication device 120 is, for example, a smartphone, but is not limited to this and may also be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, etc. The communication device 120 connects to the network 140 via wireless communication such as fifth generation mobile communication or Wi-Fi.

[0015] The network 111 includes a communication network such as the Internet or a mobile phone network, and transmits information between the vehicle 100, the communication device 120, a server (not shown), and the like.

[0016] In this information processing system 10, when a user 130 and a vehicle 100 located at a distance come close enough to be able to visually confirm a target or the like (a visual landmark), the stopping position is adjusted using spoken information and image information captured by the vehicle 100.

[0017] Before the user 130 and the vehicle 100 get close enough to visually confirm a target or the like, the vehicle 100 first moves to a stopping position that is the user's current position on a map or a destination position on a map obtained from the user's speech information. Then, when the vehicle 100 approaches the stopping position, it transmits speech information to the communication device 120 that identifies the user or asks about a location related to a visual landmark (e.g., "Are there any stores nearby?"). This allows the vehicle 100 to head toward the user's location if it can find (identify) the user from image information, or to move toward a landmark obtained from the speech information even in a situation where it is difficult to identify the user.

[0018] The location associated with the visual landmark includes, for example, the name of a target that can be identified from an image. Vehicle 100 receives speech information (e.g., "Stop in front of the vending machine") including the location associated with the visual landmark from communication device 120. Vehicle 100 then identifies the landmark from the image information and moves to the location of the landmark.

[0019] (Vehicle configuration) Next, with reference to FIG. 2, a configuration of a vehicle 100 as an example of a vehicle according to this embodiment will be described.

[0020] Fig. 2(A) shows a side view of the vehicle 100 according to this embodiment, and Fig. 2(B) shows the internal configuration of the vehicle 100. In the figure, arrow X indicates the longitudinal direction of the vehicle 100, with F indicating the front and R indicating the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of the vehicle 100.

[0021] Vehicle 100 is an electric autonomous vehicle equipped with a propulsion unit 12 and using a battery 13 as its main power source. Battery 13 is, for example, a secondary battery such as a lithium-ion battery, and vehicle 100 is propelled by propulsion unit 12 using power supplied from battery 13. Propulsion unit 12 is, for example, a four-wheeled vehicle equipped with a pair of left and right front wheels 20 and a pair of left and right rear wheels 21. Propulsion unit 12 may also be in other forms, such as a tricycle. Vehicle 100 is equipped with seating 14 for one or two people.

[0022] The propulsion unit 12 includes a steering mechanism 22. The steering mechanism 22 is a mechanism that uses a motor 22a as a drive source to change the steering angle of the pair of front wheels 20. By changing the steering angle of the pair of front wheels 20, the traveling direction of the vehicle 100 can be changed. The propulsion unit 12 also includes a drive mechanism 23. The drive mechanism 23 is a mechanism that uses a motor 23a as a drive source to rotate the pair of rear wheels 21. By rotating the pair of rear wheels 21, the vehicle 100 can move forward or backward. The propulsion unit 12 can detect and output physical quantities that represent the motion of the vehicle 100, such as the traveling speed, acceleration, steering angle, and rotational acceleration of the body of the vehicle 100.

[0023] The vehicle 100 is equipped with detection units 15 to 17 that detect targets around the vehicle 100. The detection units 15 to 17 are a group of external sensors that monitor the periphery of the vehicle 100, and in this embodiment, each is an imaging device that captures an image of the periphery of the vehicle 100, and includes, for example, an optical system such as a lens and an image sensor. In addition to the imaging device, the vehicle 100 can also employ radar or lidar (Light Detection and Ranging). Based on the image information obtained by the detection units, the vehicle 100 can acquire the position of a specific person or specific target as viewed from the coordinate system of the vehicle 100 (hereinafter referred to as relative position). The relative position can be expressed, for example, as a position 1 m to the left or 10 m ahead.

[0024] Two detection units 15 are arranged at the front of the vehicle 100, spaced apart in the Y direction, and mainly detect targets in front of the vehicle 100. Detection units 16 are arranged on the left and right sides of the vehicle 100, respectively, and mainly detect targets on the sides of the vehicle 100. Detection unit 17 is arranged at the rear of the vehicle 100, and mainly detects targets behind the vehicle 100.

[0025] FIG. 3 is a block diagram of a control system of the vehicle 100. The vehicle 100 includes a control unit (ECU) 30. The control unit 30 functions as a control device for the vehicle (mobile body). The control unit 30 includes a processor such as a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor and data used by the processor for processing. A plurality of sets of processors, storage devices, and interfaces may be provided for different functions of the vehicle 100 and configured to be able to communicate with each other.

[0026] The control unit 30 acquires the detection results of the detection units 15 to 17, input information from the operation panel 31, audio information input from the audio input device 33, speech information from the communication device 120, etc., and executes corresponding processing. The control unit 30 controls the motors 22a and 23a (travel control of the traveling unit 12), controls the display on the operation panel 31, and notifies the occupants of the vehicle 100 by audio and outputs information. The control unit 30 may further include, as a processor, a GPU or dedicated hardware suitable for executing processing of a machine learning model such as a neural network. In addition, the control unit 30 executes a stop position determination process according to this embodiment, which will be described later.

[0027] The voice input device 33 collects voices of passengers of the vehicle 100. The control unit 30 can recognize the input voices and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 34 receives GNSS signals and detects the current position of the vehicle 100.

[0028] The storage device 35 is a large-capacity storage device that stores map data and the like, including information on routes on which the vehicle 100 can travel, landmarks such as buildings, stores, and the like. The storage device 35 may also store programs executed by the processor, data used by the processor for processing, and the like. The storage device 35 may also store various parameters (e.g., trained parameters and hyperparameters of a deep neural network) of machine learning models for speech recognition and image recognition executed by the control unit 30. The storage device 35 may also be provided on a server (not shown).

[0029] The communication device 36 is a communication device that can connect to the network 140 via wireless communication such as fifth generation mobile communication or Wi-Fi.

[0030] (Software configuration for determining stopping position) Next, a software configuration for the stop position determination process in the control unit 30 will be described with reference to Fig. 4. This software configuration is realized by the control unit 30 executing a program stored in a storage medium.

[0031] The software configuration according to this embodiment includes an interaction unit 401, a vehicle control unit 402, and a database 403. The interaction unit 401 processes voice information (utterance information) exchanged with the communication device 120, processes image information acquired by the detection unit 15, etc., and performs processing to estimate a stopping position.

[0032] The vehicle control unit 402 determines the route to the stop position set by the interaction unit 401, controls each unit of the vehicle along the route, and so on. As will be described in detail later, when the vehicle 100 approaches the stop position while traveling using the relative position, the vehicle control unit 402 controls the traveling speed according to the remaining distance. For example, when the remaining distance to the stop position is greater than a predetermined value, the vehicle control unit 402 controls the vehicle to approach the stop position at a predetermined (relatively fast) first speed. Furthermore, when the remaining distance is equal to or less than a predetermined value, the vehicle control unit 402 controls the vehicle to approach the stop position at a second speed (first speed > second speed) that allows for quick stop control through safe acceleration and deceleration.

[0033] The database 403 stores various data such as map data including information on routes that the vehicle 100 can travel, landmarks such as buildings, stores, etc., and travel history information of the vehicle itself and other vehicles.

[0034] The user data acquisition unit 413 acquires utterance information and location information transmitted from the communication device 120. The user data acquisition unit 413 may store the acquired utterance information and location information in the database 403. As will be described later, the utterance information acquired by the user data acquisition unit 413 is input to a trained machine learning model to estimate the user's intention. Note that the following description will be given taking as an example a case where a user's instruction is acquired based on utterance information. However, information including the user's instruction (instruction information) is not limited to voice information, and may be other information including the user's intention, such as text information.

[0035] The voice information processing unit 414 includes a machine learning model that processes voice information and executes the inference stage processing of the machine learning model. The machine learning model of the voice information processing unit 414 performs calculations of a deep learning algorithm using, for example, a deep neural network (DNN) to recognize the content of a user's utterance and estimate the user's intention. Separate machine learning algorithms may be used for recognizing the content of a user's utterance and estimating the user's intention.

[0036] The user's intention may be estimated by a classification process that classifies utterance information into predetermined intention classes. The intention classes of utterances may be defined for each usage scenario in which the user 130 uses the vehicle 100 (e.g., before boarding, during boarding, and after disembarking). Defining an intention class for each usage scenario limits the number of classifications in intention recognition, thereby improving recognition accuracy. The usage scenario "before boarding" may be associated with an intention class such as inquiry, pick-up request, greeting, location instruction, landmark expression, agreement, denial, or repetition. The usage scenario "during boarding" may be associated with an intention class that is at least partially different from that before boarding, such as route instruction, stop instruction, acceleration instruction, deceleration instruction, agreement, denial, or repetition. Similarly, the usage scenario "after disembarking" may be associated with an intention class that is at least partially different from that before boarding and during boarding. As an example of intent class estimation, an utterance such as "Can I board now?" before boarding is classified as an intention of "inquiry." An utterance such as "Can you come soon?" is classified as an intention of "pick-up request." Furthermore, utterance information such as "in front of the vending machine" is classified as the intention of "landmark expression."

[0037] In recognizing the content of a user's utterance, for example, if the user's intention is to specify a location, the name of a place or the like included in the utterance information may be identified. In recognizing the content of a user's utterance, for example, the name of a place, a landmark such as a building, a store name, a landmark name, etc. included in the utterance information may be recognized. The landmark may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, etc. included in the utterance information.

[0038] The DNN is put into a trained state by performing the learning stage processing, and by inputting utterance information into the trained DNN, recognition processing (inference stage processing) for the utterance information can be performed. Note that, although the present embodiment will be described taking as an example a case where the vehicle 100 performs the speech recognition processing, the speech recognition processing may be performed by a server (not shown) and the recognition results may be received from the server.

[0039] The image information processing unit 415 includes a machine learning model that processes image information, and the trained machine learning model executes the inference stage processing. The machine learning model of the image information processing unit 415 performs processing to recognize targets included in the image information, for example, by performing calculations of a deep learning algorithm using a deep neural network (DNN). Targets may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, etc. included in the image. In addition, the machine learning model of the image information processing unit 415 can recognize the faces of people included in the image information, the actions of people (e.g., hand gestures), the shapes and colors of their clothes, etc.

[0040] The stop position determination unit 416 executes the operation of the stop position determination process described below in cooperation with the above-mentioned audio information processing unit 414 and image information processing unit 415. The stop position determination process will be described later.

[0041] (Outline of the stopping position determination process) 5A to 5H, an overview of the stop position determination process executed in vehicle 100 will be described. Note that the operations described in this description, with vehicle 100 as the main actor, are realized by control unit 30 of vehicle 100 executing a program and operating each part shown in FIG.

[0042] In the stop position determination process, when a user requests pickup, the user 130 and the vehicle 100, who are currently in distant locations, first approach each other to the point where the user and landmarks can be visually confirmed. The vehicle 100 then adjusts the stop position based on voice information (speech information) and image information exchanged between the user 130 and the vehicle 100, and stops at the desired location of the user 130. The reason for making the stop position adjustable in this manner is to take into account the possibility that the user may modify the stop position once determined depending on the surrounding conditions. Furthermore, by making the stop position adjustable, it is possible to easily address the possibility that the accuracy of voice recognition and image recognition may fluctuate depending on the surrounding environment, such as noise and lighting conditions. As will be described later, in this embodiment, the vehicle 100 can adjust a new stop position while moving to the currently set stop position or while stopped, allowing for a smooth merge at the new stop position.

[0043] FIG. 5A schematically illustrates a situation in which a user 130 uses speech to call a vehicle 100 parked in a waiting area in an area where the vehicle 100 is allowed to travel. First, the user 130 uses the communication device 120 to transmit utterance information 510, such as "Can you come right away?", to the vehicle 100. The vehicle 100 acquires the utterance information of the user 130 and, upon determining that the intention of the utterance information is a "pick-up request," transmits utterance information 511, such as "Yes, I'll be there right away," to the communication device 120. For example, an area in which a vehicle cannot stop exists near the current location of the user 130. Furthermore, the user 130 wants to get into the vehicle 100 while traveling toward a vending machine 500 located ahead in the direction of travel. Therefore, the user 130 transmits utterance information 512, such as "I'm in a hurry, so follow me," from the communication device 120 to the vehicle 100. Upon receiving the utterance information 512 and recognizing the intention and content of the utterance, the vehicle 100 transmits utterance information 513 saying "I understand" to the communication device 120. At this time, the user's utterance information does not include a destination designation, so the vehicle 100 acquires location information (e.g., GPS information) of the communication device 120 of the user 130 and sets the stopping position based on the location of the user 130 and map information. The vehicle 100 then calculates a route to the stopping position (currently the location of the user 130) from the map information, and moves along that route.

[0044] 5B shows a schematic example of the system status after the vehicle 100 acquires speech information 510, "Can you come soon?", and performs speech information processing, etc. The system status is shown, for example, by visual information understanding 520, which indicates the result of processing the visual information, linguistic information understanding 521, which indicates the result of processing the speech information, and likelihood 522 of the estimated stopping position.

[0045] In the visual information comprehension 520, a display 524 of the recognized target is displayed within the captured image information 523. Note that in this explanation, to ensure the visibility of the drawing, information on subjects unrelated to the explanation is omitted. For this reason, in the example shown in FIG. 5B, nothing is written in the image information 523, but in reality, there are subjects in the background of the photograph. In the example shown in FIG. 5B, "None" is superimposed, indicating that the target has not been recognized.

[0046] Linguistic information understanding 521 shows an estimation result 525 of the user's utterance intention. The horizontal bar graph indicates the probability that the estimated user's intention falls into a category (a request to pick up, an instruction to a location, etc.). In this example, it indicates that there is a high probability that the intention of the user's utterance information "Can you come right away?" is a request to pick up.

[0047] The likelihood 522 of the stop position is represented by a mesh map 526 of the area shown in FIG. 5A, and colored areas 527 are areas with a high probability (likelihood) of being estimated as a stop position. The black-filled area indicates the area with the highest probability of being a stop position. The hatched area indicates the area with the next highest probability.

[0048] FIG. 5C illustrates a situation in which the user 130 has approached close enough for the vehicle 100 to recognize the user 130 through image information. In this situation, the vehicle 100 switches its operating state from an "absolute position-based driving state" using latitude and longitude coordinates to a "relative position-based stop control state" for controlling the stop position using a relative position. Here, the absolute position refers to the position of a moving object or target relative to a specific geographic coordinate system, such as latitude and longitude. Because the relative position is determined using image information, the vehicle 100 engages in a dialogue to confirm whether the person in the image is the user 130. For example, the vehicle 100 transmits speech information 530 such as "I'm getting close. Can you wave?" The user 130 transmits speech information 531 such as "Hey, I'm here" from the communication device 120 to the vehicle 100 while waving. The vehicle 100 recognizes the user 130 waving in the image and transmits speech information 532 such as "Are you the one wearing red?" to the communication device 120. In response to this, the user 130 transmits speech information 533, which indicates agreement, "Yes, that's right," from the communication device 120 to the vehicle 100.

[0049] FIG. 5D schematically illustrates an example of a system status in which the dialogue between vehicle 100 and user 130 shown in FIG. 5C has progressed. Image information 523 illustrates recognition result 541, which recognizes a person waving, and recognition result 540, which recognizes a vending machine as a target. Furthermore, language information understanding 521 receives utterance information 533 from user 130, "Yes, that's right," and illustrates speech intention estimation result 525, which indicates a high probability that the utterance intention is "agree." Furthermore, vehicle 100 calculates likelihood 522 of the stopping position based on the relative position of user 130 obtained from recognition result 541 based on image information and the absolute position of vehicle 100. In this example, user 130 has moved, and thus the position of colored area 542 has moved to the right on map 526.

[0050] FIG. 5E shows an example of a dialogue after the vehicle 100 acquires the position of the user 130 as a relative position. The vehicle 100 transmits speech information 550, "It'll stop right in front of me," to the communication device 120. In response, the user 130 wishes to board at another location and therefore transmits speech information 551, "No, stop over there," to the vehicle 100. At this time, the user 130 is pointing at the vending machine 500. Based on the user 130's speech information 551 (including "over there") and the pointing gesture, the vehicle 100 recognizes the landmark (the vending machine) from the image information. The vehicle 100 then transmits speech information 552, "You're at the vending machine, right?" to confirm the landmark, and then receives speech information 553, "Yes, that's right," from the user 130. Through these dialogues and recognition processes, the vehicle 100 sets the position of the vending machine 500 as a stop position and controls its travel so that it stops at that position. In this way, the vehicle 100 can search for a stopping position by continuing a dialogue with the user 130 while approaching the initially set stopping position, and can flexibly reset the new stopping position even when a new stopping position is specified.

[0051] FIG. 5F schematically illustrates an example of a system status in which vehicle 100 processes utterance information 551 from user 130, "No, stop over there." Image information 523 illustrates recognition result 561, which recognizes a person pointing, and recognition result 560, which recognizes a vending machine as a target. Furthermore, language information understanding 521 illustrates utterance intention estimation result 525, which receives utterance information 551 from user 130, "No, stop over there," and indicates a high probability that the utterance intention is a "location instruction." Furthermore, vehicle 100 calculates likelihood 522 of the stop position based on the relative position of vending machine 500 obtained from image information recognition results 560 and 561 and the absolute position of vehicle 100. In this example, because the position of vending machine 500 is highly likely to be the stop position, colored region 562 has moved to the right on map 526 (to the position corresponding to vending machine 500).

[0052] FIG. 5G shows an example of a conversation between vehicle 100 and user 130 when vehicle 100 approaches the location of vending machine 500. Vehicle 100 travels while continuously measuring the relative position of the vending machine that appears in the image information, and when vehicle 100 approaches the relative position, it transmits speech information 570 saying "Sorry to keep you waiting" to communication device 120. Vehicle 100 also stops at the currently set stopping position. When vehicle 100 receives speech information 571 saying "Thank you" from user 130, it can determine that vehicle dispatch has been completed (i.e., the relative position will not be re-specified).

[0053] FIG. 5H schematically illustrates an example of the system status after vehicle 100 processes utterance information 571 of "thank you" from user 130. Image information 523 illustrates recognition result 581 of a person and recognition result 580 of a vending machine as a target. Furthermore, language information understanding 521 illustrates estimation result 525 of the utterance intention, which indicates that utterance information 571 of "thank you" from user 130 is received and that the utterance intention is highly likely to be "thank you." Furthermore, vehicle 100 calculates likelihood 522 of the stop position based on the relative position of vending machine 500 obtained from recognition result 580 based on image information and the absolute position of vehicle 100. In this example, since the position of vending machine 500 is highly likely to be the stop position, the position of region 582 indicating high probability indicates the position on map 526 corresponding to vending machine 500.

[0054] Next, with reference to Fig. 6, a state transition of the driving system control of the vehicle 100 in response to an instruction of a stop position will be described. First, when the state transition of the vehicle 100 starts, the vehicle 100 enters a "stop" state. In this state, when a transmission command such as a pick-up request is received, the vehicle 100 enters an automatic driving state. The automatic driving state roughly includes an "absolute position-based driving state," a "relative position-based stop control state," and a "stop control state."

[0055] In the "traveling state based on absolute position," as described above, when the vehicle 100 receives a pickup request from the user, the vehicle 100 first sets the initial stopping position as an absolute position. For example, if the speech information of the pickup request includes a landmark, the vehicle 100 sets the position of the landmark as an absolute position. If the speech information does not include the landmark, the vehicle 100 sets the user's position (GPS position information of the communication device 120) as an absolute position. Therefore, the vehicle 100 transitions to the "traveling state based on absolute position" of the "autonomous traveling state" and starts traveling. The vehicle 100 may confirm the stopping position with the user 130 while traveling. If the speech information of the user 130 indicates that the current stopping position is acceptable, the vehicle 100 transitions to the stop control state when it arrives at the stopping position. For example, after a predetermined time has elapsed in the stop control state, the vehicle 100 determines that the vehicle dispatch is completed and transitions to the "stopped" state.

[0056] When the vehicle 100 approaches the stop position (set as an absolute position) to the extent that the user or a landmark can be visually confirmed, the vehicle 100 transitions to a "stop position search in progress" state in the "stop control state based on relative position." During the stop position search, the vehicle 100 identifies the user or landmark and sets the stop position based on the relative position of the identified landmark. For example, the vehicle 100 transmits speech information to the user 130 to confirm the stop position. As in the above example, when the vehicle 100 receives speech information 551 such as "No, stop over there," the vehicle 100 identifies the target of the instruction (vending machine 500), sets the relative position of the target of the instruction as the stop position, and transitions to the "approach / stop control" state.

[0057] In the "Approach / Stop Control" state, the vehicle 100 moves toward the set stop position. At this time, if the vehicle 100 cannot detect the landmark that is the stop position from the image information, such as an obstacle appearing in front of it, the vehicle 100 transitions its state to "Searching for Stop Position". Also, as described above, when the vehicle 100 approaches the stop position during traveling using the relative position, the traveling speed may be controlled according to the remaining distance. The vehicle 100 stops when it arrives at the relative position and enters the stop standby state. In the stop standby state, when voice information for re-designating the relative position is acquired from the user 130, the vehicle 100 re-sets the stop position and returns the state to the "Approach / Stop Control" state. When the user 130's voice information indicates that the stop position is fine, such as "Thank you", the vehicle 100 transitions to the stop control state.

[0058] In each of the states of "Searching for Stop Position", "Approach / Stop Control", and "Stop Standby", the vehicle 100 may set driving control parameters corresponding to each state and control the driving of the vehicle according to the vehicle speed control parameters. For example, for each of the states of "Searching for Stop Position", "Approach / Stop Control", and "Stop Standby", a predetermined target (limit) vehicle speed, target acceleration / deceleration, and minimum state holding time may be held as a table, and the vehicle speed, etc. may be controlled according to the state transition. For example, in "Searching for Stop Position", since the driving route where the (new) stop position is determined is likely to change, the target vehicle speed may be lower than that in "Approach / Stop Control" where the stop position has already been determined. That is, the target vehicle speeds for "Searching for Stop Position", "Approach / Stop Control", and "Stop Standby" may be set to (A, B, 0) (where A < B), respectively. By doing so, it is possible to realize driving in accordance with the acceleration / deceleration and target vehicle speed set for each state as the state transitions. Note that the driving control of the vehicle 100 is not limited to this example. As will be described later, the vehicle speed may be controlled according to the degree of confidence in the stop position.

[0059] (A series of operations for stop position determination processing) Next, a series of operations in the stop position determination process in the vehicle 100 will be described with reference to Fig. 7. This process is realized by the control unit 30 executing a program. In the following description, for simplicity, it is assumed that the control unit 30 executes each process, but the corresponding processes are actually executed by each part of the interaction unit 401 (described above in Fig. 4) and the vehicle control unit 402.

[0060] In S701, the control unit 30 receives utterance information of a pick-up request and location information of the user from the communication device 120. The utterance information of the pick-up request includes, for example, an utterance such as "Can you come right away?" described above in FIG. 5A. The utterance information of the pick-up request may include a destination, such as "Can you come right away, before AAA?". The user's location information is, for example, location information detected by the communication device 120. In S702, the control unit 30 identifies the destination from the user utterance information of the pick-up request. For example, the control unit 30 can identify "AAA" as the destination from utterance information such as "Can you come right away, before AAA?". If the utterance information does not include a destination, such as "Can you come right away?", the control unit 30 may determine that the destination could not be identified. Alternatively, the control unit 30 may transmit utterance information inquiring about the destination to the communication device 120 and additionally acquire utterance information including the destination.

[0061] In S703, the control unit 30 determines whether the destination has been identified. For example, if the speech information of the pick-up request includes a word indicating the destination, the control unit 30 determines that the destination has been identified and proceeds to S705; otherwise, the control unit 30 proceeds to S704.

[0062] In S704, the control unit 30 sets the user's location as the stop location (when the destination cannot be identified from the speech information). At this time, the user's location information is an absolute location. By setting the user's location as the stop location, the vehicle 100 will first travel toward the user's location and follow a route that approaches the user.

[0063] In S705, the control unit 30 specifies the location of the identified destination from the map information and sets it as the stop position. For example, the control unit 30 searches the map information for the name of the destination, AAA, and sets the location information (e.g., latitude and longitude information) obtained by the search as the stop position. In this case, the location information is also an absolute position.

[0064] In S706, the control unit 30 moves the vehicle 100 to the set stop position. For example, the control unit 30 determines a travel route to the stop position based on map information, and travels according to the travel route.

[0065] In S707, the control unit 30 determines whether the vehicle 100 has approached the stop position. For example, the control unit 30 acquires current position information of the vehicle 100 and determines whether the position information is within a predetermined distance from the latitude and longitude determined as the stop position. If the current vehicle position is within the predetermined distance from the stop position, the control unit 30 determines that the vehicle has approached the stop position and proceeds to S708; if not, the control unit 30 returns to S707 (i.e., repeats the determination while traveling along the route).

[0066] In S708, the control unit 30 executes a stop position adjustment process using the relative position. Details of the stop position adjustment process will be described later with reference to Fig. 8. After completing the stop position adjustment process using the relative position, the control unit 30 then ends the series of operations.

[0067] (Operation of stop position adjustment process using relative position) Furthermore, the operation of the stop position adjustment process using the relative position in the vehicle 100 will be described with reference to Figures 8A and 8B. Note that this process is realized by the control unit 30 executing a program, similar to the process shown in Figure 7. This process is a process that specifies the relative position of a target such as a user or a landmark identified by image information acquired by the detection unit 15, etc., and controls traveling to a stop position using the relative position based on the image information. In other words, this process generally corresponds to the process in the "stop control state based on relative position" described above in Figure 6.

[0068] In S801, the control unit 30 performs object recognition processing on an image acquired by, for example, the sensing unit 15 to identify object regions (corresponding to visual landmarks) in the image.

[0069] In S802, the control unit 30 determines whether a predetermined number or more of objects are present in the surrounding area. For example, the control unit 30 determines whether the number of object regions identified in S801 is a predetermined number or more, and if the number of object regions is a predetermined number or more, the process proceeds to S804, and if not, the process proceeds to S803.

[0070] In S803, the control unit 30 identifies the user from the image information. At this time, the control unit 30 may further identify a user action such as a hand gesture or a pointing action of the user. In addition, the control unit 30 may transmit utterance information to the communication device 120 to limit the target person when identifying the user, such as "Can you wave?" or "Is that the person in the red clothes?"

[0071] In S804, the control unit 30 transmits speech information to the communication device 120 inquiring about landmarks at the stop position. As described above, when the number of regions detected in the image is equal to or greater than a certain number, it is difficult to identify the landmarks with a high degree of accuracy based solely on the image recognition results. Therefore, when the number of regions detected in the image is equal to or greater than a certain number, the control unit 30 utilizes speech information, such as transmitting speech information inquiring about landmarks at the stop position, to identify the landmarks using the speech information and image information. The control unit 30 may transmit additional speech information to narrow down the visual landmarks, such as, "Is that a red vending machine?" When the visual landmarks cannot be narrowed down to one based on the relationship between the speech information of the user 130 and the image information of the vehicle 100, the ambiguity of the visual landmarks can be reduced by obtaining additional speech information from the user. This allows for more accurate identification of the landmarks.

[0072] In S805, the control unit 30 acquires the user's speech information and identifies a landmark from the speech information. At this time, the control unit 30 may further identify user actions such as the user's hand gestures and pointing movements. For example, if the user's speech information is "stop over there," the control unit 30 identifies the demonstrative pronoun "over there." Also, if the user's speech information is "stop in front of the vending machine," the control unit 30 identifies "vending machine" as a landmark.

[0073] In S806, the control unit 30 identifies a landmark corresponding to the speech information identified in S805 from the image information and specifies its relative position. For example, if the control unit 30 identifies "over there" from the user's speech information, it recognizes the user's pointing in the image and identifies an object in that direction as a landmark. Furthermore, if the control unit 30 identifies "vending machine" from the speech information, it identifies the area of ​​the vending machine in the image information. Then, the control unit 30 specifies the relative position of the identified object. As described above, the relative distance is a position as seen from the vehicle 100, and is expressed as, for example, 1 meter to the left or 10 meters forward.

[0074] It is also possible to calculate a probability distribution indicating the probability that one or more object regions in an image correspond to a visual landmark. For example, if the landmark included in the utterance information is a "vending machine" and two or more "vending machine" regions exist in the image, the control unit 30 may calculate the probability distribution of the object region based further on a restrictive linguistic element (e.g., "blue") in the utterance content. In this case, for example, if a blue vending machine and a red vending machine exist in the image, the control unit 30 may calculate a probability distribution in which the probability of a blue vending machine is "0.90" and the probability of a red vending machine is "0.10."

[0075] If the landmark included in the speech information is a "vending machine" and two or more "vending machine" regions exist in the image, the same probability may be assigned to both object regions. In this case, the control unit 30 may further vary the probability distribution depending on the relative positional relationship between the visual landmark and the user 130. If the red vending machine is closer to the current position of the user 130 or the vehicle 100, the control unit 30 may correct the probability distribution so that the probability of the red vending machine becomes "0.6" and the probability of the blue vending machine becomes "0.4." A probability distribution can be assigned in which the probability increases in order of likelihood of being a candidate from the direction the user is approaching.

[0076] When the speech information includes a positional relationship with an object, such as "a vending machine on the left side of the building," the control unit 30 may calculate a probability distribution that takes into account the relative positional relationship as seen from the vehicle 100. For example, the probability of a vending machine area on the left side of the building may be calculated as "0.9," and the probability of a vending machine area on the right side may be calculated as "0.1."

[0077] When the control unit 30 calculates the probability distribution for the landmarks, it identifies the object with the highest probability as the landmark and determines its relative position.

[0078] In S807, the control unit 30 transmits utterance information confirming the stopping position to the communication device 120. For example, the control unit 30 transmits utterance information such as "It will stop right in front of you" to the communication device 120. The control unit 30 also receives utterance information regarding confirmation from the user 130 in response to the utterance information confirming the stopping position. For example, the control unit 30 receives utterance information such as "No, please say hello over there."

[0079] In S808, the control unit 30 determines whether the received utterance information includes a stop position designation. That is, the control unit 30 determines whether the stop position confirmation in S807 includes a designation to change the stop position. For example, if the control unit 30 determines that the utterance information of the user 130 includes a location designation such as "over there" or "in front of the vending machine," the process proceeds to S805; otherwise, the process proceeds to S809. The process proceeds to S809 when, for example, utterance information of "OK" is received from the user 130.

[0080] In S809, the control unit 30 identifies the relative position of the user or landmark identified in the image information. As described above, the relative distance is the position as seen from the vehicle 100, and is expressed as, for example, 1 m to the left or 10 m ahead.

[0081] In S810, the control unit 30 sets the identified relative position as the stop position and controls driving to the stop position. At this time, if a landmark for the stop position is identified in the image information, the stop position is updated with the position of the landmark. As described above, the control unit 30 may control the driving speed according to the remaining distance to the stop position. The driving speed may also be adjusted according to the confidence level of the stop position. For example, if multiple visual landmarks obtained from the user's speech information are present in the image (e.g., two vending machines), the control unit 30 can assign a probability distribution to each vending machine according to the user's speech and relative positions, as described above. In this case, the control unit 30 may use the value of the probability distribution as the confidence level. In other words, when the confidence level is low, the control unit 30 reduces the driving speed compared to when the confidence level is high. In this way, the vehicle can travel at a reduced speed when there is a high possibility that the stop position will change, thereby preparing for the change in the stop position.

[0082] In S811, the control unit 30 determines whether the stop position has been re-specified (for example, by receiving speech information specifying the stop position from the user 130), and if it determines that the stop position has been re-specified, proceeds to S805; if not, proceeds to S812.

[0083] In S812, the control unit 30 determines whether the landmark for the stop position can no longer be identified. If the control unit 30 determines that the landmark for the stop position can no longer be identified in the image information, the control unit 30 proceeds to process S821; otherwise, the control unit 30 proceeds to process S813. If the landmark for the stop position cannot be seen due to the presence of an object such as a truck or a tree in front of the vehicle 100, the control unit 30 will no longer be able to identify the landmark for the stop position in the image information. In such a case, the control unit 30 performs processes of S821 to S828, which will be described later, to stop the vehicle 100 in order to appropriately control the stop position of the vehicle 100.

[0084] In S813, the control unit 30 determines whether the distance to the stop position has come within a predetermined distance. If the distance to the stop position has come within the predetermined distance, the control unit 30 proceeds to S814, and if not, the control unit 30 returns to S811.

[0085] In S814, since the stop position has not been re-designated and the control unit 30 has approached the stop position, the control unit 30 decelerates, drives, and stops at the stop position. At this time, the control unit 30 transmits speech information (for example, "Sorry to have kept you waiting") to the communication device 120 to notify the arrival. The control unit 30 then ends the stop position adjustment process using the relative position and returns to the caller. Then, the control unit 30 also ends the series of processes shown in FIG. 7.

[0086] In S821, the control unit 30 sets a "lost flag" indicating that the landmark object can no longer be identified to 1. If the landmark object can be redetected from the image information while the processes of S821 to S828 are being executed, the control unit 30 resets the lost flag to 0, returns the process to S811, and can control movement to the stop position. Therefore, the processes of S822 to S828 are executed when the lost flag indicates 1.

[0087] In S822, the control unit 30 holds the stopping position when the landmark object was last identified. In S823, the control unit 30 corrects the stopping position (expressed as a relative position) based on the movement of the vehicle 100. The movement of the vehicle 100 is the acceleration, rotational acceleration, etc. of the vehicle 100 output from the traveling unit 12. The control unit 30 corrects the stopping position by predicting a change in the relative position taking into account the movement of the vehicle 100.

[0088] The control unit 30 determines whether the distance to the stop position is farther than a predetermined distance. If the control unit 30 determines that the distance to the stop position is farther than the predetermined distance, the process proceeds to S825; otherwise, the control unit 30 proceeds to S827. The predetermined distance is determined by experiment or the like, and is a distance that can be traveled with an allowable error while updating the stop position based on vehicle motion.

[0089] FIG. 8C illustrates an example of control by the control unit 30 when the distance to the stop position is greater than a predetermined distance. For simplicity's sake, this example illustrates a case where the stop position is the position of the user 130. In this example, the control unit 30 is unable to identify the user 130 in the image due to an object 850 in the traveling direction of the vehicle 100. When the distance from the vehicle 100 to the stop position is great, continuously updating the stop position based on vehicle motion can result in a large deviation in the stop position. For this reason, in this embodiment, the control unit 30 transmits a notification to the communication device 120. This notification may include speech information indicating the occurrence of a warning trigger due to the inability to identify the landmark object, and speech information 852 inquiring about a landmark for the new stop position (e.g., object 851) (e.g., "Can I park around A?"). The landmark for the new stop position may be determined by searching for an object (e.g., about 20 m away) in the image.

[0090] In this way, for example, in S825, the control unit 30 transmits speech information notifying the occurrence of a warning factor due to an inability to identify a landmark, and further transmits speech information inquiring about the landmark of the stop position to the communication device 120 in S826. Then, the control unit 30 returns the process to S805. At this time, the control unit 30 may reduce the travel speed and continue traveling until a landmark of a new stop position is specified (for example, until the control unit 30 receives speech information from the user including the landmark). In this way, if the distance from the vehicle 100 to the stop position is far, it is possible to acquire a new landmark and resume traveling to the new stop position.

[0091] FIG. 8D shows an example of control by the control unit 30 when the distance to the stop position is within a predetermined distance. Again, for simplicity of explanation, this example shows a case where the stop position is the position of the user 130. In this case, an example is shown in which the control unit 30 is unable to identify the user 130 in the image due to an object 860 in the traveling direction of the vehicle 100. When the distance to the stop position is equal to or less than the predetermined distance, even if the stop position relative to the same landmark is continuously updated based on the vehicle motion, a large deviation in the stop position is unlikely to occur. Therefore, the control unit 30 continues to move to the stop position (based on the relative position) while updating the stop position based on the vehicle motion.

[0092] In this way, the control unit 30 controls the movement to the stop position based on the relative position in S827, and determines whether the stop position has been approached in S828. If the control unit 30 determines that the stop position has been approached, the process proceeds to S814, and if not, the process returns to S828 (i.e., the stop position continues to be updated based on the vehicle motion).

[0093] As described above, the vehicle 100 first approaches the user or landmarks or the like using the absolute position to an extent that the user can visually confirm them, and then adjusts the stopping position based on the voice information (utterance information) and image information exchanged between the user 130 and the vehicle 100, thereby stopping at a position desired by the user 130. In this way, the stopping position of the vehicle (mobile body) can be flexibly adjusted between the user and the vehicle (mobile body).

[0094] Furthermore, if an object serving as a landmark for the stop position cannot be identified after determining the stop position based on image information, the stop position set based on the landmark is updated according to the vehicle's movement. That is, even if a visual landmark cannot be identified, the vehicle 100 is moved using the stop position where the landmark was last identified. This reduces the occurrence of the vehicle 100 stopping as soon as the landmark cannot be identified. Then, speech information notifying the user of the occurrence of a warning trigger is transmitted according to the distance to the stop position, and a new stop position is set using another target (landmark). This prevents a large deviation in the stop position when controlling the stop of the vehicle 100 using the relative position. Furthermore, by transmitting speech information inquiring about the landmark for the stop position to the communication device 120, a new landmark for the stop position can be acquired from the user's speech information, and the vehicle can resume traveling to another stop position.

[0095] (Variation) Modifications of the present invention will be described below. In the above embodiment, an example has been described in which the stop position determination process is executed in the vehicle 100. However, the above-mentioned stop position determination process can also be executed on the server side. In this case, the information processing system 900 is configured with a vehicle 910, a communication device 120, and a server 901, as shown in FIG. 9.

[0096] The configuration of an information processing system 900 according to this embodiment will be described with reference to FIG. 9. The information processing system 900 includes a vehicle 910, a server 901, and a communication device 120. The server 901 is configured with one or more server devices, and is capable of acquiring vehicle-related information transmitted from the vehicle 910, as well as speech information and location information transmitted from the communication device 120, via a network 111, and controlling the traveling of the vehicle 910. A server generally has more abundant computational resources available than a vehicle, for example. Furthermore, by receiving and storing image data captured by various vehicles, learning data can be collected in a wide variety of situations, enabling learning to be performed in a wider variety of situations.

[0097] For example, in an embodiment according to this modification, user utterance information is transmitted from the communication device 120 to the server 901. The server 901 also acquires image information captured by the vehicle 910 via the network 140 together with position information and the like as part of the floating data of the vehicle 910. For example, after the server 901 performs processing corresponding to S701 to S705 of the above-described stop position determination processing, the server 901 transmits a control amount, such as a traveling speed, to the vehicle 910 in S706. The vehicle 910 travels (continues to travel) in accordance with the received control amount. The server 901 then performs processing corresponding to S707 and S708. The server 901 also performs processing corresponding to S801 to S809 in the stop position adjustment processing using the relative position, and transmits a control amount, such as a traveling speed, to the vehicle 910 in S810. The vehicle 910 travels (continues to travel) in accordance with the received control amount. The server 901 then executes processes corresponding to S811 to S814 and S821 to S826. These processes by the server 901 are realized by a processor (not shown) included in the server 901 executing a program stored in a storage medium (not shown) included in the server 901.

[0098] Furthermore, the configuration of vehicle 910 may be the same as that of vehicle 100, except that control unit 30 does not execute the stop position determination process and causes the vehicle to travel in accordance with the control amount from server 901.

[0099] In this way, the server 901 uses the absolute position to move the vehicle close enough to the user or landmarks, etc., so that the user can visually confirm them. Then, the server 901 adjusts the stop position based on voice information (utterance information) and image information exchanged between the user 130 and the vehicle, thereby stopping the vehicle at the desired position of the user 130. This allows the user and the vehicle (mobile body) to flexibly adjust the stop position of the vehicle (mobile body). Furthermore, if an object serving as a landmark for the stop position cannot be identified after the stop position has been determined based on image information, the stop position set based on the landmark is updated according to the vehicle's movement. In other words, even if a visual landmark cannot be identified, the vehicle is moved using the stop position where the landmark was last identified. This reduces the occurrence of the vehicle stopping as soon as the landmark cannot be identified.

[0100] In the above embodiment, the adjustment of the stopping position of a vehicle (mobile body) between a user and the vehicle (mobile body) attempting to merge has been described, but the application of the present invention is not limited to this. For example, the present invention may be applied to a case where a user in a vehicle (mobile body) instructs a stopping position based on a landmark. For example, when a user in a mobile body utters (instructs) "Park in front of the vending machine over there," the mobile body determines the stopping position by responding with, for example, "Is that the red vending machine?" There may then be a case where the mobile body adjusts its stopping position based on a user instruction such as "Park at the convenience store over there after all."

[0101] <Summary of the embodiment> 1. In an embodiment of the present invention, A control device (e.g., 30) for a moving body (e.g., 100) that adjusts a stopping position of the moving body based on an instruction from a user, An instruction acquisition means (e.g., 413) for acquiring instruction information specifying a predetermined target from a communication device used by a user; Image acquisition means (e.g., 15 to 17) for acquiring a photographed image taken by the moving body; A determination means (e.g., 416) for determining a position of the predetermined target from the moving body as a stopping position based on the area of ​​the predetermined target identified in the captured image; a control means (e.g., 402) for controlling the movement of the moving body so that the moving body moves toward the determined stop position; A control device for a moving body is provided in which, when the specified target becomes unable to be identified in the captured image while the moving body is traveling toward the stopping position, the control means controls the traveling of the moving body to the stopping position by updating the stopping position determined for the specified target in accordance with the movement of the moving body.

[0102] This embodiment can reduce the occurrence of the mobile body stopping as soon as it is no longer possible to identify the target object, and can also reduce the problem of the distance between the users increasing due to the stopping of the mobile body while the users are moving.

[0103] 2. In the above-described embodiment, The determining means determines, based on the area of ​​the predetermined target identified in the captured image, the relative position of the predetermined target with respect to the position of the moving body as a base point, as a stop position.

[0104] According to this embodiment, the stop position can be determined using the relative position of the moving body as the base point, which is highly convenient when performing processing using images.

[0105] 3. In the above-described embodiment, The method further includes a processing means for transmitting query information to the communication device, When the specified target object cannot be identified in the captured image while the moving body is traveling toward the stop position, and the distance to the stop position determined for the specified target object is greater than a specified distance, the processing means transmits inquiry information to the communication device to set a new stop position using another target object.

[0106] According to this embodiment, when the relative position is used to control the stopping of a moving object, it is possible to prevent a large deviation from occurring in the stopping position, and it is possible to notify the user that a new stopping position will be reset.

[0107] 4. In the above-described embodiment, The inquiry information for setting the new stop position includes at least one of information indicating that the predetermined target cannot be identified and information inquiring about the target for the new stop position.

[0108] According to this embodiment, the user can quickly recognize the cause of the warning and the new stop position.

[0109] 5. In the above-described embodiment, the utterance acquisition means acquires instruction information specifying a second target for the new stop position in response to the inquiry information for setting the new stop position, The determining means determines, based on the area of ​​the second target identified in the captured image, a relative position of the second target with respect to the position of the moving body as a base point, as a new stop position.

[0110] According to this embodiment, it is possible to accept a designation of a target for a new stop position from the user and resume traveling to the new stop position.

[0111] 6. In the above-described embodiment, The processing means searches for a target for the new stop position from the captured image, and transmits inquiry information for setting the new stop position according to the search result to the communication device.

[0112] According to this embodiment, it is possible to suggest to the user that a target included in the field of view (photographed image) of the moving object be used as a landmark for a new stopping position.

[0113] 7. In the above-described embodiment, The control means reduces the moving speed of the moving body until the determination means determines the position of the second target from the moving body as a new stop position.

[0114] According to this embodiment, when a new stop position is set, the traveling direction changes, so the traveling direction can be changed safely by reducing the traveling speed.

[0115] 8. In the above-described embodiment, the determining means determines a first stop position based on location information of the communication device in response to receiving instruction information including a pick-up request; the control means controls the travel of the moving object so that the moving object travels toward the stopping position determined using the captured image in response to the moving object approaching within a predetermined distance from the initial stopping position; the first stopping position is determined as an absolute position corresponding to a position on a map; The stop position determined using the captured image is defined as the relative position of the target with the position of the moving body as the base point.

[0116] According to this embodiment, the absolute position can be used to approach the user or landmarks to the extent that they can be visually confirmed, and then the vehicle can travel to a stopping position based on the relative position obtained from the image.

[0117] 9. In the above-described embodiment, The speech acquisition means acquires instruction information designating the predetermined target after the moving object approaches a predetermined distance from the first stopping position.

[0118] According to this embodiment, the absolute position is used to approach the user or a landmark or the like to a degree that allows visual confirmation, and then the stop position can be set using the instruction information.

[0119] 10. In the above-described embodiment, The moving body is a micro-mobility vehicle.

[0120] According to this embodiment, convenient transportation can be used.

[0121] 11. In an embodiment of the present invention, A method for controlling a moving object (e.g., 100) that adjusts a stopping position of the moving object based on a user's instruction, comprising: Obtaining instruction information specifying a predetermined target from a communication device (e.g., 120) used by a user (e.g., S805, S807); Acquiring a photographed image taken by the moving object (e.g., S801); determining a position of the predetermined target from the moving object as a stopping position based on the area of ​​the predetermined target identified in the captured image (e.g., S809); and controlling the travel of the moving object so that the moving object travels toward the determined stop position (e.g., S810). The controlling method includes controlling the movement of the moving body to the stopping position by updating the stopping position determined for the specified target in accordance with the movement of the moving body when the specified target can no longer be identified in the captured image while the moving body is moving toward the stopping position.

[0122] This embodiment can reduce the occurrence of the mobile body stopping as soon as it is no longer possible to identify the target object, and can also reduce the problem of the distance between the users increasing due to the stopping of the mobile body while the users are moving. [Explanation of symbols]

[0123] 100... vehicle, 120... communication device, 30... control unit, 413... user data acquisition unit, 414... voice information processing unit, 415... image information processing unit, 416... stop position determination unit

Claims

1. A control device for a moving body that adjusts a stopping position of the moving body based on an instruction from a user, an instruction acquisition means for acquiring instruction information for designating a predetermined target from a communication device used by a user; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for identifying an area of ​​the predetermined target within the captured image and determining a stop position, which is a relative position of the predetermined target measured from the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The control device for a moving body is characterized in that, when the specified target becomes unidentifiable in the captured image while the moving body is traveling toward the stopping position, the control means controls the traveling of the moving body to the stopping position by updating the stopping position determined for the specified target in accordance with the movement of the moving body.

2. 2. The control device for a moving body according to claim 1, wherein the determining means determines the stop position based on an area of ​​the predetermined target identified in the captured image.

3. The method further includes a processing means for transmitting information of a user inquiry to the communication device, 3. The control device for a mobile body according to claim 1, wherein the processing means transmits inquiry information to the communication device to set a new stopping position using another target when the specified target cannot be identified in the captured image while the mobile body is traveling toward the stopping position and the distance to the stopping position determined for the specified target is greater than a specified distance.

4. 4. The control device for a mobile body according to claim 3, wherein the inquiry information for setting the new stopping position includes at least one of information indicating that the specified target cannot be identified and information inquiring about a target for the new stopping position.

5. the instruction acquisition means acquires instruction information specifying a second target for the new stop position in response to inquiry information for setting the new stop position, 5. The control device for a moving body according to claim 3 or 4, wherein the determination means determines a relative position of the second target with respect to the position of the moving body as a base point as a new stopping position based on an area of ​​the second target identified in the captured image.

6. 6. The control device for a moving body according to claim 3, wherein the processing means searches for a target for the new stopping position from the captured image, and transmits inquiry information for setting the new stopping position according to the search result to the communication device.

7. 6. The control device for a moving body according to claim 5, wherein the control means reduces the moving speed of the moving body until the determination means determines the position of the second target from the moving body as a new stop position.

8. the determining means determines a first stop position based on location information of the communication device in response to receiving instruction information including a pick-up request; the control means controls the travel of the moving object so that the moving object travels toward the stopping position determined using the captured image in response to the moving object approaching within a predetermined distance from the initial stopping position; the first stopping position is determined as an absolute position corresponding to a position on a map; 8. The control device for a moving body according to claim 1, wherein the stop position determined using the captured image is determined as a relative position of the predetermined target with the position of the moving body as a base point.

9. 9. The control device for a moving body according to claim 8, wherein the instruction acquisition means acquires instruction information specifying the predetermined target after the moving body has approached a predetermined distance from the initial stopping position.

10. The control device for a moving body according to any one of claims 1 to 9, wherein the moving body is a micro-mobility vehicle.

11. A method for controlling a moving object, which adjusts a stopping position of the moving object based on an instruction from a user, comprising: acquiring instruction information specifying a predetermined target from a communication device used by a user; acquiring a photographed image taken by the moving body; Identifying an area of ​​the predetermined target in the captured image and determining a stop position, which is a relative position of the predetermined target measured from the moving body; controlling the travel of the moving object so that the moving object travels toward the determined stop position; The controlling method for a moving body, characterized in that the controlling includes, when the specified target becomes unidentifiable in the captured image while the moving body is traveling toward the stopping position, controlling the traveling of the moving body to the stopping position by updating the stopping position determined for the specified target in accordance with the movement of the moving body.

12. A moving object that adjusts a stop position based on a user's instruction, an instruction acquisition means for acquiring instruction information for designating a predetermined target from a communication device used by a user; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for identifying an area of ​​the predetermined target within the captured image and determining a stop position, which is a relative position of the predetermined target measured from the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The control means controls the movement of the moving body to the stopping position by updating the stopping position determined for the specified target in accordance with the movement of the moving body when the specified target becomes unidentifiable in the captured image while the moving body is moving toward the stopping position.

13. An information processing method executed by an information processing device that adjusts a stop position of a moving object based on an instruction from a user, comprising: acquiring instruction information specifying a predetermined target from a communication device used by a user; acquiring, from the moving body, a captured image captured by the moving body and information relating to the movement of the moving body; Identifying an area of ​​the predetermined target in the captured image and determining a stop position, which is a relative position of the predetermined target measured from the moving body; transmitting a control command to the moving body to control the movement of the moving body so that the moving body moves toward the determined stop position, The information processing method characterized in that the controlling includes, when the specified target becomes unidentifiable in the captured image while the moving body is traveling toward the stopping position, controlling the traveling of the moving body to the stopping position by updating the stopping position determined for the specified target in accordance with the movement of the moving body.

14. A program for causing a computer to function as each of the means of the control device for a moving body according to any one of claims 1 to 10.

15. A storage medium storing a program for causing a computer to function as each of the means of the control device for a moving body according to any one of claims 1 to 10.

Images