MOBILE BODY CONTROL DEVICE, MOBILE BODY CONTROL METHOD, MOBILE BODY, INFORMATION PROCESSING METHOD, AND PROGRAM

Abstract

A control device for a mobile object according to the present disclosure adjusts the stop position of the mobile object on the basis of an instruction from a user. The control device comprises an instruction acquisition means that acquires instruction information from the user, an image acquisition means that acquires a captured image captured in the mobile object, a determination means that determines the stop position of the mobile object, and a control means that controls the traveling of the mobile object such that the mobile object travels toward the determined stop position. The determination means (i) determines a first stop position using position information of a communication device used by the user, or position information corresponding to the destination included in first instruction information from the user, and (ii) in response to the position of the mobile object becoming less than or equal to a predetermined distance from the first stop position due to the traveling of the mobile object, determines a second stop position on the basis of second instruction information from the user and a predetermined target region identified in the captured image.

Description

[Technical Field]

[0001] The present invention relates to a control device for a mobile body, a control method for a mobile body, a mobile body, an information processing method, and a program. [Background technology]

[0002] In recent years, electric vehicles with a seating capacity of around one or two people, known as ultra-compact mobility (also known as micromobility), have become known and are expected to become widespread as a convenient means of transportation.

[0003] In order to use autonomously driven micro-mobilities as a convenient means of transportation, it is desirable for the micro-mobility to stop at an appropriate location where the user can easily board. Patent Document 1 discloses a technology for minimizing the walking distance to the boarding location when a user leaving their home gets into an autonomously driven vehicle, thereby preventing the user from feeling inconvenienced. Furthermore, Patent Document 2 discloses a technology in which, when an autonomously driven vehicle recognizes that its occupants are in a boarding / disembarking area, it determines a stopping position within that area where the distance between the vehicle and the occupants is within a few meters, and drives to that stopping position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-91216 [Patent Document 2] Japanese Patent Publication No. 2020-142720 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when a user uses an ultra-compact mobility vehicle, there may be a use case in which the stopping position of the vehicle is dynamically adjusted while the vehicle and the user are both moving. Such a use case is effective when it becomes difficult to merge at the planned location due to congestion or restrictions, or when the stopping position needs to be finely adjusted. The above-mentioned conventional technology did not take into consideration a use case in which the vehicle's stopping position is dynamically adjusted while the vehicle and the user are both moving.

[0006] The present invention has been made in view of the above-mentioned problems, and its purpose is to realize a technology that enables flexible adjustment of the stopping position of a moving body (such as a vehicle) between a user and the moving body. [Means for solving the problem]

[0007] According to the present invention, A control device for a moving body that adjusts a stopping position of the moving body based on an instruction from a user, an instruction acquisition means for acquiring user instruction information; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for determining a stop position of the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The determining means (i) determines a first stop position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user, and (ii) determines a second stop position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the position of the moving object coming within a predetermined distance from the first stop position as a result of the moving object's travel. death, The determining means determines the second stop position by identifying the designation of the predetermined target from the second instruction information of the user and then identifying an area of ​​the predetermined target corresponding to the designation of the predetermined target from the captured image. A control device for a moving object is provided. [Effects of the Invention]

[0008] According to the present invention, the stopping position of the moving object can be flexibly adjusted between the user and the moving object. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of an information processing system according to an embodiment of the present invention. [Figure 2A] 1 is a block diagram showing an example of the hardware configuration of a vehicle according to the present embodiment; [Figure 2B] 1 is a block diagram showing an example of the hardware configuration of a vehicle according to the present embodiment; [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of a vehicle according to an embodiment of the present invention; [Figure 4] FIG. 1 is a block diagram showing an example of a functional configuration realized by a control unit according to the present embodiment. [Figure 5A] FIG. 1 is a diagram for explaining determination of a vehicle stopping position using speech and images according to the present embodiment. [Figure 5B] FIG. 2 is a diagram for explaining determination of a vehicle stopping position using speech and images according to the present embodiment. [Figure 5C] FIG. 3 is a diagram for explaining how to determine a vehicle stopping position using speech and images according to the present embodiment. [Figure 5D] FIG. 4 is a diagram for explaining how to determine a vehicle stopping position using speech and images according to the present embodiment. [Figure 5E] FIG. 5 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 5F] FIG. 6 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 5G] FIG. 7 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 5H] FIG. 8 is a diagram for explaining how to determine a vehicle's stopping position using speech and images according to the present embodiment. [Figure 6] FIG. 10 is a diagram illustrating a state transition of vehicle control in response to a stop position instruction according to the present embodiment. [Figure 7] 1 is a flowchart showing a series of operations in a stop position determination process according to the present embodiment. [Figure 8] 1 is a flowchart showing a series of operations in a stop position adjustment process using a relative position according to the present embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of an information processing system according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0011] (Configuration of information processing system) The configuration of an information processing system 10 according to this embodiment will be described with reference to Fig. 1. The information processing system 10 includes a vehicle 100 and a communication device 120.

[0012] Vehicle 100 is an example of a moving body, and is, for example, an ultra-compact mobility vehicle equipped with a battery and moving mainly by motor power. An ultra-compact mobility vehicle is a very small vehicle that is more compact than a typical automobile and has a passenger capacity of about one or two people. An ultra-compact mobility vehicle may be capable of traveling on roadways or sidewalks. In this embodiment, vehicle 100 is, for example, a four-wheeled vehicle. Note that this embodiment is not limited to vehicles and can also be applied to other moving bodies. Moving bodies are not limited to vehicles, and may include small mobility vehicles that run alongside walking users to carry luggage or lead people, and may also include other moving bodies capable of autonomous movement (for example, walking robots).

[0013] The vehicle 100 connects to the network 140 via, for example, fifth-generation mobile communication, road-to-vehicle communication, or wireless communication such as Wi-Fi. The vehicle 100 measures conditions inside and outside the vehicle (such as the vehicle's position, driving status, and surrounding object landmarks) using various sensors and accumulates the measured data. Data collected and transmitted in this manner is generally called floating data, probe data, traffic information, etc. The vehicle 100 may transmit the accumulated data to a server (not shown). When information about the vehicle is transmitted to the server, it is transmitted at regular intervals or in response to the occurrence of a specific event. The vehicle 100 can travel autonomously even when the user 130 is not on board. The vehicle 100 can start traveling from an initial stopping position toward a position where the user 130 will board, in response to speech information from the communication device 120, etc. As will be described later, the vehicle 100 connects to the network 140 The vehicle 100 adjusts the stopping position with the user by acquiring the user's speech information transmitted from the communication device 120 via the communication device 120 or transmitting the speech information to the communication device 120. The vehicle 100 stops the vehicle at the adjusted stopping position and lets the user get on.

[0014] The communication device 120 is, for example, a smartphone, but is not limited to this and may also be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, etc. The communication device 120 connects to the network 140 via wireless communication such as fifth generation mobile communication or Wi-Fi.

[0015] network 140 includes a communication network such as the Internet or a mobile phone network, and transmits information between the vehicle 100, the communication device 120, a server (not shown), and the like.

[0016] In this information processing system 10, when a user 130 and a vehicle 100 located at a distance come close enough to be able to visually confirm a target or the like (a visual landmark), the stopping position is adjusted using spoken information and image information captured by the vehicle 100.

[0017] Before the user 130 and the vehicle 100 get close enough to visually confirm a target or the like, the vehicle 100 first moves to a stopping position that is the user's current position on a map or a destination position on a map obtained from the user's speech information. Then, when the vehicle 100 approaches the stopping position, it transmits speech information to the communication device 120 that identifies the user or asks about a location related to a visual landmark (e.g., "Are there any stores nearby?"). This allows the vehicle 100 to head toward the user's location if it can find (identify) the user from image information, or to move toward a landmark obtained from the speech information even in a situation where it is difficult to identify the user.

[0018] The location associated with the visual landmark includes, for example, the name of a target that can be identified from an image. Vehicle 100 receives speech information (e.g., "Stop in front of the vending machine") including the location associated with the visual landmark from communication device 120. Vehicle 100 then identifies the landmark from the image information and moves to the location of the landmark.

[0019] (Vehicle configuration) Next, the configuration of a vehicle 100 as an example of a vehicle according to this embodiment will be described with reference to FIGS. 2A and 2B.

[0020] Fig. 2A shows a side view of vehicle 100 according to this embodiment, and Fig. 2B shows the internal configuration of vehicle 100. In the figure, arrow X indicates the longitudinal direction of vehicle 100, with F indicating the front and R indicating the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of vehicle 100.

[0021] Vehicle 100 is an electric autonomous vehicle equipped with a propulsion unit 12 and using a battery 13 as its main power source. Battery 13 is, for example, a secondary battery such as a lithium-ion battery, and vehicle 100 is propelled by propulsion unit 12 using power supplied from battery 13. Propulsion unit 12 is, for example, a four-wheeled vehicle equipped with a pair of left and right front wheels 20 and a pair of left and right rear wheels 21. Propulsion unit 12 may also be in other forms, such as a tricycle. Vehicle 100 is equipped with seating 14 for one or two people.

[0022] The propulsion unit 12 includes a steering mechanism 22. The steering mechanism 22 is a mechanism that uses a motor 22a as a drive source to change the steering angle of the pair of front wheels 20. By changing the steering angle of the pair of front wheels 20, the traveling direction of the vehicle 100 can be changed. The propulsion unit 12 also includes a drive mechanism 23. The drive mechanism 23 is a mechanism that uses a motor 23a as a drive source to rotate the pair of rear wheels 21. By rotating the pair of rear wheels 21, the vehicle 100 can move forward or backward. The propulsion unit 12 can detect and output physical quantities that represent the motion of the vehicle 100, such as the traveling speed, acceleration, steering angle, and rotational acceleration of the body of the vehicle 100.

[0023] The vehicle 100 is equipped with detection units 15 to 17 that detect targets around the vehicle 100. The detection units 15 to 17 are a group of external sensors that monitor the periphery of the vehicle 100, and in this embodiment, each is an imaging device that captures an image of the periphery of the vehicle 100, and includes, for example, an optical system such as a lens and an image sensor. In addition to the imaging device, the vehicle 100 can also employ radar or lidar (Light Detection and Ranging). Based on the image information obtained by the detection units, the vehicle 100 can acquire the position of a specific person or specific target as viewed from the coordinate system of the vehicle 100 (hereinafter referred to as relative position). The relative position can be expressed, for example, as a position 1 m to the left or 10 m ahead.

[0024] Two detection units 15 are arranged at the front of the vehicle 100, spaced apart in the Y direction, and mainly detect targets in front of the vehicle 100. Detection units 16 are arranged on the left and right sides of the vehicle 100, respectively, and mainly detect targets on the sides of the vehicle 100. Detection unit 17 is arranged at the rear of the vehicle 100, and mainly detects targets behind the vehicle 100.

[0025] FIG. 3 is a block diagram of a control system of the vehicle 100. The vehicle 100 includes a control unit (ECU) 30. The control unit 30 functions as a control device for the vehicle (mobile body). The control unit 30 includes a processor such as a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor and data used by the processor for processing. A plurality of sets of processors, storage devices, and interfaces may be provided for different functions of the vehicle 100 and configured to be able to communicate with each other.

[0026] The control unit 30 acquires the detection results of the detection units 15 to 17, input information from the operation panel 31, audio information input from the audio input device 33, speech information from the communication device 120, etc., and executes corresponding processing. The control unit 30 controls the motors 22a and 23a (travel control of the traveling unit 12), controls the display on the operation panel 31, and notifies the occupants of the vehicle 100 by audio and outputs information. The control unit 30 may further include, as a processor, a GPU or dedicated hardware suitable for executing processing of a machine learning model such as a neural network. In addition, the control unit 30 executes a stop position determination process according to this embodiment, which will be described later.

[0027] The voice input device 33 collects voices of passengers of the vehicle 100. The control unit 30 can recognize the input voices and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 34 receives GNSS signals and detects the current position of the vehicle 100.

[0028] The storage device 35 is a large-capacity storage device that stores map data and the like, including information on routes on which the vehicle 100 can travel, landmarks such as buildings, stores, and the like. The storage device 35 may also store programs executed by the processor, data used by the processor for processing, and the like. The storage device 35 may also store various parameters (e.g., trained parameters and hyperparameters of a deep neural network) of machine learning models for speech recognition and image recognition executed by the control unit 30. The storage device 35 may also be provided on a server (not shown).

[0029] The communication device 36 is a communication device that can connect to the network 140 via wireless communication such as fifth generation mobile communication or Wi-Fi.

[0030] (Software configuration for determining stopping position) Next, a software configuration for the stop position determination process in the control unit 30 will be described with reference to Fig. 4. This software configuration is realized by the control unit 30 executing a program stored in a storage medium.

[0031] The software configuration according to this embodiment includes an interaction unit 401, a vehicle control unit 402, and a database 403. The interaction unit 401 processes voice information (utterance information) exchanged with the communication device 120, processes image information acquired by the detection unit 15, etc., and performs processing to estimate a stopping position.

[0032] The vehicle control unit 402 determines the route to the stop position set by the interaction unit 401, controls each unit of the vehicle along the route, and so on. As will be described in detail later, when the vehicle 100 approaches the stop position while traveling using the relative position, the vehicle control unit 402 controls the traveling speed according to the remaining distance. For example, when the remaining distance to the stop position is greater than a predetermined value, the vehicle control unit 402 controls the vehicle to approach the stop position at a predetermined (relatively fast) first speed. Furthermore, when the remaining distance is equal to or less than a predetermined value, the vehicle control unit 402 controls the vehicle to approach the stop position at a second speed (first speed > second speed) that allows for quick stop control through safe acceleration and deceleration.

[0033] The database 403 stores various data such as map data including information on routes that the vehicle 100 can travel, landmarks such as buildings, stores, etc., and travel history information of the vehicle itself and other vehicles.

[0034] The user data acquisition unit 413 acquires utterance information and location information transmitted from the communication device 120. The user data acquisition unit 413 may store the acquired utterance information and location information in the database 403. As will be described later, the utterance information acquired by the user data acquisition unit 413 is input to a trained machine learning model to estimate the user's intention. Note that the following description will be given taking as an example a case where a user's instruction is acquired based on utterance information. However, information including the user's instruction (instruction information) is not limited to voice information, and may be other information including the user's intention, such as text information.

[0035] The voice information processing unit 414 includes a machine learning model that processes voice information and executes the inference stage processing of the machine learning model. The machine learning model of the voice information processing unit 414 performs calculations of a deep learning algorithm using, for example, a deep neural network (DNN) to recognize the content of a user's utterance and estimate the user's intention. Separate machine learning algorithms may be used for recognizing the content of a user's utterance and estimating the user's intention.

[0036] The user's intention may be estimated by a classification process that classifies utterance information into predetermined intention classes. The intention classes of utterances may be defined for each usage scenario in which the user 130 uses the vehicle 100 (e.g., before boarding, during boarding, and after disembarking). Defining an intention class for each usage scenario limits the number of classifications in intention recognition, thereby improving recognition accuracy. The usage scenario "before boarding" may be associated with an intention class such as inquiry, pick-up request, greeting, location instruction, landmark expression, agreement, denial, or repetition. The usage scenario "during boarding" may be associated with an intention class that is at least partially different from that before boarding, such as route instruction, stop instruction, acceleration instruction, deceleration instruction, agreement, denial, or repetition. Similarly, the usage scenario "after disembarking" may be associated with an intention class that is at least partially different from that before boarding and during boarding. As an example of intent class estimation, an utterance such as "Can I board now?" before boarding is classified as an intention of "inquiry." An utterance such as "Can you come soon?" is classified as an intention of "pick-up request." Furthermore, utterance information such as "in front of the vending machine" is classified as the intention of "landmark expression."

[0037] In recognizing the content of a user's utterance, for example, if the user's intention is to specify a location, the name of a place or the like included in the utterance information may be identified. In recognizing the content of a user's utterance, for example, the name of a place, a landmark such as a building, a store name, a landmark name, etc. included in the utterance information may be recognized. The landmark may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, etc. included in the utterance information.

[0038] The DNN is put into a trained state by performing the learning stage processing, and by inputting utterance information into the trained DNN, recognition processing (inference stage processing) for the utterance information can be performed. Note that, although the present embodiment will be described taking as an example a case where the vehicle 100 performs the speech recognition processing, the speech recognition processing may be performed by a server (not shown) and the recognition results may be received from the server.

[0039] The image information processing unit 415 includes a machine learning model that processes image information, and the trained machine learning model executes the inference stage processing. The machine learning model of the image information processing unit 415 performs processing to recognize targets included in the image information, for example, by performing calculations of a deep learning algorithm using a deep neural network (DNN). Targets may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, etc. included in the image. In addition, the machine learning model of the image information processing unit 415 can recognize the faces of people included in the image information, the actions of people (e.g., hand gestures), the shapes and colors of their clothes, etc.

[0040] The stop position determination unit 416 executes the operation of the stop position determination process described below in cooperation with the above-mentioned audio information processing unit 414 and image information processing unit 415. The stop position determination process will be described later.

[0041] (Outline of the stopping position determination process) 5A to 5H, an overview of the stop position determination process executed in vehicle 100 will be described. Note that the operations described in this description, with vehicle 100 as the main actor, are realized by control unit 30 of vehicle 100 executing a program and operating each part shown in FIG.

[0042] In the stop position determination process, when a user requests pickup, the user 130 and the vehicle 100, who are currently in distant locations, first approach each other to the point where the user and landmarks can be visually confirmed. The vehicle 100 then adjusts the stop position based on voice information (speech information) and image information exchanged between the user 130 and the vehicle 100, and stops at the desired location of the user 130. The reason for making the stop position adjustable in this manner is to take into account the possibility that the user may modify the stop position once determined depending on the surrounding conditions. Furthermore, by making the stop position adjustable, it is possible to easily address the possibility that the accuracy of voice recognition and image recognition may fluctuate depending on the surrounding environment, such as noise and lighting conditions. As will be described later, in this embodiment, the vehicle 100 can adjust a new stop position while moving to the currently set stop position or while stopped, allowing for a smooth merge at the new stop position.

[0043] FIG. 5A schematically illustrates a situation in which a user 130 uses speech to call a vehicle 100 parked in a waiting area in an area where the vehicle 100 is allowed to travel. First, the user 130 uses the communication device 120 to transmit utterance information 510, such as "Can you come right away?", to the vehicle 100. The vehicle 100 acquires the utterance information of the user 130 and, upon determining that the intention of the utterance information is a "pick-up request," transmits utterance information 511, such as "Yes, I'll be there right away," to the communication device 120. For example, an area in which a vehicle cannot stop exists near the current location of the user 130. Furthermore, the user 130 wants to get into the vehicle 100 while traveling toward a vending machine 500 located ahead in the direction of travel. Therefore, the user 130 transmits utterance information 512, such as "I'm in a hurry, so follow me," from the communication device 120 to the vehicle 100. Upon receiving the utterance information 512 and recognizing the intention and content of the utterance, the vehicle 100 transmits utterance information 513 saying "I understand" to the communication device 120. At this time, the user's utterance information does not include a destination designation, so the vehicle 100 acquires location information (e.g., GPS information) of the communication device 120 of the user 130 and sets the stopping position based on the location of the user 130 and map information. The vehicle 100 then calculates a route to the stopping position (currently the location of the user 130) from the map information, and moves along that route.

[0044] 5B shows a schematic example of the system status after the vehicle 100 acquires speech information 510, "Can you come soon?", and performs speech information processing, etc. The system status is shown, for example, by visual information understanding 520, which indicates the result of processing the visual information, linguistic information understanding 521, which indicates the result of processing the speech information, and likelihood 522 of the estimated stopping position.

[0045] In the visual information comprehension 520, a display 524 of the recognized target is displayed within the captured image information 523. Note that in this explanation, to ensure the visibility of the drawing, information on subjects unrelated to the explanation is omitted. For this reason, in the example shown in FIG. 5B, nothing is written in the image information 523, but in reality, there are subjects in the background of the photograph. In the example shown in FIG. 5B, "None" is superimposed, indicating that the target has not been recognized.

[0046] Linguistic information understanding 521 shows an estimation result 525 of the user's utterance intention. The horizontal bar graph indicates the probability that the estimated user's intention falls into a category (a request to pick up, an instruction to a location, etc.). In this example, it indicates that there is a high probability that the intention of the user's utterance information "Can you come right away?" is a request to pick up.

[0047] The likelihood 522 of the stop position is represented by a mesh map 526 of the area shown in FIG. 5A, and colored areas 527 are areas with a high probability (likelihood) of being estimated as a stop position. The black-filled area indicates the area with the highest probability of being a stop position. The hatched area indicates the area with the next highest probability.

[0048] FIG. 5C illustrates a situation in which the vehicle 100 has approached the user 130 sufficiently to identify the user 130 through image information. In this situation, the vehicle 100 switches its operating state from an "absolute position-based driving state" using latitude and longitude coordinates to a "relative position-based stop control state" for controlling the stop position using a relative position. Here, the absolute position refers to the position of a moving object or target relative to a specific geographic coordinate system, such as latitude and longitude. Because the relative position is determined using image information, the vehicle 100 engages in a dialogue to confirm whether the person in the image is the user 130. For example, the vehicle 100 transmits speech information 530 such as "I'm getting close. Can you wave?" The user 130 transmits speech information 531 such as "Hey, I'm here" from the communication device 120 to the vehicle 100 while waving. The vehicle 100 recognizes the user 130 waving in the image and transmits speech information 532 such as "Is that the person in the red clothes?" to the communication device 120. In response to this, the user 130 transmits speech information 533, which indicates agreement, "Yes, that's right," from the communication device 120 to the vehicle 100.

[0049] FIG. 5D schematically illustrates an example of a system status in which the dialogue between vehicle 100 and user 130 shown in FIG. 5C has progressed. Image information 523 illustrates recognition result 541, which recognizes a person waving, and recognition result 540, which recognizes a vending machine as a target. Furthermore, language information understanding 521 receives utterance information 533 from user 130, "Yes, that's right," and illustrates speech intention estimation result 525, which indicates a high probability that the utterance intention is "agree." Furthermore, vehicle 100 calculates likelihood 522 of the stopping position based on the relative position of user 130 obtained from recognition result 541 based on image information and the absolute position of vehicle 100. In this example, user 130 has moved, and thus the position of colored area 542 has moved to the right on map 526.

[0050] FIG. 5E shows an example of a dialogue after the vehicle 100 acquires the position of the user 130 as a relative position. The vehicle 100 transmits speech information 550, "It'll stop right in front of me," to the communication device 120. In response, the user 130 wishes to board at another location and therefore transmits speech information 551, "No, stop over there," to the vehicle 100. At this time, the user 130 is pointing at the vending machine 500. Based on the user 130's speech information 551 (including "over there") and the pointing gesture, the vehicle 100 recognizes the landmark (the vending machine) from the image information. The vehicle 100 then transmits speech information 552, "You're at the vending machine, right?" to confirm the landmark, and then receives speech information 553, "Yes, that's right," from the user 130. Through these dialogues and recognition processes, the vehicle 100 sets the position of the vending machine 500 as a stop position and controls its travel so that it stops at that position. In this way, the vehicle 100 can search for a stopping position by continuing a dialogue with the user 130 while approaching the initially set stopping position, and can flexibly reset the new stopping position even when a new stopping position is specified.

[0051] FIG. 5F schematically illustrates an example of a system status in which vehicle 100 processes utterance information 551 from user 130, "No, stop over there." Image information 523 illustrates recognition result 561, which recognizes a person pointing, and recognition result 560, which recognizes a vending machine as a target. Furthermore, language information understanding 521 illustrates utterance intention estimation result 525, which receives utterance information 551 from user 130, "No, stop over there," and indicates a high probability that the utterance intention is a "location instruction." Furthermore, vehicle 100 calculates likelihood 522 of the stop position based on the relative position of vending machine 500 obtained from image information recognition results 560 and 561 and the absolute position of vehicle 100. In this example, because the position of vending machine 500 is highly likely to be the stop position, colored region 562 has moved to the right on map 526 (to the position corresponding to vending machine 500).

[0052] FIG. 5G shows an example of a conversation between vehicle 100 and user 130 when vehicle 100 approaches the location of vending machine 500. Vehicle 100 travels while continuously measuring the relative position of the vending machine that appears in the image information, and when vehicle 100 approaches the relative position, it transmits speech information 570 saying "Sorry to keep you waiting" to communication device 120. Vehicle 100 also stops at the currently set stopping position. When vehicle 100 receives speech information 571 saying "Thank you" from user 130, it can determine that vehicle dispatch has been completed (i.e., the relative position will not be re-specified).

[0053] FIG. 5H schematically illustrates an example of the system status after vehicle 100 processes utterance information 571 of "thank you" from user 130. Image information 523 illustrates recognition result 581 of a person and recognition result 580 of a vending machine as a target. Furthermore, language information understanding 521 illustrates estimation result 525 of the utterance intention, which indicates that utterance information 571 of "thank you" from user 130 is received and that the utterance intention is highly likely to be "thank you." Furthermore, vehicle 100 calculates likelihood 522 of the stop position based on the relative position of vending machine 500 obtained from recognition result 580 based on image information and the absolute position of vehicle 100. In this example, since the position of vending machine 500 is highly likely to be the stop position, the position of region 582 indicating high probability indicates the position on map 526 corresponding to vending machine 500.

[0054] Next, a state transition of the driving system control of the vehicle 100 in response to a stop position instruction will be described with reference to Fig. 6. First, when the state transition of the vehicle 100 starts, the vehicle 100 enters a "stop" state. In this state, a request such as a pick-up request can be made. Departure When the command is received, the vehicle 100 enters an automatic driving state. The automatic driving state roughly includes an "absolute position-based driving state," a "relative position-based stop control state," and a "stop control state."

[0055] In the "traveling state based on absolute position," as described above, when the vehicle 100 receives a pickup request from the user, the vehicle 100 first sets the initial stopping position as an absolute position. For example, if the speech information of the pickup request includes a landmark, the vehicle 100 sets the position of the landmark as an absolute position. If the speech information does not include the landmark, the vehicle 100 sets the user's position (GPS position information of the communication device 120) as an absolute position. Therefore, the vehicle 100 transitions to the "traveling state based on absolute position" of the "autonomous traveling state" and starts traveling. The vehicle 100 may confirm the stopping position with the user 130 while traveling. If the speech information of the user 130 indicates that the current stopping position is acceptable, the vehicle 100 transitions to the stop control state when it arrives at the stopping position. For example, after a predetermined time has elapsed in the stop control state, the vehicle 100 determines that the vehicle dispatch is completed and transitions to the "stopped" state.

[0056] When the vehicle 100 approaches the stop position (set as an absolute position) to the extent that the user or a landmark can be visually confirmed, the vehicle 100 transitions to a "stop position search in progress" state in the "stop control state based on relative position." During the stop position search, the vehicle 100 identifies the user or landmark and sets the stop position based on the relative position of the identified landmark. For example, the vehicle 100 transmits speech information to the user 130 to confirm the stop position. As in the above example, when the vehicle 100 receives speech information 551 such as "No, stop over there," the vehicle 100 identifies the target of the instruction (vending machine 500), sets the relative position of the target of the instruction as the stop position, and transitions to the "approach / stop control" state.

[0057] In the "Approach / Stop Control" state, the vehicle 100 moves towards the set stop position. At this time, when the vehicle 100 cannot detect a landmark that is the stop position from the image information, such as an obstacle appearing in front of it, the vehicle 100 transitions its state to "Searching for Stop Position". Also, as described above, when the vehicle 100 approaches the stop position during traveling using the relative position, the traveling speed may be controlled according to the remaining distance. When the vehicle 100 arrives at the relative position, it stops and enters the stop standby state. In the stop standby state, when voice information for re-designating the relative position is acquired from the user 130, the vehicle 100 re-sets the stop position and returns the state to the "Approach / Stop Control" state. When the user 130's voice information indicates that the stop position is fine, such as "Thank you", the vehicle 100 transitions to the stop control state.

[0058] In each of the states of "Searching for Stop Position", "Approach / Stop Control", and "Stop Standby", the vehicle 100 may set driving control parameters corresponding to each state and control the driving of the vehicle according to the vehicle speed control parameters. For example, for each of the states of "Searching for Stop Position", "Approach / Stop Control", and "Stop Standby", a predetermined target (limit) vehicle speed, target acceleration / deceleration, and minimum state holding time may be held as a table, and the vehicle speed, etc. may be controlled according to the state transition. For example, in "Searching for Stop Position", since the driving route where the (new) stop position is determined is likely to change, the target vehicle speed may be lower than that in "Approach / Stop Control" where the stop position has already been determined. That is, the target vehicle speeds for "Searching for Stop Position", "Approach / Stop Control", and "Stop Standby" may be set to (A, B, 0) (where A < B), respectively. By doing so, it is possible to realize driving in accordance with the acceleration / deceleration and target vehicle speed set for each state as the state transitions. Note that the driving control of the vehicle 100 is not limited to this example. As described later, the vehicle speed may be controlled according to the degree of confidence in the stop position.

[0059] (A series of operations of the stop position determination process) Next, a series of operations in the stop position determination process in the vehicle 100 will be described with reference to Fig. 7. This process is realized by the control unit 30 executing a program. In the following description, for simplicity, it is assumed that the control unit 30 executes each process, but the corresponding processes are actually executed by each part of the interaction unit 401 (described above in Fig. 4) and the vehicle control unit 402.

[0060] In S701, the control unit 30 receives utterance information of a pick-up request and location information of the user from the communication device 120. The utterance information of the pick-up request includes, for example, an utterance such as "Can you come right away?" described above in FIG. 5A. The utterance information of the pick-up request may include a destination, such as "Can you come right away, before AAA?". The user's location information is, for example, location information detected by the communication device 120. In S702, the control unit 30 identifies the destination from the user utterance information of the pick-up request. For example, the control unit 30 can identify "AAA" as the destination from utterance information such as "Can you come right away, before AAA?". If the utterance information does not include a destination, such as "Can you come right away?", the control unit 30 may determine that the destination could not be identified. Alternatively, the control unit 30 may transmit utterance information inquiring about the destination to the communication device 120 and additionally acquire utterance information including the destination.

[0061] In S703, the control unit 30 determines whether the destination has been identified. For example, if the speech information of the pick-up request includes a word indicating the destination, the control unit 30 determines that the destination has been identified and proceeds to S705; otherwise, the control unit 30 proceeds to S704.

[0062] In S704, the control unit 30 sets the user's location as the stop location (when the destination cannot be identified from the speech information). At this time, the user's location information is an absolute location. By setting the user's location as the stop location, the vehicle 100 will first travel toward the user's location and follow a route that approaches the user.

[0063] In S705, the control unit 30 specifies the location of the identified destination from the map information and sets it as the stop position. For example, the control unit 30 searches the map information for the name of the destination, AAA, and sets the location information (e.g., latitude and longitude information) obtained by the search as the stop position. In this case, the location information is also an absolute position.

[0064] In S706, the control unit 30 moves the vehicle 100 to the set stop position. For example, the control unit 30 determines a travel route to the stop position based on map information, and travels according to the travel route.

[0065] In S707, the control unit 30 determines whether the vehicle 100 has approached the stop position. For example, the control unit 30 acquires current position information of the vehicle 100 and determines whether the position information is within a predetermined distance from the latitude and longitude determined as the stop position. If the current vehicle position is within the predetermined distance from the stop position, the control unit 30 determines that the vehicle has approached the stop position and proceeds to S708; if not, the control unit 30 returns to S707 (i.e., repeats the determination while traveling along the route).

[0066] In S708, the control unit 30 executes a stop position adjustment process using the relative position. Details of the stop position adjustment process will be described later with reference to Fig. 8. After completing the stop position adjustment process using the relative position, the control unit 30 then ends the series of operations.

[0067] (Operation of stop position adjustment process using relative position) Furthermore, the operation of the stop position adjustment process using the relative position in the vehicle 100 will be described with reference to Fig. 8. Note that this process is realized by the control unit 30 executing a program, similar to the process shown in Fig. 7. This process is a process that specifies the relative position of a target such as a user or a landmark identified by image information acquired by the detection unit 15, etc., and controls traveling to a stop position using the relative position based on the image information. In other words, this process generally corresponds to the process in the "stop control state based on relative position" described above in Fig. 6.

[0068] In S801, the control unit 30 performs object recognition processing on an image acquired by, for example, the sensing unit 15 to identify object regions (corresponding to visual landmarks) in the image.

[0069] In S802, the control unit 30 determines whether a predetermined number or more of objects are present in the surrounding area. For example, the control unit 30 determines whether the number of object regions identified in S801 is a predetermined number or more, and if the number of object regions is a predetermined number or more, the process proceeds to S804, and if not, the process proceeds to S803.

[0070] In S803, the control unit 30 identifies the user from the image information. At this time, the control unit 30 may further identify a user action such as a hand gesture or a pointing action of the user. In addition, the control unit 30 may transmit utterance information to the communication device 120 to limit the target person when identifying the user, such as "Can you wave?" or "Is that the person in the red clothes?"

[0071] In S804, the control unit 30 transmits speech information to the communication device 120 inquiring about landmarks at the stop position. As described above, when the number of regions detected in the image is equal to or greater than a certain number, it is difficult to identify the landmarks with a high degree of accuracy based solely on the image recognition results. Therefore, when the number of regions detected in the image is equal to or greater than a certain number, the control unit 30 utilizes speech information, such as transmitting speech information inquiring about landmarks at the stop position, to identify the landmarks using the speech information and image information. The control unit 30 may transmit additional speech information to narrow down the visual landmarks, such as, "Is that a red vending machine?" When the visual landmarks cannot be narrowed down to one based on the relationship between the speech information of the user 130 and the image information of the vehicle 100, the ambiguity of the visual landmarks can be reduced by obtaining additional speech information from the user. This allows for more accurate identification of the landmarks.

[0072] In S805, the control unit 30 acquires the user's speech information and identifies a landmark from the speech information. At this time, the control unit 30 may further identify user actions such as the user's hand gestures and pointing movements. For example, if the user's speech information is "stop over there," the control unit 30 identifies the demonstrative pronoun "over there." Also, if the user's speech information is "stop in front of the vending machine," the control unit 30 identifies "vending machine" as a landmark.

[0073] In S806, the control unit 30 identifies a landmark corresponding to the speech information identified in S805 from the image information and specifies its relative position. For example, if the control unit 30 identifies "over there" from the user's speech information, it recognizes the user's pointing in the image and identifies an object in that direction as a landmark. Furthermore, if the control unit 30 identifies "vending machine" from the speech information, it identifies the area of ​​the vending machine in the image information. Then, the control unit 30 specifies the relative position of the identified object. As described above, the relative distance is a position as seen from the vehicle 100, and is expressed as, for example, 1 meter to the left or 10 meters forward.

[0074] It is also possible to calculate a probability distribution indicating the probability that one or more object regions in an image correspond to a visual landmark. For example, if the landmark included in the utterance information is a "vending machine" and two or more "vending machine" regions exist in the image, the control unit 30 may calculate the probability distribution of the object region based further on a restrictive linguistic element (e.g., "blue") in the utterance content. In this case, for example, if a blue vending machine and a red vending machine exist in the image, the control unit 30 may calculate a probability distribution in which the probability of a blue vending machine is "0.90" and the probability of a red vending machine is "0.10."

[0075] If the landmark included in the speech information is a "vending machine" and two or more "vending machine" regions exist in the image, the same probability may be assigned to both object regions. In this case, the control unit 30 may further vary the probability distribution depending on the relative positional relationship between the visual landmark and the user 130. If the red vending machine is closer to the current position of the user 130 or the vehicle 100, the control unit 30 may correct the probability distribution so that the probability of the red vending machine becomes "0.6" and the probability of the blue vending machine becomes "0.4." A probability distribution can be assigned in which the probability increases in order of likelihood of being a candidate from the direction the user is approaching.

[0076] When the speech information includes a positional relationship with an object, such as "a vending machine on the left side of the building," the control unit 30 may calculate a probability distribution that takes into account the relative positional relationship as seen from the vehicle 100. For example, the probability of a vending machine area on the left side of the building may be calculated as "0.9," and the probability of a vending machine area on the right side may be calculated as "0.1."

[0077] When the control unit 30 calculates the probability distribution for the landmarks, it identifies the object with the highest probability as the landmark and determines its relative position.

[0078] In S807, the control unit 30 transmits utterance information confirming the stopping position to the communication device 120. For example, the control unit 30 transmits utterance information such as "It will stop right in front of you" to the communication device 120. The control unit 30 also receives utterance information regarding confirmation from the user 130 in response to the utterance information confirming the stopping position. For example, the control unit 30 receives utterance information such as "No, please say hello over there."

[0079] In S808, the control unit 30 determines whether the received utterance information includes a stop position designation. That is, the control unit 30 determines whether the stop position confirmation in S807 includes a designation to change the stop position. For example, if the control unit 30 determines that the utterance information of the user 130 includes a location designation such as "over there" or "in front of the vending machine," the process proceeds to S805; otherwise, the process proceeds to S809. The process proceeds to S809 when, for example, utterance information of "OK" is received from the user 130.

[0080] In S809, the control unit 30 identifies the relative position of the user or landmark identified in the image information. As described above, the relative distance is the position as seen from the vehicle 100, and is expressed as, for example, 1 m to the left or 10 m ahead.

[0081] In S810, the control unit 30 sets the identified relative position as the stop position and controls driving to the stop position. At this time, if a landmark for the stop position is identified in the image information, the stop position is updated with the position of the landmark. As described above, the control unit 30 may control the driving speed according to the remaining distance to the stop position. The driving speed may also be adjusted according to the confidence level of the stop position. For example, if multiple visual landmarks obtained from the user's speech information are present in the image (e.g., two vending machines), the control unit 30 can assign a probability distribution to each vending machine according to the user's speech and relative positions, as described above. In this case, the control unit 30 may use the value of the probability distribution as the confidence level. In other words, when the confidence level is low, the control unit 30 reduces the driving speed compared to when the confidence level is high. By doing so, the vehicle can travel at a reduced speed when there is a high possibility that the stop position will change, thereby preparing for the change in the stop position. On the other hand, when the confidence level is high, the vehicle can quickly approach the stop position.

[0082] for example, Final speed = (maximum set speed - minimum set speed) * Stop position confidence + minimum set speed The vehicle speed may be changed linearly according to the confidence level, as shown below. The present invention is not limited to this example, and the traveling speed according to the confidence level may be set using a non-linear function.

[0083] Furthermore, the speed may be changed depending on the progress of the interaction between the user and the vehicle. For example, Final speed = (maximum set speed - minimum set speed) * progress of dialogue + minimum set speed The vehicle speed may be set in this manner.

[0084] Alternatively, the control unit 30 may obtain the stop position together with the distribution of the confidence levels, and use an evaluation function to determine the final speed. For example, Stage cost = α * step time + β * acceleration / deceleration operation cost Terminal cost = β*divergence (distribution distance) (distribution of stopping positions, distribution of vehicle position) ·Restrictions: Upper limit vehicle speed, upper limit acceleration / deceleration By considering the above optimization problem, it is possible to obtain a target speed that minimizes the above cost within the predicted time. In this way, it is possible to perform speed control that balances acceleration / deceleration and the arrival time to the stop position for the distribution of stop positions.

[0085] In S811, the control unit 30 determines whether the stop position has been re-designated (for example, by receiving speech information designating the stop position from the user 130), and if it is determined that the stop position has been re-designated, the process proceeds to S805; otherwise, the process proceeds to S812. In S812, the control unit 30 determines whether the distance to the stop position has approached within a predetermined distance. If the distance to the stop position has approached within the predetermined distance, the control unit 30 proceeds to S813; otherwise, the process returns to S811.

[0086] In S813, since the stop position has not been re-designated and the control unit 30 has approached the stop position, the control unit 30 decelerates, drives, and stops at the stop position. At this time, the control unit 30 transmits speech information (for example, "Sorry to have kept you waiting") to the communication device 120 to notify the arrival. The control unit 30 then ends the stop position adjustment process using the relative position and returns to the caller. Then, the control unit 30 also ends the series of processes shown in FIG. 7.

[0087] As described above, the vehicle 100 first approaches the user or landmarks or the like using the absolute position to an extent that the user can visually confirm them, and then adjusts the stopping position based on the voice information (utterance information) and image information exchanged between the user 130 and the vehicle 100, thereby stopping at a position desired by the user 130. In this way, the stopping position of the vehicle (mobile body) can be flexibly adjusted between the user and the vehicle (mobile body).

[0088] (Variation) Modifications of the present invention will be described below. In the above embodiment, an example has been described in which the stop position determination process is executed in the vehicle 100. However, the above-mentioned stop position determination process can also be executed on the server side. In this case, the information processing system 900 is configured with a vehicle 910, a communication device 120, and a server 901, as shown in FIG. 9.

[0089] The configuration of an information processing system 900 according to this embodiment will be described with reference to Fig. 9. The information processing system 900 includes a vehicle 910, a server 901, and a communication device 120. The server 901 is configured with one or more server devices, and transmits information about the vehicle transmitted from the vehicle 910, and speech information and position information transmitted from the communication device 120, over a network. 140 The server can acquire the image data via the network and control the running of the vehicle 910. The server generally has more abundant computational resources than the vehicle itself. In addition, by receiving and storing image data captured by various vehicles, learning data can be collected in a wide variety of situations, enabling learning to be performed in a wider range of situations.

[0090] For example, in an embodiment according to this modification, user utterance information is transmitted from the communication device 120 to the server 901. The server 901 also acquires image information captured by the vehicle 910 via the network 140 together with position information and the like as part of the floating data of the vehicle 910. For example, after the server 901 performs processing corresponding to S701 to S705 of the above-described stop position determination processing, the server 901 transmits a control variable, such as a traveling speed, to the vehicle 910 in S706. The vehicle 910 travels (continues traveling) in accordance with the received control variable. The server 901 then performs processing corresponding to S707 and S708. The server 901 also performs processing corresponding to S801 to S809 in the stop position adjustment processing using the relative position, and transmits a control variable, such as a traveling speed, to the vehicle 910 in S810. The vehicle 910 travels (continues traveling) in accordance with the received control variable. The server 901 then performs processing corresponding to S811 to S813. These processes by the server 901 are realized by a processor (not shown) included in the server 901 executing a program stored in a storage medium (not shown) included in the server 901 .

[0091] Furthermore, the configuration of vehicle 910 may be the same as that of vehicle 100, except that control unit 30 does not execute the stop position determination process and causes the vehicle to travel in accordance with the control amount from server 901.

[0092] In this way, the server 901 uses the absolute position to bring the vehicle close enough to the user or landmarks or the like so that they can be visually confirmed, and then adjusts the stopping position based on voice information (utterance information) and image information between the user 130 and the server 901, thereby stopping the vehicle at a position desired by the user 130. In this way, it becomes possible to flexibly adjust the stopping position of the vehicle (mobile body) between the user and the vehicle (mobile body).

[0093] In the above embodiment, the adjustment of the stopping position of a vehicle (mobile body) between a user and the vehicle (mobile body) attempting to merge has been described, but the application of the present invention is not limited to this. For example, the present invention may be applied to a case where a user, while in a vehicle (mobile body), instructs the stopping position based on a landmark. For example, while in a mobile body, the user may make a speech information (instruction information) such as "Park in front of the vending machine over there," and the mobile body may determine the stopping position by responding with, for example, "Is that the red vending machine?" Thereafter, a case can be considered in which the mobile body adjusts the stopping position based on speech information (instruction information) such as "Park at the convenience store over there after all" from the user.

[0094] <Summary of the embodiment> 1. In the above-described embodiment, A control device (e.g., 30) for a moving body (e.g., 100) that adjusts a stopping position of the moving body based on an instruction from a user, An instruction acquisition means (e.g., 413) for acquiring user instruction information; Image acquisition means (e.g., 15 to 17) for acquiring a photographed image taken by the moving body; A determination means (e.g., 416) for determining a stopping position of the moving body; a control means (e.g., 402) for controlling the movement of the moving body so that the moving body moves toward the determined stop position; A control device for a mobile body is provided in which the determination means (i) determines a first stopping position using location information of a communication device used by a user or location information corresponding to a destination included in the user's first instruction information, and (ii) determines a second stopping position based on the user's second instruction information and the area of ​​a predetermined target identified in a captured image when the position of the mobile body comes within a predetermined distance from the first stopping position due to the movement of the mobile body.

[0095] According to this embodiment, the stopping position of the moving body can be flexibly adjusted between the user and the moving body.

[0096] 2. In the above-described embodiment, The determination means determines the second stop position by identifying the designation of the predetermined target from the second instruction information of the user and then identifying the area of ​​the predetermined target from the captured image.

[0097] According to this embodiment, the area of ​​a target (e.g., a vending machine) in the captured image can be identified from user instruction information such as "stop in front of the vending machine," and the position of the target (e.g., a vending machine) can be determined as the stopping position.

[0098] 3. In the above-described embodiment, The determination means identifies the user in the captured image based on the user's instruction information and the captured image, and then determines the second stopping position by identifying the area of ​​the specified target from the captured image based on the user's second instruction information and the user's behavior identified in the captured image.

[0099] According to this embodiment, the user can be identified by their speech and actions, and further, if the user points at a target (e.g., a vending machine) and says something like "stop over there," the location of the target can be determined as the stopping position based on the speech and pointing.

[0100] 4. In the above-described embodiment, The determination means determines the first stop position using position information of the communication device in response to receiving instruction information including a request for pickup.

[0101] According to this embodiment, a mobile object can be directed to the user's location simply by uttering a pickup request such as "Can you come?"

[0102] 5. In the above-described embodiment, The determining means determines the first stop location using one or more instructions including a pickup request and the destination.

[0103] According to this embodiment, the mobile body can be directed to the specified destination based on a pick-up request that includes the destination, such as "Can you come right in front of AAA?", or an additional utterance that includes the destination.

[0104] 6. In the above-described embodiment, The control means causes the moving body to travel at a travel speed reduced in accordance with a predetermined standard in response to the position of the moving body coming within a predetermined distance from the first stop position.

[0105] According to this embodiment, when the vehicle is approaching the stopping position sufficiently or when there are multiple targets and the degree of certainty is low, the traveling speed can be reduced, enabling safe acceleration and deceleration.

[0106] 7. In the above-described embodiment, The control means reduces the traveling speed in accordance with the distance from the position of the moving body to the position of the predetermined target.

[0107] According to this embodiment, when the vehicle is far from the stop position, it can quickly approach the stop position, and when the vehicle is sufficiently close to the stop position, it can safely accelerate or decelerate.

[0108] 8. In the above-described embodiment, the determining means calculates a probability distribution indicating a probability that an area of ​​one or more targets identified in the captured image is a stopping position, and determines the second stopping position based on the area of ​​the target having the highest probability; The control means reduces the traveling speed as the probability assigned to the target corresponding to the second stop position decreases.

[0109] According to this embodiment, when there are multiple targets, the stopping position can be determined from the most likely target, and even if the stopping position is changed, it can be accommodated with safe acceleration and deceleration.

[0110] 9. In the above-described embodiment, the determining means calculates a probability distribution indicating a probability that an area of ​​one or more targets identified in the captured image is a stopping position, and determines the second stopping position based on the area of ​​the target having the highest probability; The control means controls the traveling speed so that the traveling speed becomes higher than the traveling speed reduced in accordance with the predetermined standard as the probability assigned to the target corresponding to the second stop position becomes higher.

[0111] According to this embodiment, when the vehicle is sufficiently close to the stop position, the vehicle speed is reduced and the vehicle is at the stop position. probability If the speed is high, the running speed can be adjusted to improve the running speed.

[0112] According to this embodiment, when there are multiple targets, the stopping position can be determined from the most likely target.

[0113] 11. In the above embodiment, The determination means calculates a probability distribution indicating the probability that an area of ​​a plurality of targets identified in the captured image is a stopping position, and determines a second stopping position based on the distance to the candidate target, selecting a predetermined number of areas of a plurality of targets with a high probability as candidates.

[0114] According to this embodiment, when there is a distant area with the highest probability (first candidate) and a nearby area with a probability slightly lower than the maximum probability (second candidate), it becomes possible to flexibly determine the stopping position by adjusting the stopping position to the closer area.

[0115] 12. In the above-described embodiment, If the speech acquisition means acquires instruction information relating to another destination or another target while the moving body is traveling toward the determined stopping position, the determination means determines a new stopping position relating to the other destination or other target while continuing traveling.

[0116] According to this embodiment, the stop position can be adjusted while the moving body continues to travel.

[0117] 13. In the above embodiment, If the speech acquisition means acquires instruction information relating to another destination or another target while the moving body is traveling toward the determined stopping position, the determination means transmits additional instruction information to the communication device to narrow down the other destination or other target.

[0118] According to this embodiment, a dialogue can be held with the user to narrow down the targets to be identified, and the stopping position intended by the user can be specified with high accuracy.

[0119] 14. In the above-described embodiment, the first stop position is determined by an absolute position, which is the position of the target from a position based on specific geographic coordinates; The second stop position is determined by a relative position, which is the relative position of the target as viewed from the coordinate system of the moving body.

[0120] According to this embodiment, after the moving object and the user approach each other, a coordinate system that has a high affinity with the appearance of the stopping position through the image is used. in Processing can be performed.

[0121] 15. In the above-described embodiment, The instruction acquisition means acquires the instruction information based on user utterance information.

[0122] According to this embodiment, the user can easily provide instruction information by speaking. 16. In the above-described embodiment, The moving body is a micro-mobility vehicle.

[0123] According to this embodiment, convenient transportation can be used.

[0124] 17. In the above-described embodiment, A method for controlling a moving object, which adjusts a stopping position of the moving object based on an instruction from a user, comprising: Obtaining user instruction information; acquiring a photographed image taken by the moving body; determining a stopping position of the moving object; controlling the travel of the moving object so that the moving object travels toward the determined stop position; A method for controlling a moving body is provided, in which determining the stopping position of the moving body includes (i) determining a first stopping position using location information of a communication device used by a user or location information corresponding to a destination included in the user's first instruction information, and (ii) determining a second stopping position based on the user's second instruction information and the area of ​​a predetermined target identified in a captured image in response to the moving body's position coming within a predetermined distance from the first stopping position due to the moving body's movement.

[0125] According to this embodiment, the stopping position of the moving body can be flexibly adjusted between the user and the moving body. [Explanation of symbols]

[0126] 100... vehicle, 120... communication device, 30... control unit, 413... user data acquisition unit, 414... voice information processing unit, 415... image information processing unit, 416... stop position determination unit

Claims

1. A control device for a moving body that adjusts a stopping position of the moving body based on an instruction from a user, an instruction acquisition means for acquiring user instruction information; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for determining a stop position of the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The determination means (i) determines a first stop position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user, and (ii) determines a second stop position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the position of the moving body coming within a predetermined distance from the first stop position as a result of the moving body's travel; The determination means determines the second stopping position by identifying the designation of the predetermined target from the second instruction information of the user, and then identifying the area of ​​the predetermined target corresponding to the designation of the predetermined target from the captured image.

2. The control device for a moving body as described in claim 1, characterized in that the determination means determines the second stopping position by identifying the user in the captured image based on the user's instruction information and the captured image, and then identifying the area of ​​the specified target from the captured image based on the user's second instruction information and the user's behavior identified in the captured image.

3. A control device for a moving body that adjusts a stopping position of the moving body based on an instruction from a user, an instruction acquisition means for acquiring user instruction information; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for determining a stop position of the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The determination means (i) determines a first stop position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user, and (ii) determines a second stop position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the position of the moving body coming within a predetermined distance from the first stop position as a result of the moving body's travel; The determination means identifies the user in the captured image based on the user's instruction information and the captured image, and then determines the second stopping position by identifying the area of ​​the specified target from the captured image based on the user's second instruction information and the user's behavior identified in the captured image.

4. 4. The control device for a mobile body according to claim 1, wherein the determination means determines the first stop position using position information of the communication device in response to receiving instruction information including a request for pickup.

5. The control device for a mobile body according to any one of claims 1 to 3, wherein the determination means determines the first stop position using one or more pieces of instruction information including a request for pickup and the destination.

6. 6. The control device for a mobile body according to claim 1, wherein the control means causes the mobile body to travel at a travel speed reduced in accordance with a predetermined standard when the position of the mobile body comes within a predetermined distance from the first stop position.

7. 7. The control device for a moving body according to claim 6, wherein the control means reduces the traveling speed in accordance with a distance from the position of the moving body to the position of the predetermined target.

8. the determining means calculates a probability distribution indicating a probability that an area of ​​one or more targets identified in the photographed image is a stopping position, and determines the second stopping position based on the area of ​​the target having the highest probability; 7. The control device for a moving body according to claim 6, wherein the control means reduces the traveling speed as the probability assigned to the target corresponding to the second stop position decreases.

9. the determining means calculates a probability distribution indicating a probability that an area of ​​one or more targets identified in the photographed image is a stopping position, and determines the second stopping position based on the area of ​​the target having the highest probability; 7. The control device for a moving body according to claim 6, wherein the control means controls the traveling speed so that the traveling speed becomes higher than the traveling speed reduced in accordance with the predetermined standard as the probability assigned to the target corresponding to the second stop position becomes higher.

10. 8. A control device for a moving body according to claim 1, wherein the determination means calculates a probability distribution indicating the probability that an area of ​​one or more targets identified in a captured image is a stopping position, and determines the second stopping position based on the area of ​​the target having the highest probability.

11. The control device for a moving body according to any one of claims 1 to 7, characterized in that the determination means calculates a probability distribution indicating the probability that an area of ​​a plurality of targets identified in the captured image is a stopping position, and determines a second stopping position based on the distance to the candidate targets by selecting areas of a predetermined number of multiple targets with high probabilities as candidates.

12. A control device for a moving body that adjusts a stopping position of the moving body based on an instruction from a user, an instruction acquisition means for acquiring user instruction information; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for determining a stop position of the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The determination means (i) determines a first stop position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user, and (ii) determines a second stop position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the position of the moving body coming within a predetermined distance from the first stop position as a result of the moving body's travel; the control means causes the moving body to travel at a travel speed reduced in accordance with a predetermined standard in response to the moving body being positioned within a predetermined distance from the first stop position; the determining means calculates a probability distribution indicating a probability that an area of ​​one or more targets identified in the photographed image is a stopping position, and determines the second stopping position based on the area of ​​the target having the highest probability; The control device for a moving body, wherein the control means reduces the traveling speed as the probability assigned to the target corresponding to the second stop position becomes lower.

13. A control device for a moving body that adjusts a stopping position of the moving body based on an instruction from a user, an instruction acquisition means for acquiring user instruction information; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for determining a stop position of the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The determination means (i) determines a first stop position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user, and (ii) determines a second stop position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the position of the moving body coming within a predetermined distance from the first stop position as a result of the moving body's travel; the control means causes the moving body to travel at a travel speed reduced in accordance with a predetermined standard in response to the moving body being positioned within a predetermined distance from the first stop position; the determining means calculates a probability distribution indicating a probability that an area of ​​one or more targets identified in the photographed image is a stopping position, and determines the second stopping position based on the area of ​​the target having the highest probability; The control means controls the traveling speed so that the traveling speed becomes higher than the traveling speed reduced in accordance with the predetermined standard, as the probability assigned to the target corresponding to the second stop position becomes higher.

14. A control device for a mobile body described in any one of claims 1 to 13, characterized in that if the instruction acquisition means acquires instruction information related to another destination or another target while the mobile body is traveling toward the determined stopping position, the determination means determines a new stopping position related to the other destination or other target while continuing traveling.

15. A control device for a mobile body described in any one of claims 1 to 14, characterized in that if the instruction acquisition means acquires instruction information related to another destination or other target while the mobile body is traveling toward the determined stopping position, the determination means transmits additional instruction information to the communication device to narrow down the other destination or other target.

16. the first stop position is determined by an absolute position, which is the position of the target from a position based on specific geographic coordinates; 16. The control device for a moving body according to claim 1, wherein the second stop position is determined by a relative position of the target as viewed from a coordinate system of the moving body.

17. The control device for a moving body according to claim 1 , wherein the instruction acquisition means acquires the instruction information based on speech information of a user.

18. The control device for a moving body according to any one of claims 1 to 17, wherein the moving body is a micro-mobility vehicle.

19. A method for controlling a moving object, which adjusts a stopping position of the moving object based on an instruction from a user, comprising: Obtaining user instruction information; acquiring a photographed image taken by the moving body; determining a stopping position of the moving object; controlling the travel of the moving object so that the moving object travels toward the determined stop position; Determining the stopping position of the moving body includes: (i) determining a first stopping position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user; and (ii) determining a second stopping position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the moving body's position coming within a predetermined distance from the first stopping position due to the moving body's travel. A method for controlling a moving body, characterized in that determining a stopping position of the moving body includes identifying a designation of the predetermined target from the second instruction information of the user, and then identifying an area of ​​the predetermined target corresponding to the designation of the predetermined target from the captured image, thereby determining the second stopping position.

20. A moving object that adjusts a stop position based on a user's instruction, an utterance acquisition means for acquiring user instruction information; image acquisition means for acquiring a photographed image taken by the moving body; a determination means for determining a stop position of the moving body; a control means for controlling the movement of the moving body so that the moving body moves toward the determined stop position, The determination means (i) determines a first stop position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user, and (ii) determines a second stop position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the position of the moving body coming within a predetermined distance from the first stop position as a result of the moving body's travel; The determining means determines the second stopping position by identifying the designation of the predetermined target from the second instruction information of the user, and then identifying the area of ​​the predetermined target corresponding to the designation of the predetermined target from the captured image.

21. An information processing method executed by an information processing device for adjusting a stop position of a moving object based on an instruction from a user, comprising: Obtaining user instruction information; acquiring, from the moving body, a captured image captured by the moving body and position information of the moving body; determining a stopping position of the moving object; transmitting a control command to the moving body to control the movement of the moving body so that the moving body moves toward the determined stop position, Determining the stopping position of the moving body includes: (i) determining a first stopping position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user; and (ii) determining a second stopping position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the moving body's position coming within a predetermined distance from the first stopping position due to the moving body's travel. an information processing method characterized in that determining the stopping position of the moving body includes identifying the designation of the predetermined target from the second instruction information of the user, and then identifying an area of ​​the predetermined target corresponding to the designation of the predetermined target from the captured image, thereby determining the second stopping position.

22. A method for controlling a moving object, which adjusts a stopping position of the moving object based on an instruction from a user, comprising: Obtaining user instruction information; acquiring a photographed image taken by the moving body; determining a stopping position of the moving object; controlling the travel of the moving object so that the moving object travels toward the determined stop position; Determining the stopping position of the moving body includes: (i) determining a first stopping position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user; and (ii) determining a second stopping position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the moving body's position coming within a predetermined distance from the first stopping position due to the moving body's travel. A method for controlling a moving body, characterized in that determining a stopping position of the moving body includes identifying the user in the captured image based on the user's instruction information and the captured image, and then identifying the area of ​​the specified target from the captured image based on the user's second instruction information and the user's behavior identified in the captured image, thereby determining the second stopping position.

23. A method for controlling a moving object, which adjusts a stopping position of the moving object based on an instruction from a user, comprising: Obtaining user instruction information; acquiring a photographed image taken by the moving body; determining a stopping position of the moving object; controlling the travel of the moving object so that the moving object travels toward the determined stop position; Determining the stopping position of the moving body includes: (i) determining a first stopping position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user; and (ii) determining a second stopping position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the moving body's position coming within a predetermined distance from the first stopping position due to the moving body's travel. controlling the travel of the mobile body includes causing the mobile body to travel at a travel speed reduced in accordance with a predetermined standard in response to the position of the mobile body coming within a predetermined distance from the first stop position; determining the stop position of the moving object includes calculating a probability distribution indicating a probability that an area of ​​one or more targets identified in the captured image is a stop position, and determining the second stop position based on the area of ​​the target having the highest probability; A method for controlling a moving body, characterized in that controlling the moving body's traveling includes reducing the traveling speed as the probability assigned to the target corresponding to the second stop position becomes lower.

24. A method for controlling a moving object, which adjusts a stopping position of the moving object based on an instruction from a user, comprising: Obtaining user instruction information; acquiring a photographed image taken by the moving body; determining a stopping position of the moving object; controlling the travel of the moving object so that the moving object travels toward the determined stop position; Determining the stopping position of the moving body includes: (i) determining a first stopping position using location information of a communication device used by a user or location information corresponding to a destination included in first instruction information of the user; and (ii) determining a second stopping position based on second instruction information of the user and an area of ​​a predetermined target identified in a captured image in response to the moving body's position coming within a predetermined distance from the first stopping position due to the moving body's travel. controlling the travel of the mobile body includes causing the mobile body to travel at a travel speed reduced in accordance with a predetermined standard in response to the position of the mobile body coming within a predetermined distance from the first stop position; determining the stop position of the moving object includes calculating a probability distribution indicating a probability that an area of ​​one or more targets identified in the captured image is a stop position, and determining the second stop position based on the area of ​​the target having the highest probability; A method for controlling the traveling of the moving body, characterized in that controlling the traveling speed of the moving body includes controlling the traveling speed so that the traveling speed is higher than the traveling speed reduced in accordance with the predetermined standard, the higher the probability assigned to the target corresponding to the second stop position.

25. A program for causing a computer to function as each of the means of the control device for a moving body according to any one of claims 1 to 18.

26. A storage medium for storing a program for causing a computer to function as each of the means of the control device for a moving body according to any one of claims 1 to 18.

Images