MOBILE BODY CONTROL SYSTEM, MOBILE BODY, MOBILE BODY CONTROL METHOD, AND PROGRAM

Abstract

To implement control that reflects a user's intention.SOLUTION: A mobile object control apparatus includes: an acquisition unit which acquires an image including a user; a first recognition unit which recognizes an indication of a user; a second recognition unit which recognizes a first region that a mobile object is able to enter and a second region that the mobile object is not able to enter on the basis of the image; and a determination unit which determines a target position to which the mobile object will move on the basis of the indication from the user, the first region that the mobile object is able to enter, and the second region that the mobile object is not able to enter.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a control system for a moving object, a moving object, a control method for a moving object, and a program. [Background technology]

[0002] Conventionally, an item estimation system has been disclosed that estimates an item pointed to by a human among items present in the vicinity of the human based on the human's speech and actions other than speech (see, for example, Patent Document 1). When a word indicating the name of an item is recognized as a result of voice recognition, this item estimation system estimates the item pointed to by the human by prioritizing the item identified based on voice recognition over items identified based on actions other than the human's speech (the human's line of sight or pointing direction), and executes control according to the position of the estimated item. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-223172 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above techniques, there are cases where control that reflects the user's intentions cannot be achieved.

[0005] The present invention has been made in consideration of these circumstances, and one of its objectives is to provide a mobile body control system, a mobile body, a control method, and a program that can realize control that better reflects the user's intentions. [Means for solving the problem]

[0006] The moving body control system, the moving body, the moving body control method, and the program according to the present invention employ the following configuration. (1): A control system for a moving body includes an acquisition unit that acquires an image captured by a user, a first recognition unit that recognizes instructions from the user, a second recognition unit that recognizes a first area into which the moving body can enter and a second area into which the moving body cannot enter based on the image, and a determination unit that determines a target position to which the moving body will move based on the user's instructions, the first area into which the moving body can enter, and the second area into which the moving body cannot enter.

[0007] (2): In the above aspect (1), the second recognition unit detects a boundary between the first area and the second area, and the determination unit determines the target position based on the boundary.

[0008] (3): In the above aspect (2), when the first recognition unit recognizes an instruction to the target position by a gesture of the user, the determination unit determines that the target position is a position shifted a predetermined amount from the boundary toward the first region based on the instruction.

[0009] (4): In the above aspect (2) or (3), when the first recognition unit recognizes an instruction of the target position accompanied by the user's designation of a target object, the determination unit determines the target position to be a position shifted a predetermined amount from the boundary within the predetermined range in the direction of the first region based on the position of the target object and the boundary within the predetermined range from the target object.

[0010] (5): In the aspect of (4) above, the determination unit determines the target position based on a boundary obtained by shifting the boundary within the predetermined range toward the user.

[0011] (6): In any of the above aspects (2) to (5), when the first recognition unit recognizes an instruction to approach the user by a gesture of the user, the determination unit determines, based on the position of the user and the boundary within a predetermined range from the user, a position shifted a predetermined amount toward the first area from the boundary within the predetermined range as the target position.

[0012] (7): In the above aspect (6), when the target position is a stopping position, and the first recognition unit recognizes the instruction to approach the user by a gesture of the user who is stopped, the determination unit determines the position shifted by the specified amount as the stopping position.

[0013] (8): In any one of the above (1) to (7), the target position is a stop position, The determination unit determines the stop position based on the instruction.

[0014] (9): In any of the above aspects (1) to (8), the target position is a stopping position, and the determination unit causes the direction of the moving body when the moving body stops at the stopping position to face the route direction toward the user's destination.

[0015] (10): In any one of the above aspects (1) to (9), the target position is a stop position, and when the determination unit cannot determine the stop position, it moves the moving object to a position near the user.

[0016] (11): In any of the above aspects (1) to (9), the target position is a stop position, and when the determination unit cannot determine the stop position, the determination unit inquires of the user about the stop position.

[0017] (12): A moving object equipped with a control system according to any one of the above aspects (1) to (11).

[0018] (13): A method for controlling a moving body according to one embodiment of the present invention includes a computer acquiring an image captured by a user, recognizing instructions from the user, recognizing a first area into which the moving body can enter and a second area into which the moving body cannot enter based on the image, and determining a target position to which the moving body will move based on the user's instructions, the first area into which the moving body can enter, and the second area into which the moving body cannot enter.

[0019] (14): A program according to one embodiment of the present invention causes a computer to acquire an image captured by a user, recognize instructions from the user, recognize a first area into which a moving body can enter and a second area into which the moving body cannot enter based on the image, and determine a target position to which the moving body will move based on the user's instructions, the first area into which the moving body can enter, and the second area into which the moving body cannot enter. Effect of the Invention

[0020] According to (1)-(14), the control system can realize control that better reflects the user's intentions by determining a target position to which the moving object will move based on the user's instructions, a first area into which the moving object can enter, and a second area into which the moving object cannot enter.

[0021] According to (3), the control system determines the target position to be a position that is shifted a predetermined amount from the boundary toward the first area based on the instruction, thereby enabling the moving body to move and to be stopped at a position that is suitable for the user.

[0022] According to (4), the control system can improve user convenience by moving the moving object to a position based on a target specified by the user.

[0023] According to (5), the control system moves the moving object to a position where it does not overlap with the target so that the user can easily access the moving object. As a result, the user's convenience is improved.

[0024] According to (9), the control system can orient the moving body when it stops at the stopping position in the direction of a route toward the user's destination, thereby enabling the user to quickly head toward the destination when traveling together with the moving body, for example.

[0025] According to (10), the control system can stop the moving object at a location that is suitable for the user, depending on the user's behavior. [Brief description of the drawings]

[0026] [Figure 1] 1 is a diagram showing an example of a moving body M equipped with a control device according to an embodiment. [Diagram 2] 13 is a diagram showing an example of another functional configuration of the moving body M. FIG. [Diagram 3] FIG. 2 is a diagram showing an example of the behavior of a moving object M. [Figure 4] 11 is a diagram showing an example of information referred to by a moving object M. FIG. [Diagram 5] 13 is a flowchart showing an example of a process flow in which the control device 100 identifies a user. [Figure 6] 1 is a diagram for explaining a feature amount (attribute) of a user and recognition of a target. FIG. [Figure 7] FIG. 13 is a diagram illustrating an example of a user's behavior. [Figure 8] 13 is a flowchart showing an example of a process flow in which the control device 100 identifies a user. [Figure 9] FIG. 1 is a diagram showing an example of the behavior of a moving object M (part 1). [Figure 10] FIG. 2 is a diagram showing an example of the behavior of a moving object M (part 2). [Figure 11] FIG. 3 is a diagram showing an example of the behavior of a moving object M. [Figure 12] FIG. 4 is a diagram showing an example of the behavior of a moving object M. [Figure 13] FIG. 5 is a diagram showing an example of the behavior of a moving object M. [Figure 14] FIG. 6 is a diagram showing an example of the behavior of a moving object M. [Figure 15] FIG. 15 is a diagram showing an example of the concepts and information to be recognized used in the processes described above with reference to FIGS. [Figure 16] 5 is a flowchart showing an example of a flow of processing executed when the control device 100 determines a stopping position. [Figure 17]10 is a flowchart showing another example of the flow of the process executed when the control device 100 determines a stopping position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A control system for a moving object, a moving object, a control method for a moving object, and a program according to embodiments of the present invention will be described below with reference to the drawings.

[0028] <Embodiment> [Overall configuration] FIG. 1 is a diagram showing an example of a moving body M equipped with a control device according to an embodiment. The moving body M is an autonomous mobile robot. The moving body M supports the actions of a user. For example, the moving body M stops at a position specified by the user, picks up the user, and transports the user to a destination. In this embodiment, the moving body M is described as moving with the user on board, but instead (or in addition), the moving body M may transport items, lead the user and move together with the user, or follow the user and support the user's actions, and the moving body M may or may not be one that the user can ride on.

[0029] The moving body M includes a main body 2, one or more wheels 4 (4A, 4B in the figure), and a camera 10. The main body 2 is provided with an entrance, such as a door (not shown), through which a user can enter and exit the main body 2, and the user can enter the main body 2 from the entrance and board the moving body M. For example, the moving body M drives wheels 94 based on an image captured by the camera 10 to transport the user M.

[0030] In this embodiment, the user is described as riding inside main body 2, but instead (or in addition), a seating section on which the user can sit without riding inside main body 2 in order to move with the moving body M, or a step on which the user can place their feet when moving, etc. may be provided.

[0031] 2 is a diagram showing an example of other functional configurations of the moving body M. The moving body M includes, for example, a camera 10, a communication device 20, an HMI 30, a moving body sensor 40, a position identification device 50, a driving operator 80, a control device 100, a driving force output device 200, a braking device 210, and a steering device 220.

[0032] The camera 10 captures an image of the periphery of the moving body M. The camera 10 is, for example, a fisheye camera capable of capturing an image of the periphery of the moving body M at a wide angle (e.g., 360 degrees). The camera 10 is attached, for example, to the upper part of the moving body M, and captures an image of the periphery of the moving body M at a wide angle in the horizontal direction. The camera 10 may be realized by combining multiple cameras (multiple cameras capturing an image in a range of 120 degrees or 60 degrees in the horizontal direction). In addition to the camera 10, the moving body M may be equipped with a radar device or LIDAR that detects objects.

[0033] The communication device 20 is a communication interface for communicating with other devices using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like.

[0034] The HMI 30 presents various information to a user of the mobile object M and accepts input operations by the user. The HMI 30 includes various display devices, a speaker, a buzzer, a touch panel, switches, keys, and the like.

[0035] The moving body sensor 40 includes a vehicle speed sensor that detects the speed of the moving body M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, a direction sensor that detects the direction of the moving body M, and the like.

[0036] Based on the signals received from the GNSS satellites, the position determination device 50 determines the position of the moving body M. The position of the moving body M may be determined or supplemented by an INS (Inertial Navigation System) that uses the output of the moving body sensor 40.

[0037] The driving operators 80 include, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a special steering wheel, a joystick, and other operators. A sensor is attached to the driving operators 80 to detect the amount of operation or the presence or absence of operation, and the detection results are output to the control device 100, or to some or all of the driving force output device 200, the brake device 210, and the steering device 220. Note that when the moving body M is controlled only by automatic driving, the driving operators 80 may be omitted.

[0038] The control device 100 includes, for example, an acquisition unit 110, a recognition unit 120, a trajectory generation unit 130, a traveling control unit 140, an information processing unit 150, and a storage unit 180. The acquisition unit 110, the recognition unit 120, the trajectory generation unit 130 (an example of a determination unit), the traveling control unit 140, and the information processing unit 150 are each realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in a storage unit 180 (a storage device having a non-transient storage medium) such as a HDD or a flash memory, or may be stored in a removable storage medium such as a DVD or a CD-ROM, and may be installed in the HDD or the flash memory by mounting the storage medium (non-transient storage medium) in a drive device. Some of the functional units included in the control device 100 may be included in another device. For example, the acquisition unit 110 and the recognition unit 120 (one or both of a first recognition unit 122 and a second recognition unit 124 described later) may be included in another device, and the control device 100 may acquire the recognition result of the recognition unit 120 from the other device and execute various processes based on the acquired result. Moreover, the image may be provided at a location different from the moving body M, such as a roadside, instead of the camera 10. In this case, the acquisition unit 110 acquires an image captured by a camera provided at a different location.

[0039] The acquisition unit 110 acquires an image captured by the camera 10, the image being captured of a user who is not riding on the moving object M.

[0040] The recognition unit 120 includes, for example, a first recognition unit 122 and a second recognition unit 124. The first recognition unit 122 recognizes the user's position, the user's gestures, the user's actions, the direction in which the user points with his or her hand or arm, the user's instructions, and the like. The first recognition unit 122 recognizes the user's gestures, the user's actions, the direction in which the user points with his or her hand or arm, the instructions, and the like, based on, for example, an image acquired by the acquisition unit 110 and a method such as pattern matching or deep learning. In addition, the first recognition unit 122 analyzes a voice input by a microphone and recognizes an instruction included in the voice.

[0041] The second recognition unit 124 recognizes objects and the surrounding conditions of the moving body M, for example, by using a function based on AI (Artificial Intelligence) or a function based on a pre-given model, or by using both in parallel. For example, the function of "recognizing an area in which the moving body M can enter" may be realized by performing recognition of roads, sidewalks, curbs, etc. by deep learning or the like in parallel with recognition based on a pre-given condition (a signal capable of pattern matching), and scoring and comprehensively evaluating both. In addition, the second recognition unit 124 may perform a semantic segmentation process to classify each pixel in an image frame into a class (for example, an object, a first area in which entry is possible, a second area in which entry is not possible, etc.), and recognize an area in which the moving body M can enter based on the classification result. This ensures the reliability of the movement of the moving body M.

[0042] The second recognition unit 124 recognizes the position, speed, acceleration, and other states of objects around the moving body M based on the image captured by the camera 10. The position of the object is recognized as a position on absolute coordinates with a representative point of the moving body M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of the object may be expressed by a representative point such as the center of gravity or a corner of the object, or may be expressed by a represented area. The "state" of the object may include the acceleration or jerk of the object, or the "behavior state" (for example, whether or not the object is changing lanes or is about to change lanes). The second recognition unit 124 recognizes, for example, road division lines, road shoulders, curbs, median strips, guardrails, stop lines, obstacles, signals, and other road phenomena. The second recognition unit 124 recognizes the position and attitude of the moving body M.

[0043] The trajectory generating unit 130 determines a target position to which the moving body M moves based on a user's instruction, a first area into which the moving body M can enter, and a second area into which the moving body M cannot enter. The target position is, for example, a stopping position where the moving body M stops, or a traveling position where the moving body M travels (or a traveling position to which the moving body M travels).

[0044] The trajectory generating unit 130 generates a target trajectory along which the moving body M will travel automatically (without relying on the driver's operation) in the future so as to be able to respond to the surrounding conditions of the moving body M. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a sequence of points (trajectory points) to be reached by the moving body M. The trajectory points are points to be reached by the moving body M at each predetermined travel distance (for example, about several [m]) along a road, and separately, a target speed and a target acceleration at each predetermined sampling time (for example, about a few tenths [sec]) are generated as part of the target trajectory. In addition, the trajectory points may be positions to be reached by the host vehicle M at each sampling time for each predetermined sampling time. In this case, the information on the target speed and the target acceleration is expressed as an interval between the trajectory points.

[0045] The trajectory generating unit 130 performs coordinate conversion between the Cartesian coordinate system and the fisheye camera coordinate system. A one-to-one relationship is established between the coordinates of the Cartesian coordinate system and the fisheye camera coordinate system, and the relationship is stored in the storage unit 70 as correspondence information. The trajectory generating unit 130 generates a trajectory in the Cartesian coordinate system (Cartesian coordinate system trajectory) and converts the trajectory into a trajectory in the fisheye camera coordinate system (fisheye camera coordinate system trajectory). The trajectory generating unit 130 calculates the risk of the fisheye camera coordinate system trajectory. The risk is an index value indicating the likelihood that the moving body M will approach an obstacle. The risk tends to be higher the smaller the distance between the trajectory (trajectory point of the trajectory) and the obstacle, and lower the risk tends to be higher the larger the distance between the trajectory (trajectory point) and the obstacle.

[0046] If the total risk value or the risk of each trajectory point meets a preset criterion (for example, if the total risk value is less than or equal to threshold value Th1 and the risk of each trajectory point is less than or equal to threshold value Th2), the trajectory generation unit 130 adopts the trajectory that meets the criterion as the trajectory along which the moving body will move.

[0047] If the above trajectory does not satisfy the preset criteria, the trajectory generating unit 130 detects an accessible space in the fisheye camera coordinate system, and converts the detected accessible space in the fisheye camera coordinate system into an accessible space in the Cartesian coordinate system. The accessible space is a space in the moving direction of the moving body M excluding obstacles and the area around the obstacles (areas where a risk is set or areas where the risk is equal to or greater than a threshold). The trajectory generating unit 130 corrects the trajectory so that the trajectory falls within the accessible space that has been coordinate-converted into the Cartesian coordinate system. The trajectory generating unit 130 converts the Cartesian coordinate system trajectory into a fisheye camera coordinate system trajectory, and calculates the risk of the fisheye camera coordinate system trajectory based on the surrounding image and the fisheye camera coordinate system trajectory. This process is repeated to search for a trajectory that satisfies the above preset criteria.

[0048] The traveling control unit 140 causes the moving object M to travel along a trajectory that satisfies a preset criterion. The traveling control unit 140 outputs a command value to the traveling driving force output device 200 for causing the moving object M to travel along the trajectory.

[0049] The information processing unit 150 controls various devices and equipment of the moving object M. The information processing unit 150 controls, for example, the HMI 30. The information processing unit 150 acquires voice data input to a microphone and recognizes operations performed on the operation unit.

[0050] The driving force output device 200 outputs a driving force (torque) for driving the moving body M to the driving wheels. The driving force output device 200 includes, for example, an electric motor and an ECU (Electronic Control Unit) that controls the electric motor and the motor. The ECU controls the above components according to information input from the driving control unit 140 or information input from the driving operation device 80.

[0051] Brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from driving control unit 140 or information input from driving operation device 80, so that a brake torque corresponding to the braking operation is output to each wheel.

[0052] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by applying a force to, for example, a rack and pinion mechanism. The steering ECU drives the electric motor in accordance with information input from the cruise control unit 140 or information input from the driving operator 80 to change the direction of the steered wheels.

[0053] [Overview of vehicle control] FIG. 3 is a diagram showing an example of the behavior of a moving body M. The moving body M identifies a user who wishes to receive a service, travels toward a location according to the user's speech or gesture (time t), stops at the location, and lets the user board (time t+1). After the user boards the moving body M, the moving body M travels toward the user's destination (time t+2). For example, the user calls the moving body M near a location specified in advance. For example, as shown in FIG. 4, the moving body M refers to the user's clothing characteristics, physical characteristics (for example, physical characteristics provided in advance by the user to the moving body M), and gestures to identify a user who is waiting near the specified location. Then, the moving body M stops at a location according to the identified user's gestures and behavior. The user can board the stopped moving body M and head toward the destination.

[0054] [User identification process (part 1)] FIG. 5 is a flowchart showing an example of a process flow in which the control device 100 identifies a user. First, the information processing unit 150 of the control device 100 acquires specific information obtained by the mobile object M communicating with the user (step S100). The communication may be performed by voice (user's voice) or by communication with a terminal device owned by the user. The specific information is, for example, information indicating the user's intention to use the mobile object M. For example, when the user makes an utterance indicating the intention to use the mobile object M, such as "This way," or "Give me a ride," the utterance is input to a microphone (which may be the microphone of the mobile object M, or the microphone of the user's mobile terminal or roadside device, not shown). The first recognition unit 122 analyzes the utterance input to the microphone and recognizes the user's intention based on the analysis result.

[0055] Next, the recognition unit 120 recognizes the attributes of the user based on the image captured by the camera 10 (step S102). For example, the recognition unit 120 recognizes the user's joint points based on the image (performs skeleton processing). For example, the recognition unit 120 estimates the user's face, facial parts, neck, shoulders, elbows, wrists, waist, ankles, etc. from the image, and performs skeleton processing based on the positions of each estimated part. For example, the recognition unit 120 performs skeleton processing using a known method (e.g., a method such as open pose) for estimating the user's joint points and skeleton using deep learning. The recognition unit 120 identifies the user's face, upper body, lower body, etc. based on the result of the skeleton processing, and extracts feature amounts (e.g., attributes) based on the identified face, upper body, lower body, etc. The recognition unit 120 determines the user's attributes based on the result of the skeleton processing and an algorithm or pattern matching for determining the user's attributes and features.

[0056] Furthermore, when a target object is included in the specific information, the recognition unit 120 recognizes the target object (step S104). For example, when a user utters "Pick me up at the mailbox," the recognition unit 120 recognizes the corresponding mailbox. Then, in the process of step S102, the recognition unit 120 executes skeleton processing on the user who is present near the corresponding mailbox.

[0057] Next, the recognition unit 120 recognizes and determines the attributes of the user based on the result of the skeleton processing (step S106). After this processing, the process proceeds to the processing of the flowchart in FIG.

[0058] FIG. 6 is a diagram for explaining the feature quantities (attributes) of a user and the recognition of targets. The attributes of a user include clothing attributes and physical attributes. The clothing attributes include the color of the clothes and whether or not a hat is worn. The physical attributes are gender and age. A target is an object specified by a user. The recognition unit 120 identifies a user having attributes that match the attributes stored in the storage unit 180 based on the attributes of the user who uses the moving body M stored in advance in the storage unit 180 and the attributes determined based on the image.

[0059] [User identification process (part 2)] The control device 100 recognizes the behaviors shown in Fig. 7 based on the image, and identifies the user by performing the process shown in Fig. 8, which will be described later. Fig. 7 is a diagram for explaining an example of the behaviors. The behaviors are, for example, (1)-(7). (1) To stand (or stop). (2) Walking. (3) Running. (4) Wave your hand. (5) Hold up your smartphone. (6) Engaging in indefinite or present behavior (intention to show presence). (7) The face is facing in the specified direction.

[0060] 8 is a flowchart showing an example of a process flow in which the control device 100 identifies a user. After the process of step S106 in FIG. 5, the recognition unit 120 performs a matching process (step S200). For example, the recognition unit 120 recognizes a behavior of the user based on an image captured by the camera 10, and determines whether or not the recognized behavior (including a gesture) matches a behavior stored in the storage unit 180.

[0061] If the user's behavior matches the above behaviors (2)-(5), the control device 100 judges whether the user has been identified (step S202). If the user has been identified, the processing of one routine of this flowchart ends. For example, the control device 100 identifies the user by referring to the attributes of the user's clothing and physical attributes.

[0062] If the user cannot be identified in the process of step S202, the control device 100 acquires additional identification information (e.g., voice such as "this way" or "it's here") by the moving object M communicating with the user (step S204). Next, the control device 100 judges whether the user can be identified by the additional identification information (step S206). If the user cannot be identified by the additional identification information, the process returns to step S100 in Fig. 5. If the user can be identified by the additional identification information, the process of one routine of this flowchart ends.

[0063] If the user's behavior does not match the above behaviors (1)-(5) (if there is no match), the control device 100 judges whether or not a voice for communicating the user's position has been input (step S208). For example, it is judged whether or not a voice such as "Heyyy", "Over here", or "It's here" has been input. If a voice for communicating the user's position has been input, the control device 100 performs skeleton processing to evaluate time-series changes and recognize the user's behavior (step S210).

[0064] If the recognition result in step S210 indicates that the user is performing the above behavior (6) (if the user's time-series change is large), the process proceeds to step S204. If the recognition result in step S210 indicates that the user is not performing the above behavior (6) (if the user's time-series change is small), the process proceeds to step S214 or S216. Also, if the voice for communicating the user's position is not input in the above step S208, the process proceeds to step S214 or S216.

[0065] For example, if some or all of the following conditions 1-3 are satisfied, the process proceeds to step S214 or step S216 (in the following description, the process of step S214 is assumed to be performed before the process of step S216). Condition 1 is that no voice is input to communicate the user's position in step S208. Condition 2 is that the user does not perform the above behavior (6) in step S210. Condition 3: The user's behavior must match the behavior in (1) above.

[0066] If some or all of the conditions 1-3 are satisfied, the control device 100 notifies the user by voice that the moving object M has arrived near the user (step S214). After some or all of the conditions 1-3 are satisfied and the process of step S214 is completed, the control device 100 recognizes the direction of the user's face (step S216).

[0067] After step S216, the control device 100 judges whether or not the user looks in the direction of the moving object M after transmitting the voice in step S214 (step S218). If the user looks in the direction of the moving object M, the user is identified as a user who uses the moving object M, and the processing of one routine of this flowchart ends. If the user does not look in the direction of the moving object M, the processing proceeds to step S204. After step S216, the control device 100 judges whether or not the direction of the user's face has changed in accordance with the movement of the moving object M (step S220). If the direction of the user's face has changed in accordance with the movement of the moving object M, the user is identified as a user who uses the moving object M, and the processing of one routine of this flowchart ends. If the direction of the user's face has not changed in accordance with the movement of the moving object M, the processing proceeds to step S204.

[0068] As described above, the control device 100 can communicate with users even in an environment where many people are present, and can accurately identify users who plan to use the moving object M, based on the behavior and reactions of the users.

[0069] After identifying the user, the control device 100 determines one or both of a stopping position and a traveling position based on the behavior of the identified user, and controls the moving object M based on the determination result.

[0070] FIG. 9 is a diagram (part 1) showing an example of the behavior of the moving object M. The control device 100 stops the moving object M at a position (hereinafter, a stop position) indicated by pointing in the accessible area. The control device 100 recognizes the accessible area and the non-accessible area, and generates a trajectory toward the stop position in the accessible area. The control device 100 runs the moving object M along the generated trajectory, and stops the moving object M at the stop position. The stop position is, for example, a position indicated by pointing, which is a shoulder of a road. An example of a method for identifying the stop position indicated by pointing will be described later. The accessible area (first area) is an area that the moving object M can enter. The non-accessible area (second area) is an area that the moving object M cannot enter, or an area in which it is not appropriate for the moving object M to travel. The accessible area is, for example, a roadway, and the non-accessible area is, for example, a sidewalk.

[0071] Fig. 10 is a diagram (part 2) showing an example of the behavior of the moving object M. In Fig. 10, the user is pointing, but the stop position corresponding to the instruction is farther from the user than the stop position in Fig. 9. In this case, the moving object M stops at the stop position, as in Fig. 9.

[0072] FIG. 11 is a diagram (part 3) showing an example of the behavior of the moving body M. In the above-mentioned FIG. 9 and FIG. 10, the stopping position was the first side of the first and second sides of the road. The first side is the inaccessible area side where the user is present, and the second side is the inaccessible area side opposite to the side where the user is present. In FIG. 11, the stopping position is set to the second side. In addition, the user's destination is the direction D shown in FIG. 11. In this case, the moving body M stops at the stopping position taking into consideration the direction of the destination and the direction in which the user will travel after boarding.

[0073] For example, the control device 100 acquires the destination of the user in advance, and generates a trajectory for reaching the stop position based on the acquired destination and the stop position. In the example of FIG. 11, the control device 100 causes the moving body M to pass the stop position so that the moving body M can proceed toward the destination without turning after the user boards the moving body M, and then turns the moving body M to run so that the front of the moving body M faces the destination direction, and stops the moving body M at the stop position. This allows the moving body M to head toward the destination quickly and smoothly after the user boards the moving body M. The process described with reference to FIG. 11 above is an example of a process in which "the determination unit causes the moving body to turn when the moving body stops at the stop position in the direction of the route toward the user's destination."

[0074] 12 is a diagram (part 4) showing an example of the behavior of the moving object M. If the user does not specify a stopping position, the moving object M stops in an accessible area close to the user. For example, if the control device 100 identifies the user but the user is standing upright, the control device 100 stops the moving object M in a position close to the user.

[0075] FIG. 13 is a diagram (part 5) showing an example of the behavior of the moving body M. When the user does not specify a stopping position but performs a predetermined behavior indicating approaching the user, such as beckoning or waving ("when the first recognition unit recognizes an instruction to approach the user by the user's gesture"), the moving body M approaches the user (e.g., a stopped user) and stops in an accessible area close to the user. This stopping position may be, for example, the same position as the stopping position described in FIG. 12, or may be changed according to the user's behavior, such as beckoning. For example, when the moving body M arrives at the stopping position described in FIG. 12, if the user is beckoning at that time, the moving body M may further approach the user.

[0076] FIG. 14 is a diagram (part 6) showing an example of the behavior of the moving object M. When the user issues an instruction (utterance) to stop the moving object M by specifying a specific target, the control device 100 stops the moving object M at a position corresponding to the target in the accessible area. As shown in FIG. 14, when the user issues an instruction to "stop at the mailbox", the moving object M stops in the accessible area in front of the mailbox. Note that when the moving object M stops, if the distance between the moving object M and the mailbox is equal to or less than a threshold value, the control device 100 may stop the moving object M at a position in front of the mailbox (position X offset toward the user in FIG. 14) because it is difficult for the user to board the moving object M. This process is an example of the process in which "the determination unit determines the stop position based on the boundary within the predetermined range shifted toward the user".

[0077] Fig. 15 is a diagram showing an example of the concepts and information to be recognized used in the processes described above in Fig. 9 to Fig. 14. The recognition unit 120 recognizes the user's behavior, targets, and stopping positions using the following (A) to (E). (A) The recognition unit 120 performs skeleton processing to recognize the user's arms, hands, and fingers, and recognizes the positions (vertical and horizontal positions) that the recognized arms, hands, and fingers are pointing to. (B) The recognition unit 120 performs skeleton processing to recognize the positions of the user's arms, hands, fingers, and shoulders, and corrects the direction in which the recognized arms, hands, and fingers are pointing based on the height of the shoulders. (C) The recognition unit 120 performs skeleton processing to recognize the user's arms, hands, and fingers, and recognizes a gesture (for example, a beckoning gesture) made by the recognized arms, hands, and fingers. (D) The recognition unit 120 recognizes the user and the target designated by the user, and recognizes the relative position between the moving object M and the target. (E) As described above, the recognition unit 120 executes, for example, semantic segmentation processing to recognize the accessible area and the relative positions of the target and the accessible area.

[0078] [Flowchart of the process for determining the stopping position (part 1)] Fig. 16 is a flowchart showing an example of a process flow executed when the control device 100 determines a stopping position. The stopping position in step S304 of this flowchart is the stopping position corresponding to Fig. 12, and the stopping position in step S312 is the stopping position corresponding to Fig. 9-11 described above. The stopping position in step S316 of this flowchart is the stopping position corresponding to Fig. 14, and the stopping position in step S320 is the stopping position corresponding to Fig. 13 described above. The "☆" described later is an example of a stopping position based on a boundary, and the route for stopping at the "☆" described later (the route shown in Fig. 9-14) is an example of a traveling position based on a boundary.

[0079] First, the control device 100 identifies a user (step S300) as described in the flowchart of Fig. 8. Next, the information processing unit 150 determines whether or not a demonstrative term specifying a stop position is included in the user's utterance (step S302).

[0080] If the user's utterance does not include a demonstrative term specifying a stop position, the control device 100 moves the moving object M to the vicinity of the user (step S304). If the user's utterance includes a demonstrative term specifying a stop position, the recognition unit 120 acquires reference information for determining the stop position (step S306). The reference information is the information (A)-(E) described in FIG. 15.

[0081] Next, the recognition unit 120 judges whether the user is pointing at a direction (step S308). If the user is pointing at a direction, the recognition unit 120 identifies the wide-area designated area that the user is pointing at with his / her finger (step S310). Next, the recognition unit 120 estimates the user's designated position (step S312). For example, the recognition unit 120 recognizes the boundary (designated lane boundary) between the enterable area and the non-enterable area included in the wide-area designated area. The recognition unit 120 sets a predetermined position "☆" on the boundary in the wide-area designated area, and determines a position offset by a predetermined distance from this predetermined position "☆" in the direction of the enterable area as the designated position (the position the user is pointing at with his / her finger) "★". This ends the processing of this flowchart.

[0082] The processing of step S312 above is an example of processing in which "when the first recognition unit recognizes that the user is indicating the stop position with a gesture, the determination unit determines that the stop position is a position shifted a predetermined amount from the boundary toward the first region based on the instruction."

[0083] If the user does not indicate a direction by pointing, the recognition unit 120 determines whether a target object is indicated (step S314). If it is determined that a target object is indicated, the recognition unit 120 sets a predetermined position "☆" at the boundary near the indicated target object, and determines a position obtained by offsetting this predetermined position "☆" by a predetermined distance in the direction of the accessible area as the indicated position "★" (step S316). This process is an example of a process in which "if the first recognition unit recognizes that the user has specified a target object and has instructed the moving object to stop at a position based on the target object, the determination unit determines a position shifted by a predetermined amount from the boundary within the predetermined range in the direction of the first area as the stop position based on the position of the target object and the boundary within a predetermined range from the target object."

[0084] If it is determined that there is no indication of a target object, the recognition unit 120 determines whether or not there is an indication of a beckoning (step S318). If there is an indication of a beckoning, the recognition unit 120 sets a predetermined position "☆" at the boundary near the user who is beckoning, and determines a position obtained by offsetting this predetermined position "☆" by a predetermined distance in the direction of the accessible area as the indicated position "★" (step S320). This process is an example of a process in which "if the first recognition unit recognizes an indication by a user's gesture to approach the user, the determination unit determines, based on the position of the user and the boundary within a predetermined range from the user, a position shifted by a predetermined amount from the boundary within the predetermined range in the direction of the first area as the target position."

[0085] If there is no instruction to beckon, the control device 100 inquires of the user for an instruction (step S322). The inquiry may be an inquiry by voice or an inquiry by a message via communication. This ends the processing of one routine of this flowchart.

[0086] As described above, the control device 100 recognizes voices and user behavior and communicates with the user to determine the stopping position, thereby enabling control that better reflects the user's intentions.

[0087] [Flowchart of the process for determining the stopping position (part 2)] FIG. 17 is a flowchart showing another example of the flow of processing executed when the control device 100 determines a stopping position. Points different from the processing in the flowchart of FIG. 16 will be described. In this processing, the recognition processing of one or both of (B) and (C) described in FIG. 15 is omitted. In this processing, for example, the recognition of shoulder height, the correction based on the shoulder height recognition result, and the processing of recognizing a beckoning hand are omitted. A preset height is used as the shoulder height. Even when (B) and (C) are omitted in this way, the accuracy of the determination of the stopping position is guaranteed while suppressing the calculation load.

[0088] In Fig. 17, the processes of steps S318 and S322 in Fig. 16 are omitted. If no target is specified in step S314, the process of step S320 is executed. This ends the process of one routine in this flowchart.

[0089] In addition to the above-described processing, the control device 100 can more easily recognize voice and user behavior and communicate with the user to determine the stopping position, thereby achieving control that better reflects the user's intentions.

[0090] The pointing position can be estimated using the method disclosed in JP 2009-223172 A or other known methods. For example, when estimating the pointing position, various parameters such as shoulder height, shoulder width, line of sight, eye height, etc., can be taken into account in addition to the direction in which the finger is pointing.

[0091] According to the embodiment described above, the control device 100 recognizes a first area into which the moving body M can enter and a second area into which the moving body M cannot enter based on an image, and determines a stopping position where the moving body M will stop or a target position to which the moving body M will move based on the user's instructions and the first area into which the moving body M can enter and the second area into which the moving body M cannot enter, thereby realizing control that better reflects the user's intentions.

[0092] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; A user acquires a captured image, Recognizing an instruction from the user; Recognizing a first area into which the moving object can enter and a second area into which the moving object cannot enter based on the image; determining, based on the user's instruction, a first area into which the moving object can enter, and a second area into which the moving object cannot enter, one or both of a stop position at which the moving object will stop and a target position to which the moving object will move; A control device for a moving object.

[0093] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0094] M: moving body, 2: main body, 10: camera, 100: control device, 110: acquisition unit, 120: recognition unit, 122: first recognition unit, 124: second recognition unit, 130: trajectory generation unit, 140: travel control unit, 150: information processing unit, 180: storage unit

Claims

1. An acquisition unit that acquires an image captured by a user; A first recognition unit that recognizes instruction information corresponding to a position, a movement, a voice, and an instruction of the user; a second recognition unit that recognizes a first area into which the moving object can enter and a second area into which the moving object cannot enter based on the image; a determination unit that determines a target position to which the moving object will move based on instruction information from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter, When the instruction information is a voice, the determination unit determines the target position within the first area based on a target recognized from a voice, the first area, and the second area. Vehicle control system.

2. An acquisition unit that acquires an image captured by a user; A first recognition unit that recognizes an instruction from the user; a second recognition unit that recognizes a first area into which the moving object can enter and a second area into which the moving object cannot enter based on the image; a determination unit that determines a target position to which the moving object is to move based on an instruction from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter; The determination unit determines an arbitrary position pointed to by the user within the first area as a target position. Vehicle control system.

3. An acquisition unit that acquires an image captured by a user; A first recognition unit that recognizes an instruction from the user; a second recognition unit that recognizes a first area into which the moving object can enter and a second area into which the moving object cannot enter based on the image; a determination unit that determines a target position to which the moving object is to move based on an instruction from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter; The first region and the second region are adjacent to each other, and the first region and the second region extend in a reference direction. The first recognition unit recognizes an instruction of a target position by a gesture of the user, The second recognition unit detects a boundary between the first area and the second area corresponding to a target position by a gesture of the user, the determination unit determines a position shifted a predetermined amount from the boundary in a first region direction as the target position; A control system for a moving body, wherein the first area direction is a direction that intersects the reference direction and is a direction from the second area toward the first area.

4. An acquisition unit that acquires an image captured by a user; A first recognition unit that recognizes an instruction from the user; a second recognition unit that recognizes a first area into which the moving object can enter and a second area into which the moving object cannot enter based on the image; a determination unit that determines a target position to which the moving object is to move based on an instruction from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter; The first region and the second region are adjacent to each other, and the first region and the second region extend in a reference direction. The first recognition unit detects a boundary between the first region and the second region, The second recognition unit recognizes an instruction of a target position accompanied by a designation of a target by the user, the determination unit determines, as a target position, a position obtained by shifting a first position, which is the boundary within the predetermined range corresponding to the target, by a predetermined amount in a first region direction; A control system for a moving body, wherein the first area direction is a direction that intersects the reference direction and is a direction from the second area toward the first area.

5. The second recognition unit detects a boundary between the first area and the second area, The determination unit determines the target position based on the boundary.

3. A control system for a moving object according to claim 1 or 2.

6. When the first recognition unit recognizes an instruction of the target position by a gesture of the user, The determination unit determines, as the target position, a position shifted by a predetermined amount from the boundary in a first region direction based on the instruction. The control system for a moving object according to claim 5.

7. When the first recognition unit recognizes an instruction of the target position accompanied by a designation of a target by the user, The determination unit determines, based on a position of the target and the boundary within a predetermined range from the target, a position shifted by a predetermined amount from the boundary within the predetermined range in a first region direction as the target position.

7. A control system for a moving object according to claim 5 or 6.

8. The determination unit determines the target position based on a boundary obtained by shifting the boundary within the predetermined range toward the user. The control system for a moving object according to claim 7.

9. When the first recognition unit recognizes an instruction to approach the user by a gesture of the user, The determination unit determines, based on a position of the user and the boundary within a predetermined range from the user, a position shifted by a predetermined amount from the boundary within the predetermined range in a first region direction as the target position. A control system for a moving object according to any one of claims 5 to 8.

10. the target position is a stop position, When the first recognition unit recognizes the instruction to approach the user by a gesture of the user who is stopped, The determination unit determines the position shifted by the predetermined amount as the stop position. The control system for a moving object according to claim 9.

11. the target position is a stop position, The determination unit determines the stop position based on the instruction. A control system for a moving object according to any one of claims 1 to 10.

12. the target position is a stop position, the determination unit causes the direction of the moving object when the moving object stops at the stop position to be in a route direction toward the user's destination. A control system for a moving object according to any one of claims 1 to 11.

13. the target position is a stop position, When the determination unit cannot determine the stop position, the determination unit moves the moving object to a position near the user. A control system for a moving object according to any one of claims 1 to 12.

14. the target position is a stop position, When the determination unit cannot determine the stop position, the determination unit inquires of the user regarding the stop position. A control system for a moving object according to any one of claims 1 to 12.

15. A moving body equipped with a control system described in any one of claims 1 to 14.

16. The computer A user acquires a captured image, Recognizing instruction information corresponding to the user's position, movement, voice, and instruction; Recognizing a first area into which the moving object can enter and a second area into which the moving object cannot enter based on the image; determining a target position to which the moving object will move based on instruction information from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter; When the instruction information is a voice, determining the target position within the first area based on a target recognized from a voice, the first area, and the second area; A method for controlling a moving object.

17. On the computer, A user acquires a captured image, Recognizing instruction information corresponding to a position, a movement, a voice, and an instruction of the user; a first area into which the moving object can enter and a second area into which the moving object cannot enter are recognized based on the image; determining a target position to which the moving object will move based on instruction information from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter; When the instruction information is a voice, determining the target position within the first area based on a target recognized from a voice, the first area, and the second area; program.

18. The computer A user acquires a captured image, Recognizing an instruction from the user; Recognizing a first area into which the moving object can enter and a second area into which the moving object cannot enter based on the image; determining a target position to which the moving object will move based on an instruction from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter; determining an arbitrary position pointed to by the user within the first area as a target position; A method for controlling a moving object.

19. On the computer, A user acquires a captured image, Recognizing an instruction from the user; a first area into which the moving object can enter and a second area into which the moving object cannot enter are recognized based on the image; determining a target position to which the moving object will move based on an instruction from the user, a first area into which the moving object can enter, and a second area into which the moving object cannot enter; determining an arbitrary position pointed to by the user within the first area as a target position; program.

Images