Information processing device, information processing system, method, and program

Abstract

A device and method are provided which, when a tracking target tracked by a mobile device has been lost, enable efficiently rediscovering and restarting tracking of the tracked object. This device has a tracking target position estimation unit, and a tracking target search unit which searches for the tracking target. The tracking target search unit calculates one or more search paths that connect the present position of the mobile device and a position where the tracking target could be present, generates a search path list of the calculated one or more search paths in order of closeness to the tracking target estimated position, selects search paths in order from the top of the generated search path list, and searches for the tracking target by moving the mobile device in accordance with the selected search paths. It is determined whether or not the tracking target has been detected on the basis of images captured by a camera mounted on the mobile device, and if the tracking target has been detected, tracking processing is restarted.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, a method, and a program. More particularly, the present disclosure relates to an information processing device, an information processing system, a method, and a program for generating a movement path when a mobile device such as a robot follows a person, a vehicle, etc. [Background technology]

[0002] In recent years, the use of autonomous mobile objects such as robots, self-driving vehicles, and drones has increased. Among such autonomous moving bodies, there are some that are configured to set another moving body or person ahead as a "follow-up target" and move by following the follow-up target.

[0003] Typically, when a mobile device such as a robot tracks a target, it places the target within the field of view of the robot's camera or the like, confirms the target, sets a destination position at or just before the position of the target, generates a route to the set destination position, and then moves.

[0004] However, if there is a fork in the road in which the target to be tracked branches off into multiple routes and the target to be tracked suddenly moves from the fork onto a side road, the target to be tracked will move out of the robot's field of view, the robot will no longer be able to recognize the target to be tracked, and tracking processing will become impossible.

[0005] When the target to be tracked moves out of the viewing angle, it is common to estimate the position of the target to be tracked and attempt to track it based on the estimation result.

[0006] Another method is to not perform estimation processing, but to assume that the target exists at the position observed in the past, and move to a position where the target fits within the field of view. However, for example, when the target object moves out of the field of view at a crossroads and it is unclear which route the target object has taken at the crossroads, it becomes difficult to apply this method.

[0007] Incidentally, as a conventional technique that discloses a vehicle travel control process for traveling following a leading vehicle, for example, Patent Document 1 (JP 2010-203886 A) is known.

[0008] Patent Document 1 (JP 2010-203886 A) discloses a configuration for controlling the positional relationship between a leading vehicle and a following vehicle by referring to map data or the like so that the leading vehicle is always within the field of view of the following vehicle.

[0009] However, with the configuration described in Patent Document 1, for example, when a leading vehicle makes a right or left turn at an intersection, the following vehicle must wait until it approaches the leading vehicle and then make the right or left turn, which reduces processing efficiency. Furthermore, the configuration described in Patent Document 1 is based on the premise that the leading vehicle and the following vehicle are in a cooperative relationship, and there is a problem in that the configuration described in Patent Document 1 cannot be used, for example, when the leading vehicle to be followed is trying to escape from the following vehicle. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-203886 Summary of the Invention [Problem to be solved by the invention]

[0011] The present disclosure has been made in view of the above-mentioned problems, and specifically aims to provide an information processing device, an information processing system, a method, and a program that, in a configuration in which a robot follows a target while checking the target with a sensor such as a camera, enable the robot to efficiently rediscover the lost target and continue tracking it when the target goes out of the robot's field of view at, for example, an intersection. [Means for solving the problem]

[0012] A first aspect of the present disclosure provides: a tracking target position estimation unit that calculates an estimated position of a tracking target that is followed by the mobile device; a tracking target search unit that searches for the tracking target, The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; The information processing device selects a search route from the top of the generated search route list in order, and moves the mobile device along the selected search route to search for the target to be followed.

[0013] Furthermore, a second aspect of the present disclosure is An information processing system including a mobile device and an information processing device capable of communicating with the mobile device, The information processing device includes: a tracking target position estimation unit that calculates an estimated position of a tracking target that is followed by the mobile device; a tracking target search unit that searches for the tracking target, The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; Select a search route from the top of the generated search route list, The moving device is The information processing system is configured such that the information processing device moves along a selected search path to search for the target to be tracked.

[0014] Furthermore, a third aspect of the present disclosure is An information processing method executed in an information processing device, a tracking target position estimating step in which a tracking target position estimating unit calculates an estimated position of the tracking target that is to be tracked by the mobile device; The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; The information processing method includes a step of selecting a search route in order from the top of the generated search route list, and moving the mobile device along the selected search route to search for the target to be followed.

[0015] Furthermore, a fourth aspect of the present disclosure is An information processing method executed in an information processing system having a mobile device and an information processing device capable of communicating with the mobile device, The information processing device, a tracking target position estimation process for calculating an estimated position of a tracking target that is an estimated position of a tracking target that is followed by the mobile device; calculating one or more search routes connecting the current position of the mobile device and a possible position of the target to be tracked, generating a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the target to be tracked, and executing a process of selecting search routes in descending order from the top of the generated search route list; The moving device, The information processing method is such that the information processing device moves along a selected search path to search for the target to be followed.

[0016] Furthermore, a fifth aspect of the present disclosure is A program for causing an information processing device to execute information processing, a tracking target position estimating step of causing a tracking target position estimating unit to calculate an estimated position of a tracking target that is an estimated position of a tracking target that is followed by the mobile device; The target search section calculating one or more search routes connecting the current position of the mobile device and possible positions of the target to be followed; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; The program selects a search route from the top of the generated search route list in order, and moves the mobile device along the selected search route to search for the target to be followed.

[0017] The program of the present disclosure is a program that can be provided, for example, via a storage medium or communication medium in a computer-readable format to an information processing device or computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or computer system.

[0018] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0019] According to the configuration of one embodiment of the present disclosure, an apparatus and method are realized that, when a mobile device loses a target to be followed, can efficiently rediscover the target and resume tracking. Specifically, for example, the device has a tracking target position estimation unit and a tracking target search unit that searches for the tracking target. The tracking target search unit calculates one or more search routes connecting the current position of the mobile device and possible positions of the tracking target, generates a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the tracking target, selects a search route from the top of the generated search route list, and moves the mobile device along the selected search route to search for the tracking target. It determines whether the tracking target has been detected based on images captured by a camera attached to the mobile device, and if detected, resumes the tracking process. This configuration realizes an apparatus and method that, when a mobile device loses a target to be followed, can efficiently rediscover the target and resume tracking. The effects described in this specification are merely examples and are not limiting, and additional effects may also be present. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating an overview of processing executed by a mobile device according to the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an overview of processing executed by a mobile device according to the present disclosure. [Figure 3] FIG. 1 is a diagram illustrating an overview of processing executed by a mobile device according to the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating an overview of processing executed by a mobile device according to the present disclosure. [Figure 5] FIG. 1 is a diagram illustrating an overview of processing executed by a mobile device according to the present disclosure. [Figure 6] FIG. 1 is a diagram illustrating an overview of processing executed by a mobile device according to the present disclosure. [Figure 7] 1 is a block diagram showing an example of the configuration of the main components of a moving device (robot) according to the present disclosure. [Figure 8] 10A to 10C are diagrams illustrating states according to whether or not a target to be followed by a mobile device (robot) of the present disclosure can be confirmed. [Figure 9] 3A and 3B are diagrams illustrating the configuration and processing of a tracking target position estimation unit. [Figure 10] 3A and 3B are diagrams illustrating the configuration and processing of a tracking target position estimation unit. [Figure 11] 10A and 10B are diagrams illustrating a specific example of processing executed by a tracking target position estimation unit. [Figure 12] 10A and 10B are diagrams illustrating the configuration and processing of a tracking target search unit. [Figure 13] 10A and 10B are diagrams illustrating a specific example of processing executed by a follow-up target search unit. [Figure 14] 10A and 10B are diagrams illustrating a specific example of processing executed by a follow-up target search unit. [Figure 15] 10A and 10B are diagrams illustrating a specific example of processing executed by a follow-up target search unit. [Figure 16] 10A and 10B are diagrams illustrating a specific example of processing executed by a follow-up target search unit. [Figure 17]FIG. 10 is a diagram showing a flowchart illustrating a sequence of processing executed by a mobile device of the present disclosure. [Figure 18] 10A to 10C are diagrams illustrating specific examples of processing according to various branch configurations. [Figure 19] 10A to 10C are diagrams illustrating specific examples of processing according to various branch configurations. [Figure 20] FIG. 10 is a diagram illustrating an example of the configuration of a mobile device when the mobile device alone generates a search path and controls a robot. [Figure 21] 10 is a diagram illustrating an example of the configuration of a mobile device and a user terminal that can communicate with the mobile device when generating search routes and controlling a robot. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0021] The information processing device, information processing system, method, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items. 1. Overview of the process of searching for a target to be followed by a mobile device (robot) according to the present disclosure 2. Configuration Examples of the Mobile Device (Robot) of the Present Disclosure 3. Sequence of processes executed by the mobile device (robot) of the present disclosure 4. Processing examples for various branch configurations 5. Configuration Examples of the Mobile Device and Information Processing Device of the Present Disclosure 6. Summary of the Disclosure

[0022] [1. Overview of the process of searching for a target to be followed by a mobile device (robot) according to the present disclosure] First, an overview of the process of searching for a target to be followed by a mobile device (robot) according to the present disclosure will be described.

[0023] In the following examples, an autonomous robot will be used as an example of a mobile device that follows a target, but the mobile device disclosed herein includes not only such autonomous robots, but also various other mobile devices such as self-driving vehicles and drones.

[0024] As described above, when a mobile device such as a robot tracks a target, it places the target within the field of view of the robot's camera or the like, confirms the target, sets a destination position at or just before the position of the target, generates a route to the set destination position, and performs the process of moving.

[0025] However, if there is a fork in the road in which the target to be tracked branches off into multiple routes and the target to be tracked suddenly moves from the fork onto a side road, the target to be tracked will move out of the robot's field of view, the robot will no longer be able to recognize the target to be tracked, and tracking processing will become impossible.

[0026] In such a case, the present disclosure aims to efficiently find the target to be tracked and realize reliable tracking processing.

[0027] An overview of the process of searching for a target to be followed by a robot according to the present disclosure will be described with reference to FIG. FIG. 1 shows an example in which a robot 10 moves while following a target 20 to be followed. The robot 10 performs the tracking process by placing the target 20 to be tracked within the field of view of a camera 10p attached to the robot 10, and moving while checking the target 20 to be tracked.

[0028] However, when there is a fork in the road where routes L and R branch off as shown in Fig. 1, if the target 20 to be tracked moves along the target movement route 20q indicated by the dotted line in the figure, the target 20 to be tracked will move out of the field of view of the camera 10p of the robot 10, and the robot 10 will lose sight of the target. In other words, the robot 10 will not be able to bring the target 20 into the camera field of view 10p, and will lose sight of the target. In such a case, the robot 10 will not be able to follow the target 20.

[0029] In such a case, the robot 10 of the present disclosure can efficiently find the target to be followed and resume the following process. An overview of the processing executed by the robot 10 of the present disclosure will be described with reference to FIG. 2 and subsequent figures. 2 to 5 are diagrams illustrating the processes (steps S01 to S04) that are sequentially executed when the robot 10 of the present disclosure loses the target 20 to be followed. These processing steps will be explained in turn.

[0030] (Step S01) First, when the robot 10 of the present disclosure loses the target 20 to be followed, in step S01, as shown in FIG. 2, it calculates an estimated position of the target to be followed. The robot 10 performs an analysis of the captured image of the target 20 to be followed and estimates the position where the lost target 20 is most likely to be currently located.

[0031] For example, the estimated position 20b of the target to be followed is calculated as shown in Fig. 2. Note that Fig. 2 also shows the target to be followed (actual position) 20.

[0032] (Step S02) Next, in step S02, the robot 10 of the present disclosure analyzes a search path for searching for the target 20 to be followed, as shown in FIG. The search path is all the possible paths that the target 20 to be followed may take from the current position (A) of the robot 10.

[0033] The robot 10 can acquire map data from an internal storage unit or from an external source, and performs route search by referring to the acquired map data. In the example shown in FIG. 3, the robot 10 determines that the target 20 is moving along a route R or a route L based on the acquired map data. As a result, the search paths become two paths, ABC and ABD.

[0034] The search path is represented by node connection links, with the current position (A) of the robot 10 as the starting node, the branch point node (B) set at the branch point as the intermediate node, and the point on the path after passing the branch point (C or D) as the end node. In the example shown in FIG. 3, the robot 10 acquires the following two search paths (node ​​connecting links). ABC ABD

[0035] (Step S03) Next, in step S03, the robot 10 of the present disclosure generates a search route list for each of the multiple search routes (node ​​connection links) acquired in step S02, sorting the routes in order of proximity to the estimated position of the target to be tracked predicted in step S01, as shown in FIG. 4.

[0036] In the example shown in FIG. 4, of the searched route (ABC) and the searched route (ABD), the route closest to the estimated position 20b of the tracking target is the searched route (ABC), so the following list is generated. (Search Route 1) ABC (Search Route 2) ABD

[0037] (Step S04) Next, in step S04, as shown in FIG. 5, the robot 10 of the present disclosure acquires search routes in the order listed in the search route list generated in step S03, and moves along the acquired search routes to search for the target 20 to be followed.

[0038] In the example described above with reference to FIG. 4, the search path list lists the following two search paths in order: (Search Route 1) ABC (Search Route 2) ABD

[0039] First, the robot 10 selects the first search route in the list, (search route 1) ABC, and searches for the target 20 to be followed by moving along this (search route 1) ABC. The robot 10 first moves from node A to node B, then turns toward node C and analyzes the image captured by the camera in the direction of node C. The robot 10 can rediscover the target 20 to be followed by analyzing the image captured by the camera in the direction of node C. After the robot 10 finds the target 20 to be followed, the robot 10 resumes the following process.

[0040] For example, if the target 20 is not found in the search process for the target 20 according to (search route 1) ABC, the next search route recorded in the search route list generated in step S03 is acquired, and the target 20 is searched for by moving along the acquired search route. An example of this process will be described as the process shown in FIG. 6 (step S05).

[0041] (Step S05) The process of step S05 shown in FIG. 6 is executed when the target 20 is not found in the process of searching for the target 20 according to (search path 1) ABC described with reference to FIG. For example, as shown in FIG. 6, when the target 20 to be followed is moving in a direction (path L direction) that is significantly different from the estimated position 20b of the target to be followed, the process of step S05 is executed.

[0042] In this case, the robot 10 first selects the second search path in the list, (search path 2) ABD, and searches for the target 20 to be followed by moving along this (search path 2) ABD. At this point, the robot 10 has already moved from node A to node B and has turned toward node C.

[0043] In order to perform a search process according to (search path 2) ABD, the robot 10 performs a process of changing the orientation of the robot 10 toward node D. That is, the shooting direction of the camera is set toward node D. The robot 10 sets the camera's shooting direction toward node D and analyzes the captured image. The robot 10 can rediscover the target 20 to be followed by analyzing the image captured by the camera in the direction of node D. After the robot 10 finds the target 20 to be followed, the robot 10 resumes the following process.

[0044] In this way, when the robot 10 of the present disclosure loses sight of the target 20 to be followed, it sequentially executes the following processes to rediscover the target and continue the following process. (Step S01) = Calculate the estimated position of the target to be tracked (Step S02) = Obtain the search path of the target to be followed (Step S03) = A search route list is generated in which search routes for the target are arranged in order of proximity to the estimated position of the target. (Step S04) = Obtain search routes in the order listed in the generated search route list, and search for the target to be tracked according to the obtained search routes.

[0045] By performing the process according to these steps, it is possible to efficiently rediscover the target that has been lost, and continue the tracking process.

[0046] 2. Configuration Examples of the Mobile Device (Robot) Disclosed Herein Next, a configuration example of a moving device (robot) according to the present disclosure will be described.

[0047] As mentioned above, in this embodiment, an autonomous robot is used as an example of a mobile device that follows a target, but the mobile device disclosed herein includes not only such autonomous robots, but also various other mobile devices such as self-driving vehicles and drones.

[0048] For example, in the case of a drone, when it is set to follow a person moving through a hallway in a building or other indoor space, it may be difficult to determine which hallway the person being followed will take when they come to a fork in the hallway. The processing of the present disclosure can also be used to control the drone's flight path in such cases.

[0049] FIG. 7 is a block diagram showing an example of the main components of the mobile device (robot) 100 of the present disclosure. The moving device (robot) 100 shown in FIG. 7 is a block diagram showing the main configuration of the robot 10 shown in FIG.

[0050] As shown in Figure 7, the mobile device (robot) 100 has a camera 101, a distance sensor 102, a self-position estimation unit 103, a tracking target position estimation unit 104, a tracking target search unit 105, a map data storage unit 106, a robot driving information generation unit 107, and a robot driving unit 108.

[0051] The camera 101 is a camera that captures images in the direction of travel of the robot 100 and images of a target to be followed, and is a camera that captures, for example, RGB color images. The camera 101 generates and outputs image information 51, which is a captured image.

[0052] The distance sensor 102 is composed of, for example, an object detection sensor such as LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), or one of various sensors such as a stereo camera, a ToF sensor, an ultrasonic sensor, radar, sonar, or a combination of these, and generates and outputs distance information 52 including the distance to an object in the direction of travel of the robot 100, such as a target to be followed or an obstacle.

[0053] Image information 51 , which is an image captured by camera 102 , is input to self-position estimation unit 103 , target position estimation unit 104 , target search unit 105 , and robot drive information generation unit 107 . Similarly, distance information 52 generated by distance sensor 102, for example, distance information 52 including the distance to an object such as a target to be followed or an obstacle, is also input to self-position estimation unit 103, target to be followed position estimation unit 104, target to be followed search unit 105, and robot drive information generation unit 107.

[0054] The self-position estimation unit 103 calculates the current position and posture of the robot 100 . For example, the current position and orientation of the mobile device (robot) 100 is calculated by performing SLAM (Simultaneous Localization and Mapping) processing or the like, which is executed as an analysis process of images continuously captured by the camera 102 .

[0055] SLAM (Simultaneous Localization and Mapping) processing is a process that performs self-position identification (localization) and environmental map creation (mapping) in parallel.

[0056] The self-position information 53 generated by the self-position estimation unit 103 is output to the follow-up target search unit 105 .

[0057] The target position estimation unit 104 executes a process of estimating the position of the target that the robot 100 is to follow. The tracking target position estimation unit 104 receives sensor detection information from the camera 101, distance sensor 102, etc., and performs processing to estimate the position of the tracking target based on the received sensor detection information. The tracking target position estimation unit 104 executes tracking target position estimation processing according to three states depending on whether the tracking target can be confirmed based on sensor detection information from the camera 101, distance sensor 102, etc.

[0058] These three states will be explained with reference to FIG. As shown in Figure 8, the following three states are defined: (State 1) Tracking target can be confirmed (State 2) The state immediately before the target is lost (State 3) Target lost state

[0059] (State 1) A state in which the target to be followed can be confirmed is a state in which the target to be followed can be confirmed continuously based on information detected by a sensor such as a camera. (State 2) The state immediately before the target to be followed is a state in which the target to be followed cannot be confirmed for a specified time (t1) or longer based on information detected by a sensor such as a camera. (State 3) The lost state of the tracking target is a state in which the state immediately before the loss has continued for a specified time (t2) or more.

[0060] In the "(State 1) state where the target to be followed can be confirmed," the target to be followed position estimation unit 104 continuously analyzes the position of the target to be followed based on sensor detection information from a camera or the like, and outputs target to be followed position information 55 to the robot driving information generation unit 107.

[0061] On the other hand, in the "(State 2) State immediately before the target to be followed is lost" and "(State 3) State immediately before the target to be followed is lost", an estimation process of the current position of the target to be followed is performed using information detected by sensors such as cameras in the immediately preceding "(State 1) State where the target to be followed can be confirmed" and "(State 2) State immediately before the target to be followed is lost", and estimated position information 55 of the target to be followed obtained as a result of the estimation process is output to the robot driving information generation unit 107.

[0062] This position estimation process involves, for example, processing that utilizes object identification processing of the object to be tracked, processing that estimates the spatial position of the object, etc. For example, if the object to be tracked is a person, the position of the object to be tracked is estimated by performing the following processing.

[0063] (a) Object identification processing (person identification processing) = Image analysis processing that identifies people from camera images and sets the human area as the region of interest (ROI). (b) Spatial position estimation process for the object to be tracked: The human region of interest (ROI) in the image captured by the camera and the distance information for this human region are combined (fused) to estimate the 3D spatial position of the current object to be tracked (person).

[0064] In the process of estimating the 3D position of the object to be tracked, a motion model appropriate for the type of object to be tracked is used. For example, if the object to be tracked is a person, a motion model for a person is applied to estimate the position. For example, if the object to be tracked is a car, a motion model for a car is applied to estimate the position.

[0065] Details of the tracking target position estimation process executed by the tracking target position estimation unit 104 in "(State 2) State immediately before tracking target is lost" and "(State 3) State when tracking target is lost" will be described with reference to FIGS. 9 and 10.

[0066] FIG. 9 is a diagram showing an example of a detailed configuration of the target position estimation unit 104. As shown in FIG. As shown in FIG. 9, the tracking target position estimation unit 104 includes a tracking target region detection unit 121 and a tracking target three-dimensional position estimation unit 122. When the target to be tracked is a person, the target region detection unit 121 becomes a person region detection unit, and the target three-dimensional position estimation unit 122 becomes a person three-dimensional position estimation unit.

[0067] The tracking target region detection unit 121 inputs image information 51 captured by the camera 101 and extracts the region of the tracking target, for example, a person, as a region of interest (ROI). The input image information 51 is an image captured in "(State 2) State immediately before the tracking target is lost" or "(State 1) State where the tracking target can be confirmed" before "(State 2) State immediately before the tracking target is lost", and is an image captured continuously for a certain period of time.

[0068] The target region detection unit 121 extracts the target from the image information 51, for example, a human region, and outputs target region information (ROI information) 61 as a region of interest (ROI). This region of interest information (ROI information) 61 is information extracted from images continuously captured over a certain period of time, that is, data that allows the movement history of the object to be analyzed.

[0069] The three-dimensional position estimation unit 122 receives the following information: (a) Region-of-interest information (ROI information) 61 generated by the region-of-interest detection unit 121 (b) Distance information 52 generated by the distance sensor 102 The distance information 52 generated by the distance sensor 102 is distance information, such as a distance image (depth data), corresponding to successively captured images acquired at the same timing as the image capture timing applied to the process of generating the region-to-be-tracked information (ROI information) 61 in the region-to-be-tracked detection unit 121.

[0070] The three-dimensional position estimation unit 122 for the target to be tracked receives the above information (a) and (b) as input, and estimates the current three-dimensional position of the target to be tracked based on this input information. The detailed configuration of the three-dimensional position estimation unit 122 for the target object will be described with reference to FIG. As shown in FIG. 10, the three-dimensional position estimation unit 122 for the target to be tracked includes an in-image three-dimensional position calculation unit 131 for the target to be tracked and a motion model application three-dimensional position estimation unit 132 for the target to be tracked.

[0071] The in-image three-dimensional position calculation unit 131 receives the following information: (a) Region-of-interest information (ROI information) 61 generated by the region-of-interest detection unit 121 (b) Distance information 52 generated by the distance sensor 102 The distance information 52 generated by the distance sensor 102 is distance information, such as a distance image (depth data), corresponding to multiple consecutively captured images acquired at the same timing as the capture timing of the consecutively captured images applied to the process of generating the region-to-be-tracked information (ROI information) 61 in the region-to-be-tracked detection unit 121.

[0072] The in-image three-dimensional position calculation unit 131 of the target to be tracked receives the above information (a) and (b) and calculates, based on this input information, the three-dimensional position of the object to be tracked, for example, a person, within the region of region of interest information (ROI information) 61 selected from each of the continuously captured images generated by the target region detection unit 121. This calculated data becomes information indicating a change in the three-dimensional position of the target to be tracked within the continuously captured images used for analysis, i.e., the movement pattern of the target to be tracked. This information is "in-image target to be tracked (person) movement position information 62" shown in FIG. 10.

[0073] The in-image three-dimensional position calculation unit 131 outputs the generated “in-image moving position information 62 of the target (person) to be tracked” to the motion model application three-dimensional position estimation unit 132 of the target to be tracked.

[0074] The motion model application target 3D position estimation unit 132 estimates the 3D spatial position of the current target object (person) to be tracked using the "target object (person) movement position information 62 in the image" generated by the target object 3D position calculation unit 131 in the image.

[0075] In the process of estimating the 3D position of the object to be tracked, a motion model appropriate for the type of object to be tracked is used. For example, if the object to be tracked is a person, a motion model for a person is applied to estimate the position. For example, if the object to be tracked is a car, a motion model for a car is applied to estimate the position.

[0076] The three-dimensional spatial position information of the current object to be tracked (person) estimated by the motion model applying three-dimensional position estimation unit 132 is "estimated position information 55 of object to be tracked" shown in FIG. This position information corresponds to, for example, three-dimensional position information indicating the position of the "estimated position 20b of the target to be followed" described with reference to FIG.

[0077] A specific example of the processing executed by the tracking target position estimation unit 104 will be described with reference to FIG. FIG. 11 shows estimated position information 55 of the target to be tracked estimated by the target position estimation unit 104. In FIG. This estimated position information 55 of the target to be tracked is generated by a synthesis process of in-image moving position information 62 of the target to be tracked shown in FIG. 11 and an estimated moving path 63 based on a motion model.

[0078] The in-image tracking target movement position information 62 shown in FIG. 11 is the “in-image tracking target (person) movement position information 62” generated by the in-image tracking target three-dimensional position calculation unit 131 shown in FIG. 10, and corresponds to the movement trajectory of the three-dimensional position information of the region of interest (ROI) of the tracking target (person) extracted from the image of the tracking target captured by the camera 101.

[0079] The estimated movement path 63 based on the motion model shown in FIG. 11 is the estimated movement path of the target to be tracked estimated by the motion model applying three-dimensional position estimation unit 132 of the target to be tracked by applying the "in-image target (person) movement position information 62" and the human motion model, i.e., the estimated movement path up to the current time. The end point of this estimated movement route is the estimated position of the target to be followed at the current time, that is, estimated position information 55 of the target to be followed.

[0080] The “estimated position information 55 of the target to be tracked” generated by the motion model applying three-dimensional position estimation unit 132 of the target to be tracked is output to the target to be tracked search unit 105 .

[0081] Next, the follow-up target searching unit 105 shown in FIG. 7 will be described. The target search unit 105 executes a process of searching for the target that has been lost. That is, the tracking target search process described above with reference to FIGS. 3 to 6 is executed.

[0082] First, as shown in FIG. 3, a search path for searching for the target 20 to be followed is calculated. As explained with reference to Fig. 3, the search route is all the routes that the target 20 to be followed may possibly travel from the current position (A) of the robot 10. Based on the map data, all search routes that the target 20 to be followed may possibly travel are calculated. In the example of Figure 3, the search routes are two routes, ABC and ABD.

[0083] Next, as shown in FIG. 4, a search route list is generated in which the calculated multiple search routes (node-connecting links) are arranged in descending order of proximity to the estimated position of the tracking target. In the example shown in FIG. 4, of the searched route (ABC) and the searched route (ABD), the route closest to the estimated position 20b of the tracking target is the searched route (ABC), so the following list is generated. (Search Route 1) ABC (Search Route 2) ABD

[0084] Next, as described with reference to FIGS. 5 and 6, search routes are acquired in the order listed in the generated search route list, and the robot moves along the acquired search routes to search for target 20.

[0085] As shown in FIG. 7, when the target to be tracked is rediscovered as a result of the search process for the target to be tracked, the target to be tracked searching unit 105 outputs target to be tracked rediscovery information 57 to the target to be tracked position estimating unit 104. The tracking target rediscovery information 57 is information indicating that the camera 101 of the robot 100 can now capture an image of the tracking target.

[0086] The target position estimation unit 104 generates target (estimated) position information 55 in response to the input of target rediscovery information 57 and outputs it to the robot driving information generation unit 107 . The robot driving information generating unit 107 restarts the process of tracking the target to be tracked according to the (estimated) position information 55 of the target to be tracked input from the target position estimating unit 104, that is, the position information of the target to be tracked specified by the camera 101.

[0087] The detailed configuration of the follow-up target search unit 105 will be described with reference to FIG. As shown in FIG. 12, the follow-up target searching unit 105 has a searched route analyzing unit 151, a searched route determining unit 152, and a follow-up target detection confirming unit 153. When the target to be followed is a person, the target to be followed detection confirmation unit 153 functions as a person detection confirmation unit.

[0088] The search path analysis unit 151 generates a robot existence path 71, which is a path including the current position of the robot at the time the target to be followed is lost, and a possible target to be followed path 72, which is a path on which the target to be followed may exist. That is, one or more search paths connecting the current position of the robot and the possible positions of the target to be followed are calculated. The route information is acquired by referring to the map information 54 acquired from the map data storage unit 106.

[0089] The searched route analysis unit 151 receives the following data: (a) Estimated position information 55 of the target to be tracked generated by the target position estimation unit 104; (b) self-location information 53 generated by the self-location estimation unit 103; (c) map information 54 stored in the map data storage unit 106; The search path analysis unit 151 uses these data to generate a robot existence path 71 and a possible path 72 for the object to be followed.

[0090] A specific example of the processing executed by searched path analysis unit 151 will be described with reference to FIG. The search path analysis unit 151 analyzes a search path for searching for the target 20 to be followed, as shown in FIG. The searched route is all the routes that the target 20 to be followed may possibly travel from the current position (A) of the robot 10. The searched route analysis unit 151 acquires the map information 54 from the map data storage unit 106, and performs a route search by referring to the acquired map information.

[0091] In the example shown in FIG. 13, the searched route analysis unit 151 estimates that the target 20 is moving along route R or route L based on the map information 54 acquired from the map data storage unit 106. As a result, the searched path analysis unit 151 generates the following robot existence path 71 and follow-up target possible path 72. Robot existence path = AB Possible path to follow = BC, BD The search path analysis unit 151 outputs the robot existence path 71 and the possible path 72 for the target to be followed to the search path determination unit 152 .

[0092] The search route determination unit 152 uses the robot existence route 71 input from the search route analysis unit 151 and the possible route 72 where the target to be followed can exist to generate a search route list 73 in which routes that should be searched for with priority are set at the top.

[0093] The search route list 73 is a list in which routes that should be searched for with priority are placed at the top, and is generated as a search route list in which routes are arranged in order of proximity to the estimated position information 55 of the target to be tracked generated by the target to be tracked position estimation unit 104.

[0094] First, the search path determination unit 152 generates one or more search paths based on the robot existence path 71 input from the search path analysis unit 151 and the possible path 72 for the target to be followed. In the specific example shown in Fig. 13, the search path determination unit 152 generates the following two search paths (node ​​connecting links): ABC ABD

[0095] As explained above with reference to Figure 3, the search route is represented by node connection links, with the current position (A) of the robot 10 as the starting node, the branch point node (B) set at the branch point as the intermediate node, and the point on the route after passing the branch point (C or D) as the end node.

[0096] The searched route determination unit 152 generates a searched route list 73 in which the two searched routes (node ​​connecting links) are arranged in descending order of proximity to the estimated position information 55 of the target to be tracked generated by the target to be tracked position estimation unit 104. Specifically, for example, a search route list 73 shown in FIG. 14 is generated. In the example shown in FIG. 14, of the searched route (ABC) and the searched route (ABD), the route closest to the estimated position 20b of the target to be tracked is the searched route (ABC), so the following searched route list 73 is generated. (Search Route 1) ABC (Search Route 2) ABD

[0097] The searched route determination unit 152 acquires searched routes from the top of the generated searched route list 73 in order, and outputs the acquired searched routes to the robot driving information generation unit 107 as searched route information 56 .

[0098] The robot driving information generation unit 107 generates robot driving information 58 for moving the robot according to the search path information 56 input from the search path determination unit 152 as shown in FIG. 7, and outputs it to the robot driving unit 108. The robot driving unit 108 moves the robot 10 in accordance with the robot driving information 58 input from the robot driving information generating unit 107 to search for a target to follow.

[0099] A specific example of the search process will be described with reference to FIG. The search path list 73 lists the following two search paths in order: (Search Route 1) ABC (Search Route 2) ABD The searched route determination unit 152 selects the first searched route in the list, (searched route 1) ABC, and outputs this (searched route 1) ABC to the robot driving information generation unit 107 as searched route information 56.

[0100] The robot driving information generation unit 107 generates robot driving information 58 for moving the robot according to the search route information 56 "(search route 1) ABC" input from the search route determination unit 152, and outputs it to the robot driving unit 108. The robot driving unit 108 moves the robot 10 along "(search path 1) ABC" in accordance with the robot driving information 58 input from the robot driving information generating unit 107, and performs processing to search for a target to follow.

[0101] A specific example of the search process of the robot 10 according to "(search route 1) ABC" is shown in FIG. The robot 10 first moves from node A to node B, then turns toward node C and analyzes the image captured by the camera in the direction of node C. The robot 10 can rediscover the target 20 to be followed by analyzing the image captured by the camera in the direction of node C.

[0102] The target detection confirmation unit 153 of the target search unit 105 shown in FIG. 12 receives the image information from the camera 101 and the distance information 52 from the distance sensor 102, and confirms whether or not the target has been detected. If a target is detected, the target detection information 56 shown in FIG. In this case, the target position estimation unit 104 analyzes the position of the target in the image captured by the camera 101, for example, generates target position information 55 shown in FIG. After that, the normal tracking process for tracking the target captured in the image captured by the camera 101 is resumed.

[0103] On the other hand, if the tracking target detection confirmation unit 153 determines that the tracking target has not been detected even after inputting and analyzing the image information from the camera 101 and the distance information 52 from the distance sensor 102, it outputs tracking target detection failure information 74 shown in Figure 12 to the search path determination unit 152. In this case, the searched route determination unit 152 acquires the next searched route recorded in the searched route list 73, moves along the acquired searched route, and searches for the follow-up target. An example of this process will be described with reference to FIG.

[0104] The process shown in FIG. 16 is executed when the target 20 is not found in the process of searching for the target 20 according to (search path 1) ABC described with reference to FIG. 15, for example. For example, as shown in FIG. 16, if the target 20 to be followed is moving in a direction (toward route L) that is significantly different from the estimated position 20b of the target to be followed, the target 20 to be followed cannot be found in the search process for the target 20 to be followed according to (search route 1) ABC.

[0105] In this case, the searched route determination unit 152 selects the next searched route recorded in the searched route list 73, (searched route 2) ABD, and outputs this (searched route 2) ABD to the robot driving information generation unit 107 as searched route information 56.

[0106] The robot driving information generation unit 107 generates robot driving information 58 for moving the robot according to the search route information 56 "(search route 2) ABD" input from the search route determination unit 152, and outputs it to the robot driving unit 108. At this point, the robot 10 has already moved from node A to node B and has turned toward node C.

[0107] The robot driving information generation unit 107 generates robot driving information 58 for changing the orientation of the robot 10 toward node D to perform a search process according to (search path 2) ABD, and outputs it to the robot driving unit 108. The robot driver 108 sets the orientation of the robot 10 in the direction of node D.

[0108] As a result, camera 101 succeeds in capturing an image of target 20 to be tracked on the path L side. Thereafter, target detection confirmation unit 153 of target searching unit 105 shown in Fig. 12 outputs target detection information 56 to be tracked shown in Fig. 12 to target position estimation unit 104. Target position estimation unit 104 analyzes the position of the target to be tracked in the image captured by camera 101, generates target position information 55 to be tracked shown in Fig. 7, and outputs it to robot driving information generation unit 107. After that, the normal tracking process for tracking the target captured in the image captured by the camera 101 is resumed.

[0109] Specific examples of the configuration and processing of the mobile device (robot) 100 of the present disclosure have been described above with reference to FIGS. In this way, when the mobile device (robot) 100 of the present disclosure loses sight of the target 20 to be followed, it sequentially executes the following processes to rediscover the target and continue the following process. (Step S01) = Calculate the estimated position of the target to be tracked (Step S02) = Obtain the search path of the target to be followed (Step S03) = A search route list is generated in which search routes for the target are arranged in order of proximity to the estimated position of the target. (Step S04) = Obtain search routes in the order listed in the generated search route list, and search for the target to be tracked according to the obtained search routes.

[0110] By performing the process according to these steps, it is possible to efficiently rediscover the target that has been lost, and continue the tracking process.

[0111] 3. Sequence of Processing Executed by the Mobile Device (Robot) of the Present Disclosure Next, a processing sequence executed by the mobile device (robot) 100 of the present disclosure will be described with reference to the flowchart shown in FIG.

[0112] 17 and subsequent flowcharts can be executed, for example, according to a program stored in a storage unit of the mobile device (robot) 100. For example, the processing is executed under the control of a data processing unit (control unit) having a CPU or the like with a program execution function. The processing of each step in the flowchart shown in FIG. 17 will be explained below in order.

[0113] (Step S101) First, the state in which the mobile device (robot) 100 can confirm the target to be followed is determined. This state determination process is the process of determining the following three states, which has been described above with reference to FIG. (State 1) Tracking target can be confirmed (State 2) The state immediately before the target is lost (State 3) Target lost state

[0114] (State 1) A state in which the target to be followed can be confirmed is a state in which the target to be followed can be confirmed continuously based on information detected by a sensor such as a camera. (State 2) The state immediately before the target to be followed is a state in which the target to be followed cannot be confirmed for a specified time (t1) or longer based on information detected by a sensor such as a camera. (State 3) The lost state of the tracking target is a state in which the state immediately before the loss has continued for a specified time (t2) or more.

[0115] In step S101, the mobile device (robot) 100: (State 1) Tracking target can be confirmed If it is determined that the state is (State 1), the process proceeds to step S110.

[0116] On the other hand, in step S101, the mobile device (robot) 100: (State 2) The state immediately before the target is lost (State 3) Target lost state If it is determined that the state is either (State 2) or (State 3), the process proceeds to step S102.

[0117] (Step S102) In step S101, the mobile device (robot) 100: (State 2) The state immediately before the target is lost (State 3) Target lost state If it is determined that the state is either (State 2) or (State 3), the process proceeds to step S102.

[0118] In this case, the mobile device (robot) 100 executes a process of estimating the position of the target to be followed in step S102. This process is executed by the tracking target position estimation unit 104 shown in FIG. As previously described with reference to FIGS. 9 to 11, the target position estimation unit 104 performs the target position estimation process by utilizing the object identification process for the target and the spatial position estimation process for the object.

[0119] For example, when the object to be tracked is a person, the following process is carried out to estimate the position of the object to be tracked. (a) Object identification processing (person identification processing) = Image analysis processing that identifies people from camera images and sets the human area as the region of interest (ROI). (b) Spatial position estimation process for the object to be tracked: The human region of interest (ROI) in the image captured by the camera and the distance information for this human region are combined (fused) to estimate the 3D spatial position of the current object to be tracked (person).

[0120] In this tracking target position estimation process, a motion model according to the type of tracking target object is used. For example, if the tracking target object is a person, a motion model for a person is applied to estimate the position. For example, if the tracking target object is a car, a motion model for a car is applied to estimate the position.

[0121] By the position estimation process of the target to be tracked in step S102, target position estimation information 55 as shown in FIG. 11, for example, is generated.

[0122] (Step S103) Next, in step S103, the mobile device (robot) 100 analyzes a search path for searching for a target to be followed. This process is executed by the follow-up target searching unit 105 shown in FIG. Specifically, this is executed by the search path analysis unit 151 of the follow-up target search unit 105 shown in FIG.

[0123] As previously described with reference to FIGS. 12 and 13, the searched route analysis unit 151 receives the following data: (a) Estimated position information 55 of the target to be tracked generated by the target position estimation unit 104; (b) self-location information 53 generated by the self-location estimation unit 103; (c) map information 54 stored in the map data storage unit 106; The search path analysis unit 151 uses these data to analyze a search path for searching for the target 20 to be followed.

[0124] That is, one or more search paths connecting the current position of the robot and the possible positions of the target to be followed are calculated. Specifically, the robot existence path 71 and the possible path 72 of the target to be followed, which have been described with reference to FIGS. 12 and 13, are generated.

[0125] (Step S104) Next, in step S104, the mobile device (robot) 100 generates a search path list for searching for a target to be followed.

[0126] This process is executed by the follow-up target searching unit 105 shown in FIG. Specifically, this is executed by the search path determination unit 152 of the follow-up target search unit 105 shown in FIG.

[0127] As explained above with reference to Figures 12 and 14, the search route determination unit 152 uses the robot existence route 71 input from the search route analysis unit 151 and the possible route 72 where the target to be tracked may exist to generate a search route list 73 in which routes that should be searched for with priority are set at the top. The search route list 73 is a list in which routes that should be searched for with priority are placed at the top, and is a search route list in which routes are arranged in order of proximity to the estimated position information 55 of the target to be tracked generated by the target to be tracked position estimation unit 104.

[0128] (Step S105) Next, in step S105, the mobile device (robot) 100 selects a search route from the top of the search route list generated in step S104.

[0129] This process is executed by the follow-up target searching unit 105 shown in FIG. Specifically, this is executed by the search path determination unit 152 of the follow-up target search unit 105 shown in FIG.

[0130] As previously explained with reference to Figures 15 and 16, the search route determination unit 152 acquires search routes from the top of the search route list generated in step S104, and outputs the acquired search routes to the robot driving information generation unit 107 as search route information 56.

[0131] (Step S106) Next, in step S106, the mobile device (robot) 100 generates robot driving information for moving the robot along the searched route selected from the top of the searched route list in step S105.

[0132] This process is executed by the robot driving information generating unit 107 shown in FIG. The robot driving information generation unit 107 generates robot driving information 58 for moving the robot according to the search path information 56 input from the search path determination unit 152 as shown in FIG. 7, and outputs it to the robot driving unit 108.

[0133] (Step S107) Next, in step S107, the mobile device (robot) 100 drives the robot in accordance with the robot driving information generated by the robot driving information generating unit 107 in step S106.

[0134] This process is executed by the robot driving unit 108 shown in FIG. As shown in FIG. 7, the robot driving unit 108 moves the robot 10 in accordance with the robot driving information 58 input from the robot driving information generating unit 107 to perform processing for searching for a target to follow.

[0135] (Step S108) Next, in step S108, the mobile device (robot) 100 determines whether or not a target to be followed has been found.

[0136] This process is executed by the follow-up target searching unit 105 shown in FIG. Specifically, this is executed by the target detection confirmation unit 153 of the target search unit 105 shown in FIG.

[0137] The target detection confirmation unit 153 of the target search unit 105 shown in FIG. 12 receives the image information from the camera 101 and the distance information 52 from the distance sensor 102, and confirms whether or not the target has been detected. If a target to be followed is detected, the determination in step S108 becomes Yes, and the process proceeds to step S110. On the other hand, if the target to be followed is not detected, the determination in step S108 becomes No, and the process proceeds to step S109.

[0138] (Step S109) Next, in step S109, the mobile device (robot) 100 determines whether or not the search process has been completed for all search routes in the search route list generated in step S104.

[0139] This process is executed by the follow-up target searching unit 105 shown in FIG. Specifically, this is executed by the search path determination unit 152 of the follow-up target search unit 105 shown in FIG.

[0140] The searched route determination unit 152 determines whether or not search processing has been completed for all searched routes in the generated searched route list. If it is determined that the search is completed, it is determined that the search for the target to be followed has failed, and the process ends. On the other hand, if it is not complete, the process returns to step S105, and the next search route in the search route list is selected, and the process from step S106 onwards is executed. That is, the robot is driven along the next search route, and detection of the target to be followed is confirmed.

[0141] (Step S110) In step S101, the mobile device (robot) 100: (State 1) Tracking target can be confirmed If it is determined that the state is (State 1), the process proceeds to step S110.

[0142] If the Yesk determination is made in step S108, that is, if a target object is detected in the search process along the search path, the process proceeds to step S110.

[0143] In these cases, the mobile device (robot) 100 executes a process of following the object to be followed in step S110. This tracking process is a normal tracking process based on the detection information of the camera 101 and the distance sensor 102. That is, it is a process of tracking a target captured in an image captured by the camera 101, for example.

[0144] (Step S111) Finally, the mobile apparatus (robot) 100 determines whether or not to end the following process in step S111. This determination is made based on a predetermined rule, such as ending tracking when the target to be tracked goes outside a predetermined tracking range.

[0145] If it is determined in step S111 that the tracking process should not be ended, the process returns to step S101, and the processes from step S101 onwards are repeated. On the other hand, if it is determined in step S111 that the tracking process should be ended, the process ends.

[0146] By executing processing according to this flow, the mobile device (robot) disclosed herein can efficiently perform a search process for the target to be followed, even if the target to be followed is lost, and can rediscover the target to be followed in a short time and continue the tracking process.

[0147] [4. Processing examples for various branching configurations] Next, examples of processing according to various branch configurations will be described.

[0148] In the above-described embodiment, an example of processing when the target vehicle is lost, i.e., lost sight of, at a T-junction as shown in FIG. 1 has been described. However, there are various configurations of routes in which the target vehicle may be lost, i.e., lost sight of. The mobile device (robot) of the present disclosure is not limited to a specific route configuration, and is capable of quickly rediscovering the target to be followed when the target is lost in any route configuration.

[0149] Below, an example of processing corresponding to a plurality of different route configurations will be described. FIG. 18 shows a T-junction similar to that shown in FIG. 1, but the route BD is set to be impassable due to an obstacle.

[0150] In such a case, the robot 10 sets only one search path ABD as the search path and executes the search process. In other words, the route BD that constitutes the search route ABD is an impassable route, and it can be determined that there is no possibility of the target to be followed entering, so it is excluded from the search targets.

[0151] Even if an obstacle between B and D is not listed in the map information, the presence of the obstacle can be detected by the camera of the robot 10 and the distance sensor 101, and at this obstacle detection stage, it is possible to perform processing to delete the route A, B, and D from the search route.

[0152] Figure 19 shows the configuration of a Y-junction. Furthermore, although the target 20 to be followed is located in the direction of the route R, the estimated position 20b of the target to be followed estimated by the robot 10 is set on the route L side as shown in the figure.

[0153] In this case, the search route list generated by the robot 10 will be generated as a list in the following order: (Search Route 1) ABD (Search Route 2) ABC

[0154] First, the robot 10 searches for an object to be followed according to (search path 1) ABD. However, the target 20 cannot be detected by the target search process according to this (search path 1) ABD.

[0155] In this case, the following (search path 2) processing for searching for the target to be followed according to ABC is executed: In this (search path 2) processing for searching for the target to be followed according to ABC, the target to be followed 20 can be detected. In this example, since the estimated position 20b of the target to be tracked is significantly different from the actual position of the target to be tracked 20, it takes a long time to detect the target to be tracked, but the target to be tracked 20 can eventually be rediscovered and the tracking process can be continued.

[0156] 5. Configuration Examples of the Mobile Device and Information Processing Device of the Present Disclosure Next, configuration examples of the mobile device and information processing device of the present disclosure will be described.

[0157] The mobile device of the present disclosure includes not only the robot described in the above-mentioned embodiments, but also various other mobile devices such as self-driving vehicles and drones.

[0158] In addition, calculation of the search path and drive control such as movement control and orientation control of the mobile device (robot, etc.) may be performed by a data processing unit of an information processing device provided inside the mobile device such as a robot, or by an external information processing device capable of communicating with the robot.

[0159] 20 is a diagram showing an example of the configuration of a mobile device 500 when the mobile device (robot, etc.) calculates a search path and performs drive control such as movement control and orientation control of the mobile device (robot, etc.) by itself. That is, it is a diagram showing an example of the configuration of a mobile device 500 when an information processing device provided inside the mobile device calculates a search path and performs drive control such as movement control and orientation control of the mobile device (robot, etc.).

[0160] FIG. 21 is a diagram showing an example configuration of a mobile device 500 and a user terminal 700 when a user terminal capable of communicating with the mobile device, such as a controller, PC, or smartphone, calculates a search path and performs drive control such as movement control and orientation control of the mobile device (robot, etc.).

[0161] First, with reference to FIG. 20, a configuration example of a mobile device 500 in which the mobile device alone calculates a search path and performs drive control such as movement control and orientation control of the mobile device (robot, etc.).

[0162] As shown in FIG. 20, a mobile device 500 includes a camera 501, a sensor 502, a data processing unit 503, a position information acquisition unit 504, an input unit 505, an output unit 506, a communication unit 507, and a storage unit (memory) 508.

[0163] The camera 501 captures images in the direction of travel of the mobile device 500 and images of the target to be followed. The sensor 502 is an object detection sensor configured by, for example, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) or the like. It measures the distance to a target to be tracked or an obstacle. Note that the sensor 502 is not limited to LiDAR, and may be, for example, a stereo camera, a ToF sensor, an ultrasonic sensor, radar, sonar, or the like.

[0164] The data processing unit 503 executes the processing according to the above-described embodiment, that is, the calculation of a search path and drive control processing such as movement control and orientation control of a moving device (robot, etc.).

[0165] The main components of the mobile device (robot) 100 described above with reference to FIG. 7 , such as the self-position estimation unit 103, the target position estimation unit 104, the target search unit 105, and the robot driving information generation unit 107, are included in the data processing unit 503.

[0166] The data processing unit 503 has a processor such as a CPU having a program execution function, and executes processes according to the flowcharts explained in the above-mentioned embodiments. The program is stored in the storage unit 508.

[0167] The location information acquisition unit 504 communicates with, for example, a GPS satellite 600, analyzes the current location (latitude, longitude, altitude) of the mobile device 500 based on the communication information with the GPS satellite 600, and outputs the analysis information to the data processing unit 503.

[0168] The input unit 505 is, for example, a unit operated by a user, and is used for various processes, such as inputting user requests such as starting and stopping the vehicle. The output unit 506 includes an audio output unit, an image output unit, and the like. A communication unit 507 executes communication with a user terminal and an external server.

[0169] The storage unit (memory) 508 is used as a storage area and a work area for programs executed by the data processing unit 503. It is also used as a storage area for various parameters applied to processing. The storage unit (memory) 106 is composed of RAM, ROM, etc.

[0170] Next, referring to Figure 21, we will explain the configuration of a mobile device 500 and a user terminal 700 when a user terminal that can communicate with the mobile device, such as a controller, PC, or smartphone, calculates a search path and performs drive control such as movement control and orientation control of a mobile device (robot, etc.).

[0171] The moving device 500 has the same configuration as that described with reference to FIG. It communicates with the user terminal 700 via the communication unit 507 .

[0172] The following describes the configuration of the user terminal 700. As shown in the figure, the user terminal 700 has a data processing unit 701, a storage unit (memory) 702, a communication unit 703, an input unit 704, an output unit 705, and a display unit 706.

[0173] The data processing 701 calculates a search path for the mobile device 500 and executes drive control processing such as movement control and orientation control of the mobile device (robot, etc.).

[0174] The processing performed by the self-position estimation unit 103, the target position estimation unit 104, the target search unit 105, the robot driving information generation unit 107, and the like, which are the main components of the mobile device (robot) 100 described above with reference to FIG. 7, is executed in a data processing unit 701.

[0175] A data processing unit 701 of the user terminal 700 generates searched route information and drive control information such as movement control and orientation control of a mobile device (robot, etc.), and transmits these to the mobile device 500 via a communication unit 703 . The mobile device 500 moves in accordance with the control information received from the user terminal 700 .

[0176] The data processing unit 701 has a processor such as a CPU having a program execution function, and executes processes according to the flowcharts described in the above-mentioned embodiments. The program is stored in the storage unit 702 .

[0177] The storage unit (memory) 702 is used as a storage area and a work area for programs executed by the data processing unit 701. It is also used as a storage area for various parameters applied to processing. The storage unit (memory) 204 is composed of RAM, ROM, etc. The communication unit 703 executes communication with the mobile device 500 and an external server.

[0178] The input unit 704 is a unit operated by the user, and is used for various processes, for example, inputting user requests such as starting and ending control of the mobile device 500 . The output unit 705 includes an audio output unit, an image output unit, and the like.

[0179] The display unit 706 is used to display images captured by the camera of the mobile device 500, maps stored in the storage unit 702, route information generated by the data processing unit 701, and the like.

[0180] In an information processing system using a mobile device 500 and a user terminal 700 as shown in FIG. 21, for example, the following processing is possible. The data processing unit of the user terminal 700 calculates an estimated position of the target to be tracked, which is the estimated position of the target to be tracked by the mobile device 500, and further generates a searched route list and selects searched routes in order from the top of the searched route list. The mobile device 500 moves along the search route selected by the user terminal 700 to search for the target to be followed.

[0181] Furthermore, when the mobile device 500 moves along the search route selected by the user terminal 700 to search for the target to be tracked, the mobile device 500 inputs images captured by a camera attached to the mobile device 500 and transmits the captured images to the user terminal 700. The user terminal 700 determines, based on the captured image received from the mobile device 500, whether or not a target to be followed has been detected. Using the result of this determination, the mobile device 400 is caused to resume the tracking process or to execute a search process along the next search route. For example, the following processing is possible.

[0182] 6. Summary of the Disclosure The embodiments of the present disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0183] The technology disclosed in this specification can be configured as follows. (1) a target position estimation unit that calculates an estimated target position that is an estimated position of a target that is followed by a mobile device; a tracking target search unit that searches for the tracking target, The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; An information processing device that selects a search route from the top of the generated search route list in order, and moves the mobile device along the selected search route to search for the target to be followed.

[0184] (2) The tracking target search unit When searching for the target by moving the mobile device along the selected search path, The information processing device according to (1), wherein an image captured by a camera attached to the mobile device is input and it is determined whether a target to be tracked has been detected.

[0185] (3) The tracking target position estimation unit The information processing device according to (1) or (2), wherein the information processing device inputs a past image captured by a camera attached to the mobile device, in which the target to be tracked is captured, and calculates the estimated position of the target to be tracked.

[0186] (4) The tracking target position estimation unit The information processing device according to any one of (1) to (3), wherein the estimated position of the target to be followed is calculated by applying a motion model corresponding to the type of the target to be followed.

[0187] (5) The tracking target search unit The information processing device according to any one of (1) to (4), wherein the current position of the mobile device estimated by a self-position estimation unit is input, and one or more search routes connecting possible positions of the target to be followed are calculated.

[0188] (6) The self-location estimation unit The information processing device according to (5), wherein the current position of the mobile device is calculated by SLAM (Simultaneous Localization and Mapping) processing.

[0189] (7) The moving device is a sensor for detecting the target to be followed; The tracking target search unit The information processing device according to any one of (1) to (6), which executes a process of searching for the target to be followed when the sensor does not detect the target to be followed.

[0190] (8) The information processing device according to (7), wherein the sensor includes a camera.

[0191] (9) The information processing device a sensor for detecting the target to be followed; In a first state in which the sensor detects the target, executes a tracking process for the target to be tracked in accordance with the detection information of the sensor; A second state in which a period in which the sensor does not detect the target occurs for a specified time (t1); or In a third state in which the second state continues for a specified time (t2), The information processing device according to any one of (1) to (8), wherein the target searching unit executes a process of searching for the target.

[0192] (10) The tracking target search unit The information processing device according to any one of (1) to (9), wherein map information is referenced to calculate one or more search routes connecting the current position of the mobile device and possible positions of the target to be followed.

[0193] (11) The information processing device The information processing device according to any one of (1) to (10), which is provided in the mobile device.

[0194] (12) The information processing device The information processing device according to any one of (1) to (10), which is an information processing device capable of communicating with the mobile device.

[0195] (13) An information processing system having a mobile device and an information processing device capable of communicating with the mobile device, The information processing device includes: a tracking target position estimation unit that calculates an estimated position of a tracking target that is followed by the mobile device; a tracking target search unit that searches for the tracking target, The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; Select a search route from the top of the generated search route list, The moving device is An information processing system in which the information processing device moves along a selected search route to search for the target to be tracked.

[0196] (14) The moving device is When the information processing device moves along the selected search route to search for the target to be tracked, inputting an image captured by a camera attached to the mobile device and transmitting the captured image to the information processing device; The information processing device includes: The information processing system according to (13), wherein it is determined whether or not a target to be tracked has been detected based on the captured image received from the mobile device.

[0197] (15) An information processing method executed in an information processing device, a tracking target position estimating step in which a tracking target position estimating unit calculates an estimated position of the tracking target that is to be tracked by the mobile device; The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; An information processing method that executes the steps of selecting a search route in order from the top of the generated search route list, and moving the mobile device along the selected search route to search for the target to be tracked.

[0198] (16) An information processing method executed in an information processing system having a mobile device and an information processing device capable of communicating with the mobile device, The information processing device, a tracking target position estimation process for calculating an estimated position of a tracking target that is an estimated position of a tracking target that is followed by the mobile device; calculating one or more search routes connecting the current position of the mobile device and a possible position of the target to be tracked, generating a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the target to be tracked, and executing a process of selecting search routes in descending order from the top of the generated search route list; The moving device, An information processing method in which the information processing device moves along a selected search path to search for the target to be tracked.

[0199] (17) A program for causing an information processing device to execute information processing, a tracking target position estimating step of causing a tracking target position estimating unit to calculate an estimated position of a tracking target that is an estimated position of a tracking target that is followed by the mobile device; The target search section calculating one or more search routes connecting the current position of the mobile device and possible positions of the target to be followed; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; A program that causes a search route to be selected in order from the top of the generated search route list, and causes the mobile device to move along the selected search route to search for the target to be followed.

[0200] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from the recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0201] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are all located in the same housing. [Industrial Applicability]

[0202] As described above, according to the configuration of one embodiment of the present disclosure, an apparatus and method are realized that, when a mobile device loses a target to be followed, can efficiently rediscover the target and resume tracking. Specifically, for example, the device has a tracking target position estimation unit and a tracking target search unit that searches for the tracking target. The tracking target search unit calculates one or more search routes connecting the current position of the mobile device and possible positions of the tracking target, generates a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the tracking target, selects a search route from the top of the generated search route list, and moves the mobile device along the selected search route to search for the tracking target. It determines whether the tracking target has been detected based on images captured by a camera attached to the mobile device, and if detected, resumes the tracking process. This configuration realizes an apparatus and method that, when a mobile device loses a target to be followed, can efficiently rediscover the target and resume tracking. [Explanation of symbols]

[0203] 10. Robot 20 Follow-up target 100 Mobile Device (Robot) 101 Camera 102 Distance Sensor 103 Self-position estimation part 104 Tracking target position estimation unit 105 Tracking target search unit 106 Map data storage unit 107 Robot driving information generation unit 108 Robot drive unit 121 Tracking target area detection unit 122 3D position estimation unit for tracking target 131 3D position calculation unit for tracking target in image 132 Motion model applied tracking target 3D position estimation unit 151 Search Path Analysis Unit 152 Search path determination unit 153 Tracking target detection confirmation unit 500 Mobile Device 501 Camera 502 Sensors 503 Data Processing Unit 504 Location information acquisition unit 505 Input section 506 Output section 507 Communications Department 508 Memory 600 GPS satellites 700 user terminals 701 Data Processing Unit 702 Memory 703 Communications Department 704 Input section 705 Output Section 706 Display section

Claims

1. a tracking target position estimation unit that calculates an estimated position of a tracking target that is followed by the mobile device; a tracking target search unit that searches for the tracking target, The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the tracking target; selecting a search route in order from the top of the generated search route list, and moving the mobile device along the selected search route to search for the target to be tracked; an information processing device that, if the target to be followed cannot be detected on the selected search route, determines whether or not there are any unselected search routes in the search route list, and if there are, selects the topmost search route from the unselected search routes in the search route list, and moves the mobile device along the selected search route to search for the target to be followed.

2. The tracking target search unit When searching for the target by moving the mobile device along the selected search path, The information processing apparatus according to claim 1 , wherein an image captured by a camera attached to the mobile device is input, and whether or not a target to be followed is detected is determined.

3. The tracking target position estimation unit The information processing apparatus according to claim 1 , wherein an image of the target to be tracked that is a past image captured by a camera attached to the mobile device is input to calculate the estimated position of the target to be tracked.

4. The tracking target position estimation unit The information processing apparatus according to claim 1 , wherein the estimated position of the target to be followed is calculated by applying a motion model corresponding to the type of the target to be followed.

5. The tracking target search unit The information processing device according to claim 1 , wherein the current position of the mobile device estimated by a self-position estimation unit is input, and one or more search routes connecting possible positions of the target to be followed are calculated.

6. The self-location estimation unit The information processing device according to claim 5 , wherein the current position of the mobile device is calculated by SLAM (Simultaneous Localization and Mapping) processing.

7. The moving device is a sensor for detecting the target to be followed; The tracking target search unit The information processing apparatus according to claim 1 , further comprising: a process for searching for the target to be followed when the sensor does not detect the target to be followed.

8. The information processing device according to claim 7 , wherein the sensor includes a camera.

9. The information processing device includes: a sensor for detecting the target to be followed; In a first state in which the target is detected by the sensor, executes a tracking process for the target to be tracked in accordance with the detection information of the sensor; A second state in which a period in which the sensor does not detect the target occurs for a specified time (t1); or In a third state in which the second state occurs continuously for a specified time (t2), The information processing apparatus according to claim 1 , wherein the target searching unit executes a process of searching for the target.

10. The tracking target search unit The information processing device according to claim 1 , wherein map information is referenced to calculate one or more search routes connecting the current position of the mobile device and possible positions of the target to be followed.

11. The information processing device includes: The information processing device according to claim 1 , wherein the information processing device is provided within the mobile device.

12. The information processing device includes: The information processing device according to claim 1 , wherein the information processing device is capable of communicating with the mobile device.

13. An information processing system including a mobile device and an information processing device capable of communicating with the mobile device, The information processing device includes: a tracking target position estimation unit that calculates an estimated position of a tracking target that is followed by the mobile device; a tracking target search unit that searches for the tracking target, The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the tracking target; Select a search route from the top of the generated search route list, The moving device is the information processing device moves along the selected search path to search for the target to be tracked; If the target to be tracked cannot be detected on the search route selected by the mobile device, The information processing device includes: determining whether or not there is an unselected search route in the search route list, and if there is, selecting the highest-ranking search route from the unselected search routes in the search route list; The moving device is An information processing system that searches for the target to be tracked by moving the mobile device along a selected search route.

14. The moving device is When the information processing device moves along the selected search route to search for the target to be tracked, inputting an image captured by a camera attached to the mobile device and transmitting the captured image to the information processing device; The information processing device includes: The information processing system according to claim 13 , wherein it is determined whether or not a target to be tracked has been detected based on the captured image received from the mobile device.

15. An information processing method executed in an information processing device, a tracking target position estimating step in which a tracking target position estimating unit calculates an estimated position of the tracking target that is to be tracked by the mobile device; The tracking target search unit calculating one or more search paths connecting the current position of the mobile device and possible positions of the target to be tracked; generating a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the tracking target; selecting a search route in order from the top of the generated search route list, and moving the mobile device along the selected search route to search for the target to be tracked; An information processing method in which, if the target to be tracked cannot be detected on the selected search route, it is determined whether or not there are any unselected search routes in the search route list, and if there are, the topmost search route is selected from the unselected search routes in the search route list, and the mobile device is moved along the selected search route to search for the target to be tracked.

16. An information processing method executed in an information processing system having a mobile device and an information processing device capable of communicating with the mobile device, The information processing device, a tracking target position estimation process for calculating an estimated position of a tracking target that is an estimated position of a tracking target that is followed by the mobile device; calculating one or more search routes connecting the current position of the mobile device and a possible position of the target to be tracked, generating a search route list in which the calculated one or more search routes are arranged in descending order of proximity to the estimated position of the target to be tracked, and executing a process of selecting search routes in descending order from the top of the generated search route list; The moving device, the information processing device moves along the selected search path to search for the target to be tracked; If the target to be tracked cannot be detected on the search route selected by the mobile device, The information processing device includes: determining whether or not there is an unselected search route in the search route list, and if there is, selecting the highest-ranking search route from the unselected search routes in the search route list; The moving device is An information processing method for searching for the target to be followed by moving the mobile device along a selected search route.

17. A program for causing an information processing device to execute information processing, a tracking target position estimating step of causing a tracking target position estimating unit to calculate an estimated position of a tracking target that is an estimated position of a tracking target that is followed by the mobile device; The target search section calculating one or more search routes connecting the current position of the mobile device and possible positions of the target to be followed; generating a search route list in which the one or more calculated search routes are arranged in descending order of proximity to the estimated position of the tracking target; Selecting a search route from the top of the generated search route list in order, and moving the mobile device along the selected search route to search for the target to be tracked; A program for determining whether or not there are any unselected search routes in the search route list if the target to be tracked cannot be detected on the selected search route, and if there are any, selecting the topmost search route from the unselected search routes in the search route list, and moving the mobile device along the selected search route to search for the target to be tracked.

Images