Method for controlling a mobile object, control device, mobile object, program, and storage medium

Abstract

This technology provides the ability for a moving object to take effective actions to locate a specific individual if it fails to recognize that individual. [Solution] The method for controlling a moving body according to the present disclosure is a method for controlling a moving body that moves with a specific person, and includes: recognizing a specific person and other pedestrians among pedestrians around the moving body using images taken around the moving body in a first period; predicting the future position of a first candidate among pedestrians around the moving body relative to the moving body using images taken after the first period, in response to the fact that the specific person can no longer be recognized in images taken after the first period; and controlling the movement of the moving body to move from its current position to a specific position for recognizing the first candidate, in accordance with the predicted future position of the first candidate.

Description

Technical Field

[0001] The present invention relates to a moving body control method, a control device, a moving body, a program, and a storage medium.

Background Art

[0002] Moving bodies such as robots that travel autonomously by following a user or leading a user are known. Patent Document 1 discloses a technology in which a robot travels in front of a pre-recognized specific person (guidance target) to guide the specific person, and reduces the moving speed to re-recognize the specific person when recognition of the specific person using face tracking or human body tracking processing fails.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described conventional technology, when the moving body reduces its moving speed, the specific person may accidentally enter the image and the area of the specific person becomes suitable for recognition, so that the specific person may be re-recognized. However, if such a situation does not occur, the specific person cannot be re-recognized, and the moving body has no choice but to stop moving. In addition, when a plurality of pedestrians are present around the moving body, the specific person may be hidden behind other pedestrians, or may move away from the moving body to avoid other pedestrians or obstacles. For this reason, when recognition of the specific person fails, it is desirable for the moving body to take an effective action to locate the specific person.

[0005] The present invention has been made in view of the above problems, and an object thereof is to provide a technology that enables a moving body to take an effective action to locate a specific person when recognition of the specific person fails. [Means for solving the problem]

[0006] According to the present invention, A method for controlling a mobile object that moves with a specific person, Using images taken of the area around the moving object during the first period, the system recognizes the specific person and other pedestrians among the pedestrians around the moving object. In the images taken after the first period, if the specific person can no longer be recognized, Using images captured after the first period, predict the future position of a first candidate among the pedestrians surrounding the moving object relative to the moving object, A method for controlling a moving body is provided, which includes controlling the movement of the moving body to move from its current position to a specific position for recognizing the first candidate, depending on the predicted future position of the first candidate. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technology that enables a moving object to take effective actions to find a specific person if it fails to recognize that person. [Brief explanation of the drawing]

[0008] [Figure 1] A perspective view showing an example of the overall configuration of the mobile body of this embodiment. [Figure 2] A schematic plan view illustrating an example of a mobile unit for the mobile body of this embodiment. [Figure 3] A block diagram showing an example of the control system for the mobile body of this embodiment. [Figure 4] A block diagram showing an example of the functional configuration of the control unit in this embodiment. [Figure 5A] Figure (1) illustrates the process for finding a specific person in this embodiment. [Figure 5B] Figure (2) illustrates the process for finding a specific person in this embodiment. [Figure 5C]Figure (3) illustrates the process for finding a specific person in this embodiment. [Figure 6] A flowchart illustrating a series of operations for locating a specific person in this embodiment. [Figure 7] A flowchart illustrating the series of operations of the pedestrian detection process in this embodiment. [Figure 8] A flowchart illustrating the series of operations involved in moving to the viewpoint position in this embodiment. [Figure 9] This diagram illustrates an example of using the degree of open space in this embodiment as a cost. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features from the multiple features described in the embodiments may be combined arbitrarily. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0010] <Embodiment> The following embodiments will be described using a micromobility as an example of a mobile body. A micromobility is, for example, a small electric vehicle that can carry luggage and follow, lead, or guide a user (also referred to as a specific person). However, this embodiment may be applied not only to vehicles that carry luggage but also to vehicles that can carry people, and not only to vehicles but also to autonomously mobile robots. For example, it can be applied to a robot that can walk autonomously using its feet without having wheels. Furthermore, the following description will be based on an example of a mobile body having one driven wheel, but the mobile body does not necessarily have to have a driven wheel, and more than one driven wheel may be used. This embodiment is not limited to mobile bodies that are electrically powered but can also be applied to mobile bodies that move using other power sources.

[0011] <Configuration of Mobile Object> The configuration of the mobile object 100 according to the embodiment will be described with reference to Figure 1. Figure 1 is a perspective view showing the overall configuration of the mobile object. As shown in Figure 1, the mobile object 100 of the embodiment includes a lower housing 111, an upper housing 112, a front display unit 113, a plurality of detection units 114, an audio output unit 115, a light emitting unit 116, two drive wheels 120, and a driven wheel 121. Note that the side where the drive wheels 120 are provided is defined as the front side.

[0012] The lower housing 111 rotatably holds the drive wheels 120 and the driven wheel 121, and accommodates a traveling unit including the drive wheels 120 and the driven wheel 121. The upper part of the lower housing 111 has a loading space 118 capable of carrying a user's luggage. The lower housing 111 accommodates the light emitting unit 116. The region of the lower housing 111 that accommodates the light emitting unit 116 is made of a material that can transmit light.

[0013] The upper housing 112 is configured to extend upward from the rear part of the lower housing 111. The upper housing 112 supports the front display unit 113, the detection units 114, and the audio output unit 115 at its upper end portion.

[0014] The front display unit 113 notifies the user of information via images. The front display unit 113 may be a display device that displays images, such as a liquid crystal display device or an organic EL (Electro Luminescence) display device. The front display unit 113 is provided on the front side of the upper end of the upper housing 112. The front display unit 113 faces forward. The front display unit 113 notifies the user of information by displaying images imitating human eyes or a human face, characters, and the like.

[0015] The plurality of detection units 114 detect targets such as people and objects around the mobile body 100. The detection units 114 may be imaging apparatuses each having an optical system such as a lens, an image sensor, and the like. The detection units 114 capture images of the surroundings and output image data. The detection units 114 may include devices capable of detecting objects, such as radar and LiDAR (Light Detection and Ranging), instead of or in addition to the imaging apparatuses. The plurality of detection units 114 are provided, for example, on the front, rear, left, and right of the upper end portion of the upper housing 112. Thereby, the plurality of detection units 114 can capture images in all directions (that is, 360° around the mobile body 100) to generate an image. Note that the installation locations and the number of the detection units 114 are not particularly limited, and may be changed as appropriate.

[0016] The audio output unit 115 notifies surrounding people such as users of information by audio. The audio output unit 115 may be an audio output device such as a speaker.

[0017] The light emitting unit 116 notifies surrounding people such as users of information by light. The light emitting unit 116 includes, for example, one or a plurality of light emitting elements such as LEDs (Light Emitting Diodes). The light emitting unit 116 is provided so as to surround the lower housing 111. The light emitting unit 116 may emit a plurality of colors of light. The light emitting unit 116 can emit light of different colors for each region or direction. That is, the light emitting unit 116 emits light of different colors in each of, for example, the front region 116F, the left region 116L, and the right region. Further, the light emitting unit 116 can switch between light emission and light extinction for each region. Note that the light emission of the regions may be controlled in finer sub-regions.

[0018] The two drive wheels 120 are provided on the left and right of the front and lower portion of the lower housing 111. The drive wheels 120 rotate around the axle when drive force is transmitted from the traveling unit.

[0019] The driven wheel 121 is located at the rear and center of the lower housing 111. The driven wheel 121 is held so as to be able to pivot around a vertical axis that extends in the vertical direction.

[0020] Figure 2 is a schematic plan view illustrating the travel unit 125 of the mobile body 100. The travel unit 125 of the mobile body 100 will be described with reference to Figure 2.

[0021] The travel unit 125 is located at the bottom of the lower housing 111. The travel unit 125 has two drive wheels 120, a driven wheel 121, a drive mechanism 122, a motor 123a, a motor 123b, and a battery 124.

[0022] When power is supplied from the battery 124, motors 123a and 123b function as a drive source and supply driving force to the drive mechanism 122. The drive mechanism 122 transmits the driving force supplied from motors 123a and 123b to the drive wheels 120. This causes the drive mechanism 122 to rotate the drive wheels 120, moving the mobile body 100 forward or backward. The drive mechanism 122 causes the direction of travel of the mobile body 100 to be either left or right by creating a rotational difference between motors 123a and 123b. By changing the magnitude of the rotational difference, the drive mechanism 122 enables various turns of the mobile body 100, such as pivot turns, slow turns, and super-pivot turns.

[0023] Figure 3 is a block diagram of the control system of the mobile unit 100. The control system of the mobile unit 100 will be described with reference to Figure 3. As shown in Figure 3, the mobile unit 100 further comprises a control unit 130, an operation unit 132, a top display unit 131, an audio input unit 133, a GNSS sensor 134, a storage device 135, and a communication unit 136.

[0024] The control unit 130 is an example of a control device, also called an ECU (Electronic Control Unit), and may be a computer. The control unit 130 is responsible for the overall control of the mobile body 100. The control unit 130 includes one or more processors, such as a CPU (Central Processing Unit), memory devices such as semiconductor memory, and interfaces with external devices.

[0025] The processor may be an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), or QPU (Quantum Processing Unit) in addition to or instead of a CPU. The processor realizes various functions of the control unit 130 by reading and executing computer programs stored in memory devices or the like.

[0026] Some or all of the functions of the control unit 130 may be implemented by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0027] The memory device may be, for example, RAM (Random Access Memory). The memory device stores programs executed by the processor and data used by the processor for processing. Multiple sets of processors, memory devices, and interfaces may be provided according to the function of the mobile unit 100, and each set may be configured to communicate with one another.

[0028] The control unit 130 acquires output such as image information output by the detection unit 114, user input information acquired by the operation unit 132, and voice information output by the voice input unit 133, and performs processing according to each piece of information. The control unit 130 performs behavioral control, including the movement of the mobile body 100, by controlling the motors 123a and 123b of the driving unit 125. The control unit 130 notifies the user or people in the vicinity of information. For example, the control unit 130 notifies the user of information by controlling the display of images displayed on the front display unit 113 and the top display unit 131. The control unit 130 may also notify the user of information by emitting light from the light-emitting unit 116. The control unit 130 may also notify the user of information by outputting voice from the voice output unit 115. The control unit 130 may perform processing on the image information from the detection unit 114 using a machine learning model for image recognition (e.g., a deep neural network). Furthermore, the control unit 130 may perform processing on the audio information from the audio input unit 133 using a machine learning model for speech recognition (e.g., a deep neural network).

[0029] The control unit 130 can recognize feature points in images captured by the detection unit 114 using a machine learning model. Furthermore, the control unit 130 can estimate the self-position of the moving object 100 using the recognized feature points and map information pre-stored in the memory device 135. The control unit 130 can also recognize people (e.g., pedestrians) in the image using a machine learning model and assign a unique identifier (ID) to each pedestrian. Then, the control unit 130 can track (track) the pedestrians assigned identifiers within subsequent images.

[0030] The top display unit 131 may be an image display device. The top display unit 131 is provided on the upper surface of the upper end of the upper housing 112. Based on image information acquired from the control unit 130, the top display unit 131 displays images and characters related to information to be notified to the user. For example, the top display unit 131 displays an operation screen for accepting user input.

[0031] The operation unit 132 may be an input device such as a touch panel or keyboard. The operation unit 132 is, for example, provided on the display surface of the top display unit 131. The operation unit 132 receives input from the user and outputs it to the control unit 130.

[0032] The audio input unit 133 may be a microphone or the like. The audio input unit 133 picks up the voice of the user and the surroundings of the mobile unit 100 and outputs it to the control unit 130.

[0033] The GNSS sensor 134 receives GNSS signals from satellites to detect the current position of the mobile object 100. GNSS stands for Global Navigation Satellite System.

[0034] The storage device 135 includes a non-volatile recording medium for storing various data. The storage device 135 may be an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device 135 may store programs executed by the processor, parameters necessary for program execution, and data to be processed by the processor. The storage device 135 may also store various parameters of machine learning models for speech recognition and image recognition executed by the control unit 130 (e.g., trained parameters and hyperparameters of a deep neural network). The storage device 135 can store map information of the space in which the mobile body 100 travels. The map information is pre-provided map information, and the mobile body 100 may change (update) part of the map information while traveling. The map information may be represented, for example, in the form of an occupied grid map, where each grid can represent an area in which the mobile body 100 can travel or an area of ​​obstacles. If the current position of the mobile body 100 is placed in the center of the occupied grid map, the position of obstacles can be identified from the current position of the mobile body 100.

[0035] The communication unit 136 communicates with external devices such as the user's communication terminal 140 via wireless communication, such as Wi-Fi and fifth-generation mobile communication.

[0036] Figure 4 is a functional block diagram showing the functions of the control unit 130. The functions of the control unit 130 will be explained with reference to Figure 4.

[0037] The control unit 130 comprises a recognition unit 401, an action prediction unit 402, an exploration action decision unit 403, a view action decision unit 404, a path generation unit 405, and a drive control unit 406. The control unit 130 may, for example, realize the functions of each part of the control unit 130 by reading a program from the storage device 135 and executing it.

[0038] The recognition unit 401 recognizes features of the environment surrounding the moving object 100 from images captured by the detection unit 114, which capture the area around the moving object 100. The recognition unit 401 may use one or more trained machine learning models to recognize features of the surrounding environment from multiple images taken from various directions of the moving object 100. Features of the surrounding environment include, for example, features in the images that can be associated with roads, pathways, buildings, walls, shops, signs, markers, obstacles, pedestrians, and vehicles.

[0039] Furthermore, the recognition unit 401 recognizes pedestrians around the moving object 100 from the image captured by the detection unit 114, which shows the area around the moving object 100. If the user's (specific person's) face and entire body have been captured in advance and the user's features have been registered, the recognition unit 401 can use a machine learning model to recognize pedestrians in the image captured by the detection unit 114, and distinguish between the specific person and other pedestrians. In addition, the recognition unit 401 assigns a unique identifier to each of the recognized specific person and other pedestrians, and tracks the specific person and each pedestrian in the image thereafter. If, as a result of recognizing pedestrians in the image, a new pedestrian (a pedestrian whose features do not match those of a pedestrian to which an identifier has already been assigned) appears, the recognition unit 401 assigns a new identifier to that pedestrian.

[0040] The behavior prediction unit 402 uses images captured by the detection unit 114 to predict the future position of each pedestrian relative to the moving object 100. Known techniques can be used for predicting the future position of pedestrians using images. For example, the behavior prediction unit 402 outputs information indicating that pedestrian A will be located at a position (x, y) on the plane (with the position of the moving object 100 as the origin) on which the moving object 100 is traveling, after a time Δt has elapsed from the current time.

[0041] The search action decision unit 403 determines an action to search for the specific person in order to re-recognize them, in response to the loss of recognition of the specific person. The search action decision unit 403 selects a candidate for the search target from among the pedestrians recognized in the image and determines an action to move from the current position to a specific location for recognizing the candidate (for example, a position that is larger than a predetermined size and can capture the entire body of the candidate). Furthermore, if the search action decision unit 403 does not find the specific person among the pedestrians surrounding the moving object 100, it determines an action to move to a hidden area from the moving object 100 and search for the specific person. The hidden area is, for example, an area that is a blind spot from the moving object due to an obstacle. Details of the processing by the search action decision unit 403 will be described later.

[0042] If the View Action Decision Unit 404 determines that the specific person is not present even after taking the action determined by the Search Action Decision Unit 403, it decides to take action to a predetermined location (viewing position) suitable for finding the specific person. Then, the View Action Decision Unit 404 recognizes the specific person using the image captured at that predetermined location. This viewing position can also be described as a position where the specific person can easily see the moving object 100. The processing by the View Action Decision Unit 404 will be described in detail later.

[0043] The path generation unit 405 generates a path for the mobile body 100 to travel. If a user is present in the travel area, the path generation unit 405 generates a path to follow the user or to lead the user. It also generates a path for the actions determined by the exploration action decision unit 403 and the view action decision unit 404.

[0044] The drive control unit 406 moves the mobile body 100 along the path generated by the path generation unit 405 by controlling, for example, motor 123a and motor 123b. The drive control unit 406 may also make motor 123a and motor 123b rotate in opposite directions to perform a pivot turn or a pivot turn (also called a turn in place) without moving the mobile body 100. The drive control unit 406 may also turn the mobile body 100 by a turn other than a pivot turn.

[0045] <Overview of the process for locating a specific individual> Next, with reference to Figures 5A to 5C, an overview of the process for finding a specific person according to this embodiment will be described. Figure 5A schematically shows the mobile body 100, pedestrians, and obstacles on a plane on which the mobile body 100 is traveling. Pedestrians 501 to 506 are moving around the mobile body 100, and walls 530 and 531 are present around the mobile body 100 and pedestrians 501 to 506. Pedestrian 501 was recognized and tracked as a specific person in images taken previously (for example, in the first period). However, at the timing shown in Figure 5A, the control unit 130 is no longer able to recognize this specific person (for example, because the specific person is partially hidden in the shadow of pedestrian 502). Therefore, the control unit 130 recognizes pedestrian 501 as a new pedestrian. Pedestrians 502 to 504 were recognized and tracked in images taken previously (in the first period) (each has been assigned a unique identifier). Pedestrians 505 and 506 are pedestrians that were recognized after the control unit 130 was no longer able to recognize the specific individuals.

[0046] The control unit 130 predicts the future position of each pedestrian relative to the mobile body 100. In Figure 5A, the arrows emanating from each pedestrian represent vectors from the pedestrian's current position to their predicted future position (position after time Δt). The circle 521 surrounding the future position of pedestrian 505 represents a specific position for the mobile body 100 to recognize pedestrian 505. Circle 521 represents, for example, a distance of 3 meters from pedestrian 505. Similarly, the circle 522 surrounding the future position of pedestrian 501 indicates a specific position for the mobile body 100 to recognize pedestrian 501. Furthermore, the circle 523 surrounding the future position of pedestrian 506 indicates a specific position for the mobile body 100 to recognize pedestrian 506.

[0047] The control unit 130 selects candidates for the specific person. For example, the control unit 130 excludes pedestrians (pedestrians 501 to 506) included in images taken after the first period that were distinguished from the specific person in the images taken during the first period. Then, the control unit 130 selects candidates for the specific person from the remaining pedestrians. This is done because, in the previously taken images, pedestrians 502 to 504 were distinguished from the specific person and tracked, so it can be determined that these pedestrians are not the specific person.

[0048] The control unit 130 can, for example, select a pedestrian (in this case, pedestrian 505) that is closest to the moving object 100 among the predicted future positions as the first candidate. After moving to a specific position (for example, position 510 on circle 521) corresponding to the first candidate's predicted future position, the control unit 130 recognizes a specific person using an image that includes the first candidate. This allows for efficient searching of pedestrians who are likely to be the specific person.

[0049] This position 510 is, for example, a position from which more parts of the first candidate's entire body (e.g., face, both shoulders, both arms, waist, and both legs) can be recognized from the image than the current position of the moving object 100. If more parts of the pedestrian's entire body can be recognized, the features of the registered specific person and the features of the pedestrian in the image can be compared with accuracy. To put this example in other words, position 510 is, for example, a position from which the recognition accuracy of the first candidate's key points is higher than that of the current position of the moving object 100 when recognizing the key points of the first candidate from the image. If the recognition accuracy of the pedestrian's key points is high, the features of the registered specific person and the features of the pedestrian in the image can be compared with accuracy.

[0050] If the control unit 130 determines that the specific person is not present using the image containing the first candidate, it identifies the pedestrian whose predicted future position is next closest to the moving object 100 (in this case, 501) as the second candidate. Depending on the predicted future position of the second candidate, the control unit 130 moves to a specific position for recognizing the second candidate (for example, position 511 on circle 522), and then recognizes the specific person using the image containing the second candidate. In the example shown in Figure 5A, since pedestrian 501 is the specific person, the control unit 130 can recognize the specific person using the image captured at position 511.

[0051] The control unit 130 may estimate a path to sequentially traverse predicted future positions and select the first pedestrian to traverse the path as the first candidate. In the example shown in Figure 5A, the control unit 130 estimates a path to sequentially traverse positions 510, 511, and 512. When traveling along this path, the first pedestrian to traverse the path will be pedestrian 505, so the control unit 130 can select pedestrian 505 as the first candidate.

[0052] If it is determined that the specific person is not present using an image containing the first candidate, the mobile object 100 can move to a hidden area from the mobile object 100 based on an occupation grid map that shows the locations of surrounding obstacles. For example, in the occupation grid map, wall 531 is shown as an obstacle. Also, the area opposite wall 531 as seen from the mobile object 100 shown in Figure 5A is a hidden area (for example, a drivable area that is a blind spot from the mobile object 100 due to an obstacle).

[0053] Furthermore, the mobile device 100 may output an audio or display indicating that it is searching for a specific person when it can no longer recognize the specific person in the captured image. This audio or display may include, for example, the name of the specific person, or it may simply indicate that it is searching. In this way, it becomes easier for the specific person to find the mobile device 100.

[0054] Figure 5B shows an example in which the mobile device 100 travels through a hidden area of ​​wall 531 to search for a user. The mobile device 100 identifies a hidden area based on the occupied grid map described above and moves into that hidden area. The mobile device 100 uses the image captured after moving into the hidden area to recognize a specific person. The mobile device 100 uses the image captured while traveling through the hidden area to predict the future positions of pedestrians 541 and 542. For example, the mobile device 100 selects pedestrian 541 as a third candidate and moves to a specific position (position 551 on circle 561) to recognize that third candidate. The mobile device 100 uses the image captured at position 551, which includes pedestrian 541, to determine whether the specific person is present. If it is determined that the specific person is not present, the mobile device 100 moves to a specific position (position on circle 562) to recognize pedestrian 542 and recognizes pedestrian 542.

[0055] As shown in Figure 5C, if the mobile body 100 determines that the specific person is not present based on the image captured after moving into the hidden area, it may travel to a predetermined position (viewing position) suitable for finding the specific person. In this case, the mobile body 100 recognizes the specific person using the image captured at the predetermined position.

[0056] In the above-described embodiment, the case in which the mobile body 100 moves to a predetermined position after moving to the hidden area is explained as an example, but the mobile body 100 may also move to the predetermined position after the operation shown in Figure 5A.

[0057] Figure 5C shows an example in which the mobile device 100 determines a viewing position and recognizes a specific person from within a flow of people. In the example shown in Figure 5C, pedestrians 571 to 576 are moving, forming a flow of people. Region 581 indicates an area where the distance to the pedestrians forming the flow is close, and the entire body of the pedestrians cannot be captured within the field of view. Region 582, adjacent to region 581, is a position where it is possible to capture approximately the entire body of the pedestrians forming the flow within the field of view. In this area, the density of pedestrians is below a predetermined threshold. On the other hand, as the distance from the position of the flow of people moves to line 583, the accuracy of pedestrian recognition using the image decreases. In other words, the viewing position is a position where the density of pedestrians is below a predetermined threshold, and includes a position where it is possible to image an area where the density of pedestrians is higher than that position.

[0058] Region 584 represents an area where shadows are formed due to the position of the sun and buildings, resulting in lower illumination than the surrounding area. Note that, in addition to this example, region 584 may also be an area that is backlit when imaging the direction of pedestrian flow from region 584, for example, due to the position of the sun and buildings.

[0059] The mobile device 100 can determine the viewing position using cost superposition. For example, the mobile device 100 can use a first cost corresponding to the distance from the pedestrian. This cost increases as the mobile device approaches the pedestrian from the boundary A between region 581 and region 582. On the other hand, this cost decreases as the straight line from boundary A toward line 583 approaches the center of region 582 (position 585), and increases as the line moves toward line 583 from position 585. In other words, when imaging from a direction perpendicular to the direction of pedestrian flow, the cost is set to be low for positions where almost the entire body of the pedestrian can be imaged, and high when the mobile device is too close or too far from the pedestrian.

[0060] Furthermore, the mobile device 100 can use a cost that corresponds to the brightness. For example, the cost within region 584 can be set higher than the cost in other regions. In this case, the mobile device 100 is controlled not to image the flow of people outside region 584.

[0061] Furthermore, the mobile device 100 can use a cost that corresponds to the density of pedestrians in the captured image. For example, this cost will be higher in areas where there is pedestrian flow within region 581 because the density of pedestrians is high, and lower in other areas.

[0062] The mobile unit 100 may use a cost that corresponds to the degree of openness of the space. The degree of openness of the space may be calculated, for example, from the graph shown in Figure 9. The graph shown in Figure 9 is an aggregate of the distance from a certain grid (location) to an obstacle grid on the occupied grid map, in each direction from that grid. This indicates that the space is open if there are obstacles far away from a certain grid at all angles. In other words, the area of ​​region 901 represents the degree of openness of the space, and the larger the area, the more open the space. Here, the vertical axis of Figure 9 is set so that the scale becomes shorter as the distance from the grid to the obstacle increases. In this way, the area of ​​region 901 becomes larger the less obstacle there is (the more open) closer to the grid. That is, it is possible to construct a cost such that the absence of nearby obstacles is more important than the absence of distant obstacles. In the example shown in Figure 5C, for example, locations close to wall 530 or wall 531 will have a higher cost, and locations further away from these obstacles will have a lower cost.

[0063] The mobile unit 100 can determine the viewing position using the superposition of the costs described above. For example, the mobile unit 100 can set the position with the minimum cost among the superpositioned costs as the viewing position and generate a path to the viewing position. For example, the area around wall 530 and within area 584 of area 582 have high costs. Therefore, the mobile unit 100 can, for example, set the position below area 584 within area 582 as the viewing position.

[0064] When the mobile device 100 arrives at a viewing location and begins imaging of the flow of people, it may output an audio or display indicating that it is searching for a specific person. This audio or display may include, for example, the name of the specific person, or it may simply indicate that it is searching.

[0065] Referring to Figure 6, a series of operations performed by the control unit 130 to locate a specific person will be described. This series of operations is achieved by the control unit 130 executing a program stored in the memory device 135, and each part within the control unit 130 executing its own processing. This series of operations can also be performed while the mobile body 100 is autonomously moving.

[0066] In S601, the recognition unit 401 acquires images from the detection unit 114 that were captured around the moving object 100 during a first period. In S602, the recognition unit 401 uses the images captured during the first period to recognize a specific person and other pedestrians among the pedestrians around the moving object 100 and tracks them within the images. As described above, the recognition unit 401 can recognize pedestrians in the images captured by the detection unit 114 using a machine learning model, and distinguish between a specific person and other pedestrians. The recognition unit 401 also assigns identifiers to the specific person and other pedestrians and tracks them within the images.

[0067] In S603, the recognition unit 401 acquires further images captured after the first period and determines whether a state has occurred in which the specific person cannot be recognized in the images. If the recognition unit 401 cannot recognize the specific person in the images, it proceeds to S604; otherwise, it repeats the process in S603.

[0068] In S604, the search action decision unit 403 executes pedestrian search processing (processing to search for a specific person in order to re-recognize them). Details of this process will be described later with reference to Figure 7.

[0069] In S605, the recognition unit 401 acquires further images and determines whether it has recognized a specific person in the images. If the recognition unit 401 determines that it has not recognized a specific person in the images, it proceeds to S606; otherwise, it terminates the process.

[0070] In S606, the view action decision unit 404 executes a process to move to the view location. Details of this process will be described later with reference to Figure 8. In S607, the recognition unit 401 acquires an image captured at the view location and executes a process to recognize a specific person in the image. In S608, the recognition unit 401 determines whether it has recognized a specific person in the image captured at the view location. If the recognition unit 401 determines that it has not recognized a specific person, it proceeds to S606; otherwise, it terminates the process.

[0071] Referring to Figure 7, a series of operations of the pedestrian detection process performed by the control unit 130 will be described. The series of operations of the pedestrian detection process is realized when the control unit 130 executes a program stored in the memory device 135, and each part within the control unit 130 executes its own processing. This process starts when the process in S604 is executed.

[0072] In S701, the search action decision unit 403 identifies (one or more) pedestrians that were recognized after the point at which the specific person could no longer be recognized. As described above, the search action decision unit 403 identifies pedestrians 501, 505, and 506, among the pedestrians shown in Figure 5A, that were recognized after the specific person could no longer be recognized.

[0073] In S702, the behavior prediction unit 402 predicts the future position of the recognized (one or more) pedestrians. As described above, the behavior prediction unit 402 predicts the future position of each of the pedestrians, for example, pedestrian 501, pedestrian 505, and pedestrian 506, relative to the moving object 100.

[0074] In S703, the search action decision unit 403 selects the pedestrian whose predicted future position is closest to the moving object 100 as a candidate. In the example described above, for example, the search action decision unit 403 selects the pedestrian (pedestrian 505) whose predicted future position is closest to the moving object 100 as a candidate.

[0075] In S704, the search action decision unit 403 determines a specific location for recognizing a candidate, and the path generation unit 405 generates a path to move from the current location to that specific location. In S705, the drive control unit 406 controls the movement of the mobile body 100 along the generated path to the specific location for recognizing the candidate.

[0076] In S706, the recognition unit 401 determines whether it has recognized a specific person using an image captured at a specific location for candidate recognition. If the recognition unit 401 determines that it has recognized a specific person, it returns to the caller S604; otherwise, it proceeds to S707.

[0077] In the example shown in Figure 5A, pedestrian 501 is a pedestrian who can no longer be recognized as a specific person. Therefore, when the processing for the pedestrian identified in S701 is performed sequentially, if pedestrian 501 is recognized using the image captured at position 511 (a specific position for recognizing pedestrian 501), the recognition unit 401 determines that a specific person has been recognized.

[0078] In S707, the search action decision unit 403 determines whether it has performed processing on all pedestrians identified in S701 (pedestrian 501, pedestrian 505, and pedestrian 506). If the search action decision unit 403 has performed processing on all pedestrians, it proceeds to S708; otherwise, it returns to S703.

[0079] In this embodiment, the process is described as being repeated for all pedestrians identified in S701. However, this embodiment is not limited to this example. The search action decision unit 403 may proceed to S708 if the path taken to recognize each pedestrian (in the example of Figure 5A, the path to position 512 via positions 510 and 511) exceeds a predetermined length.

[0080] In S708, the search action decision unit 403 moves to a hidden area from the mobile body 100 based on the occupied grid map showing the positions of surrounding obstacles. As described above, the search action decision unit 403 moves to a drivable area that is a blind spot from the mobile body 100 due to obstacles, for example. Then, the recognition unit 401 recognizes a specific person using the image captured in the hidden area (determines whether a specific person is present in the image). The search action decision unit 403 may perform the same processing as in S702 to S706 for pedestrians present in the hidden area. Once the processing in S708 is complete, the search action decision unit 403 returns the process to the caller in S604.

[0081] Referring to Figure 8, a series of operations for moving to the viewpoint position, performed by the control unit 130, will be described. This series of operations for moving to the viewpoint position is achieved by the control unit 130 executing a program stored in the memory device 135, and each part within the control unit 130 executing its own processes. This process begins when the process in S606 is executed.

[0082] In S801, the viewing behavior determination unit 404 calculates, for example, the cost of each position (around the moving object 100) according to the brightness from the captured image. In the example described above in Figure 5C, for example, the cost may be such that the darker the position (e.g., region 584), the higher the cost, and the brighter the position (e.g., outside region 584), the lower the cost.

[0083] In S802, the view behavior determination unit 404 calculates the cost of each location according to the density of pedestrians from the captured image, for example. In the example shown in Figure 5C, for example, the cost is high at locations where there is pedestrian flow within area 581 because the density of pedestrians is high, while the cost is low at other locations.

[0084] In S803, the view behavior determination unit 404 calculates the cost of each location according to the degree of openness of the space. In the example shown in Figure 5C, locations around wall 531 have a high cost, while locations further away from wall 531 have a lower cost.

[0085] In S804, the view action determination unit 404 weights and adds the costs calculated in S801 to S803 to the cost corresponding to the distance from the pedestrian. In S805, the view action determination unit 404 generates a route to the view location. For example, the view action determination unit 404 sets the location with the minimum cost among the superimposed costs as the view location and generates a route to the view location. In S806, the drive control unit 406 controls the movement of the mobile body 100 according to the route to the view location. When the processing in S806 is complete, the view action determination unit 404 returns the process to the caller S606.

[0086] In the above embodiment, all costs calculated in S801 to S803 were used. However, it is also possible to weight the first cost, which is calculated based on the distance from the pedestrian, by adding a second cost, which is calculated based on at least one of the following: a cost based on brightness, a cost based on the density of pedestrians in the captured image, and a cost based on the degree of openness of the space.

[0087] In the above-described embodiment, the control device was explained as being the control unit 130 of the mobile body 100. However, the control device may be included in, for example, an information processing server located outside the mobile body 100. In this case, the control device may perform some or all of the processing performed by the control unit 130 while communicating with the mobile body 100. In other words, this embodiment is also applicable when the mobile body 100 is remotely controlled from a device separate from the mobile body 100.

[0088] As described above, in the above embodiment, the control unit 130 uses images taken around the mobile body 100 during a first period to recognize a specific person and other pedestrians (pedestrians 501-504) among the pedestrians around the mobile body 100. If the control unit 130 can no longer recognize the specific person in the images taken after the first period, it uses the images taken after the first period to predict the future position of a first candidate (e.g., pedestrian 505) among the pedestrians around the mobile body 100 relative to the mobile body 100. The control unit 130 also controls the movement of the mobile body 100 to move from its current position to a specific position (e.g., position 510) for recognizing the first candidate, according to the predicted future position of the first candidate. In this way, if recognition of the specific person fails, the mobile body can take effective actions to find the specific person.

[0089] Furthermore, in this embodiment, the control unit 130 identifies a hidden area based on the occupied grid map and moves the mobile body 100 into the hidden area, and recognizes a specific person using the image captured in the hidden area. The control unit 130 also determines the viewing position using cost superposition and recognizes a specific person using the image captured from the viewing position. By further combining these behavioral controls, if the recognition of a specific person fails, the mobile body can take effective actions to find the specific person.

[0090] <Summary of Embodiments> (Composition 1) A method for controlling a mobile object that moves with a specific person, Using images taken of the area around the moving object during the first period, the system recognizes the specific person and other pedestrians among the pedestrians around the moving object. In the images taken after the first period, if the specific person can no longer be recognized, Using images captured after the first period, predict the future position of a first candidate among the pedestrians surrounding the moving object relative to the moving object, A method for controlling a moving body, comprising controlling the movement of the moving body to move from its current position to a specific position for recognizing the first candidate, depending on the predicted future position of the first candidate.

[0091] In this embodiment, if the recognition of a specific person fails, the moving object can take effective actions to find that person.

[0092] (Configuration 2) The control method according to configuration 1, wherein the specific location is a location from which more parts of the first candidate's entire body can be recognized from the image than the current location.

[0093] In this embodiment, if more parts of the pedestrian's entire body can be recognized, the feature quantities of a registered specific person can be accurately compared with the feature quantities of the pedestrian in the image.

[0094] (Composition 3) The control method according to configuration 1, wherein the specific location is a location where the recognition accuracy of the key points of the first candidate is higher or higher than that of the current location when recognizing the key points of the first candidate from the image.

[0095] In this embodiment, if the recognition accuracy of key points of pedestrians is high, the feature quantities of a registered specific person and the feature quantities of pedestrians in the image can be compared with high accuracy.

[0096] (Composition 4) Predicting the future position of the first candidate relative to the moving object is: Using images taken after the first period, predict the future positions of pedestrians around the moving object relative to the moving object, The control method according to configuration 1, which includes selecting a pedestrian who is closest to the moving object among the predicted future positions as the first candidate.

[0097] In this embodiment, it is possible to efficiently search for pedestrians who are highly likely to be a specific person.

[0098] (Composition 5) Predicting the future position of the first candidate relative to the moving object is: Using images taken after the first period, predict the future positions of pedestrians around the moving object relative to the moving object, A control method according to configuration 1, comprising: estimating a path to sequentially visit predicted future positions; and selecting the first pedestrian to visit while following the path as the first candidate.

[0099] In this embodiment, the search for pedestrians who are highly likely to be a specific person can be optimized.

[0100] (Composition 6) Predicting the future position of the first candidate relative to the moving object is: Other pedestrians who were recognized as distinct from the specific person in the images captured during the first period are excluded from the pedestrians included in the images captured after the first period. The control method according to configuration 1, which includes selecting the first candidate from among the remaining pedestrians excluding the pedestrian in question.

[0101] In this embodiment, pedestrians who are unlikely to be the specific person can be excluded from the search range, thereby reducing the computational cost of searching for the specific person.

[0102] (Composition 7) Predicting the future position of the first candidate relative to the moving object is: Other pedestrians who were recognized as distinct from the specific person in the images captured during the first period are excluded from the pedestrians included in the images captured after the first period. For the remaining pedestrians excluding the pedestrian in question, predict their future positions relative to the moving object, The control method according to configuration 1, which includes selecting a pedestrian who is closest to the moving object among the predicted future positions as the first candidate.

[0103] In this embodiment, pedestrians who are unlikely to be the specified person can be excluded from the search range, thereby reducing the computational cost of predicting the future location of pedestrians.

[0104] (Composition 8) The control method according to configuration 1, further comprising moving to the specific location corresponding to the predicted future location of the first candidate, and then recognizing the specific person using an image containing the first candidate.

[0105] In this embodiment, the moving object can take effective actions to locate a specific person.

[0106] (Composition 9) If, after moving to the specific location corresponding to the predicted future location of the first candidate, it is determined that the specific person does not exist using an image containing the first candidate, Using images taken after the first period, predict the future position of a second candidate among the pedestrians surrounding the moving object relative to the moving object, The control method according to configuration 1, further comprising controlling the movement of the moving body to move to a specific location for recognizing the second candidate, depending on the predicted future location of the second candidate.

[0107] In this embodiment, pedestrians who are highly likely to be a specific person can be searched for sequentially.

[0108] (Composition 10) If, after moving to the specific location corresponding to the predicted future location of the first candidate, it is determined that the specific person does not exist using an image containing the first candidate, Based on the occupied grid map showing the location of obstacles around the moving body, the moving body moves into a hidden area from the moving body. The control method according to configuration 1, further comprising moving to the hidden area and recognizing the specific person using the image captured.

[0109] In this embodiment, the search range can be expanded from around the moving object to another area where a specific person may be present.

[0110] (Composition 11) If, after moving to the specific location corresponding to the predicted future location of the first candidate, it is determined that the specific person does not exist using an image containing the first candidate, To drive to a designated location for finding the aforementioned specific person, The control method according to configuration 1, further comprising recognizing the specific person using an image captured at the predetermined location.

[0111] In this embodiment, if a specific person cannot be found around the moving object, the object can be searched for at a suitable vantage point for finding the specific person.

[0112] (Composition 12) The control method according to configuration 11, wherein the predetermined location for finding the specific person is a location where the density of pedestrians is below a predetermined threshold, and includes a location where an area with a higher density of pedestrians than that location can be imaged.

[0113] In this embodiment, it is possible to search for a specific person within a crowd from a location away from the crowd.

[0114] (Composition 13) If, after moving to the hidden area, it is determined that the specific person does not exist using the image taken after moving to the hidden area, To drive to a designated location for finding the aforementioned specific person, The control method according to configuration 10, further comprising recognizing the specific person using an image captured at the predetermined location.

[0115] In this embodiment, the search range is expanded to include areas where a specific person may be present, and then the specific person can be searched for at a viewpoint suitable for finding them.

[0116] (Composition 14) The control method according to configuration 11, further comprising determining a predetermined position for finding the specific person using a cost obtained by weighting and adding a first cost for each position around the moving body according to the distance from the pedestrian, with a second cost of at least one of the following: the cost for each position around the moving body according to the brightness, the cost for each position around the moving body according to the density of pedestrians in the captured image, and the cost for each position around the moving body according to the degree of openness of the space.

[0117] In this embodiment, the mobile body can be moved to a significant location for detecting a specific person.

[0118] (Composition 15) The control method according to configuration 1, further comprising outputting an audio or display indicating that the specific person is being searched for in an image taken after the first period, in response to the inability to recognize the specific person.

[0119] In this embodiment, it becomes easier for a specific person to spot the moving object.

[0120] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention. [Explanation of Symbols]

[0121] 100 ···Mobile unit, 113 ···Front display unit, 115 ···Audio output unit, 116 ···Light-emitting unit, 130 ···Control unit, 140 ···Communication terminal, 401 ···Recognition unit, 402 ···Action prediction unit, 403 ···Search action decision unit, 404 ···View action decision unit, 405 ···Route generation unit, 406 ···Drive control unit

Claims

1. A method for controlling a mobile object that moves with a specific person, Using images taken of the area around the moving object during the first period, the system recognizes the specific person and other pedestrians among the pedestrians around the moving object. In the images taken after the first period, if the specific person can no longer be recognized, Using images taken after the first period, predict the future position of a first candidate among the pedestrians surrounding the moving object relative to the moving object, A method for controlling a moving body, comprising controlling the movement of the moving body to move from its current position to a specific position for recognizing the first candidate, depending on the predicted future position of the first candidate.

2. The control method according to claim 1, wherein the specific location is a location from which more parts of the first candidate's entire body can be recognized from the image than the current location.

3. The control method according to claim 1, wherein the specific position is a position where the recognition accuracy of the key points of the first candidate is higher or higher than that of the current position when recognizing the key points of the first candidate from the image.

4. Predicting the future position of the first candidate relative to the moving object is: Using images taken after the first period, predict the future positions of pedestrians around the moving object relative to the moving object, The control method according to claim 1, further comprising selecting a pedestrian who is closest to the moving object among the predicted future positions as the first candidate.

5. Predicting the future position of the first candidate relative to the moving object is: Using images taken after the first period, predict the future positions of pedestrians around the moving object relative to the moving object, The control method according to claim 1, comprising estimating a path to sequentially visit predicted future locations, and selecting the first pedestrian to visit along the path as the first candidate.

6. Predicting the future position of the first candidate relative to the moving object is: Other pedestrians who were recognized as distinct from the specific person in the images captured during the first period are excluded from the pedestrians included in the images captured after the first period. The control method according to claim 1, further comprising selecting the first candidate from among the remaining pedestrians excluding the pedestrian in question.

7. Predicting the future position of the first candidate relative to the moving object is: Other pedestrians who were recognized as distinct from the specific person in the images captured during the first period are excluded from the pedestrians included in the images captured after the first period. For the remaining pedestrians excluding the pedestrian in question, predict their future positions relative to the moving object, The control method according to claim 1, further comprising selecting a pedestrian who is closest to the moving object among the predicted future positions as the first candidate.

8. The control method according to claim 1, further comprising moving to the specific location corresponding to the predicted future location of the first candidate, and then recognizing the specific person using an image containing the first candidate.

9. If, after moving to the specific location corresponding to the predicted future location of the first candidate, it is determined that the specific person does not exist using an image containing the first candidate, Using images taken after the first period, predict the future position of a second candidate among the pedestrians surrounding the moving object relative to the moving object, The control method according to claim 1, further comprising controlling the movement of the moving body to move to a specific location for recognizing the second candidate, depending on the predicted future location of the second candidate.

10. If, after moving to the specific location corresponding to the predicted future location of the first candidate, it is determined that the specific person does not exist using an image containing the first candidate, Based on the occupied grid map showing the location of obstacles around the moving body, the moving body moves into a hidden area from the moving body. The control method according to claim 1, further comprising moving to the hidden area and recognizing the specific person using an image captured.

11. If, after moving to the specific location corresponding to the predicted future location of the first candidate, it is determined that the specific person does not exist using an image containing the first candidate, To drive to a designated location for finding the aforementioned specific person, The control method according to claim 1, further comprising recognizing the specific person using an image captured at the predetermined position.

12. The control method according to claim 11, wherein the predetermined location for discovering the specific person is a location where the density of pedestrians is below a predetermined threshold, and includes a location where an area with a higher density of pedestrians than that location can be imaged.

13. If, after moving to the hidden area, it is determined that the specific person does not exist using the image taken after moving to the hidden area, To drive to a designated location for finding the aforementioned specific person, The control method according to claim 10, further comprising recognizing the specific person using an image captured at the predetermined location.

14. The control method according to claim 11, further comprising determining a predetermined position for finding the specific person using a cost obtained by weighting and adding a first cost for each position around the moving body according to the distance from the pedestrian, with a second cost of at least one of the following: the cost for each position around the moving body according to the brightness, the cost for each position around the moving body according to the density of pedestrians in the captured image, and the cost for each position around the moving body according to the degree of openness of the space.

15. The control method according to claim 1, further comprising outputting an audio or display indicating that the specific person is being searched for in an image taken after the first period, in response to the inability to recognize the specific person.

16. A control device for a mobile body that moves with a specific person, A recognition means that uses images taken of the area around the moving object during a first period to recognize the specific person and other pedestrians among the pedestrians around the moving object, If, in an image taken after the first period, the specific person can no longer be recognized, A prediction means that uses images taken after the first period to predict the future position of a first candidate among pedestrians around the moving object relative to the moving object, A control device for a moving body, comprising: control means for controlling the movement of the moving body to move from its current position to a specific position for recognizing the first candidate, depending on the predicted future position of the first candidate.

17. A mobile entity that travels with a specific person, A recognition means that uses images taken of the area around the moving object during a first period to recognize the specific person and other pedestrians among the pedestrians around the moving object, If, in an image taken after the first period, the specific person can no longer be recognized, A prediction means that uses images taken after the first period to predict the future position of a first candidate among pedestrians around the moving object relative to the moving object, A mobile body comprising control means for controlling the movement of the mobile body to move from its current position to a specific position for recognizing the first candidate, depending on the predicted future position of the first candidate.

18. A program for causing a computer to perform each step of the control method described in any one of claims 1 to 15.

19. A storage medium storing a program for causing a computer to perform each step of the control method described in any one of claims 1 to 15.

Images