Mobile body control system, mobile body, and mobile body control method

Abstract

To provide a mobile object control system which efficiently respond to remaining persons of a disaster area, etc. by using a mobile object, the mobile object and a mobile object control method.SOLUTION: In a mobile object control system, a mobile object 30 calls a remaining person and approaches an object detected as the remaining person to determine whether or not the object is a person when the remaining person does not evacuate. Then, the mobile object 30 further approaches the remaining person to determine whether or not the person is lying when the object determined as the person is the remaining person. The mobile object 30 performs a predetermined operation instruction when the person is not lying and notifies a mobile object control device of a rescue request since there are possibilities of a machine trouble, or that the person cannot perform a scheduled operation or cannot understand notification content due to a problem of a language, etc. when the person does not perform a predetermined operation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a mobile object control system, a mobile object, and a mobile object control method that can efficiently respond to people remaining in a disaster area or the like using a mobile object. [Background technology]

[0002] Conventionally, when releasing water from a dam, prior warnings are given to downstream residents using sirens and electronic signs, in accordance with the Specified Multipurpose Dam Act. However, even with such warnings, there is a possibility that people may remain in areas at risk of flooding due to the rise in river water caused by the dam's release. For this reason, local governments, even in the midst of a tight disaster response operation, are patrolling river channels to monitor those remaining, regardless of the risk. Such patrols of those remaining are carried out not only when dams release water, but also when there is a possibility of a disaster occurring due to natural phenomena such as earthquakes or storms.

[0003] In order to avoid such a situation, it is conceivable to use multiple unmanned aerial vehicles (drones) to patrol areas where a disaster may occur (hereinafter referred to as "disaster areas") and provide guidance to remaining people, etc. For example, Patent Document 1 discloses a technology for using multiple unmanned aerial vehicles (drones) to track targets such as suspects. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-061082 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, since information about the tracking target needs to be handed over between base stations, there is a problem that the remaining person cannot be tracked in places where there are no base stations, such as a river channel located in an area downstream before the dam releases water. Also, even if the remaining person who is the tracking target is detected from the image and a call is made to urge the remaining person to evacuate, there is no guarantee that the remaining person will understand the call.

[0006] For example, there may be cases where the remaining people cannot understand the call due to some reason such as the language used, or are unable to evacuate due to injury, or where a human-shaped signboard or the like is mistakenly detected as a remaining person and a call is made to them, or where the call does not reach the remaining people in the first place due to surrounding noise, etc. Unless these situations are judged and a prompt response is taken, it will be impossible to efficiently and reliably guide the remaining people to evacuate, and their safety cannot be ensured.

[0007] Given these circumstances, how to use drones to efficiently respond to those remaining in disaster areas has become an important issue. This issue arises not only when using drones, but also when using mobile devices such as robots, and when responding to earthquakes and other disasters rather than dam releases.

[0008] The present invention has been made to solve the problems (issues) associated with the prior art described above, and aims to provide a mobile object control system, a mobile object, and a mobile object control method that can efficiently respond to people remaining in disaster areas, etc. using mobile objects. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the present invention provides a mobile body control system having a mobile body that patrols a predetermined area and a mobile body control device that controls the mobile body, wherein the mobile body is area an imaging means for capturing an image; and a person determination means for determining whether or not a person is included in the image captured by the imaging means; a posture estimation means for estimating a posture of a person when the person determination means determines that the image includes a person;a notification means for issuing a notification including a request for a predetermined action to the person when it is determined that the image includes a person by at least the person determination means; a motion determination means for determining whether or not the motion of the person included in the image captured by the imaging means is the predetermined motion; R and a control means for performing control according to the determination result. The notification means issues a notification including a request for a predetermined action to the person when the posture estimation means estimates that the person has not fallen. It is characterized by:

[0012] In addition, in the above invention, the present invention is characterized in that the control means controls the mobile body control device to request rescue for the person when the posture estimation means estimates that the person has fallen.

[0013] The present invention also provides A mobile body control system having a mobile body that patrols a predetermined area and a mobile body control device that controls the mobile body, wherein the mobile body comprises: an imaging means that captures an image of the predetermined area; a person determination means that determines whether or not a person is included in the image captured by the imaging means; a notification means that issues a notification to the person, including a request for the person to perform a predetermined action, on condition that at least the person determination means has determined that the image contains a person; an action determination means that determines whether or not the action of the person included in the image captured by the imaging means is the predetermined action; and a control means that performs control in accordance with the determination result by the action determination means. The control means is characterized in that, when the movement determination means determines that the movement of the person is the specified movement, it controls the aircraft and issues an alert to guide the person to evacuate from the specified area.

[0014] The present invention also provides A mobile body control system having a mobile body that patrols a predetermined area and a mobile body control device that controls the mobile body, wherein the mobile body comprises: an imaging means that captures an image of the predetermined area; a person determination means that determines whether or not a person is included in the image captured by the imaging means; a notification means that issues a notification to the person, including a request for the person to perform a predetermined action, on condition that at least the person determination means has determined that the image contains a person; an action determination means that determines whether or not the action of the person included in the image captured by the imaging means is the predetermined action; and a control means that performs control in accordance with the determination result by the action determination means. The control means controls the mobile object control device to request assistance when the movement determination means determines that the movement of the person is not the predetermined movement.

[0015] The present invention also provides a mobile body for patrolling a predetermined area, the mobile body comprising: area an imaging means for capturing an image; and a person determination means for determining whether or not a person is included in the image captured by the imaging means; a posture estimation means for estimating a posture of a person when the person determination means determines that the image includes a person; a notification means for issuing a notification including a request for a predetermined action to the person when it is determined that the image includes a person by at least the person determination means; a motion determination means for determining whether or not the motion of the person included in the image captured by the imaging means is the predetermined motion; R and a control means for performing control according to the determination result. The notification means issues a notification including a request for a predetermined action to the person when the posture estimation means estimates that the person has not fallen. It is characterized by: The present invention also provides a mobile body that patrols a predetermined area, comprising: an imaging means for capturing an image of the predetermined area; a person determination means for determining whether or not a person is included in the image captured by the imaging means; a notification means for issuing a notification to the person including a request for the person to perform a predetermined action, on condition that at least the person determination means has determined that the image contains a person; a motion determination means for determining whether or not the motion of the person included in the image captured by the imaging means is the predetermined motion; and a control means for performing control in accordance with the determination result by the motion determination means, wherein the control means controls the vehicle and issues a notification to guide the person to evacuate from the predetermined area when the motion determination means has determined that the motion of the person is the predetermined motion. The present invention also provides a mobile body that patrols a predetermined area, comprising an imaging means for capturing an image of the predetermined area, a person determination means for determining whether or not a person is included in the image captured by the imaging means, a notification means for issuing a notification to the person including a request for the person to perform a predetermined action, on condition that at least the person determination means has determined that the image contains a person, a motion determination means for determining whether or not the motion of the person included in the image captured by the imaging means is the predetermined motion, and a control means for performing control in accordance with the determination result by the motion determination means, wherein the control means controls to request rescue when the motion determination means determines that the motion of the person is not the predetermined motion.

[0016] The present invention also provides a mobile body control method in a mobile body control system having a mobile body that patrols a predetermined area and a mobile body control device that controls the mobile body, wherein the mobile body patrols a predetermined area. area an imaging step of capturing an image; and a person determination step of the moving body determining whether or not a person is included in the image captured by the imaging step; a posture estimation step of estimating a posture of the person if it is determined that the image includes a person by the person determination step; a notification step in which the moving body issues a notification including a request for a predetermined action to the person on condition that it has been determined that a person is included in the image by at least the person determination step; a motion determination step in which the moving body determines whether or not the motion of the person included in the image captured by the imaging step is the predetermined motion; R and a control step for performing control according to the determination result. and the notifying step, when it is estimated by the posture estimating step that the person has not fallen, issues a notification to the person including a request to perform a predetermined action. It is characterized by: The present invention also provides a mobile body control method in a mobile body control system having a mobile body that patrols a predetermined area and a mobile body control device that controls the mobile body, the method including: an imaging process in which the mobile body captures an image of the predetermined area; a person determination process in which the mobile body determines whether a person is included in the image captured by the imaging process; a notification process in which the mobile body issues a notification to the person including a request for the person to perform a predetermined action, on the condition that at least the person determination process has determined that the image contains a person; a motion determination process in which the mobile body determines whether the motion of the person included in the image captured by the imaging process is the predetermined motion; and a control process in which the mobile body performs control in accordance with the determination result of the motion determination process, wherein the control process controls the vehicle and issues a notification to guide the person to evacuate from the predetermined area when the motion determination process has determined that the motion of the person is the predetermined motion. The present invention also provides a mobile body control method in a mobile body control system having a mobile body that patrols a predetermined area and a mobile body control device that controls the mobile body, comprising: an imaging process in which the mobile body captures an image of the predetermined area; a person determination process in which the mobile body determines whether a person is included in the image captured by the imaging process; a notification process in which the mobile body issues a notification to the person including a request for the person to perform a predetermined action, on the condition that at least the person determination process has determined that the image contains a person; an action determination process in which the mobile body determines whether the action of the person included in the image captured by the imaging process is the predetermined action; and a control process in which the mobile body performs control in accordance with the determination result of the action determination process, wherein the control process controls the mobile body control device to request rescue when the action determination process determines that the action of the person is not the predetermined action. [Effects of the Invention]

[0017] According to the present invention, it is possible to efficiently respond to people remaining in disaster areas, etc., using mobile vehicles. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is an explanatory diagram for explaining an overview of a mobile object control system according to this embodiment. [Figure 2] FIG. 2 is a diagram showing the system configuration of the mobile object control system shown in FIG. [Figure 3] FIG. 3 is a functional block diagram showing the configuration of the moving body shown in FIG. [Figure 4] FIG. 4 is an explanatory diagram for explaining the determination of a person using the trained model for person detection shown in FIG. [Figure 5] FIG. 5 is a flowchart showing the processing procedure of the moving body shown in FIG. [Figure 6]FIG. 6 is a flowchart showing the procedure of the person detection process. [Figure 7] FIG. 7 is a flowchart (part 1) showing the procedure of the re-call process. [Figure 8] FIG. 8 is a flowchart (part 2) showing the processing procedure of the re-call process. [Figure 9] FIG. 9 is a flowchart showing the procedure of the collapse detection process. [Figure 10] FIG. 10 is a flowchart showing the procedure of the motion detection process. [Figure 11] FIG. 11 is a flowchart showing the procedure of the evacuation guidance process. DETAILED DESCRIPTION OF THE INVENTION

[0019] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a mobile object control system, a mobile object, and a mobile object control method according to the present invention will be described in detail with reference to the accompanying drawings.

[0020] <Overview of the mobile control system> First, an overview of the mobile object control system according to this embodiment will be described. FIG. 1 is an explanatory diagram for explaining the overview of the mobile object control system according to this embodiment. As shown in FIG. 1, this mobile object control system is a system that can efficiently respond to people remaining in areas where a disaster such as a river may occur using a mobile object. Here, the explanation will be centered on the case where an unmanned aerial vehicle called a drone is used as the mobile object.

[0021] The mobile body 30 of this embodiment is an unmanned aerial vehicle that can fly autonomously under the control of a mobile body control device (not shown). In other words, the movement of the mobile body 30 is not controlled by an operator, but is autonomously controlled in response to instructions from the mobile body control device.

[0022] The mobile object 30 receives a movement instruction to the disaster area from a mobile object control device (not shown) and moves to the disaster area. The mobile object 30 then determines whether or not there are any remaining people, and if it determines that there are any remaining people, calls out to the remaining people for evacuation guidance (S1).

[0023] Thereafter, if the remaining person does not move, the moving body 30 approaches the remaining person and determines whether the object detected as the remaining person is a person (S2). If the moving body 30 determines that the detected object is a person, it approaches the remaining person further and determines whether the person has fallen (S3).

[0024] If the mobile body 30 determines that the person has not collapsed, it instructs the remaining person to take a predetermined action (S4). If the remaining person takes the predetermined action, it guides the remaining person to evacuate, but if the remaining person does not take the predetermined action, it notifies the mobile body control device of a request for rescue (S5).

[0025] <System configuration of mobile control system> Next, the system configuration of the mobile object control system according to this embodiment will be described. Fig. 2 is a diagram showing the system configuration of the mobile object control system shown in Fig. 1. As shown in Fig. 2, in this mobile object control system, a server device 10, a mobile object control device 20, and a plurality of mobile objects 30a, 30b, and 30c (hereinafter, sometimes collectively referred to as "mobile objects 30") are connected to a network N.

[0026] The server device 10 is a device for performing supervised learning of a trained model for human detection and a trained model for action detection that are mounted on a moving body 30, which will be described later.

[0027] The mobile body control device 20 transmits travel instructions including travel route data and voice messages for evacuation guidance to the mobile body 30 in the event of a disaster. It also receives a notice of a rescue request from the mobile body 30 and processes the notice for display.

[0028] When the mobile body 30 receives a movement instruction from the mobile body control device 20, it moves to the disaster area according to the movement route data included in the movement instruction. When the mobile body 30 arrives at the disaster area, it captures images of the surrounding area using its own imaging unit and determines whether any remaining people are present. If any remaining people are present, the mobile body 30 provides evacuation guidance to the remaining people. The mobile body 30 also performs the following processes: (1) determining its own position information using signals from an altitude sensor and a GNSS (Global Navigation Satellite System) unit and controlling the rotors to move; (2) capturing images of the surrounding area using the imaging unit and detecting people; (3) issuing instructions to the remaining people and determining whether they have performed the specified actions; and (4) providing evacuation guidance to the remaining people.

[0029] <Configuration of moving body 30a> Next, the configuration of the mobile object 30a shown in Fig. 2 will be described. Fig. 3 is a functional block diagram showing the configuration of the mobile object 30a shown in Fig. 2. Note that the mobile objects 30b and 30c also have a similar configuration. As shown in Fig. 3, the mobile object 30a has an imaging unit 31, an altitude sensor 32, a GNSS unit 33, a notification unit 34, a wireless communication unit 35, rotors 36, a memory unit 37, and a control unit 38.

[0030] The imaging unit 31 is an imaging device that uses a camera to capture images of the disaster area. The camera mounted on the imaging unit 31 may be capable of simultaneously capturing visible light images and infrared light images of the same imaging range, or may be capable of capturing only one of them.

[0031] The altitude sensor 32 is a sensor that measures the altitude of the mobile unit 30a. For example, a barometric pressure sensor can be used as this altitude sensor 32, or a barometric pressure sensor and an ultrasonic sensor can be combined. This allows the mobile unit 30a to stably maintain its altitude close to the ground. The GNSS unit 33 is a unit that receives signals from multiple GNSS satellites and identifies the position coordinates of its own device.

[0032] The notification unit 34 is a processing unit that notifies the remaining people of evacuation guidance data 37d received from the evacuation guidance unit 38b (described later). For example, the notification unit 34 can notify the remaining people with a synthesized alarm message sound, an alarm sound, and / or light emission. The notification unit 34 is also a processing unit that notifies the remaining people with a message of a predetermined action instructing the remaining people, received from the notification processing unit 38d (described later).

[0033] The wireless communication unit 35 is an interface unit for wireless communication with the mobile body control device 20. The multiple rotors 36 rotate individually under the control of the control unit 38, enabling attitude control and movement of the mobile body 30a.

[0034] The memory unit 37 is a storage device such as a hard disk drive or non-volatile memory, and stores a trained model for person detection 37a, a trained model for action detection 37b, movement route data 37c, evacuation guidance data 37d, and image data 37e.

[0035] The trained model for person detection 37a is a trained model that determines whether or not a person is present in an image captured by the imaging unit 31. The trained model for motion detection 37b is a trained model that determines whether or not a person is performing a predetermined motion in an image captured by the imaging unit 31. The trained model for motion detection 37b also determines whether or not a person has fallen, based on the image captured by the imaging unit 31. The trained model for person detection 37a and the trained model for motion detection 37b are generated by performing training in the server device 10 using a large amount of training data and correct answer data.

[0036] The travel route data 37c is travel route data received from the mobile object control device 20, and the mobile object 30a travels according to this travel route data 37c. The evacuation guidance data 37d is data such as a voice message for providing evacuation guidance received from the mobile object control device 20. The image data 37e is image data captured by the imaging unit 31.

[0037] The control unit 38 is a control unit that controls the entire moving body 30a, and has an imaging control unit 38a, an evacuation guidance unit 38b, a person determination unit 38c, a notification processing unit 38d, a motion determination unit 38e, and a movement control unit 38f. In practice, by loading these programs into a CPU and executing them, the imaging control unit 38a, the evacuation guidance unit 38b, the person determination unit 38c, the notification processing unit 38d, the motion determination unit 38e, and the movement control unit 38f will execute their respective corresponding processes.

[0038] The imaging control unit 38a controls the imaging unit 31 to capture images of the disaster area, and stores the captured images in the storage unit 37 as image data 37e.

[0039] The evacuation guidance unit 38b receives data such as a voice message for evacuation guidance transmitted from the mobile object control device 20, and stores the data as evacuation guidance data 37d in the storage unit 37. The evacuation guidance unit 38b also outputs the stored evacuation guidance data 37d to the notification unit 34.

[0040] The person determination unit 38c performs processing to determine whether or not a person is present in an image based on an image captured by controlling the imaging unit 31. Specifically, as shown in Fig. 4, image data 37e captured by the imaging unit 31 is input to the trained model for person detection 37a, and the person probability (0 to 1) output from the trained model for person detection 37a is obtained. Based on this value, if the person probability is greater than a predetermined value (explained as 0.8 in this embodiment), it is determined to be a person, and if the person probability is 0.8 or less, it is determined to be not a person.

[0041] The notification processing unit 38d is a processing unit that determines that an object detected as a remaining person by the moving body 30a is a person, and if the action determination unit 38e described below determines that the object has not fallen, it notifies the person by voice with action instructions to have them perform a specified action.

[0042] The movement determination unit 38e estimates the posture of the person in the image based on the image data 37e captured by controlling the imaging unit 31, and determines whether the person has fallen. Furthermore, when the notification processing unit 38d instructs the person to move, it performs processing to determine whether the person in the image has performed a predetermined movement (the instructed movement).

[0043] The movement control unit 38f is a control unit that controls the movement of the aircraft by controlling the multiple rotors 36 using the movement route data 37c, the altitude information output by the altitude sensor 32, and the position information output by the GNSS unit 33. Specifically, the movement control unit 38f controls the aircraft to move according to the movement route data 37c. Furthermore, if the remaining people have not evacuated, the movement control unit 38f determines that there is a possibility that the evacuation guidance message or the like has not been received, and controls the aircraft to lower its altitude to approach the remaining people so that the remaining people can easily hear the alarm sound from the alarm unit 34.

[0044] <Processing Procedure of Moving Body 30a> Next, a description will be given of the processing procedure of the moving body 30a shown in Fig. 3. Fig. 5 is a flowchart showing the processing procedure of the moving body 30a shown in Fig. 3. Note that the same processing is performed for the moving bodies 30b and 30c.

[0045] As shown in Fig. 5, the mobile object 30a moves to the disaster area according to the travel route data 37c stored in the storage unit 37 (step S101). Then, the mobile object 30a detects whether or not there are any people remaining in the disaster area (step S102). Specifically, the mobile object 30a inputs the image captured by the imaging unit 31 into the trained model for person detection 37a to determine whether or not a person is present in the image, and acquires a person probability. If the person probability is greater than 0.8, it is determined that a person is present, and if the person probability is 0.8 or less, it is determined that a person is not present.

[0046] If it is determined that no person is present (step S103; No), the moving object 30a returns to the moving route, continues patrolling (step S107), and ends the series of processes.

[0047] On the other hand, if the mobile body 30a determines that a person is present (step S103; Yes), it determines that the person is a remainder and notifies the notification unit 34 of the evacuation guidance data 37d (step S104). Then, the mobile body 30a determines whether the remainder has evacuated (step S105). Specifically, after notifying the evacuation guidance data 37d, it tracks the remainder, and if the change in the position information acquired from the GNSS 33 is in the direction of the evacuation location, it determines that the remainder has evacuated, and if the change is not in the direction of the destination location or there is no change, it determines that the remainder has not evacuated.

[0048] If the moving object 30a determines that the person has not evacuated (step S105; No), it performs a re-call process (step S108) and proceeds to step S107.

[0049] On the other hand, if the moving body 30a determines that the people have evacuated (step S105; Yes), it determines whether the evacuation of the remaining people has been completed (step S106). Specifically, it determines whether the evacuation of the remaining people has been completed based on whether the location information acquired from the GNSS 33 indicates that the remaining people have moved to an evacuation site.

[0050] If the remaining people have not completed evacuation (step S106; No), the moving body 30a proceeds to step S104 and continues evacuation guidance. On the other hand, if the moving body 30a determines that the remaining people have completed evacuation (step S106; Yes), it returns to the movement route, continues patrol (step S107), and ends the series of processes.

[0051] <Person detection processing procedure> Next, the person detection process will be described. Fig. 6 is a flowchart showing the procedure of the person detection process. As shown in Fig. 6, the moving object 30a controls the imaging unit 31 to capture an image (step S201).

[0052] Then, the moving body 30a calculates the probability that a person is included in the image based on the captured image (step S202). Specifically, the captured image is input to the trained model for person detection 37a to obtain the person probability. Thereafter, the moving body 30a determines whether the person probability output from the trained model for person detection 37a is greater than a predetermined value (step S203).

[0053] If the person probability is greater than a predetermined value (step S203; Yes), the moving body 30a determines that a person is included in the image (step S204) and returns to the caller. On the other hand, if the person probability is equal to or less than a predetermined value (step S203; No), the moving body 30a determines that a person is not included in the image (step S205) and returns to the caller.

[0054] <Recall processing procedure> Next, the re-call process will be described. Figures 7 and 8 are flowcharts showing the processing procedure of the re-call process. As shown in Figure 7, the moving body 30a calls the remaining people to evacuate, and if the remaining people do not move, it lowers its altitude and changes the shooting angle (step S301).

[0055] Thereafter, the moving body 30a performs a person detection process to determine whether the object recognized as a remaining person is a person (step S302). Specifically, the moving body 30a inputs the captured image captured by the imaging unit 31 into the trained model for person detection 37a to determine whether a person is present in the image and obtains a person probability. If the person probability is greater than 0.8, it is determined that a person is present, and if the person probability is 0.8 or less, it is determined that a person is not present. Then, if it is determined that a person is not included as a result of the person detection process (step S303; No), it is determined that an object that is not a person has been erroneously detected as a person, and the process proceeds to step S107 in FIG. 5.

[0056] On the other hand, if it is determined that a person is included as a result of person detection (step S303; Yes), it determines that the person is a remainder, but that there is a possibility that the remainder has not received the evacuation guidance message, and lowers the altitude of the moving body 30a (step S304). Thereafter, the moving body 30a performs a fall detection process to determine whether the remainder has fallen, in order to determine the possibility that the remainder has fallen and is unable to move (step S305). Specifically, the image captured by the imaging unit 31 is input to the trained model for motion detection 37b, and the probability that the remainder has fallen is obtained. If the probability of falling is greater than a predetermined value, it is determined that the remainder has fallen, and if the probability of falling is equal to or less than the predetermined value, it is determined that the remainder has not fallen.

[0057] If the moving body 30a determines that a remaining person has fallen as a result of the fall detection (step S306; Yes), it notifies the moving body control device 20 of a rescue request (step S307) and proceeds to S107 in FIG.

[0058] On the other hand, if it is determined as a result of the fall detection that the remaining person has not fallen (step S306; No), the moving body 30a further lowers its altitude (step S308). Then, the moving body 30a performs a motion detection process including issuing motion instructions to the remaining person (step S309). Specifically, the moving body 30a instructs the remaining person to perform a predetermined motion, such as waving their hand. Then, the image captured by the imaging unit 31 is input to the trained model for motion detection 37b, and the probability that the predetermined motion was performed in accordance with the instruction is obtained. If the probability of performing the predetermined motion is greater than a predetermined threshold, the moving body 30a determines that the predetermined motion was performed, and if the probability of performing the predetermined motion is equal to or less than the predetermined threshold, the moving body 30a determines that the predetermined motion was not performed.

[0059] When it is determined that the moving object 30a has performed a predetermined movement as a result of the movement detection (step S310; Yes), the moving object 30a performs evacuation guidance processing (step S311) and returns to step S107 in FIG.

[0060] On the other hand, if the mobile body 30a determines as a result of the motion detection that the specified motion is not being performed (step S310: No), it notifies the mobile body control device 20 of a request for rescue (step S312) and proceeds to S107 in Figure 5, since it is highly likely that the remaining person is unable to perform the specified motion due to equipment trouble or injury, which prevents the alarm unit 34 from making a voice call, or that the remaining person is unable to understand the content of the alarm due to language or other issues.

[0061] <Fall detection processing procedure> Next, the procedure of the fall detection process will be described. Fig. 9 is a flowchart showing the procedure of the fall detection process. As shown in Fig. 9, the moving body 30a controls the imaging unit 31 to capture an image (step S401).

[0062] Then, the moving body 30a calculates the probability that the remaining person in the image has fallen based on the captured image (step S402). Specifically, the captured image is input to the trained model for motion detection 37b, and the posture of the remaining person is determined based on skeletal information, etc., to obtain the probability that the remaining person has fallen. Thereafter, the moving body 30a determines whether the probability that the remaining person has fallen output from the trained model for motion detection 37b is greater than a predetermined value (step S403).

[0063] If the probability that the remaining person has fallen is greater than a predetermined value (step S403; Yes), the moving body 30a determines that the remaining person in the image has fallen (step S404), and returns to step S306 in Fig. 7. On the other hand, if the probability that the remaining person has fallen is equal to or less than a predetermined value (step S403; No), the moving body 30a determines that the remaining person in the image has not fallen (step S405), and returns to step S306 in Fig. 7.

[0064] <Procedure for motion detection processing> Next, the processing procedure of the motion detection process will be described. Fig. 10 is a flowchart showing the processing procedure of the motion detection process. As shown in Fig. 10, the moving object 30a makes a call including a predetermined motion instruction (step S501). Here, the predetermined motion instruction is, for example, an instruction such as "Wave both hands above your head."

[0065] The moving body 30a controls the imaging unit 31 to capture an image (step S502). Then, the moving body 30a calculates the probability that a person in the image has performed a predetermined action based on the captured image (step S503). Specifically, the captured image is input to the trained model for action detection 37b, and the probability that a remaining person in the image has performed the predetermined action is obtained. Thereafter, the moving body 30a determines whether the probability of performing the predetermined action output from the trained model for action detection 37b is greater than a predetermined value (step S504).

[0066] If the probability that the remaining person performed the predetermined action is greater than a predetermined value (step S504; Yes), the moving body 30a determines that the remaining person in the image performed the predetermined action because they heard the message and understood its contents (step S505), and returns to step S310 in Fig. 8. On the other hand, if the probability that the remaining person performed the predetermined action is equal to or less than a predetermined value (step S504; No), the moving body 30a determines that the remaining person in the image did not perform the predetermined action because the message did not reach the remaining person due to equipment trouble, inability to move due to injury, or inability to understand the language, or because the remaining person did not understand the message (step S506), and returns to step S310 in Fig. 8.

[0067] <Procedure for evacuation guidance> Next, the procedure of the evacuation guidance process will be described. Fig. 11 is a flowchart showing the procedure of the evacuation guidance process. As shown in Fig. 11, the moving object 30a notifies the evacuation guidance data 37d from the notifying unit 34 (step S601).

[0068] The mobile body 30a determines whether the remaining people are evacuating (step S602). Specifically, after reporting the evacuation guidance data 37d, the mobile body 30a tracks the remaining people, and if a change in the position information acquired from the GNSS 33 indicates the direction of an evacuation site, the mobile body 30a determines that the remaining people are evacuating. Then, if the mobile body 30a determines that the remaining people are evacuating (step S602; Yes), the mobile body 30a determines whether the evacuation of the remaining people is complete (step S603). Specifically, the mobile body 30a determines whether the evacuation of the remaining people is complete depending on whether the position information acquired from the GNSS 33 indicates that the remaining people have moved to an evacuation site.

[0069] On the other hand, if the moving body 30a determines that the remaining people have not evacuated (step S602; No), it notifies the moving body control device 20 of a rescue request (step S604) and returns to step S107 in FIG.

[0070] If the person has not completed evacuation (step S603; No), the moving body 30a proceeds to step S601 and continues evacuation guidance. On the other hand, if the moving body 30a determines that the person has completed evacuation (step S603; Yes), the moving body 30a returns to step S107 in FIG.

[0071] As described above, in this embodiment, the mobile body 30a calls out to the remaining people, and if the remaining people do not evacuate, the mobile body 30a approaches the object detected as the remaining person and determines whether it is a person. If it is determined to be a person, that is, if it is a remaining person, the mobile body 30a approaches the remaining person and determines whether it has collapsed. If the person has not collapsed, a predetermined action instruction is given. If the person does not perform the predetermined action, there is a possibility that there is an equipment problem, the person is unable to perform the scheduled action, or the content of the notification cannot be understood due to language or other issues. Therefore, a rescue request is notified to the mobile body control device 20, and therefore it is possible to efficiently respond to remaining people in disaster areas, etc., using mobile bodies.

[0072] In the above embodiment, the trained model for person detection 37a and the trained model for action detection 37b are described as trained models using deep learning, but the present invention is not limited to this, and trained models that have undergone supervised machine learning can also be used.

[0073] The configurations illustrated in the above embodiments are merely functional schematics and are not necessarily physically configured as shown. In other words, the distribution and integration of each device is not limited to that illustrated, and all or part of the devices can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. [Industrial Applicability]

[0074] As described above, the mobile object control system, mobile object, and mobile object control method according to the present invention are suitable for efficiently responding to people remaining in a disaster area or the like using a mobile object. [Explanation of symbols]

[0075] 10 Server device 20 Mobile control device 30, 30a, 30b, 30c mobile 31 Imaging unit 32 Altitude sensor 33 GNSS unit 34 Information Department 35 Radio Communication Department 36 Rotor 37 Memory section 37a Pre-trained model for person detection 37b Pre-trained model for motion detection 37c Travel route data 37d Evacuation Guidance Data 37e Image data 38 Control Unit 38a Imaging control unit 38b Evacuation guidance section 38c Person recognition section 38d Notification processing section 38e Operation judgment section 38f Movement control unit N Network

Claims

1. A mobile object control system having a mobile object that patrols a predetermined area and a mobile object control device that controls the mobile object, The moving body is an imaging means for capturing an image of the predetermined area; a person determination means for determining whether or not a person is included in the image captured by the imaging means; a posture estimation means for estimating a posture of a person when the person determination means determines that the image includes a person; a notification means for issuing a notification to the person, including a request to perform a predetermined action, when the person determination means has determined that the image includes a person; a motion determination means for determining whether or not a motion of the person included in an image captured by the imaging means is the predetermined motion; a control means for performing control in accordance with the determination result by the operation determination means; Equipped with The notification means A mobile object control system characterized in that, when the posture estimation means estimates that the person has not fallen, a notification including a request for the person to perform a predetermined action is issued to the person.

2. The control means 2. The mobile body control system according to claim 1, wherein when the posture estimation means estimates that the person has fallen, the mobile body control device is controlled to request help for the person.

3. A mobile object control system having a mobile object that patrols a predetermined area and a mobile object control device that controls the mobile object, The moving body is an imaging means for capturing an image of the predetermined area; a person determination means for determining whether or not a person is included in the image captured by the imaging means; a notification means for issuing a notification to the person, including a request to perform a predetermined action, when the person determination means has determined that the image includes a person; a motion determination means for determining whether or not a motion of the person included in an image captured by the imaging means is the predetermined motion; a control means for performing control in accordance with the determination result by the operation determination means; Equipped with The control means A mobile control system characterized by controlling the aircraft and issuing an alert to guide the person to evacuate from the specified area when the movement determination means determines that the person's movement is the specified movement.

4. A mobile object control system having a mobile object that patrols a predetermined area and a mobile object control device that controls the mobile object, The moving body is an imaging means for capturing an image of the predetermined area; a person determination means for determining whether or not a person is included in the image captured by the imaging means; a notification means for issuing a notification to the person, including a request to perform a predetermined action, when the person determination means has determined that the image includes a person; a motion determination means for determining whether or not a motion of the person included in an image captured by the imaging means is the predetermined motion; a control means for performing control in accordance with the determination result by the operation determination means; Equipped with The control means A mobile object control system characterized in that, when the motion determination means determines that the motion of the person is not the predetermined motion, the mobile object control device is controlled to request help.

5. A mobile body that patrols a predetermined area, an imaging means for capturing an image of the predetermined area; a person determination means for determining whether or not a person is included in the image captured by the imaging means; a posture estimation means for estimating a posture of a person when the person determination means determines that the image includes a person; a notification means for issuing a notification to the person, including a request to perform a predetermined action, when the person determination means has determined that the image includes a person; a motion determination means for determining whether or not a motion of the person included in an image captured by the imaging means is the predetermined motion; a control means for performing control in accordance with the determination result by the operation determination means; Equipped with The notification means A moving body characterized in that, when the posture estimation means estimates that the person has not fallen, a notification including a request for the person to perform a predetermined action is issued to the person.

6. A mobile body that patrols a predetermined area, an imaging means for capturing an image of the predetermined area; a person determination means for determining whether or not a person is included in the image captured by the imaging means; a notification means for issuing a notification to the person, including a request to perform a predetermined action, when the person determination means has determined that the image includes a person; a motion determination means for determining whether or not a motion of the person included in an image captured by the imaging means is the predetermined motion; a control means for performing control in accordance with the determination result by the operation determination means; Equipped with The control means A mobile body characterized in that, when the movement determination means determines that the movement of the person is the specified movement, the mobile body controls the vehicle and issues an alert to guide the person to evacuate from the specified area.

7. A mobile body that patrols a predetermined area, an imaging means for capturing an image of the predetermined area; a person determination means for determining whether or not a person is included in the image captured by the imaging means; a notification means for issuing a notification to the person, including a request to perform a predetermined action, when the person determination means has determined that the image includes a person; a motion determination means for determining whether or not a motion of the person included in an image captured by the imaging means is the predetermined motion; a control means for performing control in accordance with the result of the determination by the operation determination means; Equipped with The control means A moving body characterized in that, when the movement determining means determines that the movement of the person is not the predetermined movement, the moving body is controlled to request help.

8. A mobile object control method in a mobile object control system having a mobile object that patrols a predetermined area and a mobile object control device that controls the mobile object, an imaging step in which the moving body captures an image of the predetermined area; a person determination step in which the moving object determines whether or not a person is included in the image captured by the imaging step; a posture estimation step of estimating a posture of the person if it is determined that the image includes a person by the person determination step; a notification step of the moving object issuing a notification to the person, including a request to perform a predetermined action, on condition that the image is determined to include a person by at least the person determination step; a motion determination step in which the moving body determines whether or not a motion of the person included in the image captured in the imaging step is the predetermined motion; a control step of controlling the moving body in accordance with the determination result of the motion determination step; Including, The notification step includes: A mobile object control method, characterized in that, when it is estimated by the posture estimation step that the person has not fallen, a notification including a request for the person to perform a predetermined action is issued to the person.

9. A mobile object control method in a mobile object control system having a mobile object that patrols a predetermined area and a mobile object control device that controls the mobile object, comprising: an imaging step in which the moving body captures an image of the predetermined area; a person determination step in which the moving object determines whether or not a person is included in the image captured by the imaging step; a notification step of the moving object issuing a notification to the person, including a request to perform a predetermined action, on condition that the image is determined to include a person by at least the person determination step; a motion determination step in which the moving body determines whether or not a motion of the person included in the image captured in the imaging step is the predetermined motion; a control step of controlling the moving body in accordance with the determination result of the motion determination step; Including, The control step includes: A mobile object control method characterized by controlling the aircraft and issuing an alert to guide the person to evacuate from the specified area when the movement determination process determines that the person's movement is the specified movement.

10. A mobile object control method in a mobile object control system having a mobile object that patrols a predetermined area and a mobile object control device that controls the mobile object, comprising: an imaging step in which the moving body captures an image of the predetermined area; a person determination step in which the moving object determines whether or not a person is included in the image captured by the imaging step; a notification step of the moving object issuing a notification to the person, including a request to perform a predetermined action, on condition that the image is determined to include a person by at least the person determination step; a motion determination step in which the moving body determines whether or not a motion of the person included in the image captured in the imaging step is the predetermined motion; a control step of controlling the moving body in accordance with the determination result of the motion determination step; Including, The control step includes: A mobile object control method, characterized in that, when the motion determination step determines that the motion of the person is not the predetermined motion, the mobile object control method performs control to request help.

Images