Mobile object control system, mobile object control device, and mobile object control method

Abstract

To accurately detect a target object from picture images or videos taken by an infrared imaging device when taking an image of an area including the target object using a moving body with an on-board infrared imaging device.SOLUTION: A moving body control system comprises a moving body control device 20 that controls a moving body 30a to acquire multiple infrared images of a disaster area, and generates a thermal map based on the acquired infrared images. The moving body control device 20 also includes: performing a routing operation so as to move to the points where remainers can possibly exist in the disaster area; correcting the moving route based on the thermal map so as to detect the remainers using the infrared images for the routing operation; and notifying the moving body 30a of the corrected moving route data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a movement control system, a movement control device, and a movement control method capable of efficiently detecting an object from an image or video captured by an infrared imaging device when imaging a region including the object using a moving body equipped with the infrared imaging device.

Background Art

[0002] Conventionally, a technique of mounting a camera for imaging visible light video on a flying body such as a drone or an unmanned aircraft and taking an aerial photo of the ground with this camera is known. For example, the ground is imaged with a camera of a drone flying over a disaster area, survivors remaining in the disaster area are detected, and evacuation information is notified to the detected survivors.

[0003] In such a case, if imaging is performed under backlight, there is a possibility that survivors included in the video cannot be detected. For this reason, Patent Document 1 discloses an imaging method that enables imaging without the light source being reflected by checking whether a light source is located within the imaging range even when imaging is performed under backlight.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the invention described in Patent Document 1 above is based on the premise of capturing images with a visible light camera. Therefore, it cannot solve the problem that arises when using an infrared camera that visualizes infrared rays, which increase in accordance with the temperature of an object, as an image or video, where a person cannot be detected if their temperature is close to the temperature of their surroundings. In particular, when attempting to detect people remaining in a disaster area and guide them to evacuation, if the people remaining cannot be detected, it becomes impossible to notify them of evacuation information and ensure the safety of the people remaining.

[0006] Therefore, when imaging using an aircraft equipped with an infrared camera, a crucial challenge is how to distinguish between humans and non-humans. This challenge arises not only when detecting humans using an aircraft, but also when imaging using various moving objects to detect various types of objects that are the target of detection.

[0007] The present invention has been made to solve the problems (issues) of the above-mentioned prior art, and aims to provide a mobile body control system, a mobile body control device, and a mobile body control method that can accurately detect an object from an image or video captured by an infrared imaging device when imaging an area in which an object may exist using a mobile body equipped with an infrared imaging device. [Means for solving the problem]

[0008] To solve the above-mentioned problems and achieve the objective, the present invention provides a mobile body control system comprising a mobile body equipped with an infrared imaging device for capturing infrared images, and a mobile body control device for controlling the mobile body. ,before The mobile control device is ,before The temperature data generated based on multiple infrared images, which are collected in advance by the mobile device and which capture the same region from multiple imaging directions, is associated with the identification information of the said region. warmThe system is characterized by comprising: a temperature map generation unit that generates a temperature map; a movement path setting unit that sets a movement path including the imaging direction to be captured by the infrared imaging device of the moving body; a movement path correction unit that, by referring to at least the temperature map, estimates an infrared image captured from a predetermined imaging direction, and corrects the movement path to capture an infrared image from a different imaging direction than the predetermined imaging direction if the temperature indicated by each pixel of the estimated infrared image is close to the temperature of the human body; and a movement path notification unit that notifies the moving body of the movement path corrected by the movement path correction unit.

[0009] Furthermore, the present invention is characterized in that, in the above invention, the mobile body control device further comprises an imaging instruction notification means for notifying the mobile body of imaging path data including the imaging start position, imaging end position and imaging direction of the mobile body, and imaging instructions, and the mobile body further comprises an infrared image notification unit that, upon receiving the imaging instruction, performs imaging with the infrared imaging device based on the imaging path data and notifies the mobile body control device of the captured infrared image.

[0011] Furthermore, the present invention is characterized in that, in the above invention, the temperature map generation unit generates a plurality of temperature maps for each time period or weather condition, and the movement path correction unit corrects the movement path, including the imaging direction, based on the temperature map that matches the movement date and time or weather conditions.

[0012] Furthermore, the present invention is characterized in that, in the above invention, the mobile body control device receives an infrared image captured by the infrared imaging device mounted on the mobile body moving along the movement path, and if it determines that a change in the movement path is necessary based on the received infrared image, the movement path correction unit corrects the movement path again and notifies the mobile body of the corrected movement path.

[0013] Furthermore, the present invention is characterized in that, in the above invention, when the moving body determines that it is necessary to change the moving path based on the infrared image captured by the infrared imaging device while moving along the moving path, it requests the moving body control device to change the moving path; when the moving path correction unit receives the request to change the moving path from the moving body, it corrects the moving path based on the temperature map generated by the temperature map generation unit; and the moving path notification unit notifies the moving body of the moving path corrected by the moving path correction unit.

[0014] Furthermore, the present invention relates to a mobile body control device that controls a mobile body equipped with an infrared imaging device for capturing infrared images, and associates temperature data generated based on multiple infrared images collected in advance by the mobile body and captured from multiple imaging directions of the same region with identification information of the region. warm The system is characterized by comprising: a temperature map generation unit that generates a temperature map; a movement path setting unit that sets a movement path including the imaging direction to be captured by the infrared imaging device of the moving body; a movement path correction unit that, by referring to at least the temperature map, estimates an infrared image captured from a predetermined imaging direction, and corrects the movement path to capture an infrared image from a different imaging direction than the predetermined imaging direction if the temperature indicated by each pixel of the estimated infrared image is close to the temperature of the human body; and a movement path notification unit that notifies the moving body of the movement path corrected by the movement path correction unit.

[0015] Furthermore, the present invention relates to a mobile body control method in a mobile body control system having a mobile body equipped with an infrared imaging device for capturing infrared images and a mobile body control device for controlling the mobile body, wherein the mobile body control device associates temperature data generated based on a plurality of infrared images collected in advance by the mobile body and captured from a plurality of imaging directions of the same region with identification information of the region. warmA temperature map generation step for generating a temperature map, a movement path setting step for setting a movement path including an imaging direction imaged by the infrared imaging device of the moving body, and at least by referring to the temperature map, an infrared image imaged from a predetermined imaging direction is analogized, and when the temperature indicated by each pixel of the analogized infrared image is close to the human body temperature, a movement path correction step for correcting the movement path so as to image an infrared image from another imaging direction different from the predetermined imaging direction, and a movement path notification step for notifying the moving body of the movement path corrected by the movement path correction step.

Effect of the Invention

[0016] According to the present invention, when imaging an area where an object may exist using a moving body equipped with an infrared imaging device, the object can be accurately detected from an image or video imaged by the infrared imaging device.

Brief Description of the Drawings

[0017] [Figure 1] FIG. 1 is a diagram showing an overview of a movement control system according to Embodiment 1. [Figure 2] FIG. 2 is a diagram showing the system configuration of the movement control system shown in FIG. 1. [Figure 3] FIG. 3 is a functional block diagram showing the configuration of the movement control device shown in FIG. 2. [Figure 4] FIG. 4 is a functional block diagram showing the configuration of the moving body shown in FIG. 2. [Figure 5] FIG. 5 is a diagram showing an example of an imaging path along which the moving body moves. [Figure 6] FIG. 6 is a diagram showing an example of the temperature map shown in FIG. 3. [Figure 7] FIG. 7 is an explanatory diagram for explaining the correction of the movement path. [Figure 8] FIG. 8 is a sequence diagram showing the processing procedure of the movement control system. [Figure 9] FIG. 9 is a diagram showing an overview of a movement control system according to Embodiment 2. [Figure 10] Figure 10 is a functional block diagram showing the configuration of the mobile body shown in Figure 9. [Figure 11] Figure 11 is a functional block diagram showing the configuration of the mobile control device shown in Figure 9. [Figure 12] Figure 12 is a sequence diagram (part 1) showing the processing procedure of the mobile control system shown in Figure 9. [Figure 13] Figure 13 is a sequence diagram (part 2) showing the processing procedure of the mobile control system shown in Figure 9. [Modes for carrying out the invention]

[0018] Embodiments of the mobile object control system, mobile object control device, and mobile object control method according to the present invention will be described in detail below with reference to the drawings. In the embodiments shown below, a drone will be used as the mobile object.

[0019] [Embodiment 1] <Overview of the Mobile Control System> First, an overview of the mobile control system according to Embodiment 1 will be described. Figure 1 is a diagram showing an overview of the mobile control system according to Embodiment 1. As shown in Figure 1, this mobile control system is a system that can accurately detect persons remaining in areas where disasters are likely to occur (hereinafter collectively referred to as "disaster areas") using infrared images captured by an infrared imaging device 31 mounted on a mobile body 30a.

[0020] The mobile body 30a shown in Figure 1 is a drone capable of autonomous flight under the control of the mobile body control device 20. This mobile body 30a does not perform takeoff, landing, attitude changes, or flight through manual operation by an operator, but rather is a device that autonomously controls its movement in response to instructions from the mobile body control device 20.

[0021] The mobile body control device 20 is a control device that controls the mobile body 30a. It notifies the mobile body 30a of imaging path data (including the imaging start position, imaging end position, and imaging direction) to capture infrared images of the area along the imaging path, and receives the captured infrared images from the mobile body 30a. It also generates a temperature map from the infrared images, corrects the movement path using this temperature map, and notifies the mobile body 30a of the movement path data including the corrected movement path.

[0022] In this mobile control system, the mobile control device 20 notifies the waiting mobile body 30a of a movement instruction including imaging path data for capturing infrared images of the disaster area (S1). This imaging path data includes the imaging start position, imaging end position, and imaging direction.

[0023] When the mobile device 30a receives a movement instruction including imaging path data from the mobile device control device 20, it moves to the imaging start position (S2). The imaging start position is determined by latitude, longitude, and altitude. The mobile device 30a does not need to acquire infrared images until it moves to the imaging start position.

[0024] Subsequently, the mobile unit 30a moves according to the imaging path data while capturing infrared images at predetermined time intervals (for example, 0.1 seconds) (S3). In some cases, the mobile unit 30a may hover in the air and only change the imaging direction. If the imaging direction included in the imaging path data is other than the reference direction (the direction in which the infrared imaging device 31 is attached), the mobile unit 30a will either tilt its body or change the field of view of the infrared imaging device 31. When the mobile unit 30a reaches the imaging completion position, it stops capturing infrared images (S4) and moves to a predetermined standby position (S5). After that, the mobile unit 30a transmits the captured infrared images to the mobile unit control device 20 (S6).

[0025] The mobile device control unit 20 generates a temperature map based on multiple infrared images received from the mobile device 30a (S7). A detailed explanation of this temperature map will be given later, but it is data that associates multiple infrared images taken of the same region from multiple imaging directions with identification information of that subregion.

[0026] Subsequently, the mobile device control unit 20 sets movement path data for the mobile device 30a to detect remaining persons located in the disaster area (S8). Then, the mobile device control unit 20 corrects the movement path data based on the temperature map (S9). By referring to the temperature map, it can infer an infrared image obtained when the disaster area is imaged from a certain imaging direction, and if it is determined that the temperature indicated by each pixel of the inferred infrared image is close to the temperature of the human body and it is difficult to detect remaining persons, it can correct the movement path to acquire an infrared image taken from a different imaging direction.

[0027] Subsequently, the mobile device control unit 20 issues a movement instruction to the mobile device 30a, including the corrected movement path data, and the mobile device 30a begins moving to search for the remaining person. If the infrared image captured by the mobile device 30a along its movement path includes a remaining person, an alarm sound or voice guidance is emitted to guide the remaining person to leave the disaster area.

[0028] <System configuration of the mobile control system> Next, the system configuration of the mobile control system according to this embodiment 1 will be described. Figure 2 is a diagram showing the system configuration of the mobile control system shown in Figure 1. As shown in Figure 2, this mobile control system has a configuration in which a disaster management device 10, a mobile control device 20, and a plurality of mobile bodies 30a, 30b, and 30c (hereinafter collectively referred to as "mobile bodies 30") are connected to a network N.

[0029] Disaster management device 10 is a management device that manages disaster-related information operated by government agencies. Examples include the disaster prevention information provision center operated by the Ministry of Land, Infrastructure, Transport and Tourism, and management devices operated by local governments that manage dam discharge information.

[0030] The mobile device control unit 20 performs processes such as generating a temperature map based on infrared images previously collected by the mobile device 30, setting a movement path including the imaging direction to be captured by the infrared imaging device 31 of the mobile device 30, correcting the movement path including the imaging direction based on at least the temperature map, and notifying the mobile device 30 of the corrected movement path.

[0031] The mobile unit 30 performs processes such as acquiring its own position information from signals from an altitude sensor and a GNSS (Global Navigation Satellite System) unit, controlling its rotor blades to move, capturing infrared images with an infrared imaging device 31 at predetermined time intervals, and transmitting the infrared images to the mobile unit control device 20.

[0032] <Configuration of the mobile control device 20> Next, the configuration of the mobile device control unit 20 shown in Figure 2 will be described. Figure 3 is a functional block diagram showing the configuration of the mobile device control unit 20 shown in Figure 2. As shown in Figure 3, the mobile device control unit 20 includes a display unit 21, an input unit 22, a communication I / F unit 23, a wireless communication unit 24, a storage unit 25, and a control unit 26.

[0033] The display unit 21 is a display device such as a liquid crystal panel or a display device, and the input unit 22 is an input device such as a keyboard or mouse. The communication I / F unit 23 is an interface unit for communicating with other devices such as the disaster management device 10. The wireless communication unit 24 is an interface unit for wireless communication with the mobile device 30.

[0034] The storage unit 25 is a storage device such as a hard disk drive or non-volatile memory, and stores disaster data 25a, imaging path data 25b, infrared image data 25c, temperature map 25d, and movement path data 25e, etc.

[0035] Disaster data 25a is data that includes disaster area information, weather information, dam discharge information, etc., acquired from the disaster management device 10. Imaging path data 25b is a path for acquiring infrared images of the disaster area in advance, and includes the imaging start position (latitude, longitude, altitude), imaging end position (latitude, longitude, altitude), and imaging direction.

[0036] The infrared image data 25c is data from multiple infrared images received from the mobile device 30. The temperature map 25d is temperature data of the disaster area generated based on the infrared images, and is data that associates temperature data generated from multiple infrared images of the same area captured from multiple imaging directions with identification information of that area. A detailed explanation of the temperature map 25d will be given later.

[0037] The movement path data 25e is corrected movement path data obtained by correcting the movement path taken when moving through the disaster area to detect the presence or absence of lingering persons using the temperature map 25d. The reason for this correction is that if the difference between the temperature of the lingering persons and the ambient temperature is less than a predetermined threshold, it becomes impossible to detect the lingering persons included in the infrared image. In such cases, if the imaging direction for capturing the infrared image is changed, the difference between the temperature of the lingering persons and the ambient temperature may exceed the predetermined threshold, making it possible to detect the lingering persons included in the infrared image. For this reason, the movement path is corrected using the temperature map 25d.

[0038] The control unit 26 is a control unit that controls the entire mobile device control device 20, and includes a disaster data acquisition unit 26a, an imaging path identification unit 26b, a temperature map generation unit 26c, a movement path setting unit 26d, a movement path correction processing unit 26e, and a movement path notification unit 26f. In practice, by loading these programs into the CPU and executing them, the disaster data acquisition unit 26a, the imaging path identification unit 26b, the temperature map generation unit 26c, the movement path setting unit 26d, the movement path correction processing unit 26e, and the movement path notification unit 26f will each execute the corresponding processes.

[0039] The disaster data acquisition unit 26a acquires disaster data based on weather warnings and advisories such as heavy rain and storms, or dam discharge information, from the disaster management device 10, and stores it in the storage unit 25 as disaster data 25a. The disaster data 25a includes information indicating the type of disaster and the disaster area. In this embodiment, for example, if information on a disaster area where a disaster is likely to occur is acquired one week in advance, the mobile device 30 is moved to this disaster area to capture infrared images and create a temperature map 25d. Alternatively, a disaster may be simulated during normal times, an area where a disaster is likely to occur is designated as a disaster area, and the mobile device 30 is moved to this disaster area to capture infrared images and create a temperature map 25d.

[0040] The imaging path identification unit 26b identifies the imaging path of the moving object 30 that captures infrared images based on the information of the disaster area contained in the disaster data 25a. This imaging path includes the movement route, as well as the imaging start position (latitude, longitude, altitude), the imaging end position (latitude, longitude, altitude), and the imaging angle.

[0041] The temperature map generation unit 26c generates a temperature map 25d based on the infrared image data 25c stored in the storage unit 25 and stores it in the storage unit 25. Specifically, it generates a temperature map 25d that associates temperature data generated from multiple infrared images of the same region captured from multiple imaging directions with identification information of that region. A description of such a temperature map 25d will be given later.

[0042] The movement route setting unit 26d is a processing unit that sets a route for the moving object 30 to move through the disaster area based on disaster data 25a. For example, it sets a route to move in the order of points A, B, C, and D in the disaster area, where there is a high possibility of people remaining behind.

[0043] The movement path correction processing unit 26e is a processing unit that corrects the movement path based on the temperature map 25d. For example, when moving from point A to point C via point B, if point B is reached via the shortest path, due to factors such as sunlight reflection, the temperature difference between the person remaining at point B and the surrounding area will disappear, and the person remaining at point B may not be detectable in the infrared image taken during the shortest path. In such cases, it may be possible to detect the person remaining at point B using an infrared image taken from a different imaging direction at point B. For this reason, the movement path is corrected using the temperature map 25d.

[0044] If multiple temperature maps 25d have been created for each time period or weather condition, the user can select the temperature map 25d that best suits the time period or weather conditions of the movement and use the selected temperature map 25d. The movement route notification unit 26f is a processing unit that transmits movement route data 25e to the mobile body 30 via the wireless communication unit 24.

[0045] <Configuration of mobile unit 30a> Next, the configuration of the mobile body 30a shown in Figure 2 will be described. Figure 4 is a functional block diagram showing the configuration of the mobile body 30a shown in Figure 2. Mobile bodies 30b and 30c have similar configurations. As shown in Figure 4, the mobile body 30a includes an infrared imaging device 31, an altitude sensor 32, a GNSS unit 33, a wireless communication unit 34, a rotor blade 35, a memory unit 36, and a control unit 37.

[0046] The infrared imaging device 31 is an infrared camera that visualizes infrared radiation emitted from an object as an infrared image. This infrared imaging device 31 may be attached to a mobile body 30a and have a controllable field of view.

[0047] The altitude sensor 32 is a sensor that measures the altitude of the mobile body 30a. For example, a barometric pressure sensor can be used as this altitude sensor 32. Alternatively, a barometric pressure sensor and an ultrasonic sensor can be combined. This allows the mobile body 30a to maintain a stable altitude close to the ground. The GNSS unit 33 is a unit that receives signals from multiple GNSS satellites to determine the position coordinates of its own device.

[0048] The wireless communication unit 34 is an interface unit for wireless communication with the mobile body control device 20. The multiple rotor blades 35 rotate individually under control from the control unit 37, enabling attitude control and movement of the mobile body 30a. Therefore, even if the field of view of the infrared imaging device 31 attached to the mobile body 30a cannot be controlled, by controlling the attitude of the mobile body 30a, it becomes possible to capture images not only directly below, but also in the lower right and lower left directions.

[0049] The storage unit 36 ​​is a storage device such as a hard disk drive or non-volatile memory, and stores imaging path data 36a, infrared image data 36b, movement path data 36c, and notification data 36d. The imaging path data 36a is imaging path data notified in advance by the mobile device control device 20 for imaging infrared images of the disaster area, and the mobile device 30a moves according to this imaging path data 36a.

[0050] Infrared image data 36b is infrared image data captured by the mobile body 30a while it is moving along the imaging path. This infrared image data 36b is transmitted to the mobile body control device 20 after the mobile body 30a has moved to a standby position. Movement path data 36c is data of the movement path used to detect people remaining in the disaster area. Notification data 36d is notification data, such as voice data, to notify people remaining in the disaster area.

[0051] The control unit 37 is a control unit that controls the entire moving body 30a and includes a movement control unit 37a, an imaging control unit 37b, a detection unit 37c, and a notification unit 37d. In practice, these programs are loaded into the CPU and executed, causing the movement control unit 37a, the imaging control unit 37b, the detection unit 37c, and the notification unit 37d to execute the processes corresponding to them.

[0052] The movement control unit 37a is a control unit that controls the movement of the aircraft by controlling multiple rotor blades 35 using imaging path data 36a or movement path data 36c, altitude information output by the altitude sensor 32, and position information output by the GNSS unit 33. Specifically, the movement control unit 37a controls the aircraft to move along the imaging path of the imaging path data 25b or the movement path of the movement path data 36c.

[0053] The imaging control unit 37b is a control unit that controls the acquisition of infrared images by the infrared imaging device 31. Specifically, when the moving object 30a receives imaging path data from the moving object control device 20 and moves along this imaging path, it controls the movement to acquire infrared images at predetermined intervals (for example, 0.1 seconds). When the moving object 30a receives movement path data from the moving object control device 20 and moves along this movement path, it also controls the movement to acquire infrared images at predetermined intervals (for example, 0.1 seconds). If the infrared imaging device 31 is configured to allow control of the field of view, the imaging control unit 37b controls the field of view according to the imaging direction included in the imaging path data.

[0054] The detection unit 37c is a processing unit that detects a person remaining in the infrared image captured while the moving object 30a is moving along its path. When detecting a person remaining in the infrared image, for example, the person can be detected by performing template matching processing using a human template on the infrared image. Alternatively, the presence or absence of a person can be detected by inputting the infrared image into a trained model obtained by deep learning. When using deep learning, it is necessary to generate a trained model by performing supervised learning using infrared images containing people and their results as training data. It is also possible to use machine learning other than deep learning.

[0055] The notification unit 37d is a notification unit that, when a person remaining in the building is detected from an infrared image, notifies the person of an evacuation request via voice message or other means. Alternatively, it may output an alarm sound or a warning light instead of voice.

[0056] <About the imaging path data> Next, we will explain the imaging path data when the mobile device 30a captures infrared images of the disaster area. Figure 5 shows an example of the imaging path taken by the mobile device 30a. Here, we will explain using the example of moving the same disaster area in three ways, as shown in Figures 5(a), 5(b), and 5(c), and imaging areas A2 to F2 from three directions.

[0057] As shown in Figure 5(a), the moving object 30a moves in the following order: region A3, region B3, region C3, region D3, region E3, region F3, region F2, region E2, region D2, region C2, region B2, region A2, region A1, region B1, region C1, region D1, region E1, and region F1. During this time, the infrared imaging device 31 captures infrared images of the region directly below the moving object 30a. That is, when moving over region A3, it captures an infrared image of region A3, and when moving over region B3, it captures an infrared image of region B3.

[0058] Subsequently, as shown in Figure 5(b), the moving object 30a moves again in the order of region A3, region B3, region C3, region D3, region E3, region F3, region F2, region E2, region D2, region C2, region B2, region A2, region A1, region B1, region C1, region D1, region E1, and region F1. During this time, the infrared imaging device 31 captures infrared images of the region in the lower left direction of the moving object 30a. That is, when moving over region A3, it captures an infrared image of region A2, and when moving over region B3, it captures an infrared image of region B2.

[0059] Subsequently, as shown in Figure 5(c), the moving object 30a moves again in the order of region A3, region B3, region C3, region D3, region E3, region F3, region F2, region E2, region D2, region C2, region B2, region A2, region A1, region B1, region C1, region D1, region E1, and region F1. During this time, the infrared imaging device 31 captures infrared images of the region in the lower right direction of the moving object 30a. That is, when moving over region A1, it captures an infrared image of region A2, and when moving over region B1, it captures an infrared image of region B2.

[0060] As a result, when focusing on region B2, three infrared images can be obtained: one taken from directly above, one taken from the upper right, and one taken from the upper left. Furthermore, by equipping the mobile body 30a with multiple infrared imaging devices 31 and having each device image in a different direction, or by using an infrared imaging device 31 capable of capturing infrared images in all directions, infrared images taken from different directions can be obtained in a single movement. Also, while imaging from three directions is described, it is not limited to this.

[0061] <About Temperature Map 25d> Next, we will explain the temperature map 25d shown in Figure 3. Figure 6 shows an example of the temperature map 25d shown in Figure 3. The temperature map 25d associates temperature data generated based on infrared images collected at multiple locations along imaging paths for a single disaster area. The fact that the temperature indicated by each pixel in the infrared image is close to the temperature of the human body means that the temperature difference between the background other than the survivors and the survivors is difficult to discern, and there is a possibility of overlooking survivors.

[0062] As shown in Figure 6, for the time period "AM" in region "B2", the temperature data "t" was generated based on the infrared image of region "B2" taken from position B1. 21a "and temperature data "t" generated based on the infrared image of region "B2" taken from position B2. 22a " and temperature data "t" generated based on the infrared image of region "B2" taken from position B3. 32a This is the correspondence between the two.

[0063] Similarly, for the time period "afternoon" in region "B2", temperature data "t" is generated based on the infrared image of region "B2" taken from position B1. 21p "and temperature data "t" generated based on the infrared image of region "B2" taken from position B2. 22p " and temperature data "t" generated based on the infrared image of region "B2" taken from position B3. 32p This is the correspondence between the two.

[0064] The movement path correction processing unit 26e of the mobile device control device 20 corrects the movement path based on these temperature maps 25d. For example, when region B2 is imaged from position B2, that is, when region B2 is imaged from directly above, the temperature data "t 22aIf the number of pixels indicating a temperature close to that of a human body exceeds a predetermined threshold, it is determined that it may not be possible to detect the residual person. Then, the movement path is corrected so that imaging of region B2 is performed not from directly above, but from another position (such as position B1 or B3) where the number of pixels indicating a temperature close to that of a human body is minimized. Note that although data for "morning" and "afternoon" has been described here, it is also possible to generate temperature maps 25d for each time of day or for each weather condition. Furthermore, temperature maps 25d may also be generated for each season.

[0065] <Regarding correction of travel paths> Next, the correction of the movement path of the mobile device control device 20 will be explained. Figure 7 is an explanatory diagram for explaining the correction of the movement path. Figure 7(a) shows the situation when the mobile device 30a has imaged a predetermined area of ​​the disaster area from imaging position A. In this figure, if the riverbed area is at a temperature close to that of the human body, the remaining persons on the riverbed in the infrared image may be buried and indistinguishable from the riverbed, making it impossible to detect the remaining persons.

[0066] In contrast, Figure 7(b) shows the same area as the area captured at imaging position A, but captured from imaging position B, which is different from imaging position A. As shown in this figure, by changing the imaging position, the river, in addition to the riverbed, becomes the background for the remaining persons. Since the river is usually cooler than the human body, even if the riverbed area is at a temperature close to that of the human body, some of the remaining persons whose background is the river can be distinguished from the river, making it possible to detect the remaining persons. The movement path correction processing unit 26e of the mobile object control device 20 corrects the movement path data, including the imaging angle of the infrared imaging device 31 of the mobile object 30a, based on the disaster data 25a and the temperature map 25d.

[0067] <Processing procedure for mobile object control system> Next, the processing procedure of the mobile control system shown in Figure 2 will be described. Figure 8 is a sequence diagram showing the processing procedure of the mobile control system. As shown in Figure 8, the mobile control device 20 acquires disaster data, including information on the disaster area, from the disaster management device 10 (step S101).

[0068] Subsequently, the mobile device control unit 20 identifies an imaging path for capturing infrared images (step S102). For example, as shown in Figures 5(a), 5(b), and 5(c), it is set to move through the disaster area multiple times while changing the imaging angle. Then, the mobile device control unit 20 transmits the imaging path data 25b to the mobile device 30a (step S103).

[0069] When the mobile unit 30a receives imaging path data 25b from the mobile unit control device 20 (step S104), it moves to the imaging start position and captures an infrared image while moving according to the imaging path (step S105). When the mobile unit 30a moves to the imaging end position included in the imaging path data 25b, it moves to the standby position and transmits the infrared image to the mobile unit control device 20 (step S106).

[0070] The mobile device control unit 20 receives infrared images, generates a temperature map based on the received multiple infrared images (step S107), and stores the generated temperature map as temperature map 25d in the storage unit 25 (step S108). Subsequently, if the mobile device control unit 20 determines, based on disaster data acquired from the disaster management device 10, that evacuation of remaining persons located in the disaster area is necessary, it sets movement path data to detect the remaining persons located in the disaster area (step S109). Then, the mobile device control unit 20 corrects the movement path data based on temperature map 25d (step S110) and transmits the corrected movement path data to the mobile device 30a. When the mobile device 30a receives the movement path data from the mobile device control unit 20, it moves according to the movement path, captures infrared images, and transmits the infrared images to the mobile device control unit 20.

[0071] As described above, in the mobile control system according to this embodiment 1, the mobile control device 20 controls the mobile body 30 to capture multiple infrared images of the disaster area and generates a temperature map 25d based on the captured infrared images. The mobile control device 20 then sets a movement path to move to locations in the disaster area where there is a high possibility of survivors remaining, corrects the movement path data of this movement path data based on the temperature map 25d so that survivors can be accurately detected using infrared images, and notifies the mobile body 30 of the corrected movement path data 25e, thereby enabling accurate detection of survivors in the disaster area.

[0072] In the above embodiment 1, the correction of the movement path data by the mobile object control device 20 was shown to be performed based on the temperature map 25d. However, the present invention is not limited to this, and it is also possible to correct the movement path data using 3D data of buildings, groves of trees, etc. in the disaster area so that these obstacles do not get between the mobile object 30 and the area to be imaged.

[0073] Furthermore, the mobile object control device 20 can receive infrared images from the mobile object 30 as it moves along its path, and if it determines that the temperature indicated by each pixel forming the infrared image is close to the temperature of a human body and that it may not be possible to detect a residual person, it can recorrect the movement path data, including the imaging angle of the mobile object 30, based on the temperature map 25d while the mobile object 30 is moving, and notify the mobile object 30 of this correction.

[0074] [Embodiment 2] By the way, in the above embodiment 1, the movement path was not changed while the mobile body 30 was moving, but the present invention is not limited thereto. In this embodiment 2, the mobile body 40a, while moving through a disaster area, has pixels that show temperatures close to the temperature of the human body, but has not detected any lingering persons from this infrared image. In this case, the mobile body 40a determines that the background temperature of the area shown in the infrared image is close to the temperature of the human body, and that it may not be possible to detect lingering persons. The mobile body 40a then notifies the mobile body control device 50 of a request to change the movement path, and the mobile body control device 50, upon receiving the request to change the path, corrects the movement path data including the imaging angle, and notifies the mobile body 40a of the corrected new movement path data.

[0075] The outline of the mobile control system according to this second embodiment will now be described. Figure 9 is a diagram showing the outline of the mobile control system according to the second embodiment. As shown in Figure 9, the mobile control device 50 receives disaster data from a disaster management device 10 (not shown) (S11).

[0076] The mobile device control unit 50 sets a movement route for the mobile device 40a in the disaster area and notifies the mobile device 40a of a movement instruction including the set movement route data 25e (S12).

[0077] If the mobile body 40a receives a movement instruction from the mobile body control device 50, including movement path data 25e, it moves to the starting position based on the movement path included in the received movement path data (S13). The mobile body 40a moves along the movement path while detecting remaining persons based on the captured infrared image (S14).

[0078] If the mobile body 40a determines that the temperature indicated by the pixels forming the captured infrared image is close to the temperature of the human body and that it may not be possible to detect a residual person, it sends a request to the mobile body control device 50 to change its movement path (S15). The temperature indicated by the pixels forming such an infrared image is the same as that described in Embodiment 1. When the mobile body control device 50 receives a request to change its movement path from the mobile body 40a, it corrects the movement path data, including the imaging angle, based on the temperature map 25d (S16). Then, the mobile body control device 50 notifies the mobile body 40a of an update instruction that includes the corrected movement path data, including the imaging angle (S17).

[0079] The mobile body 40a moves based on the movement path data, including the updated imaging angle, in accordance with the update instruction, and captures an infrared image to detect the remaining person (S18).

[0080] <Configuration of mobile unit 40a> Next, the configuration of the mobile body 40a shown in Figure 9 will be described. Figure 10 is a functional block diagram showing the configuration of the mobile body 40a shown in Figure 9. Note that functional parts similar to those of the mobile body 30a shown in Figure 4 are given the same reference numerals, and their detailed explanations are omitted. As shown in Figure 10, the mobile body 40a includes an infrared imaging device 31, an altitude sensor 32, a GNSS unit 33, a wireless communication unit 34, a rotor blade 35, a memory unit 46, and a control unit 47.

[0081] The storage unit 46 is a storage device such as a hard disk drive or non-volatile memory, and stores infrared image data 36b, movement path data 36c, and notification data 36d. The movement path data 36c is data of the movement path of the moving object 40a, which has been corrected based on the temperature map 25d, using the movement path data of the disaster area from the disaster data 25a.

[0082] The control unit 47 is a control unit that controls the entire moving body 40a and includes a movement control unit 37a, an imaging control unit 37b, a detection unit 37c, a notification unit 37d, and a change request notification unit 47a. In practice, by loading these programs into the CPU and executing them, the processes corresponding to the movement control unit 37a, imaging control unit 37b, detection unit 37c, notification unit 37d, and change request notification unit 47a are made to execute, respectively.

[0083] The change request notification unit 47a is a processing unit that sends a request to the mobile object control device 50 to change the movement path if there is a pixel in the infrared image captured by the mobile object 40a that shows a temperature close to that of a human body, and no residual person has been detected. This is because, even though no residual person has been detected from the infrared image, if there is a pixel in the infrared image that shows a temperature close to that of a human body, it is possible that the residual person and the background cannot be distinguished because the background is close to that of a human body, and the residual person may be lost in the background and not detected. In such a case, a request to change the movement path data is sent to the mobile object control device 50, the corrected movement path data is received, and the movement is carried out based on the corrected movement path data, thereby reducing the possibility of missing the residual person.

[0084] <Configuration of the mobile control device 50> Next, the configuration of the mobile control device 50 will be described. Figure 11 is a functional block diagram showing the configuration of the mobile control device 50 shown in Figure 9. Note that functional parts similar to those of the mobile control device 20 shown in Figure 3 are denoted by the same reference numerals, and their detailed explanations are omitted. As shown in Figure 11, the mobile control device 50 includes a display unit 21, an input unit 22, a communication I / F unit 23, a wireless communication unit 24, a storage unit 25, and a control unit 56.

[0085] The control unit 56 is a control unit that controls the entire mobile device control device 50, and includes a disaster data acquisition unit 26a, an imaging path identification unit 26b, a temperature map generation unit 26c, a movement path setting unit 26d, a movement path notification unit 26f, and a movement path correction unit 56a. In practice, by loading these programs into the CPU and executing them, the processes corresponding to the disaster data acquisition unit 26a, the imaging path identification unit 26b, the temperature map generation unit 26c, the movement path setting unit 26d, the movement path notification unit 26f, and the movement path correction unit 56a are executed, respectively.

[0086] The movement path correction unit 56a is a processing unit that corrects the movement path based on the temperature map 25d when it receives a change request notified from the moving body 40a.

[0087] <Processing procedure for mobile object control system> Next, the processing procedure of the mobile control system will be described. Figures 12 and 13 are sequence diagrams showing the processing procedure of the mobile control system shown in Figure 9. As shown in these figures, the mobile control device 50 acquires disaster data 25a from a disaster management device 10 (not shown) (step S201).

[0088] Then, the mobile device control unit 50 sets the shortest movement route data from the disaster area information contained in the disaster data 25a (step S202). Then, the mobile device control unit 50 notifies the mobile device 40a of a movement instruction including the movement route data 25e (step S203).

[0089] When the mobile unit 40a receives a movement instruction from the mobile unit control device 50, including movement path data 25e (step S204), it stores this as movement path data 36c and begins moving according to the movement path data 36c (step S205). The mobile unit 40a takes infrared images of the disaster area each time and processes the infrared images to detect any remaining persons (step S206). For example, template matching can be used.

[0090] The mobile device 40a determines whether or not it has detected a person remaining in the area if there is a pixel whose temperature is close to the temperature of a human body as indicated by each pixel forming the infrared image. If a person remaining in the area is detected, the pixel is treated as having captured an image of the person remaining in the area, and processing such as evacuation guidance is performed for that person remaining in the area. However, if no person remaining in the area is detected, it may be background temperature rather than a human body, so the mobile device control unit 50 is notified of a request to change the movement path (step S207). If the mobile device control unit 50 receives a change request from the mobile device 40a (step S208), it corrects the movement path data, including the imaging angle, based on the temperature map 25d (step S209). Then, the mobile device control unit 50 notifies the mobile device 40a of an update instruction including the corrected movement path data 25e (step S210).

[0091] If the mobile unit 40a receives a change instruction including movement path data (step S211), it moves based on the movement path data while detecting remaining persons (step S212), and terminates the above series of processes when the movement is complete.

[0092] As described above, the mobile body control system according to Embodiment 2 notifies the mobile body control device 50 of a request to change the movement path when the temperature indicated by each pixel forming the infrared image of the mobile body 40a is close to the temperature of a human body, but no lingering person has been detected. Upon receiving the change request, the mobile body control device 50 corrects the movement path data based on the temperature map 25d. Subsequently, the mobile body control device 50 notifies the mobile body 40a of a change instruction including the corrected movement path data, and the mobile body 40a moves based on the corrected movement path data. In other words, if there is a pixel in the captured infrared image that is close to the temperature of a human body, and that pixel does not detect a lingering person, the system determines that the background temperature of the area shown in the captured infrared image is close to the temperature of a human body, and the system is configured to capture that area from a different position, thereby enabling accurate detection of lingering persons in the disaster area.

[0093] In the embodiments 1 and 2 described above, the case in which infrared images (still images) are captured by the infrared imaging device 31 has been explained, but it can also be applied to the case in which infrared video is captured. In this case, the same processing can be performed by extracting infrared images from the infrared video at regular time intervals.

[0094] The configurations illustrated in each of the above embodiments are functional schematics and do not necessarily have to be physically represented as shown. In other words, the distributed and integrated forms of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. [Industrial applicability]

[0095] The mobile object control system, mobile object control device, and mobile object control method according to the present invention are suitable for accurately detecting an object from an image or video captured by an infrared imaging device when imaging an area including an object using a mobile object equipped with an infrared imaging device. [Explanation of Symbols]

[0096] 10 Disaster management device 20, 50 Mobile control devices 21 Display section 22 Input section 23 Communication I / F Section 24 Wireless Communication Section 25 Memory section 25a Disaster data 25b Imaging path data 25c infrared image data 25d temperature map 25e Travel Path Data 26 Control Unit 26a Disaster Data Acquisition Unit 26b Imaging path identification unit 26c Temperature map generation unit 26d Movement path setting unit 26e Movement path correction processing unit 26f Movement Route Notification Unit 30, 30a, 30b, 30c, 40a Mobile 31 Infrared imaging device 32 Advanced Sensors 33 GNSS units 34 Wireless Communication Section 35 Rotary Wings 36 Memory section 36a Imaging path data 36b Infrared image data 36c Travel path data 36d Reporting Data 37 Control Unit 37a Movement control unit 37b Imaging control unit 37c Detection unit 37d News Department 46 Memory section 47 Control Unit 47a Change Request Notification Section 56 Control Unit 56a Movement path correction processing unit

Claims

1. A mobile body control system comprising a mobile body equipped with an infrared imaging device for capturing infrared images, and a mobile body control device for controlling the mobile body, The aforementioned mobile device control device is A temperature map generation unit generates a temperature map that associates temperature data generated based on multiple infrared images collected in advance by the moving body and taken from multiple imaging directions of the same region with identification information of the region, A movement path setting unit sets a movement path that includes the imaging direction to be captured by the infrared imaging device of the moving body, A movement path correction unit that, by referring to at least the temperature map, estimates an infrared image captured from a predetermined imaging direction, and corrects the movement path to capture an infrared image from a different imaging direction than the predetermined imaging direction if the temperature indicated by each pixel of the estimated infrared image is close to the temperature of the human body, A movement path notification unit notifies the moving body of the movement path corrected by the movement path correction unit. A mobile control system characterized by having the following features.

2. The aforementioned mobile device control device is The system further comprises imaging instruction notification means for notifying the moving body of imaging path data including the imaging start position, imaging end position, and imaging direction, as well as imaging instructions. The aforementioned moving body is The mobile body control system according to claim 1, further comprising an infrared image notification unit that, upon receiving the imaging instruction, performs imaging using the infrared imaging device based on the imaging path data and notifies the mobile body control device of the captured infrared image.

3. The temperature map generation unit, Generate multiple temperature maps for each time period or weather condition. The aforementioned movement path correction unit, A mobile object control system according to claim 1 or 2, characterized in that it corrects the movement path, including the imaging direction, based on a temperature map that matches the date and time of movement or weather conditions.

4. The aforementioned mobile device control device is A mobile body control system according to any one of claims 1 to 3, characterized in that it receives an infrared image captured by the infrared imaging device mounted on a mobile body moving along the aforementioned movement path, and if it determines that a change in the movement path is necessary based on the received infrared image, the movement path correction unit corrects the movement path again and notifies the mobile body of the corrected movement path.

5. The aforementioned moving body is When it is determined that a change in the movement path is necessary based on the infrared image captured by the infrared imaging device while moving along the aforementioned movement path, a request to change the movement path is made to the mobile body control device. The aforementioned movement path correction unit, When a request to change the movement path is received from the moving body, the movement path is corrected based on the temperature map generated by the temperature map generation unit. The aforementioned movement path notification unit, The movement path corrected by the movement path correction unit is notified to the moving body. A mobile body control system according to any one of the features 1 to 3.

6. A mobile device control device that controls a mobile body equipped with an infrared imaging device for capturing infrared images, A temperature map generation unit generates a temperature map that associates temperature data generated based on multiple infrared images collected in advance by the moving body and taken from multiple imaging directions of the same region with identification information of the region, A movement path setting unit sets a movement path that includes the imaging direction to be captured by the infrared imaging device of the moving body, A movement path correction unit that, by referring to at least the temperature map, estimates an infrared image captured from a predetermined imaging direction, and corrects the movement path to capture an infrared image from a different imaging direction than the predetermined imaging direction if the temperature indicated by each pixel of the estimated infrared image is close to the temperature of the human body, A movement path notification unit notifies the moving body of the movement path corrected by the movement path correction unit. A mobile device control device characterized by being equipped with the following features.

7. A mobile body control system comprising a mobile body equipped with an infrared imaging device for capturing infrared images, and a mobile body control device for controlling the mobile body, wherein the mobile body control method is described above. The mobile body control device includes a temperature map generation step, which generates a temperature map by associating temperature data generated based on multiple infrared images collected in advance by the mobile body and taken from multiple imaging directions of the same region with identification information of the region, A movement path setting step, which sets a movement path including the imaging direction to be imaged by the infrared imaging device of the moving body, A movement path correction step which involves, at least by referring to the temperature map, inferring an infrared image captured from a predetermined imaging direction, and correcting the movement path to capture an infrared image from a different imaging direction than the predetermined imaging direction if the temperature indicated by each pixel of the inferred infrared image is close to the temperature of the human body, A movement path notification step in which the movement path corrected by the movement path correction step is notified to the moving body; A method for controlling a mobile object, characterized by including the following:

Images