Anomaly detection system and mobile body

Abstract

To properly grasp abnormality of a region within a determination image.SOLUTION: A disaster detection system 100 comprises: an image feature extraction part 203 for extracting a feature amount of a region in an image; an arrangement relation extraction part 204 which on the basis of a feature amount of a predetermined region in the image extracted by the image feature extraction part and a feature amount of a peripheral region around the predetermined region, extracts an arrangement relation between the predetermined region and the peripheral region; an arrangement relation difference calculation part 902 which calculates a difference between an arrangement relation between a plurality of learning images and that of a determination image; and an abnormality determination part 904 which on the basis of the difference calculated by the arrangement relation difference calculation part, determines abnormality of a region in the determination image.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an abnormality detection system and a moving object.

Background Art

[0002] In a disaster-stricken area, from the perspective of saving lives, rapid grasping of the damage situation is required. It is ideal to identify the disaster locations from images obtained by imaging a city from a high altitude using a high-resolution camera equipped on an artificial satellite or the like. However, the arrival cycle of an artificial satellite is at least one day even at its fastest, and it cannot quickly arrive directly above the disaster area. Therefore, artificial satellites are not suitable for quickly grasping disasters. Furthermore, there are problems such as bad weather often accompanying disasters and clouds hiding the ground surface, making it difficult to recognize using satellite images. On the other hand, a UAV (unmanned aerial vehicle) is suitable for quickly grasping the damage situation in that it can be immediately flown in the disaster area. Moreover, since it flies under the clouds, it is easy to acquire information on the ground surface as an image.

[0003] Patent Document 1 discloses extracting a building area from an image taken from above, and paying attention to the image texture inside the area of each building to identify the damage situation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] For example, in large-scale disasters such as major earthquakes or tornadoes that cause buildings to collapse, it is possible to identify the extent of damage to a building from the image textures inside the building's area. On the other hand, in the case of floods, for example, the building itself does not collapse, so it is impossible to tell from an aerial view whether the building has been damaged. To determine whether a building has been damaged, it is necessary to judge not only the texture of the building's area but also the texture of the ground surface in the surrounding area.

[0006] Therefore, the present invention has been made in view of the above problems, and its purpose is to provide a technology for appropriately identifying abnormalities in regions within a judgment image. [Means for solving the problem]

[0007] To solve the above objectives, the present invention includes: an image feature extraction unit that extracts feature quantities of regions within an image; a configuration relationship extraction unit that extracts the configuration relationship between a predetermined region and a surrounding region based on the feature quantities of a predetermined region within the image extracted by the image feature extraction unit and the feature quantities of a surrounding region around the predetermined region; a configuration relationship difference calculation unit that calculates the difference between the configuration relationship of a plurality of training images and the configuration relationship of a judgment image; and an anomaly determination unit that determines an anomaly in a region within the judgment image based on the difference calculated by the configuration relationship difference calculation unit. [Effects of the Invention]

[0008] According to the present invention, abnormalities in the region within the judgment image can be appropriately identified. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing the hardware configuration of the disaster detection system according to Example 1. [Figure 2] Functional block diagram of the disaster detection system according to Example 1. [Figure 3] A flowchart showing the preliminary preparations for the learning process of the disaster detection system according to Example 1. [Figure 4] Flowchart of the learning process of the disaster detection system according to Example 1. [Figure 5] A flowchart illustrating the arrangement relationship extraction process according to Example 1. [Figure 6] A diagram illustrating the arrangement relationship extraction process in Example 1. [Figure 7] A diagram illustrating the arrangement relationship extraction process in Example 1. [Figure 8] A table showing the positional relationship between a building and other objects in Example 1. [Figure 9] Functional block diagram of the disaster detection system during operation according to Example 1. [Figure 10] A flowchart illustrating the processing during operation of the disaster detection system according to Example 1. [Figure 11] A diagram illustrating the quantitative evaluation of the disaster detection system during operation according to Example 1. [Figure 12] A flowchart illustrating the operational refinement process related to Example 1. [Figure 13] Functional block diagram during operation according to Example 2. [Figure 14] A flowchart illustrating the processing during operation according to Example 2. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, the following embodiments are illustrative for the purpose of explaining the present disclosure, and are not intended to limit the scope of the present disclosure only to those embodiments. A person skilled in the art can implement the present disclosure in various other ways without departing from the scope of the present disclosure. Also, in the configuration of the invention described below, the same reference numerals are commonly used between different drawings for the same part or parts having the same or similar functions, and duplicate descriptions may be omitted. Also, when there are a plurality of elements having the same or similar functions, they may be described with different subscripts attached to the same reference numeral. However, when it is not necessary to distinguish a plurality of elements, the subscript may be omitted in the description. Also, the positions, sizes, shapes, and ranges of each configuration shown in the drawings and the like may not represent the actual positions, sizes, shapes, and ranges in order to facilitate the understanding of the present disclosure. Therefore, the present disclosure is not limited to the positions, sizes, shapes, and ranges disclosed in the drawings and the like. Also, components represented in the singular form in this specification may be plural unless otherwise clearly indicated in the context.

[0011] The disaster detection system performs anomaly detection on the peripheral area of a building by focusing on the difference in the arrangement relationship between the disaster time and the non-disaster time. Specifically, from the image dataset constructed by collecting only the images taken from above in an environment where no disaster exists, the arrangement relationship of the surface textures that can be arranged in the peripheral area of the building is learned as the "normal state". When an image of the disaster area is actually input during operation, the degree of deviation from the "normal state" during learning is evaluated for the arrangement relationship of the surface textures in the peripheral area of the building in the input image, and when the deviation is large, it is determined as an "abnormal state".

Embodiment

[0012] <Overall System Configuration> FIG. 1 is a block diagram showing the hardware configuration of the disaster detection system according to Embodiment 1.

[0013] As an example of the "abnormality detection system" shown in FIG. 1, the disaster detection system 100 is a system for grasping the situation in the disaster area. The disaster detection system 100 may be mounted on a vehicle as an example of a "mobile body" equipped with a driving mechanism and move to the disaster area. The disaster detection system 100 includes a sensor unit 110 and an information processing device 120. The sensor unit 110 may be a mobile sensor capable of acquiring images in the disaster area, such as a drone or a satellite.

[0014] The sensor unit 110 has an imaging unit 111, a communication unit 112, a driving unit 113, and an attitude sensor 114. The imaging unit 111, the communication unit 112, the driving unit 113, and the attitude sensor 114 may be connected to each other. The imaging unit 111 is a sensor such as a camera. The communication unit 112 has a function of being able to transmit the data acquired by the imaging unit 111 to the information processing device 120 and a function of being able to receive a signal from the information processing device 120. The attitude sensor 114 is a sensor such as a gyro. The data acquired by the attitude sensor 114 may be transmitted to the information processing device 120 via the communication unit 112. The driving unit 113 is, for example, a motor. The driving unit 113 has a function of driving based on a signal received from the information processing device 120 by the imaging unit 111, for example, via the communication unit 112.

[0015] The information processing device 120 can be configured by a general server. For example, the information processing device 120 can be configured by a server including an input device, an output device, a processing device, and a storage device as hardware. The information processing device 120 realizes various functions by the processing device reading a program stored in the storage device and executing the read program, and cooperating with other hardware as necessary. Note that the information processing device 120 may be configured by a single server, or any part of the input device, the output device, the processing device, and the storage device may be configured by another computer system connected by a network.

[0016] The information processing device 120 includes, for example, a communication unit 121, a storage device 122, a processor 123, a display unit 124, and a control unit 125 as its functional configuration. The communication unit 121, the storage device 122, the processor 123, the display unit 124, and the control unit 125 may be connected to each other.

[0017] The communication unit 121 may have the function of sending and receiving data or signals arriving from the sensor unit 110. The storage device 112 may be, for example, RAM (Random Access memory). The processor 123 may be, for example, an FPGA (Field Programmable Gate Array) or a GPU (graphic processing unit). The disaster detection method described later may be implemented on the storage device 122 and the processor 123. The display unit 124 may display, for example, the image acquired by the imaging unit 111 received from the sensor unit 110 via the communication unit 121, the information from the attitude sensor 114, and the disaster area estimation result estimated by the disaster detection method described later based on these, in a form such as a GUI. The control unit 125 may control the drive unit 113 via the communication unit 121 and the communication unit 112.

[0018] Figure 2 is a functional block diagram of the disaster detection system according to Example 1.

[0019] The disaster detection system 100 includes a normal city image dataset 201 as an example of "training images", a region segmentation unit 202, an image feature extraction unit 203, a local feature storage unit 208, a placement relationship extraction unit 204, a placement relationship storage unit 205, an image restoration unit 206, and a restoration error calculation unit 207.

[0020] The normal city image dataset 201 may be a set of images taken from above by a satellite or UAV of a city where no disaster has occurred (ground level during non-disaster situations), and for simplicity, such images will be referred to as city images in this specification.

[0021] The region segmentation unit 202 is an algorithm that has a function called region segmentation, such as Mask RCNN or FCN (Fully convolutional network). In this embodiment, the region segmentation unit 202 may use an algorithm that has been pre-trained to estimate, for example, building regions or road regions from urban images.

[0022] The image feature extraction unit 203 extracts image features from regions within the image. The image feature extraction unit 203 may use, for example, a CNN (convolutional neural network) with learnable parameters, or it may introduce a configuration such as a ViT (Vision transformer) with learnable parameters.

[0023] The local feature storage unit 208 may have a function to store the image feature quantities extracted by the image feature extraction unit 203.

[0024] The arrangement relationship extraction unit 204 extracts the arrangement relationship between a predetermined region and surrounding regions based on the feature quantities of a predetermined region in the image extracted by the image feature extraction unit 203 and the feature quantities of a predetermined surrounding region. The arrangement relationship extraction unit 204 may have a structure equipped with learnable parameters. Details of the arrangement relationship extraction unit 204 will be described later. The arrangement relationship extraction unit 204 may have a function to extract combinations of feature quantities that are likely to exist in surrounding regions, in addition to the feature quantities of each region itself, for each image feature quantity extracted by the image feature extraction unit 203 for each region in the image. Specifically, for example, if a building region is used as an example, the arrangement relationship extraction unit 204 may extract, in addition to the feature quantities of the building region itself, that road regions tend to appear around buildings in general, and that road regions tend to be in close proximity to building regions.

[0025] The arrangement relationship storage unit 205 may have a function of storing the arrangement relationship feature quantities extracted by the arrangement relationship extraction unit 204.

[0026] The image reconstruction unit 206 may have the function of reconstructing the original input city image by estimating it from, for example, arrangement relation features and image features extracted by the image feature extraction unit 203. Specifically, the image reconstruction unit 206 may have a structure with learnable parameters such as a linear layer or a deconvolutional layer.

[0027] The reconstruction error calculation unit 207 may perform learning by calculating the error between the image reconstructed by the image reconstruction unit 206 and the actual input city image using a function such as L2 loss, and backpropagating the error. The targets for backpropagation of the error may be the image feature extraction unit 203, the arrangement relationship extraction unit 204, and the image reconstruction unit 206.

[0028] Figure 3 is a flowchart showing the pre-preparation steps for the learning process of the disaster detection system according to Example 1.

[0029] In S301, the disaster detection system 100 prepares a city image dataset 201. Specifically, if the sensor unit 110 is a UAV, the disaster detection system 100 collects city images captured by a similar UAV. The city images to be collected should be from various cities, preferably with a wide variety of images, such as those from different seasons. However, the city image dataset 201 must not contain any disasters that are the target of detection by the disaster detection system 100.

[0030] In S302, the disaster detection system 100 trains the region segmentation unit 202. As mentioned above, the region segmentation unit 202 is a known algorithm such as FCN. The region segmentation unit 202 may use other urban images in addition to the urban image dataset 201 during training. The region segmentation unit 202 performs a known training method to estimate areas such as building regions or road regions, for example, by providing annotations to the urban image.

[0031] Figure 4 is a flowchart of the learning process of the disaster detection system according to Example 1.

[0032] The learning process performed by this disaster detection system 100 is applied to all city images that make up the city image dataset 201. However, for the sake of explanation, the following explanation will focus on a specific city image X1.

[0033] In S401, the region division unit 202 estimates, for example, building regions or road regions from the city image X1 and outputs them as masks M.

[0034] In S402, the image feature extraction unit 203 extracts a group of image features F from the city image X1 (image feature extraction process). When using ViT, for example, the image feature extraction unit 203 may pre-divide the city image X1 into h vertical sections and w horizontal sections, and for each divided region, the CNN included in ViT extracts image features f having, for example, dimension d. As a result, R = h × w image features f, corresponding to the total number of regions, are extracted. The image feature extraction unit 203 may consider these R = h × w image features f as the image feature group F. The image feature extraction unit 203 may receive a mask M at the same time as the input city image X1. Furthermore, since the city image X1 and the mask M have the same vertical and horizontal dimensions, the image feature extraction unit 203 may receive them superimposed in the channel direction, for example. Furthermore, the image feature extraction unit 203 may, for example, receive a city image X1 as input to a CNN, and input a mask M to the CNN using a known algorithm such as AdaIN (Adaptive Instance Normalization). As a result, the image feature extraction unit 203 can extract image features f by taking into account the information of the mask M.

[0035] In S403, the image feature extraction unit 203 stores each image feature f that constitutes the image feature group F extracted in S402 in the local feature storage unit 208. The normal city image dataset 201 contains multiple city images. Therefore, the local feature storage unit 208 stores the image feature f in a way that is associated with each city image. The flowchart in Figure 4 is a loop process, and the image feature f may be updated in the local feature storage unit 208 after each loop process.

[0036] In S404, the arrangement relationship extraction unit 204 extracts arrangement relationship features C for each feature f that constitutes the image feature group F, in relation to the R image feature f that constitute the image feature group F (arrangement relationship extraction process). A specific example of the process in S404 will be described later.

[0037] In S405, the arrangement relationship extraction unit 204 stores the arrangement relationship feature quantity C extracted in S404 in the arrangement relationship storage unit 205. In this storage process, as in S403, the arrangement relationship feature quantity stored in the arrangement relationship storage unit 205 may be updated each time the loop process is completed.

[0038] In S406, the image reconstruction unit 206 receives an integrated feature vector, which is a combination of the spatial relationship feature vector C extracted in S404 and the image feature vector f. The image reconstruction unit 206 then performs the process of reconstructing the input image, the city image X1, and generates a reconstructed image Y.

[0039] In S407, the reconstruction error calculation unit 207 calculates the error between the input image (city image) X1 and the reconstructed image Y using, for example, a known L2 loss function. Subsequently, the reconstruction error calculation unit 207 backpropagates the obtained error to the image feature extraction unit 203, the arrangement relationship extraction unit 204, and the image reconstruction unit 206, which are equipped with learnable parameters, and updates the parameters. This reconstruction learning is self-supervised learning, and the image feature quantities f extracted by the image feature extraction unit 203 are expected to be features that are easy to identify in subsequent processing using methods such as K-means, for example, if extracted from a forest area in the image, they are image feature quantities that represent a forest.

[0040] In S408, the reconstruction error calculation unit 207 decides whether to continue learning. For example, if the error calculated in S407 is smaller than a pre-set value, the reconstruction error calculation unit 207 may decide to terminate learning; otherwise, it returns to S401 and resumes the learning process.

[0041] <Specific Explanation of Arrangement Relationship Extraction Process in S404> FIG. 5 is a flowchart showing the arrangement relationship extraction process according to the first embodiment, and FIGS. 6 and 7 are diagrams for explaining the arrangement relationship extraction process of FIG. 5 according to the first embodiment.

[0042] Image 601 is an example of the urban image X1. For convenience, in the drawing, the building is drawn as if viewed from the side. However, it should be noted that in reality, the building obtained by aerial photography mainly consists of the roof.

[0043] Images 602 and 603 are examples of masks showing the building and road areas in the mask M, respectively. Images 602 and 603 may be obtained by the estimation of the region division unit 202.

[0044] Image 604 shows a state where a grid A obtained by dividing the urban image X1 into 6 equal parts in the vertical direction and 7 equal parts in the horizontal direction is superimposed as an example when using ViT in the image feature extraction process in S402. The image feature f is extracted from the region corresponding to each grid of Image 604. In this division example, an example is shown in which a total of 42 image features f can form the image feature group F.

[0045] First, in S501, the arrangement relationship extraction unit 204 selects, for example, the building Z from the mask M related to the building area of Image 602. Here, the specific process when the building Z depicted in FIG. 601 is selected will be described.

[0046] In S502, the arrangement relationship extraction unit 204 may newly set the size of the grid B according to, for example, the average size of the buildings existing in the mask M. This is an example of a solution to address the fact that when the sensor for acquiring the urban image is a UAV, the imaging altitude for each urban image is different, resulting in different sizes of the buildings reflected in the urban image.

[0047] In S503, the arrangement relationship extraction unit 204 performs a process to reassociate the image feature quantities f of grid A with each grid cell of grid B, which was newly set in S502. As an example of this process, consider the case shown in Figure 7, where image 701 is grid B, image 702 is grid A, and image 703 is an image in which both images 701 and 702 overlap. In this case, the arrangement relationship extraction unit 204 calculates a distance of four units between the center B1 of a grid cell constituting grid B and the four points A1, A2, A3, and A4 that constitute grid A closest to the center B1. Next, the arrangement relationship extraction unit 204 may determine the image feature quantity f of the center B1 by centroid calculation based on the image feature quantities of points A1, A2, A3, and A4 and the calculated distance.

[0048] Image 605 is an image obtained by superimposing the newly defined grid B onto the city image X1. In grid B, new features are defined for each grid cell through the processing in S503.

[0049] In S504, the arrangement relationship extraction unit 204 extracts the arrangement relationship between building Z in the city image X1 and surrounding objects as an arrangement relationship feature C. The arrangement relationship extraction unit 204 may use, for example, the arrangement relationship kernel K shown in Figure 606. Here, as an example, the arrangement relationship kernel K is shown as having 5 vertical and 5 horizontal squares. The size of each grid in the arrangement relationship kernel K may match that of grid B. The size of the arrangement relationship kernel K can be any size, such as 7 vertical and 7 horizontal squares, and will serve as a hyperparameter when training and operating this disaster detection method. Furthermore, each grid in the arrangement relationship kernel K may be distinguished on the arrangement relationship kernel K as K0, K1, ... as shown in Figure 606.

[0050] Figure 8 is a table showing the positional relationship between a building and other objects in Example 1.

[0051] Table 801 shows a specific example of the arrangement of urban image X1 when building Z is placed at the central position K0 of the arrangement kernel K. In Table 801, identification classes are defined as, for example, trees, water, grass, roads, houses, cars, etc. However, these identification classes may be explicitly defined in advance as in Table 801 using the classes estimated by the mask M output by the region segmentation unit 202, or they may be implicitly classified by clustering processing such as K-means.

[0052] In Table 801, for example, since there is a water region in the upper right area of ​​building Z, the corresponding region (k2, k3, k15, k4, k16) can be considered a water region.

[0053] The house may be described as being located in (k8,k24), (k20), and (k22), respectively, with respect to the other three independent houses.

[0054] Table 801 may be converted into arrangement relation feature quantities C relating to the area surrounding building Z by processing as described later. Table 801 may be stored in the arrangement relation feature quantity storage unit 205.

[0055] As an example of feature quantification for Table 801, for each classification class in Table 801, since the arrangement relation kernel K has 25 cells, a 25-dimensional vector with a correspondence relationship for each cell may be defined.

[0056] Furthermore, if the pre-defined classification classes are only six, for example, water, trees, grass, roads, houses, and cars, then the spatial relation feature C can be a 25*6=150-dimensional vector.

[0057] Furthermore, as an example of representing the water class in Table 801 as a 25-dimensional vector, the 2nd, 3rd, 15th, 4th, and 16th elements of the 25-dimensional vector, where the water region exists, may be set to 1, and the other elements to 0.

[0058] Furthermore, as an example of representing the tree class in Table 801 as a 25-dimensional vector, the 1st, 8th, 24th, and 25th elements of the 25-dimensional vector, where tree regions exist, may be set to 1, and the other elements to 0. In the example of the car class in Table 801, since there is no car region, it may be a 25-dimensional vector with all elements being 0. In the example in Table 801, for example, the vectorization process described above may be performed on each class, and the 150-dimensional vector obtained by concatenating all 6 classes may be used as the surrounding arrangement relation feature C.

[0059] Here, the specific processing for building Z was explained. Furthermore, in S505, the arrangement relationship extraction unit 204 performs the same processing for all other buildings present in mask M. This processing was explained using the building area as an example for the city image X1. However, the same processing is also performed for the area of ​​grid B. Specifically, for example, the arrangement relationship with the surrounding area may be extracted as a feature, centered on the road area.

[0060] The extracted spatial relationship feature C is extracted with respect to grid B. However, the spatial relationship extraction unit 204 may reassign the spatial relationship feature C to grid A by performing the same processing as in S503 for subsequent processing.

[0061] Figure 9 is a functional block diagram of the disaster detection system according to Example 1 during operation.

[0062] As described above, the disaster detection system 100 includes a region division unit 202, an image feature extraction unit 203, a placement relationship extraction unit 204, a placement relationship storage unit 205, and a local feature storage unit 208. Furthermore, the disaster detection system 100 includes an imaging unit 111, a local feature difference calculation unit 901, a placement relationship difference calculation unit 902, a difference addition unit 903, and an anomaly determination unit 904. Here, the image feature extraction unit 203, the placement relationship extraction unit 204, the placement relationship storage unit 205, and the local feature storage unit 208 may be learned by the learning method described above.

[0063] The imaging unit 111 flies over the disaster area and captures a city image X2 as an example of a "judgment image".

[0064] The local feature difference calculation unit 901 calculates, for each region, the L2 distance between the image feature quantity f and the element closest to the image feature quantity f among the individual feature quantities stored in the local feature storage unit 208.

[0065] The placement relationship difference calculation unit 902 calculates the difference between the placement relationship feature quantity C and the feature quantity stored in the placement relationship storage unit 205.

[0066] The difference addition unit 903 calculates the added difference by adding the difference calculated by the local feature difference calculation unit 901 and the difference calculated by the arrangement relationship difference calculation unit 902.

[0067] The abnormality determination unit 904 compares the added difference calculated by the difference addition unit 903 with a preset threshold, and determines that an abnormality exists if the difference exceeds the threshold.

[0068] Figure 10 is a flowchart showing the processing during operation of the disaster detection system according to Example 1.

[0069] In S1001, the imaging unit 111 flies over the disaster area and captures an image of the city X2.

[0070] In S1002, the region division unit 202 estimates a mask M for the city image X2.

[0071] In S1003, the image feature extraction unit 203 estimates the image feature set F from the city image X2 using the same processing as in S402.

[0072] In S1004, the arrangement relationship extraction unit 204 extracts arrangement relationship features C for each region from each image feature f that constitutes the image feature group F, using the same processing as in S404.

[0073] In S1005, the local feature difference calculation unit 901 calculates, for example, the L2 distance for each region by comparing the image feature quantity f with the element closest to the image feature quantity f among the individual feature quantities stored in the local feature memory unit 208. Since the disaster detection system 100 of this specification is trained by reconstruction learning during training, regions that did not exist during training are difficult to reconstruct. Therefore, if a region includes something that did not exist during training, such as flames or smoke, those regions are difficult to reconstruct, and since there are no image feature quantities close to the image feature quantity f stored in the local feature memory unit 208, the difference calculated by the local feature difference calculation unit 901 becomes large.

[0074] At the same time, the arrangement relationship difference calculation unit 902 may calculate the difference between the arrangement relationship feature quantity C and the features stored in the arrangement relationship storage unit 205. For example, the arrangement relationship difference calculation unit 902 may add, for example, the L2 distance to the elements of the arrangement relationship feature quantity C that are close in distance from the arrangement relationship feature quantity C, and use that as the difference.

[0075] In S1006, the difference addition unit 903 calculates the added difference by adding the difference calculated by the local feature difference calculation unit 901 and the difference calculated by the arrangement relationship difference calculation unit 902.

[0076] Furthermore, the abnormality determination unit 904 compares the added difference calculated by the difference addition unit 903 with a preset threshold and determines that the region exceeding the threshold is an abnormal region.

[0077] Figure 11 is a diagram illustrating the quantitative evaluation of the disaster detection system during operation according to Example 1.

[0078] Here, the processing of the arrangement relationship difference calculation unit 902 will be explained visually. The disaster detection system 100 is equipped with a database 1100. The database 1100 is a visual representation of the arrangement relationship between the water region and, for example, the central region being a building, from the arrangement relationship storage unit 205, according to the arrangement relationship kernel K. In this example, the database 1100 has arrangement relationships 1101 to 1106. Here, a rule may be added that rotated versions of each element constituting the database 1100 are considered identical, and specifically, arrangement relationship 1101 and arrangement relationship 1102 may be considered identical.

[0079] The placement relationship feature difference calculation unit 902 may, for example, when a placement relationship 1107 is extracted in the relationship between a building area and a water area during actual operation, compare it with the most similar placement relationship 1102 in the database 1100 and evaluate the difference to be equivalent to 2 squares. On the other hand, when a placement relationship 1108 is extracted, the placement relationship difference calculation unit 902 may compare it with the most similar placement relationship 1102 and evaluate the difference to be equivalent to 8 squares. In this example, intuitively, the database 1100 has learned that buildings exist that are located near rivers or ponds. However, cases like placement relationship 1108, where water areas exist on two sides of a building, indicate a large difference from the database 1100.

[0080] In S1006, the arrangement relationship difference calculation unit 902 compares the difference for each region with a preset threshold, and determines that it is abnormal if it exceeds the threshold.

[0081] In the above-described operational processing, the arrangement relationship difference calculation unit 902 may perform processing on the image group captured by the sensor unit 110. Furthermore, the information processing device 120, which receives the images acquired by the sensor unit 110 during flight via the communication units 112 and 121, may process them in real time. Furthermore, the display unit 124 may display the areas determined to be abnormal. Furthermore, the control unit 125 may send a signal to the drive unit 113 to cause the sensor unit 110 to approach the areas determined to be abnormal, and then present the details of the affected area to the user through the imaging unit 111 and the display unit 124. This makes it possible to contribute to understanding the disaster situation.

[0082] Figure 12 is a flowchart showing the operational refinement process according to Example 1.

[0083] Next, we will explain how to retrain the algorithm that the disaster detection system 100 learned when it made a false detection.

[0084] This disaster detection system 100 uses a method to detect disasters based on combinations of object placements that do not exist in the urban image dataset 201. Therefore, even if there is no actual disaster, the disaster detection system 100 may issue an incorrect disaster determination (false positive) if it receives input for an object placement that does not exist in the urban image dataset 201. The processing described here is an example of a process to correct false positive images.

[0085] In S1201, the imaging unit 111 actually captures an image of the city X2 using, for example, a UAV.

[0086] In S1202, the disaster detection system 100 performs disaster detection on the city image X2.

[0087] In S1204, the disaster detection result is checked, for example, by a human visual inspection or by another disaster detection system 100 to determine whether it is a false positive. If a disaster has been determined when no disaster exists, in S1205, the disaster detection system 100 may update its knowledge of the combination by adding the city image X2 to the city image dataset 201 and running the learning process again.

[0088] The disaster detection system 100 includes an image feature extraction unit 203, a placement relationship extraction unit 204, a placement relationship difference calculation unit 902, and an anomaly determination unit 904. The image feature extraction unit 203 extracts feature quantities of regions within urban images X1 and X2. The placement relationship extraction unit 204 extracts the placement relationship between a predetermined region and a surrounding region based on the feature quantities of a predetermined region within images X1 and X2 extracted by the image feature extraction unit 203 and the feature quantities of a predetermined surrounding region. The placement relationship difference calculation unit 902 calculates the difference between the placement relationships of multiple urban images X1 and the placement relationship of urban image X2. The anomaly determination unit 904 determines an anomaly in a region within urban image X2 based on the difference calculated by the placement relationship difference calculation unit 902.

[0089] This allows for the proper identification of anomalies within areas of the urban image X2. In particular, in disaster-stricken areas, even if the building itself is not damaged, it is possible to estimate whether or not the building has been damaged based on the surrounding area. [Examples]

[0090] Embodiment 2 of the present invention will be described with reference to Figures 13 and 14. In this embodiment, only the differences from Embodiment 1 will be described.

[0091] In Example 2, we will describe an example of a different method for detecting a disaster using the disaster detection method learned in Example 1.

[0092] Figure 13 is a functional block diagram of the operation according to Example 2.

[0093] The disaster detection system 200 detects disasters using a pre-trained disaster detection method. The disaster detection system 200 may be used as an alternative to the method shown in Figure 9, and as an example, it comprises an imaging unit 111, a region division unit 202, an image feature extraction unit 203, a placement relationship extraction unit 204, an image reconstruction unit 206, and a reconstruction error calculation unit 207.

[0094] Figure 14 is a flowchart showing the processing during operation according to Example 2.

[0095] The disaster detection system 200 performs disaster detection using the functional configuration shown in Figure 13. S1401, S1402, S1403, and S1404 may perform the same processing as S1001, S1002, S1003, and S1004, respectively.

[0096] In S1405, the same process as in S406 is performed. That is, the image reconstruction unit 206 reconstructs the reconstructed image Y based on a feature obtained by combining, for example, the image feature quantity f extracted in S1403 and the arrangement relation feature quantity C extracted in S1404.

[0097] In S1406, the reconstruction error calculation unit 207 may, for example, measure the L2 distance for each pixel between the city image X2 and the reconstructed image Y, and use the L2 distance as an evaluation value to determine anomalies for each pixel. During training, the reconstructed image Y is reconstructed based on an integrated feature obtained by combining the image feature f and the placement relation feature C. Therefore, since combinations of image feature f and placement relation feature C that did not exist during training cannot be well reconstructed on the reconstructed image Y, the degree of anomaly can be evaluated.

[0098] Furthermore, the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described.

[0099] For example, in the above embodiment 1, the abnormality determination unit 904 determined the abnormal region based on the added difference calculated by the difference addition unit 903. However, the abnormality determination unit 904 may also determine the abnormal region based on the difference calculated by the arrangement relationship difference calculation unit 902. [Explanation of Symbols]

[0100] 100...Disaster detection system, 124...Display unit, 200...Disaster detection system, 201...Normal city image dataset, 202...Region segmentation unit, 203...Image feature extraction unit, 204...Placement relationship extraction unit, 205...Placement relationship storage unit, 206...Image restoration unit, 207...Restoration error calculation unit, 901...Local feature difference calculation unit, 902...Placement relationship difference calculation unit, 904...Anomaly determination unit

Claims

1. An image feature extraction unit that extracts feature quantities from regions within an image, Based on the feature quantities of a predetermined region in the image extracted by the image feature extraction unit and the feature quantities of the surrounding region around the predetermined region, the arrangement relationship extraction unit extracts the arrangement relationship between the predetermined region and the surrounding region. A configuration relationship difference calculation unit calculates the difference between the configuration relationships of multiple training images and the configuration relationships of the judgment image, An anomaly detection system comprising: an anomaly determination unit that determines an anomaly in a region within the determination image based on the difference calculated by the arrangement relationship difference calculation unit.

2. The aforementioned set of training images consists of images taken from above of a city during a non-disaster situation. The image feature extraction unit and the arrangement relationship extraction unit learn the image group, The anomaly detection system according to claim 1.

3. An image restoration unit that restores the learning image based on the feature quantities of the predetermined region within the learning image and the feature quantities of the surrounding region, The system includes a restoration error calculation unit that calculates the error between the training image and the image restored by the image restoration unit, The reconstruction error calculation unit backpropagates the error to the image feature extraction unit, the arrangement relationship extraction unit, and the image reconstruction unit. The anomaly detection system according to claim 1.

4. The image feature extraction unit has a region segmentation unit that inputs the inference results of the region segmentation algorithm. The anomaly detection system according to claim 1.

5. The arrangement relationship extraction unit classifies the predetermined region and the surrounding region into predefined identification classes and extracts the arrangement relationship between the class of the predetermined region and the class of the surrounding region. The anomaly detection system according to claim 1.

6. The arrangement relationship extraction unit sets the size of the predetermined region for each of the training images based on the size of the object inferred by the region segmentation algorithm. The anomaly detection system according to claim 1.

7. The arrangement relationship extraction unit sets the size of the peripheral area based on the size of the predetermined area or the size of the peripheral area relative to the predetermined area that has been set in advance. The anomaly detection system according to claim 1.

8. Having an arrangement relationship storage unit that stores the aforementioned arrangement relationship, The anomaly detection system according to claim 1.

9. The system includes a local feature difference calculation unit that calculates the difference between predetermined regions in the plurality of training images and predetermined regions in the judgment image. The abnormality determination unit determines an abnormality in the region within the determination image based on the difference calculated by the arrangement relationship difference calculation unit and the difference calculated by the local feature difference calculation unit. The anomaly detection system according to claim 1.

10. An image restoration unit that restores the learning image based on the feature quantities of the predetermined region within the learning image and the feature quantities of the surrounding region, The system includes a restoration error calculation unit that calculates the error between the image restored by the image restoration unit and the training image, The restoration error calculation unit determines an abnormality in the region within the judgment image based on the difference calculated by the restoration error calculation unit. The anomaly detection system according to claim 1.

11. The system has a display unit that displays the determination result from the abnormality determination unit. The anomaly detection system according to claim 1.

12. If the abnormality detection unit determines that a detection image from a non-disaster situation is abnormal, it adds that detection image to the learning image. The anomaly detection system according to claim 1.

13. An anomaly detection system according to any one of claims 1 to 12, A mobile body comprising the aforementioned anomaly detection system and a movable drive mechanism.

Images