Detection device, detection method, and detection program

Abstract

A detection device (10) includes a determination unit (132) and a decoding unit (133). The determination unit (132) determines whether or not the difference between a first feature obtained by encoding a first image and a second feature obtained by encoding a second image exceeds a threshold. When it is determined that the difference exceeds the threshold, the decoding unit (133) decodes the first feature, the second feature, a third feature obtained by encoding the first feature, and a fourth feature obtained by encoding the second feature.

Description

Detection device, detection method, and detection program

[0001] The present invention relates to a detection device, a detection method, and a detection program.

[0002] A technique called change detection is known, which compares images of the same location taken at different times. Change detection is used to monitor changes in environments, cities, etc. In addition, a technique is known that uses machine learning algorithms such as Transformers to improve the accuracy and efficiency of change detection (see, for example, Non-Patent Document 1).

[0003] Bandara, W. G. C., & Patel, V. M. (2022, July). "A transformer based siamese network for change detection.", In IGARSS 2022 2022 IEEE International Geoscience and Remote Sensing Symposium (pp.207-210). IEEE.

[0004] However, change detection may not be easily implemented using conventional techniques.

[0005] For example, change detection may be performed on terrestrial imagery captured by satellites, and change detection using machine learning models requires significant computing resources.

[0006] However, satellite-mounted computers may not have sufficient computing resources, and in such cases, it is necessary to transmit images from the satellite to a high-performance computer on the ground to perform change detection.

[0007] In order to solve the above-mentioned problems and achieve the object, the detection device is characterized by having a determination unit that determines whether a difference between a first feature obtained by encoding a first image and a second feature obtained by encoding a second image exceeds a threshold, and a decoding unit that, when it is determined that the difference exceeds the threshold, decodes the first feature, the second feature, a third feature obtained by encoding the first feature, and a fourth feature obtained by encoding the second feature.

[0008] According to the present invention, change detection can be easily performed.

[0009] FIG. 1 is a diagram illustrating an example of the configuration of a detection device according to a first embodiment. FIG. 2 is a diagram illustrating conventional change detection. FIG. 3 is a diagram illustrating change detection according to the first embodiment. FIG. 4 is a diagram illustrating the processing flow of the first embodiment. FIG. 5 is a diagram illustrating the distance between feature amounts. FIG. 6 is a diagram illustrating the distance between feature amounts. FIG. 7 is a flowchart illustrating the processing flow of the detection device according to the first embodiment. FIG. 8 is a diagram illustrating the results of an experiment. FIG. 9 is a diagram illustrating the results of the experiment. FIG. 10 is a diagram illustrating the results of the experiment. FIG. 11 is a diagram illustrating an example of a computer that executes a detection program.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a detection device, a detection method, and a detection program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0011] One objective of this embodiment is to easily perform change detection on images captured by a satellite. For example, in this embodiment, a satellite captures images of the Earth (e.g., the sea or land). A detection device mounted on the satellite performs change detection on images of the same location captured at different times. In this case, the detection device may terminate the process midway (Early Exit) as necessary without performing all the processing related to change detection. Furthermore, the detection device performs change detection using a model including an encoder and decoder, with the decoder being lighter than conventional models.

[0012] Conventional motion detection will be described using Fig. 2. Fig. 2 is a diagram illustrating conventional motion detection. As shown in Fig. 2, the model has an encoder 30a and a decoder 40a. An image 201 and an image 202 are input to the model. The image 202 is an image captured at the same location as the image 201 but at a later date and time than the image 201.

[0013] The encoder 30a extracts information (intermediate features) by encoding the features of the image. The decoder 40a decodes the intermediate features extracted by the encoder 30a and creates a change map 210a. The change map is a map that highlights areas where changes have occurred in the image. In particular, the decoder 40a performs expensive processing, and there are cases where it cannot be implemented by a computer onboard a satellite.

[0014] 3 is a diagram illustrating change detection according to the first embodiment. As shown in FIG. 3, the model according to the first embodiment includes an encoder 30 and a decoder 40. In the model, intermediate features extracted by the encoder 30 are decoded by the decoder 40 to create a change map 210.

[0015] The encoder 30 has a plurality of encoders (encoder 301, encoder 304, etc.) arranged in multiple stages. If a condition is met, an early exit is performed midway through the encoder 30, and processing is stopped. For example, if no significant difference is found between the image 201 and the image 202, an early exit is performed, and unnecessary processing is omitted.

[0016] Satellite images often contain areas where no significant events are occurring. For example, more than half of the images in the change detection dataset presented in Reference 1 showed no change.

[0017] Reference 1: Hao Chen and Zhenwei Shi, “A spatial temporal attention based method and a new dataset for remote sensing image change detection,” Remote Sensing, vol. 12, no. 10, pp. 1662, 2020

[0018] [Configuration of First Embodiment] The configuration of a detection device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a detection device according to the first embodiment. The detection device 10 is a computer mounted on an artificial satellite.

[0019] As shown in FIG. 1 , the detection device 10 includes a communication unit 11 , a storage unit 12 , and a control unit 13 .

[0020] The communication unit 11 performs data communication with other devices via a network. For example, the communication unit 11 is a communication module for performing satellite communication.

[0021] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. Note that the storage unit 12 may also be a data-rewritable semiconductor memory such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The storage unit 12 stores an operating system (OS) and various programs executed by the detection device 10.

[0022] The storage unit 12 stores model information 121. The model information 121 is information related to a model. For example, the model information 121 is parameters such as weights and biases for constructing a model that is a neural network. For example, the model includes an encoder 30 and a decoder 40.

[0023] The control unit 13 controls the entire detection device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control unit 13 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

[0024] The control unit 13 functions as various processing units by running various programs, and includes, for example, an encoding unit 131, a determining unit 132, a decoding unit 133, and a map creating unit .

[0025] Here, the encoder 30 and decoder 40 of this embodiment may have the same configuration as the Hierarchical Transformer Encoder and Lightweight MLP Decoder included in the Transformer described in Non-Patent Document 1, respectively.

[0026] The processing of each part of the control unit 13 will be described with reference to Fig. 4. Fig. 4 is a diagram illustrating the flow of processing in the first embodiment.

[0027] The encoding unit 131 encodes the features of an image using the encoder 30. The encoder 30 is a multi-stage encoder and includes an encoder 301, an encoder 302, an encoder 303, and an encoder 304. Each encoder included in the encoder 30 further encodes the output of the encoder in the previous stage.

[0028] The encoder 301 encodes the feature of the image 201 to obtain the feature F 1 pre The encoder 302 outputs the feature F 1 pre Encode the feature F 2 preThe encoder 303 outputs the feature F 2 pre Encode the feature F 3 pre The encoder 304 outputs the feature F 3 pre Encode the feature F 4 pre Output.

[0029] The encoder 301 encodes the feature of the image 202 and generates the feature F 1 post The encoder 302 outputs the feature F 1 post Encode the feature F 2 post The encoder 303 outputs the feature F 2 post Encode the feature F 3 post The encoder 304 outputs the feature F 3 post Encode the feature F 4 post Output.

[0030] Image 201 is an example of a first image. Image 202 is an example of a second image. Feature F 3 pre is an example of the first feature amount. 3 post is an example of the second feature amount. 4 pre is an example of the third feature amount. 4 post is an example of the fourth feature amount.

[0031] The number of stages of the encoder 30 is not limited to four. If the encoder 30 has N stages, the first feature and the second feature can be said to be the feature output by the (N-1)th stage encoder. Furthermore, the third feature and the fourth feature can be said to be the feature output by the Nth stage encoder.

[0032] The determination unit 132 determines the feature F 3 preand the feature F 3 post The determination unit 132 determines whether the difference between and exceeds a threshold. If the difference exceeds the threshold, the detection device 10 performs Early Exit. The determination unit 132 makes the determination using a determination module 1321.

[0033] If it is determined that the difference exceeds the threshold, the decoding unit 133 decodes the feature F 3 pre and feature F 3 post and feature F 3 pre The encoded feature F 4 pre and feature F 3 post The encoded feature F 4 post and decode.

[0034] The specific processing contents of the decoding unit 133 will be described. The decoding unit 133 decodes the feature quantity F 3 pre and feature F 3 post The absolute value of the difference between is calculated as in equation (1).

[0035]

[0036] The decoding unit 133 similarly decodes the feature F 4 pre and feature F 4 post Calculate the absolute value of the difference between

[0037] The decoding unit 133 uses a block included in the MLP-Mixer (see Reference 2) to calculate V in equation (2). l and Y in equation (3) l Calculate.

[0038]

[0039]

[0040] Reference 2: Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy, Mlpmixer: An all mlp architecture for vision,” 2021

[0041] MPL l token is the token-mixing layer of MLP-Mixer. l channel is the channel-mixing layer of MLP-Mixer.

[0042] 1 is the stage of the encoder 30. For example, Y 3 is the feature F output from the encoder 303. pre 3 and feature F post 3 It is calculated based on, for example, Y 4 is the feature F output from the encoder 304. pre 4 and feature F post 4 It is calculated based on the

[0043] The decoding unit 133 3 and Y 4 As shown in equation (4), the combination is a vector concatenation, a convolution (conv2d 1x1 ), including batch normalization (BN).

[0044]

[0045] In this way, the decoding unit 133 decodes the feature F 3 pre and feature F 3 postThe absolute value of the difference between 3 and feature F 4 pre and feature F 4 post The absolute value of the difference between 4 Decode the value Z obtained by combining and.

[0046] The determination module 1321 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 and Fig. 6 are diagrams illustrating the distance between feature quantities. When there is a large change between the image 201 and the image 202 (when there is an event), the encoder 303 determines the feature quantity F 3 pre and feature F 3 post If the distance between and becomes large and there is no significant change, the feature F 3 pre and feature F 3 post The distance between the two is learned in advance so that it becomes small.

[0047] The encoder 303 is trained using Triplet Loss. The loss function of Triplet Loss is as shown in equation (5). a is an anchor vector. p is a positive vector. n is a negative vector. For example, the feature amount of a reference image is a, the feature amount of an image with an event relative to the reference image is n, and the feature amount of an image with no event relative to the reference image is p. d is a function for calculating Euclidean distance. margin is a hyperparameter.

[0048]

[0049] By learning using Triplet Loss, the encoder 303 performs encoding so that the distance between features becomes closer when there is no event between two images, and so that the distance between features becomes greater when there is an event between the two images. Note that the encoder 303 may be trained using a loss function other than Triplet Loss.

[0050] The determination module 1321 determines the feature F 3 pre and feature F 3 post and outputs whether the Euclidean distance between the image 201 and the image 202 is equal to or greater than a threshold value θ. If the distance is less than the threshold value θ, the determination unit 132 determines that there is no significant change between the image 201 and the image 202, and performs Early Exit. Note that θ may be determined in advance by learning.

[0051] [Processing Flow of First Embodiment] The processing flow of the detection device 10 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the processing flow of the detection device according to the first embodiment.

[0052] 7, first, the encoding unit 131 acquires two images captured at different dates and times (step S101). Next, the encoding unit 131 executes the first half of encoding (encoder 301, encoder 302, encoder 303) on the feature quantities of each image (step S102).

[0053] The determination unit 132 determines whether there is a significant change between the images based on the distance between the feature amounts (step S103). If there is no significant change (step S103, No), the detection device 10 stops processing (step S104).

[0054] If there is a large change (Yes in step S103), the encoding unit 131 executes the second half of encoding (encoder 304) on the intermediate feature (step S105).

[0055] Next, the decoding unit 133 combines the multiple feature quantities obtained by encoding (step S106).The decoding unit 133 then decodes the combined value (step S107).The map creation unit 134 creates and outputs a change map based on the values ​​obtained by decoding (step S108).

[0056] [Evaluation of the embodiment] Experiments were carried out to evaluate the embodiment using two types of encoders 30, ScratchFormer (see Reference 3) and ChangeFormer (see Non-Patent Document 1).

[0057] Reference 3: Mubashir Noman, Mustansar Fiaz, Hisham Cholakkal, Sanath Narayan, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan, Remote Sensing Change Detection with Transformers Trained from Scratch

[0058] The variations of the decoder 40 are as follows: Decoder-A: Feature F 3 pre and feature F 3 post Pairs and feature F 4 pre and feature F 4 post Decode using MLP-Mixer with the same weights for the pair of Decoder-B: Feature F 3 pre and feature F 3 post Pairs and feature F 4 pre and feature F 4 post Decode using MLP-Mixer with different weights for the pair of . Decoder-C: Feature F 3 pre and feature F 3 post Pairs and feature F 4 pre and feature F 4 post and decode it by combining the pair (method of equation (4)).

[0059] The results of measuring the Intersection over Union (IoU) and F1 score for combinations of two types of encoders 30 and three variations of the decoder 40 are shown in Figure 8. Figure 8 shows the experimental results. The values ​​in parentheses indicate the difference from the baseline. In many combinations, performance is improved compared to the baseline.

[0060] The amount of data transmission reduction achieved by Early Exit is shown in Fig. 9. Fig. 9 shows the experimental results. Fig. 9 also shows the Precision (Equation (6)) and Recall (Equation (7)) of change detection.

[0061]

[0062]

[0063] TP (True Positive) is the number of cases where the detection result was true when the correct answer was true. FP (False Positive) is the number of cases where the detection result was true when the correct answer was false. FN (False Negative) is the number of cases where the detection result was false when the correct answer was false.

[0064] The runtime metrics for each combination are shown in Fig. 10. Fig. 10 is a diagram showing the results of the experiment. According to the embodiment, the execution speed of change detection was improved, and the memory usage and inference execution time were reduced.

[0065] [Program] In one embodiment, the detection device 10 can be implemented by installing a detection program that executes the above-described processes as package software or online software on a desired computer. For example, by having an information processing device execute the above-described detection program, the information processing device can function as the detection device 10. The information processing device referred to here includes desktop and notebook personal computers. Other examples of information processing devices include smartphones, tablet terminals, and the like.

[0066] 8 is a diagram showing an example of a computer that executes a detection program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0067] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0068] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the detection device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the detection device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD.

[0069] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0070] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0071] REFERENCE SIGNS LIST 10 Detector 11 Communication unit 12 Storage unit 13 Control unit 30, 301, 302, 303, 304 Encoder 40 Decoder 121 Model information 131 Encoding unit 132 Determination unit 133 Decoding unit 134 Map creation unit

Claims

1. A detection device comprising: a determination unit that determines whether a difference between a first feature obtained by encoding a first image and a second feature obtained by encoding a second image exceeds a threshold; and a decoding unit that, when it is determined that the difference exceeds the threshold, decodes the first feature, the second feature, a third feature obtained by encoding the first feature, and a fourth feature obtained by encoding the second feature.

2. The detection device according to claim 1, characterized in that the decoding unit decodes a value obtained by combining a value obtained by processing the absolute value of the difference between the first feature amount and the second feature amount using an MLP-Mixer and a value obtained by processing the absolute value of the difference between the third feature amount and the fourth feature amount using an MLP-Mixer.

3. A detection method executed by a detection device, comprising: a determination step of determining whether a difference between a first feature obtained by encoding a first image and a second feature obtained by encoding a second image exceeds a threshold; and a decoding step of decoding the first feature, the second feature, a third feature obtained by encoding the first feature, and a fourth feature obtained by encoding the second feature, if it is determined that the difference exceeds the threshold.

4. A detection program that causes a computer to execute the following steps: a determination step of determining whether a difference between a first feature obtained by encoding a first image and a second feature obtained by encoding a second image exceeds a threshold; and a decoding step of decoding the first feature, the second feature, a third feature obtained by encoding the first feature, and a fourth feature obtained by encoding the second feature when it is determined that the difference exceeds the threshold.

Images