Detection device, detection method, and detection program

Abstract

A detection device (10) comprises a division unit (131), a determination unit (132), and a transmission control unit (133). The division unit (131) divides an image into a plurality of regions. The determination unit (132) determines, for each of the plurality of regions, whether an event is present or not. The transmission control unit (133) transmits, to an external device, an image of a region in which the event is determined to be present among the plurality of regions.

Description

Detection device, detection method, and detection program

[0001] The present invention relates to a detection device, a detection method, and a detection program.

[0002] Recent advances in space-related hardware have increased the computing resources onboard artificial satellites, enabling them to perform some of the processing of acquired data, such as images, in real time (see, for example, Non-Patent Document 1).

[0003] Yuanyuan Wang, Chao Wang, Hong Zhang, Yingbo Dong, and Sisi Wei, “A sar dataset of ship detection for deep learning under complex backgrounds,” Remote Sensing, vol. 11, no. 7, 2019.

[0004] However, with conventional techniques, there are cases where it is not possible to efficiently process data acquired by satellites.

[0005] For example, because processing using machine learning models places a very heavy processing load on the system, the computing resources onboard a satellite may not be able to process data efficiently using the machine learning models.

[0006] It is also conceivable to transmit data acquired by satellites to computers on the ground for processing, but this would incur costs for transmitting the data.

[0007] In order to solve the above-mentioned problems and achieve the objectives, the detection device is characterized by having a division unit that divides an image into multiple regions, a determination unit that determines whether or not an event has occurred for each of the multiple regions, and a transmission control unit that transmits images of regions among the multiple regions that have been determined to have an event to an external device.

[0008] According to the present invention, data acquired by an artificial satellite can be processed efficiently.

[0009] FIG. 1 is a diagram illustrating an example of the configuration of a detection device according to a first embodiment. FIG. 2 is a diagram illustrating image segmentation. FIG. 3 is a flowchart illustrating the processing flow of the detection device according to the first embodiment. FIG. 4 is a diagram illustrating precision versus recall for Faster RCNN. FIG. 5 is a diagram illustrating reduction rate versus recall for Faster RCNN. FIG. 6 is a diagram illustrating precision versus recall for Yolov7. FIG. 7 is a diagram illustrating reduction rate versus recall for Yolov7. FIG. 8 is a diagram illustrating an example of a computer that executes a detection program.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a detection device, a detection method, and a detection program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0011] Here, satellites may use Synthetic Aperture Radar (SAR) to take images. SAR imaging has the advantage of being less sensitive to various weather conditions than other optical imaging methods. This makes SAR imaging particularly suitable for continuous monitoring, which is susceptible to changes in the atmosphere.

[0012] Furthermore, it is known that imaging of water surfaces is easily affected by weather conditions such as clouds, fog, and darkness. Due to the aforementioned characteristics, SAR imaging methods are well suited to imaging water surfaces.

[0013] One objective of this embodiment is to efficiently process data acquired using SAR. For example, in this embodiment, a satellite uses SAR to capture images of water surfaces (e.g., oceans) on Earth. A detection device mounted on the satellite transmits a portion of the captured image data to a computer on the ground. The computer on the ground performs a detailed analysis of the received images. Note that the computer on the ground has more abundant computing resources than the satellite.

[0014] [Configuration of First Embodiment] The configuration of a detection device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a detection device according to the first embodiment. The detection device 10 is a computer mounted on an artificial satellite.

[0015] As shown in FIG. 1 , the detection device 10 includes a communication unit 11 , a storage unit 12 , and a control unit 13 .

[0016] The communication unit 11 performs data communication with other devices via a network. For example, the communication unit 11 is a communication module for performing satellite communication.

[0017] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. Note that the storage unit 12 may also be a data-rewritable semiconductor memory such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The storage unit 12 stores an operating system (OS) and various programs executed by the detection device 10.

[0018] The storage unit 12 stores model information 121. The model information 121 is information related to a model. For example, the model information 121 is parameters such as weights and biases for constructing a model that is a neural network. For example, the model is Faster RCNN (see Reference 1) or YoloV7 (see Reference 2).

[0019] Reference 1: Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, “Faster r cnn : Towards real time object detection with region proposal networks,” Advances in neural information processing systems, vol. 28.

[0020] Reference 2: Chien-Yao Wang, Alexey Bochkovskiy, and Hong Yuan Mark Liao, “Yolov7: Trainable bag of freebies sets new state of the art for real time object detectors,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp.7464 7475.

[0021] The control unit 13 controls the entire detection device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The control unit 13 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

[0022] The control unit 13 functions as various processing units by running various programs. For example, the control unit 13 includes a dividing unit 131, a determining unit 132, and a transmission control unit 133.

[0023] The dividing unit 131 divides an image into a plurality of regions. In this example, the dividing unit 131 divides an image of the ocean captured by an imaging method using SAR.

[0024] 2 is a diagram illustrating image division. As shown in FIG. 2, the division unit 131 divides the image into a plurality of rectangular regions. In the following description, each divided region may be referred to as a tile.

[0025] The determination unit 132 determines whether or not an event occurs for each of the multiple regions, i.e., for each tile. The occurrence of an event may mean that the image satisfies a condition.

[0026] Here, the determination unit 132 determines that an event has occurred if a ship is captured in a tile. The determination unit 132 uses a model (machine learning model) for object detection to determine whether a ship is captured in each tile. The determination unit 132 makes the determination using, for example, Faster RCNN or YoloV7, which are constructed based on the model information 121.

[0027] The model outputs a score based on the input image according to the probability that an object is captured in the image. If the score output by the model for a tile exceeds a threshold, the determination unit 132 determines that an object is captured in the tile. In this case, the object is a ship.

[0028] Faster RCNN consists of two modules: a Region Proposal Network (RPN) that predicts object regions, and a Fast RCNN detector that performs classification. Faster RCNN also extracts features from images using Resnet50 and a Feature Pyramid Network (FPN) (see Reference 3).

[0029] Reference 3: Tsung Yi Lin, Piotr Doll ´ ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie, “Feature pyramid networks for object detection,” 2017.

[0030] Models including Faster RCNN, or YoloV7, were trained on the SARship dataset (see Non-Patent Document 1), which consists of 102 China Gaofen-3 images and 108 Sentinel-1 images. The images in the SARship dataset contain 39,729 ships with numerous annotations of different scales and backgrounds.

[0031] The model is evaluated by Precision (Equation (1)), Recall (Equation (2)), and F1 Score (Equation (3)) for the training data.

[0032]

[0033]

[0034]

[0035] If the score for a tile output by the model exceeds a preset threshold, the determination unit 132 determines that a ship is depicted in the tile, i.e., that the result is True. On the other hand, if the score for a tile output by the model is equal to or less than the threshold, the determination unit 132 determines that a ship is not depicted in the tile, i.e., that the result is False. The truth or falsity of the correct answer in the learning data (the aforementioned SARship dataset) is known.

[0036] The total number of cases is the number of tiles obtained by dividing one or more images included in the training data. TP (True Positive) is the number of cases where the judgment result was true when the correct answer was true. FP (False Positive) is the number of cases where the judgment result was true when the correct answer was false. FN (False Negative) is the number of cases where the judgment result was false when the correct answer was false. Note that TP, FP, and FN are not included in the training data, but are values ​​calculated based on the training data.

[0037] Higher precision and higher recall are both desirable. However, there is a trade-off between precision and recall. If you increase the threshold, precision increases and recall decreases. If you decrease the threshold, precision decreases and recall increases.

[0038] Therefore, the determination unit 132 uses a threshold value set to maximize the F1 score. That is, the determination unit 132 determines that an event has occurred when the score calculated by the model for each of the multiple regions exceeds the threshold value set to maximize the F1 score of the model for the training data.

[0039] The transmission control unit 133 transmits images (tiles) of areas where it has been determined that an event has occurred, among the multiple areas, to an external device (for example, a computer on the ground).

[0040] [Processing Flow of First Embodiment] The processing flow of the detection device 10 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the processing flow of the detection device according to the first embodiment.

[0041] 3, first, the dividing unit 131 acquires data (step S101). For example, the data is an image of the ocean captured using SAR. The dividing unit 131 divides the acquired image into multiple regions (step S102).

[0042] The determination unit 132 selects an unselected area from among the divided image areas (tiles) (step S103). For example, the determination unit 132 selects areas from the areas shown in FIG. 2 in order from the northwest.

[0043] The determination unit 132 determines whether or not there is a significant change in the selected area, i.e., whether or not there is an event (step S104). If the score obtained by inputting the tile into the model exceeds the threshold, the determination unit 132 determines Yes in step S104.

[0044] If there is a significant change in the selected area (step S104, Yes), the transmission control unit 133 transmits the selected area (tile) (step S105). The transmission control unit 133 transmits the tile to, for example, a computer on the ground. If there is no significant change in the selected area (step S104, No), the transmission control unit 133 does not transmit the tile.

[0045] If there is an unselected area (step S106, Yes), the determination unit 132 returns to step S103 and repeats the process. On the other hand, if there is no unselected area (step S106, No), the determination unit 132 ends the process.

[0046] [Example of Threshold Values] Here, the indexes of each model when threshold values ​​are actually set in the embodiment are shown.

[0047] FIG. 4 is a diagram showing the precision versus recall of Faster RCNN. FIG. 5 is a diagram showing the reduction rate versus recall of Faster RCNN. When using the aforementioned training data, the F1 score for Faster RCNN was maximized when the threshold was 0.6. In this case, the recall was 0.867 (dotted lines in FIGS. 4 and 5). In this case, the precision was 0.515 (see FIG. 4), and the reduction rate (data reduction rate) was 58.7% (see FIG. 5). The reduction rate is the proportion of tiles for which the determination unit 132 did not determine that an event exists. The larger the reduction rate, the greater the reduction cost of data transmission.

[0048] Thus, Faster RCNN achieved high recall and a sufficient reduction rate, demonstrating its robustness in detecting small ships, for example, when the background of the image is a large ocean.

[0049] Figure 6 shows the precision versus recall for Yolov7. Figure 7 shows the reduction rate versus recall for Yolov7. When using the training data described above, the F1 score for Yolov7 was maximized when the threshold was 0.4. In this case, the recall was 0.683 (dotted lines in Figures 6 and 7). In this case, the precision was 0.83 (see Figure 6), and the reduction rate was 80% (see Figure 7).

[0050] As such, Yolov7 achieved high precision, sufficient recall, and a high reduction rate, making it suitable for ship detection when images are captured in locations where ships are sparsely present.

[0051] Furthermore, we measured runtime metrics for each model to evaluate their practicality and deployability, and confirmed that they are capable of real-time processing and decision-making in maritime surveillance scenarios. Runtime metrics include model size, inference time, and Intersection over Union (IoU).

[0052] Faster RCNN had a model size of 158MB, an inference time per image of 29ms, and an IoU of 0.39 during validation. YOLOv7 had a model size of 142MB, an inference time per image of 9ms, and an IoU of 0.48 during validation. For both models, the runtime metrics were favorable, considering that processing was performed onboard a satellite.

[0053] Furthermore, this embodiment not only reduces the cost of transmitting data, but also enables a fast and effective response in critical scenarios such as disaster detection.

[0054] [Program] In one embodiment, the detection device 10 can be implemented by installing a detection program that executes the above-described processes as package software or online software on a desired computer. For example, by having an information processing device execute the above-described detection program, the information processing device can function as the detection device 10. The information processing device referred to here includes desktop and notebook personal computers. Other examples of information processing devices include smartphones, tablet terminals, and the like.

[0055] 8 is a diagram showing an example of a computer that executes a detection program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0056] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0057] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the detection device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the detection device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD.

[0058] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0059] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0060] REFERENCE SIGNS LIST 10 Detection device 11 Communication unit 12 Storage unit 13 Control unit 121 Model information 131 Division unit 132 Determination unit 133 Transmission control unit

Claims

1. A detection device comprising: a division unit that divides an image into a plurality of regions; a determination unit that determines whether or not an event has occurred in each of the plurality of regions; and a transmission control unit that transmits an image of a region that has been determined to have an event to an external device.

2. The detection device according to claim 1, characterized in that the determination unit determines that an event has occurred if the score calculated by the model for each of the plurality of regions exceeds a threshold set so as to maximize the F1 score of the model for the training data.

3. A detection method executed by a detection device, comprising: a division step of dividing an image into a plurality of regions; a determination step of determining whether or not an event has occurred in each of the plurality of regions; and a transmission control step of transmitting an image of a region among the plurality of regions for which it has been determined that an event has occurred to an external device.

4. A detection program that causes a computer to execute the following steps: a division step of dividing an image into a plurality of regions; a determination step of determining whether or not an event has occurred in each of the plurality of regions; and a transmission control step of transmitting, to an external device, an image of a region among the plurality of regions that has been determined to have an event.

Images