Object detection device, object detection method, program, and monitoring system

Abstract

The present invention provides an object detection device that can prevent false detections caused by feature point extraction from captured images. [Solution] The object detection device (2) includes a data acquisition unit (21) that acquires video information of the water surface and temperature information in the area of ​​the video information, and an inference unit (22) that infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area shown by the video information, as well as the difference in temperature information between the water surface and an object on the water surface.

Description

Technical Field

[0001] The present disclosure relates to an object detection device, an object detection method, a program, and a monitoring system.

Background Art

[0002] Object detection technology based on image processing is used for monitoring flowing substances such as in rivers. For example, Patent Document 1 describes a monitoring method for monitoring floating objects on the water surface using object detection technology based on image processing. In this monitoring method, the movement of a predetermined number of feature points set in the first frame image among a plurality of frame images of a waterway is detected for each feature point over the plurality of frame images. Then, it is determined whether each pixel constituting one of the plurality of frame images corresponds to a floating object in the waterway, and based on this determination result, the ratio of the number of pixels determined to have floating objects in one frame image is calculated. The monitoring method described in Patent Document 1 determines whether a floating object exists in the waterway based on the magnitude relationship between the number of feature points and the predetermined number, and the ratio of the number of pixels.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional object detection technology, the extraction of feature points in an image plays an important role, but there is a problem that the feature points of an object cannot be appropriately detected due to the characteristics of the image, resulting in false detection. For example, an image taken in the dark such as at night has low contrast and unclear edges of the subject, so false detection is likely to occur due to insufficient or inaccurate extraction of feature points derived from the object to be detected. Furthermore, in bright environments such as during the daytime, objects that are not intended for detection, such as flying objects over the water surface or objects moving along waterways, may be reflected on the water surface. In such images of the water surface, feature points originating from objects not intended for detection are difficult to distinguish from feature points originating from objects that are intended for detection, making false detections likely. Thus, the instability of feature point extraction, which depends on the characteristics of the image, can sometimes impair the reliability of object detection. This problem can also occur in the conventional technology described in Patent Document 1.

[0005] This disclosure aims to solve the above-mentioned problems and to provide an object detection device that can prevent false detections caused by feature point extraction from captured images. [Means for solving the problem]

[0006] The object detection device according to this disclosure includes a data acquisition unit that acquires video information of the water surface and temperature information in the area of ​​the video information, and an inference unit that infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area shown by the video information, as well as the difference in temperature information between the water surface and an object on the water surface. [Effects of the Invention]

[0007] The object detection device according to this disclosure infers whether or not an object to be detected exists in the shooting area based on the difference in temperature information between the water surface and the object on the water surface, in addition to the image of the shooting area indicated by the video information. As a result, the object detection device according to this disclosure can prevent false detections caused by feature point extraction from the captured image. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of the monitoring system according to Embodiment 1. [Figure 2] This is a block diagram showing an example configuration of an object detection device according to Embodiment 1. [Figure 3]This is a flowchart showing the object detection method according to Embodiment 1. [Figure 4] This is a block diagram showing a modified configuration of the object detection device according to Embodiment 1. [Figure 5] Figures 5A and 5B show the hardware configuration that realizes the function of the object detection device according to Embodiment 1. [Figure 6] This is a block diagram showing an example configuration of an object detection device according to Embodiment 2. [Figure 7] This is a flowchart showing the object detection method according to Embodiment 2. [Figure 8] This is a block diagram showing a modified configuration of the object detection device according to Embodiment 2. [Modes for carrying out the invention]

[0009] Embodiment 1. Embodiment 1 describes a monitoring system, an object detection device, and an object detection method that infer whether or not an object to be detected exists in the shooting area based on the difference in temperature information between the water surface and an object on the water surface, in addition to the image of the shooting area indicated by the video information.

[0010] (Overview of conventional object detection) In supervised learning for object detection on water surfaces, training data is collected. This involves collecting image data of the water surface where objects are present. Supervised learning assigns a label to each image indicating whether or not an object is present on the water surface, and if so, its location. The trained model learns from this labeled data as the "ground truth."

[0011] During training, feature points are extracted from the training data, which are images. For example, feature points of objects in an image are extracted based on the difference in color or brightness between the object on the water surface and the background water surface. Alternatively, feature points of edges, which are the boundaries of the object with respect to the water surface, may be extracted from the image, or shape features specific to the object to be detected may be extracted.

[0012] In supervised learning, a model for predicting the presence and location of an object is constructed based on the features extracted from an image. For example, a convolutional neural network (hereinafter referred to as CNN), YOLO (You Only Look Once), or SSD (Single Shot MultiBox Detector) is used. CNN extracts feature points for each small region of the image and infers whether there is an object on the water surface based on the extracted feature points. YOLO or SSD is an algorithm specialized for object detection, which analyzes the entire image at once and simultaneously predicts the position and class of the object.

[0013] The performance of the trained model is evaluated using test data. Evaluation metrics such as accuracy, recall, F1 score, mean average precision (mAP), etc. are used. In particular, in object detection, how accurate the position of the bounding box in the image is is also an important evaluation target.

[0014] In supervised learning for object detection on the water surface, first, features are extracted from the images which are the training data. Further, using a trained model such as CNN or YOLO, it is inferred whether there is an object on the water surface. If it is inferred that there is an object, the position of the object is detected. By repeating this learning and evaluation, it is possible to create a model with high accuracy.

[0015] (Problems of Object Detection) As described above, in conventional object detection, the extraction of feature points in the image plays an important role, but there are cases where the feature points of the object cannot be appropriately detected depending on the characteristics of the image. For example, an image taken in the dark has low contrast and unclear edges of the subject, so the feature points derived from the object to be detected are insufficiently or inaccurately extracted. Also, in a photographed image of the water surface in which an object outside the detection target is reflected, it is difficult to distinguish the features derived from the object outside the detection target from the features derived from the object to be detected, and false detection is likely to occur.

[0016] (Overview of Object Detection Using Temperature Information) On the other hand, there are differences between the temperature information of the water surface and the temperature information of the objects on the water surface. For example, there are differences in heat capacity, radiation characteristics, temperature change patterns, or reflection characteristics that define the temperature of the water surface and the objects on the water surface. For example, water has a very large specific heat capacity, so the temperature change of the water surface is gentle. Also, the heat absorbed by the water surface during the day is retained until night and slowly released. On the other hand, objects on the water surface generally have a smaller specific heat capacity than water, so the temperature change becomes rapid. The temperature of the object tends to rise rapidly due to the heat absorbed by the object during the day and rapidly cool at night.

[0017] Moreover, there are differences in radiation characteristics between the water surface and the objects on the water surface. For example, water emits radiation in the region of long wavelengths such as infrared light, and has a radiation spectrum close to blackbody radiation according to the water temperature. Objects on the water surface perform thermal radiation according to their temperature, but the radiation spectrum varies depending on the material or surface state of the object. Generally, objects on the water surface emit radiation in a wider wavelength range than water.

[0018] Furthermore, there are differences in the temperature change patterns between the water surface and the objects on the water surface. The water surface tends to have a small daily change and a large seasonal change, while objects on the water surface have a large daily change and are directly affected by the air temperature or solar radiation.

[0019] Furthermore, there are differences in reflection characteristics between the water surface and the objects on the water surface. The water surface has the properties of both specular reflection and diffuse reflection, while for objects on the water surface, if the surface state is rough, the property of diffuse reflection is strongly manifested, and if it is smooth, the property of specular reflection becomes strong. Due to these differences, the water surface and the objects on the water surface show different temperature change patterns and radiation characteristics, and can often be clearly distinguished in thermal images and the like.

[0020] Therefore, in the object detection device according to Embodiment 1, in addition to the image of the water surface, the device infers whether or not an object to be detected exists in the shooting area based on the difference in temperature information between the water surface and the object on the water surface. In other words, in Embodiment 1, in addition to feature points extracted from the image of the water surface, the device detects objects on the water surface by considering the difference in temperature information between the water surface and the object on the water surface. This makes it possible to prevent false detections caused by the extraction of feature points from the image as described above.

[0021] Next, a monitoring system equipped with an object detection device according to Embodiment 1 will be described. By incorporating the object detection device according to Embodiment 1, this monitoring system can prevent false detections caused by feature point extraction from captured images, enabling highly accurate monitoring.

[0022] (Basic configuration of the monitoring system) Figure 1 is a block diagram showing an example configuration of the monitoring system 1 according to Embodiment 1. In Figure 1, the monitoring system 1 is a system that monitors whether or not an object to be detected exists on the surface of a water body such as a river or waterway. In particular, it detects objects on the water surface using video information captured from the water surface and temperature information in the area captured by this video information. The objects to be detected are objects present on the water surface, such as floating debris, sediment, driftwood, drift ice, living organisms, or chemical substances such as oil.

[0023] As shown in Figure 1, the monitoring system 1 is configured to include an object detection device 2, an imaging device 3, a temperature sensor 4, and a monitoring device 5. The object detection device 2 infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area indicated by the video information, as well as the difference in temperature information between the water surface and the object on the water surface. For example, object detection device 2 uses a trained model to infer whether or not an object to be detected exists in the captured area. Here, the trained model is a model that, when an image of the captured area containing temperature information is input, outputs an inference result indicating whether or not an object to be detected exists in the captured area.

[0024] The object detection device 2 outputs notification information, a notification image, and inference results to the monitoring device 5. The inference result includes information such as detection flag information indicating whether or not the object to be detected is present on the water surface. If the detection flag information indicates the presence of an object, the inference result also includes a notification image. The object detection device 2 also stores past inference results and reads the inference result indicated by the request information from the monitoring device 5 and outputs it to the monitoring device 5. Notification information is information used to notify the monitor of the presence of an object when the detection flag information indicates the presence of an object. For example, it is alarm control information that causes the monitoring device 5 to output an alarm. The notification image is an image created by superimposing information indicating the detected object onto a still image that constitutes video information of the water surface, when the detection flag information indicates the presence of an object. Here, the information indicating the detected object may include, for example, the image region in which the object exists, or text information indicating the object within the captured image.

[0025] The camera 3 is a visible light camera that photographs the surface of water, such as a river, and is positioned, for example, beside a river. The camera 3 is also adjusted so that the area for detecting objects on the water surface fits within its field of view. Furthermore, the camera 3 is connected to the object detection device 2 via wired or wireless communication. The video information of the water surface captured by the camera 3 is transmitted to the object detection device 2.

[0026] The temperature sensor 4 is a sensor that detects temperature information of an area including the water surface captured by the imaging device 3, and is, for example, a far-infrared camera. The temperature detection range of the temperature sensor 4 is adjusted to fit within the imaging area of ​​the imaging device 3. For example, if the temperature sensor 4 is a far-infrared camera, the field of view of the imaging device 3 and the field of view of the far-infrared camera are matched. In this case, the field of view may be matched physically, or it may be matched in software based on a reference point existing in real space. The temperature sensor 4 is connected to the object detection device 2 via wired or wireless communication. The temperature information of the imaging area detected by the temperature sensor 4 is transmitted to the object detection device 2.

[0027] The monitoring device 5 is a device that monitors the object to be detected in the shooting area based on information output from the object detection device 2. For example, the monitoring device 5 includes an alarm device that outputs an alarm according to alarm control information transmitted as notification information, a display device that displays a notification image, and an input device for inputting request information to request inference results from the object detection device 2.

[0028] Furthermore, while the example shows the monitoring system 1 consisting of separate devices—an object detection device 2, a camera 3, a temperature sensor 4, and a monitoring device 5—it is not limited to this. For example, the monitoring system 1 may be a device that possesses some or all of the functions of the object detection device 2, camera 3, temperature sensor 4, and monitoring device 5.

[0029] (Basic configuration of an object detection device) Figure 2 is a block diagram showing an example configuration of the object detection device 2 according to Embodiment 1. The object detection device 2 acquires video information of the water surface and temperature information in the area of ​​this video information, and infers whether or not an object to be detected exists in the area of ​​the image based on the image of the area of ​​the image and the difference in temperature information between the water surface and the object on the water surface. In Figure 2, the object detection device 2 includes a data acquisition unit 21, an inference unit 22, a storage unit 23, a notification processing unit 24, an overlay processing unit 25, and a storage processing unit 26.

[0030] The data acquisition unit 21 acquires video information and temperature information. The video information is video of the surface of a river or other body of water captured by the camera 3. For example, the video information is video including the water surface captured at a predetermined frame rate by a visible light camera placed next to the river or other body of water. In this case, any frame image can be extracted from the video information as a still image. The visible light camera is connected to the object detection device 2 via wired or wireless communication, and the data acquisition unit 21 acquires video information from the visible light camera via wired or wireless communication. The temperature information is information indicating the temperature of each object in the image area of ​​the video information, as detected by the temperature sensor 4. For example, the temperature information is a thermal image showing the temperature distribution in the image area, including the water surface, captured by the far-infrared camera at a predetermined frame rate. The far-infrared camera is positioned next to a river or the like so that its field of view matches that of the imaging device 3. In this case, a thermal image of any frame can be extracted as a still image from the temperature information. The data acquisition unit 21 acquires the temperature information from the far-infrared camera via wired or wireless communication.

[0031] Furthermore, the data acquisition unit 21 executes the following processes (1) to (3) and outputs the processed data generated by these processes to the inference unit 22. Process (1) is the process of generating a still image from video information. For example, the data acquisition unit 21 extracts a frame image from among multiple frame images included in the video information at a predetermined time and outputs it as a still image to the inference unit 22. The time can be specified by the monitoring device 5 or directly to the object detection device 2. The data acquisition unit 21 may also extract frame images at a fixed period or at a fixed frame image interval. Furthermore, the data acquisition unit 21 may extract frame images from the video information that match the conditions specified by the inference unit 22. Possible conditions include, for example, selecting images in which pixel values ​​such as brightness in a specific area of ​​the image are higher than a predetermined threshold. Process (2) is a process that maps temperature information to the still image obtained in process (1). For example, if the temperature information consists of multiple frame images that visualize the temperature distribution, the data acquisition unit 21 extracts the frame image at the time specified in process (1) from among the multiple frame images included in the temperature information, and maps the extracted image to the still image obtained in process (1). Here, "mapping" refers to the process of associating the image that visualizes the temperature distribution with the still image, for example, the process of superimposing the image that visualizes the temperature distribution onto the still image. Process (3) is the process of outputting the still images obtained in process (1) and process (2) to the inference unit 22. In other words, the still images obtained in process (1) and process (2) are the processed data described above.

[0032] Although the above examples show the data acquisition unit 21 performing processes (1) to (3), the explanation is not limited to this. For example, at least one of processes (1) to (3) may be performed by the imaging device 3 or the temperature sensor 4. Alternatively, a processing device for performing data processing may be provided between the imaging device 3 or the temperature sensor 4 and the object detection device 2.

[0033] The inference unit 22 infers whether or not an object to be detected exists in the shooting area, based on the image of the shooting area indicated by the video information, as well as the difference in temperature information between the water surface and the object on the water surface. For example, the inference unit 22 uses the trained model 231 read from the memory unit 23 to infer whether or not an object to be detected exists in the imaging area. In this embodiment, the trained model 231 is a model that, when an image of a shooting area containing temperature information is input, outputs an inference result of whether or not an object to be detected exists in the shooting area. The image of the shooting area containing temperature information is, for example, the image obtained in process (2).

[0034] If the inference unit 22 infers that an object to be detected exists in the shooting area, it executes the following processes (1A) and (2A). Process (1A) is the process of setting the detection flag information to "on". The detection flag information is information that is included in the inference result output from the inference unit 22 to the notification processing unit 24, the superimposing processing unit 25, and the storage processing unit 26. Process (2A) is the process of generating position information indicating the position of the object inferred to exist in the captured image, and outputting this position information and the still image obtained in process (1) to the superimposing processing unit 25.

[0035] If the inference unit 22 infers that there is no object to be detected in the shooting area, it performs the following processes (1B) and (2B). Process (1B) is the process of setting the detection flag information to off. The inference result, including the detection flag information, is output from the inference unit 22 to the notification processing unit 24, the superimposition processing unit 25, and the storage processing unit 26. Process (2B) is the process of outputting only the still image obtained in process (1), which was acquired from the data acquisition unit 21, to the superimposition processing unit 25.

[0036] The memory unit 23 includes a trained model 231 and a storage unit 232. In Figure 2, the object detection device 2 is equipped with the memory unit 23, but the memory unit 23 may be provided outside the object detection device 2. In this case, the memory unit 23 and the object detection device 2 are connected via wired or wireless communication, and the inference unit 22 inputs data to the trained model 231 stored in the memory unit 23 via wired or wireless communication and obtains the inference results output from the trained model 231. Similarly, the storage processing unit 26 exchanges data with the storage unit 232 provided in the memory unit 23 via wired or wireless communication.

[0037] The pre-trained model 231 is a model constructed by pre-training an algorithm that, when an image obtained in process (2) is input, infers whether or not a target object exists in the captured area. For example, the training data is an image of the captured area with temperature information of the target object mapped to it, and the pre-trained model 231 is a program obtained by supervised learning using this training data. CNN, YOLO, or SSD can be used to construct the pre-trained model 231. By using images of still images of rivers with temperature information mapped onto them as training data, a trained model can be obtained that outputs inference results with reduced false positives, even when inputting images with few features, such as those taken at night, or images where objects not to be detected are reflected on the water surface.

[0038] The storage unit 232 stores the inference results from the inference unit 22. For example, the storage unit 232 stores the inference results, including detection flag information and an image (still image) of the captured area, associated with the time of capture. If the detection flag information is set to "on," the image area of ​​the object to be detected is superimposed on the image of the captured area.

[0039] When the notification processing unit 24 obtains an inference result indicating the presence of a target object in the imaging area, it generates notification information to notify the presence of the target object. For example, when the notification processing unit 24 obtains detection flag information set to ON from the inference unit 22, it generates notification information to notify the monitor of the presence of the target object. The notification information is, for example, alarm control information that causes the monitoring device 5 to output an alarm. The notification information generated by the notification processing unit 24 is transmitted to the monitoring device 5 by wired or wireless communication.

[0040] The superimposition processing unit 25 generates a notification image. The notification image is an image in which the image area of ​​the detected object is superimposed on the image of the captured area. When the superimposition processing unit 25 obtains position information indicating the location of the object whose presence has been detected by the inference unit 22 and the still image obtained in process (1), it performs a process to include the area indicated by the position information in the still image. For example, the superimposition processing unit 25 generates an image of the area indicated by the position information and superimposes the generated image onto the still image. The image of the area indicated by the position information may be an image in which the image of the object is embedded in that area, or it may be an image in which the boundary of the area is indicated by an accent line. The notification image generated by the superimposition processing unit 25 is transmitted to the monitoring device 5 via wired or wireless communication. Furthermore, the superimposing processing unit 25 outputs detection flag information and a notification image or still image to the storage processing unit 26.

[0041] The storage processing unit 26 stores the inference results from the inference unit 22 in the storage unit 232, reads the inference results according to the request from the storage unit 232, and outputs them to the requester. For example, the storage processing unit 26 stores the image and detection flag information acquired from the superimposition processing unit 25, linked to the time of capture, in the storage unit 232 as an inference result. The storage processing unit 26 searches the storage unit 232 based on the time indicated by the request information from the monitoring device 5, and transmits the inference results of the search results to the monitoring device 5 via wired or wireless communication.

[0042] Although the object detection device 2 shown includes all of the notification processing unit 24, the superimposing processing unit 25, and the storage processing unit 26, it is not limited to this configuration. For example, at least one of the notification processing unit 24, the superimposing processing unit 25, and the storage processing unit 26 may be provided by an external device separate from the object detection device 2. The external device may be a monitoring device 5, or it may be a device that is connected to the object detection device 2 and the monitoring device 5 via wired or wireless communication.

[0043] Next, the object detection method according to Embodiment 1 will be described. Figure 3 is a flowchart illustrating the object detection method according to Embodiment 1, showing a series of processes performed by the object detection device 2. First, the data acquisition unit 21 acquires video information and temperature information (step ST1). For example, the data acquisition unit 21 acquires video information captured by the camera 3 and temperature information detected by the temperature sensor 4 via wired or wireless communication. The data acquisition unit 21 extracts still images and thermal images from the video information and temperature information that correspond to the time information specified by the monitoring device 5. For example, if a monitor monitoring video of a water surface wants to see video from a desired time, they input time information indicating the above time using the input device provided in the monitoring device 5. The monitoring device 5 transmits the input time information to the object detection device 2. The data acquisition unit 21 of the object detection device 2 acquires still images and thermal images corresponding to the time information received from the monitoring device 5. The data acquisition unit 21 then generates a superimposed image by superimposing an image region showing the temperature distribution in the thermal image onto the still image, and outputs the still image and superimposed image as processed data to the inference unit 22.

[0044] The inference unit 22 infers whether or not a target object exists in the shooting area based on the image of the shooting area indicated by the video information, as well as the difference in temperature information between the water surface and the object on the water surface (step ST2). For example, the inference unit 22 inputs the processed data acquired from the data acquisition unit 21 into the trained model 231 and obtains the inference result output from the trained model 231. If the inference result indicates that a target object exists, the inference unit 22 sets the detection flag information to "on" and identifies the location information of the region of the temperature distribution in the thermal image that corresponds to the object. For example, the inference unit 22 utilizes the fact that the temperature distributions of the object and the water surface are different on the water surface where the target object exists, and identifies the location information of the region of the temperature distribution that corresponds to the object based on a predetermined temperature threshold. Subsequently, the inference unit 22 executes processes (1A) and (2A) if it infers that an object to be detected exists in the imaging area, and executes processes (1B) and (2B) if it infers that an object to be detected does not exist in the imaging area.

[0045] Next, a modified example of the object detection device 2 according to Embodiment 1 will be described. Figure 4 is a block diagram showing the configuration of object detection device 2A, which is a modified version of object detection device 2. Object detection device 2A is a device that integrates a shooting function that photographs the water surface in visible light and a temperature detection function that visualizes the temperature distribution of the shooting area. Object detection device 2A may be fixedly placed beside a river or the like, it may be a device mounted on a mobile vehicle such as a drone, or it may be a mobile terminal such as a smartphone.

[0046] Similar to object detection device 2, object detection device 2A acquires video information of the water surface and temperature information in the area of ​​the video information. Based on the image of the area of ​​capture, as well as the difference in temperature information between the water surface and objects on the water surface, it infers whether or not an object to be detected exists in the area of ​​capture. As shown in Figure 4, in addition to the shooting unit 27 and temperature detection unit 28, object detection device 2A includes a data acquisition unit 21A, an inference unit 22, a storage unit 23, a notification processing unit 24, an overlay processing unit 25, and an accumulation processing unit 26.

[0047] The data acquisition unit 21A acquires video information from the imaging unit 27 and temperature information from the temperature detection unit 28. The data acquisition unit 21A also executes the above processes (1) to (3) and outputs the processed data generated by these processes to the inference unit 22. The imaging unit 27 is a visible light camera that photographs the surface of water such as rivers. The object detection device 2A is positioned so that the area to be detected for objects on the water surface falls within the field of view of the imaging unit 27. Furthermore, the video information of the water surface captured by the imaging unit 27 is acquired by the data acquisition unit 21A.

[0048] The temperature detection unit 28 is a sensor that detects temperature information of the area including the water surface captured by the imaging unit 27, and is, for example, a far-infrared camera. The temperature detection range of the temperature detection unit 28 is adjusted to fit within the imaging area of ​​the imaging unit 27. For example, if the temperature detection unit 28 is a far-infrared camera, the field of view of the imaging unit 27 and the field of view of the temperature detection unit 28 may be physically matched, or the field of view may be matched in software using a reference point existing in real space. The temperature information detected by the temperature detection unit 28 is acquired by the data acquisition unit 21A.

[0049] Furthermore, the data acquisition unit 21A executes the above processes (1) to (3) and outputs the processed data generated by these processes to the inference unit 22. Although the example shows the data acquisition unit 21A performing processes (1) to (3), it is not limited to this. For example, at least one of processes (1) to (3) may be performed by the imaging unit 27 or the temperature detection unit 28.

[0050] Next, we will describe the hardware configuration that enables the functionality of object detection device 2. The data acquisition and inference functions of the object detection device 2 are implemented by a processing circuit. Specifically, the object detection device 2 includes a processing circuit for executing the processes from step ST1 to step ST2 shown in Figure 3. The processing circuit may be dedicated hardware, or it may be a CPU (Central Processing Unit) that executes a program stored in memory.

[0051] Figure 5A is a block diagram showing the hardware configuration that realizes the functions of the object detection device 2. Figure 5B is a block diagram showing the hardware configuration that runs the software that realizes the functions of the object detection device 2. In Figures 5A and 5B, the object detection device 2 acquires video information and temperature information from external devices via the input interface 100. The external devices are, for example, the imaging device 3 and temperature sensor 4 shown in Figure 1. The object detection device 2 also acquires request information from the monitoring device 5 via the input interface 100. The object detection device 2 outputs notification information, notification images, and inference results to the monitoring device 5 via the output interface 101.

[0052] If the processing circuit is a dedicated hardware processing circuit 102 as shown in Figure 5A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. Each function of the object detection device 2 may be implemented by separate processing circuits, or these functions may be implemented together by a single processing circuit.

[0053] When the processing circuit is the processor 103 shown in Figure 5B, each function of the object detection device 2 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 104.

[0054] The processor 103 realizes each function of the object detection device 2 by reading and executing a program stored in the memory 104. For example, the object detection device 2 includes a memory 104 for storing a program that, when executed by the processor 103, results in the execution of steps ST1 to ST2 shown in Figure 3. These programs cause the computer to execute the object detection procedure or method performed by the object detection device 2. The memory 104 may be a computer-readable storage medium that stores a program for causing the computer to function as the object detection device 2.

[0055] Memory 104 includes, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM) (registered trademark), as well as magnetic disks, flexible disks, optical disks, compact disks, minidiscs, and DVDs.

[0056] Some of the functions of the object detection device 2 may be implemented by dedicated hardware, while other functions may be implemented by software or firmware. For example, the functions of the data acquisition unit 21 may be implemented by a processing circuit 102, which is dedicated hardware, and the functions of the inference unit 22 may be implemented by the processor 103 reading and executing a program stored in memory 104. In this way, the processing circuit can implement the above functions by hardware, software, firmware, or a combination thereof.

[0057] Furthermore, the data acquisition unit 21A, inference unit 22, notification processing unit 24, superposition processing unit 25, and storage processing unit 26 functions of the object detection device 2A are also realized by the dedicated processing circuit 102 shown in Figure 5A, or by the processor 103 and memory 104 shown in Figure 5B, which are provided by the computer. Similar to Embodiment 1, the memory 104 stores programs that constitute information processing applications for realizing the functions of the data acquisition unit 21A, inference unit 22, notification processing unit 24, superposition processing unit 25, and storage processing unit 26. The processor 103 executes the information processing application read from the memory 104, thereby realizing the functions of the data acquisition unit 21A, the inference unit 22, the notification processing unit 24, the superposition processing unit 25, and the storage processing unit 26.

[0058] As described above, the object detection device 2 or 2A according to Embodiment 1 includes a data acquisition unit 21 or 21A that acquires video information of the water surface and temperature information in the shooting area of ​​the video information, and an inference unit 22 that infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area shown by the video information and the difference in temperature information between the water surface and objects on the water surface. Since the object detection device 2 or 2A infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area and the difference in temperature information between the water surface and objects on the water surface, the object detection device 2 or 2A can prevent false detections caused by feature point extraction from captured images. For example, the object detection device 2 or 2A can accurately infer whether or not an object to be detected exists even if the video information is unclear, such as when a river is photographed at night, or if the video information contains reflections of objects other than those to be detected on the water surface.

[0059] In the object detection device 2 or 2A according to Embodiment 1, the inference unit 22 uses a trained model 231 to infer whether or not an object to be detected exists in the shooting area. The trained model 231 is a model that, when an image of the shooting area including temperature information is input, outputs an inference result of whether or not an object to be detected exists in the shooting area. Since the inference of whether or not an object to be detected exists in the shooting area is based not only on the image of the shooting area but also on the difference in temperature information between the water surface and the object on the water surface, the object detection device 2 or 2A can prevent false detections caused by feature point extraction from the captured image.

[0060] In the object detection device 2 or 2A according to Embodiment 1, the temperature information is an image showing the temperature distribution of the shooting area. The trained model 231 is a model that, when an image with the temperature distribution associated with the image of the shooting area is input, outputs an inference result of whether or not an object to be detected exists in the shooting area. In addition to the image of the shooting area, it is possible to infer whether or not an object to be detected exists in the shooting area based on the difference in temperature information between the water surface and objects on the water surface.

[0061] The object detection device 2 or 2A according to Embodiment 1 includes a notification processing unit 24. When the notification processing unit 24 obtains an inference result indicating that an object to be detected is present in the shooting area, it generates notification information to notify the presence of the object to be detected. The notification information allows the monitoring device 5 to be notified that an object to be detected is present in the shooting area.

[0062] The object detection device 2 or 2A according to Embodiment 1 includes an overlay processing unit 25. The overlay processing unit 25 generates a notification image by overlaying the image area of ​​the object to be detected onto the image of the captured area. The notification image allows the monitoring device 5 to visually notify the presence of the object to be detected.

[0063] The object detection device 2 or 2A according to Embodiment 1 includes a storage processing unit 26. The storage processing unit 26 stores the inference results from the inference unit 22 in a storage unit 232, reads the inference results according to the request from the storage unit 232 and outputs them to the requester. The history of the inference results stored in the storage unit 232 can be obtained.

[0064] The object detection device 2A according to Embodiment 1 includes an imaging unit 27 for capturing video information and a temperature detection unit 28 for detecting temperature information. The data acquisition unit 21A acquires video information from the imaging unit 27 and temperature information from the temperature detection unit 28. Since the object detection device 2A integrates the imaging unit 27 and the temperature detection unit 28, the object detection device 2A can be provided not only as a device fixedly positioned beside a river or the like, but also as a mobile device or a portable terminal.

[0065] In the object detection method according to Embodiment 1, the data acquisition unit 21 or 21A performs a step ST1 in which it acquires video information of the water surface and temperature information in the shooting area of ​​the video information, and the inference unit 22 performs a step ST2 inference whether or not an object to be detected exists in the shooting area, based on the image of the shooting area shown by the video information and the difference in temperature information between the water surface and an object on the water surface. The inference unit 22 uses a trained model 231 to infer whether or not an object to be detected exists in the shooting area. The trained model 231 is a model that, when an image of the shooting area including temperature information is input, outputs an inference result of whether or not an object to be detected exists in the shooting area. By having the object detection device 2 or 2A execute this method, false detections caused by feature point extraction from the captured image can be prevented.

[0066] The computer executing the program according to Embodiment 1 functions as an object detection device 2 or 2A. This makes it possible to provide an object detection device 2 or 2A that can prevent false detections caused by feature point extraction from captured images.

[0067] The monitoring system 1 according to Embodiment 1 includes an object detection device 2 or 2A and a monitoring device 5 that monitors the object to be detected in the shooting area based on the information output from the object detection device 2 or 2A. This makes it possible to provide a monitoring system 1 that can accurately infer whether or not the object to be detected is present, even if the image information is unclear, such as when a river is photographed at night, or if the image information contains reflections of objects other than the object to be detected on the water surface.

[0068] Embodiment 2. Embodiment 2 describes an object detection device that infers whether or not an object to be detected exists in the shooting area based on an image of the shooting area, and determines whether or not an object to be detected exists in the shooting area based on the inference result using temperature information.

[0069] Figure 6 is a block diagram showing an example configuration of the object detection device 2B according to Embodiment 2. In Figure 6, components that function similarly to the object detection device 2 shown in Figure 2 are denoted by the same reference numerals. The object detection device 2B is a device included in the monitoring system 1 shown in Figure 1. The object detection device 2B acquires video information of the water surface and temperature information in the area of ​​this video information, and infers whether or not an object to be detected exists in the area of ​​the image, based on the image of the area of ​​the image and the difference in temperature information between the water surface and the object on the water surface.

[0070] As shown in Figure 6, the object detection device 2B includes a data acquisition unit 21, an inference unit 22A, a storage unit 23A, a notification processing unit 24, an overlay processing unit 25, and a storage processing unit 26. For example, the functions of the data acquisition unit 21, inference unit 22A, notification processing unit 24, overlay processing unit 25, and storage processing unit 26 of the object detection device 2B are realized by a dedicated processing circuit 102 shown in Figure 5A, or by a processor 103 and memory 104 shown in Figure 5B of a computer. Similar to Embodiment 1, the memory 104 stores programs that constitute information processing applications for realizing the functions of the data acquisition unit 21, inference unit 22A, notification processing unit 24, overlay processing unit 25, and storage processing unit 26. The processor 103 executes the information processing application read from the memory 104, thereby realizing the functions of the data acquisition unit 21, the inference unit 22A, the notification processing unit 24, the superposition processing unit 25, and the storage processing unit 26.

[0071] The data acquisition unit 21 acquires video information and temperature information. The data acquisition unit 21 executes processes (1C) and (2C) and outputs the processed data generated by these processes to the inference unit 22A. Process (1C) is the process of generating a still image from video information. For example, the data acquisition unit 21 extracts a frame image from among multiple frame images contained in the video information at a predetermined time and outputs it as a still image to the inference unit 22A. The time can be specified by the monitoring device 5 or directly to the object detection device 2B. Furthermore, the data acquisition unit 21 may extract frame images from the video information that match the conditions specified by the inference unit 22A. Possible conditions include, for example, selecting images in which pixel values ​​such as brightness in a specific area of ​​the image are higher than a predetermined threshold. Process (2C) is the process of outputting the still image obtained in process (1C) to the inference unit 22A. In other words, the still image obtained in process (1C) is the processed data described above.

[0072] The example shown illustrates the case where the data acquisition unit 21 performs processes (1C) and (2C), but it is not limited to this. For example, process (1C) may be performed by the imaging device 3 or the temperature sensor 4. Alternatively, a processing device that performs data processing may be provided between the imaging device 3 or the temperature sensor 4 and the object detection device 2B.

[0073] The inference unit 22A uses the trained model 231A to infer whether or not an object to be detected exists in the shooting area, and determines whether or not an object to be detected exists in the shooting area based on the temperature information. As shown in Figure 6, the inference unit 22A comprises a first inference unit 221 and a determination unit 222. The first inference unit 221 uses the trained model 231A to infer whether or not an object to be detected exists in the shooting area.

[0074] The pre-trained model 231A is a model that, upon input of an image of a captured area, outputs an inference result indicating whether or not a target object exists in that captured area. Specifically, the pre-trained model 231A is a model constructed by pre-training an algorithm that, upon input of a still image of the captured area obtained in the above process (1C), infers whether or not a target object exists in the captured area. The training data is image data containing the target object on the water surface of the captured area, and the pre-trained model 231A is a program obtained by supervised learning using this training data. CNN, YOLO, or SSD, etc., can be used to construct the pre-trained model 231A.

[0075] The determination unit 222 determines whether or not an object to be detected exists in the shooting area based on the temperature information and the inference result output from the trained model 231A, and outputs this determination result as the inference result from the inference unit 22A. For example, the determination unit 222 maps the temperature information to the still image obtained as the inference result, and determines whether or not an object to be detected exists by comparing the image area of ​​the object in the still image with the area estimated to be the temperature distribution of the object to be detected. For example, the determination unit 222 determines that the object has been accurately detected if the region in which the object is inferred to exist in a still image inferred by the inference unit 22A matches the region estimated to be the temperature distribution of the object to be detected. Furthermore, the determination unit 222 determines that an object has been falsely detected if there is no area in the still image that is estimated to be the temperature distribution of the object to be detected. Furthermore, the determination unit 222 determines that there was a failure to detect the target object if an area estimated to be the temperature distribution of the target object is superimposed on an area other than the location where the object was inferred to exist in the still image. Thus, the determination process performed by the determination unit 222 can be described as a process that filters the correct detection result from the inference result of the inference unit 22A using temperature information. The temperature distribution filtering applied to the inference results may be AND, OR, or temperature distribution-first processing. AND processing assumes the presence of the target object if both the inference result and the temperature distribution result indicate the presence of the object. OR processing assumes the presence of the target object if either the inference result or the temperature distribution result indicates the presence of the object. Temperature distribution-first processing assumes the presence of the target object if the temperature distribution result indicates the presence of the object. AND processing reduces false positives but increases missed detections. OR processing increases false positives but reduces missed detections. These processes can be selected according to the monitoring level. For example, at high monitoring levels, the focus should be on reducing false positives, while at low monitoring levels, the focus should be on reducing missed detections. Furthermore, at night, it may be important to prioritize reducing detection failures, while during the daytime, when reflections occur, it may be important to prioritize reducing false detections.

[0076] If the inference unit 22A infers that an object to be detected exists in the shooting area, it performs the following processes (1D) and (2D). Process (1D) is the process of setting the detection flag information to "on". The detection flag information is information that is included in the inference result output from the inference unit 22A to the notification processing unit 24, the superimposing processing unit 25, and the storage processing unit 26. Process (2D) is the process of generating position information indicating the position of the object inferred to exist in the captured image, and outputting this position information and the still image obtained in process (1C) to the superimposing processing unit 25.

[0077] If the inference unit 22A infers that there is no object to be detected in the imaging area, it performs the following processes (1E) and (2E). Process (1E) is the process of setting the detection flag information to off. The inference result, including the detection flag information, is output from the inference unit 22A to the notification processing unit 24, the superimposition processing unit 25, and the storage processing unit 26. Process (2E) is the process of outputting only the still image obtained in process (1C) to the superimposition processing unit 25.

[0078] The memory unit 23A stores the program that constitutes the trained model 231A, and is also provided with a memory area that functions as a storage unit 232. The storage unit 232 stores the inference results from the inference unit 22A. For example, the storage unit 232 stores the inference results, including detection flag information and an image (still image) of the captured area, associated with the time of capture. If the detection flag information is set to "on", the image area of ​​the object to be detected is superimposed on the image of the captured area.

[0079] In Figure 6, the object detection device 2B is equipped with a storage unit 23A, but the storage unit 23A may be provided outside the object detection device 2B. In this case, the storage unit 23A and the object detection device 2B are connected via wired or wireless communication, and the inference unit 22A inputs data to the trained model 231A stored in the storage unit 23A via wired or wireless communication and obtains the inference results output from the trained model 231A. Similarly, the storage processing unit 26 exchanges data with the storage unit 232 provided in the storage unit 23A via wired or wireless communication.

[0080] When the notification processing unit 24 obtains an inference result indicating the presence of a target object in the imaging area, it generates notification information to notify the presence of the target object. For example, when the notification processing unit 24 obtains detection flag information set to "on" from the inference unit 22A, it generates notification information to notify the monitor of the presence of the target object. The notification information is, for example, alarm control information that causes the monitoring device 5 to output an alarm. The notification information generated by the notification processing unit 24 is transmitted to the monitoring device 5 via wired or wireless communication.

[0081] The superimposition processing unit 25 generates a notification image. The notification image is an image in which the image area of ​​the detected object is superimposed on the image of the captured area. When the superimposition processing unit 25 obtains position information indicating the location of the object whose presence has been detected by the inference unit 22 and the still image obtained in processing (1C), it performs a process to include the area indicated by the position information in the still image. For example, the superimposition processing unit 25 generates an image of the region indicated by the location information and superimposes the generated image onto the still image. The image of the region indicated by the location information may be an image in which an image of an object is embedded in that region, or it may be an image in which the boundary of the region is indicated by a highlighting line. The notification image generated by the superimposition processing unit 25 is transmitted to the monitoring device 5 via wired or wireless communication. Furthermore, the superimposing processing unit 25 outputs detection flag information and a notification image or still image to the storage processing unit 26.

[0082] The storage processing unit 26 stores the inference results from the inference unit 22A in the storage unit 232, reads the inference results according to the request from the storage unit 232, and outputs them to the requester. For example, the storage processing unit 26 stores the image and detection flag information acquired from the superimposition processing unit 25 in the storage unit 232 as an inference result, linked to the time of capture. The storage processing unit 26 searches the storage unit 232 based on the time indicated by the request information from the monitoring device 5 and transmits the inference result of the search result to the monitoring device 5 via wired or wireless communication.

[0083] Although the object detection device 2B is shown, which includes all of the notification processing unit 24, the superimposing processing unit 25, and the storage processing unit 26, it is not limited to this. For example, at least one of the notification processing unit 24, the superimposing processing unit 25, and the storage processing unit 26 may be provided by an external device separate from the object detection device 2B. The external device may be the monitoring device 5, or it may be a device that is connected to the object detection device 2B and the monitoring device 5 via wired or wireless communication.

[0084] Next, the object detection method according to Embodiment 2 will be described. Figure 7 is a flowchart illustrating the object detection method according to Embodiment 2, showing a series of processes performed by the object detection device 2A. The data acquisition unit 21 acquires video information and temperature information (step ST1A). For example, the data acquisition unit 21 acquires video information captured by the camera 3 and temperature information detected by the temperature sensor 4 via wired or wireless communication. The data acquisition unit 21 extracts still images and thermal images from the video information and temperature information that correspond to the time information specified by the monitoring device 5. For example, if a monitor monitoring video of the water surface wants to see video from a desired time, they input time information indicating the above time using the input device provided in the monitoring device 5. The monitoring device 5 transmits the input time information to the object detection device 2. The data acquisition unit 21 of the object detection device 2 acquires still images and thermal images corresponding to the time information received from the monitoring device 5, and outputs the still images and temperature information as processed data to the inference unit 22A.

[0085] The first inference unit 221 infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area indicated by the video information (step ST2A). For example, the first inference unit 221 inputs a still image acquired from the data acquisition unit 21 to the trained model 231A and obtains the inference result output from the trained model 231A.

[0086] The determination unit 222 determines, based on the temperature information, whether or not an object to be detected exists in the imaging area based on the inference result from the first inference unit 221 (step ST3A). For example, as described above, the determination unit 222 maps temperature information to the still image obtained as an inference result and compares the image region of the object in the still image with the region estimated to be the temperature distribution of the object to be detected, thereby determining whether or not the object to be detected exists.

[0087] If the determination unit 222 determines that the object to be detected exists, the inference unit 22A sets the detection flag information to "on" and identifies the location information of the region of the temperature distribution in the thermal image that corresponds to the object. For example, the inference unit 22A utilizes the fact that the temperature distribution of the object and the water surface are different on the water surface where the object to be detected exists, and identifies the location information of the region of the temperature distribution that corresponds to the object based on a predetermined temperature threshold. Subsequently, if the inference unit 22A infers that an object to be detected exists in the imaging area, it executes processes (1D) and (2D). If it infers that an object to be detected does not exist in the imaging area, it executes processes (1E) and (2E).

[0088] Next, a modified example of the object detection device 2B according to Embodiment 2 will be described. Figure 8 is a block diagram showing the configuration of object detection device 2C, which is a modified version of object detection device 2B. Object detection device 2C is a device that integrates a shooting function that captures images of the water surface in visible light and a temperature detection function that visualizes the temperature distribution of the shooting area. Object detection device 2C may be fixedly placed beside a river or the like, it may be a device mounted on a mobile vehicle such as a drone, or it may be a mobile terminal such as a smartphone.

[0089] Similar to object detection device 2B, object detection device 2C acquires video information of the water surface and temperature information in the area of ​​the video information. Based on the image of the area of ​​capture, as well as the difference in temperature information between the water surface and objects on the water surface, it infers whether or not an object to be detected exists in the area of ​​capture. As shown in Figure 8, in addition to the shooting unit 27 and temperature detection unit 28, object detection device 2C includes a data acquisition unit 21A, an inference unit 22A, a storage unit 23A, a notification processing unit 24, an overlay processing unit 25, and an accumulation processing unit 26.

[0090] The data acquisition unit 21A acquires video information from the imaging unit 27 and temperature information from the temperature detection unit 28. The data acquisition unit 21A also executes processes (1C) and (2C) and outputs the processed data generated by these processes to the inference unit 22A. The imaging unit 27 is a visible light camera that photographs the surface of water such as rivers. The object detection device 2C is positioned so that the area to be detected for objects on the water surface falls within the field of view of the imaging unit 27. Furthermore, the video information of the water surface captured by the imaging unit 27 is acquired by the data acquisition unit 21A.

[0091] The temperature detection unit 28 is a sensor that detects temperature information of the area including the water surface captured by the imaging unit 27, and is, for example, a far-infrared camera. The temperature detection range of the temperature detection unit 28 is adjusted to fit within the imaging area of ​​the imaging unit 27. For example, if the temperature detection unit 28 is a far-infrared camera, the field of view of the imaging unit 27 and the field of view of the temperature detection unit 28 may be physically matched, or the field of view may be matched in software using a reference point existing in real space. The temperature information detected by the temperature detection unit 28 is acquired by the data acquisition unit 21A.

[0092] Furthermore, the data acquisition unit 21A executes processes (1C) and (2C) and outputs the processed data generated by these processes to the inference unit 22A. Although the example shows the data acquisition unit 21A performing processes (1C) and (2C), it is not limited to this. For example, process (1C) may be performed by the imaging unit 27 or the temperature detection unit 28.

[0093] The data acquisition unit 21A, inference unit 22A, notification processing unit 24, superposition processing unit 25, and storage processing unit 26 functions of the object detection device 2C are also realized by the dedicated processing circuit 102 shown in Figure 5A, or by the processor 103 and memory 104 shown in Figure 5B, which are provided by the computer. Similar to Embodiment 1, the memory 104 stores programs that constitute information processing applications for realizing the functions of the data acquisition unit 21A, inference unit 22A, notification processing unit 24, superposition processing unit 25, and storage processing unit 26. The processor 103 executes the information processing application read from the memory 104, thereby realizing the functions of the data acquisition unit 21A, the inference unit 22A, the notification processing unit 24, the superposition processing unit 25, and the storage processing unit 26.

[0094] Furthermore, while we have shown a case where the first inference unit 221 uses the trained model 231A to infer whether or not an object to be detected exists in the imaging area, the object detection device 2B or 2C is not limited to this. For example, the first inference unit 221 may determine whether or not an object to be detected exists in the shooting area based on the results of image analysis of the still image of the shooting area obtained in processing (1C), without using the trained model 231A. Image analysis may include processes such as preprocessing, feature point extraction, shape analysis, and labeling of objects. Preprocessing includes, for example, grayscale conversion of color images to monochrome, noise reduction using smoothing filters, and adjustment of brightness or contrast. Feature point extraction includes processes such as detecting object edges using edge detection algorithms, binarization using thresholds, and identifying candidate objects in multiple sections within an image. Shape analysis determines the shape and size of the object identified by feature point extraction. Object labeling involves assigning a label to the region of an object in an image once its shape analysis determines it to be the target object. By defining the boundaries of the region identified by this label, the object detection results are visualized. The determination unit 222, based on temperature information, determines whether or not an object to be detected exists in the shooting area based on the determination result from image analysis, thereby preventing false detections caused by feature point extraction from the captured image.

[0095] As described above, in the object detection device 2B or 2C according to Embodiment 2, the inference unit 22A comprises a first inference unit 221 and a determination unit 222. The first inference unit 221 uses a trained model 231A to infer whether or not an object to be detected exists in the shooting area. The trained model 231A is a model that, when an image of the shooting area is input, outputs an inference result of whether or not an object to be detected exists in the shooting area. The determination unit 222 determines whether or not an object to be detected exists in the shooting area based on the inference result output from the trained model 231A, and outputs the determination result as the inference result. Since the inference result using the image of the shooting area is used to determine whether or not an object to be detected exists in the shooting area based on the difference in temperature information between the water surface and the object on the water surface, the object detection device 2B or 2C can prevent false detections caused by feature point extraction from the captured image. For example, object detection devices 2B or 2C can accurately infer whether or not a target object is present even if the image information is unclear, such as when the river is photographed at night, or if the image information contains reflections of objects other than those to be detected on the water's surface. Furthermore, object detection devices 2B or 2C can be implemented simply by updating the software to add a judgment processing function to object detection devices that detect objects using only captured images.

[0096] The object detection device 2C according to Embodiment 2 includes an imaging unit 27 for capturing video information and a temperature detection unit 28 for detecting temperature information. The data acquisition unit 21A acquires video information from the imaging unit 27 and temperature information from the temperature detection unit 28. Since the object detection device 2C integrates the imaging unit 27 and the temperature detection unit 28, the object detection device 2C can be provided not only as a device fixedly positioned beside a river or the like, but also as a mobile device or a portable terminal.

[0097] In the object detection method according to Embodiment 2, the data acquisition unit 21 or 21A performs step ST1A of acquiring video information of the water surface and temperature information in the shooting area of ​​the video information, and the inference unit 22A performs step ST2A of inferring whether or not an object to be detected exists in the shooting area based on the image of the shooting area shown by the video information and the difference in temperature information between the water surface and an object on the water surface. The inference unit 22A uses a trained model 231A to infer whether or not an object to be detected exists in the shooting area. The trained model 231 is a model that, when an image of the shooting area is input, outputs an inference result of whether or not an object to be detected exists in the shooting area. The determination unit 222 determines whether or not an object to be detected exists in the shooting area based on the inference result output from the trained model 231A, and outputs the determination result as the inference result. By having the object detection device 2B or 2C execute this method, false detections caused by feature point extraction from the captured image can be prevented.

[0098] The computer executing the program according to Embodiment 2 functions as an object detection device 2B or 2C. This makes it possible to provide an object detection device 2B or 2C that can prevent false detections caused by feature point extraction from captured images.

[0099] The monitoring system 1 according to Embodiment 2 includes an object detection device 2B or 2C and a monitoring device 5 that monitors the object to be detected in the shooting area based on the information output from the object detection device 2B or 2C. This makes it possible to provide a monitoring system 1 that can accurately infer whether or not the object to be detected is present, even if the image information is unclear, such as when a river is photographed at night, or if the image information contains reflections of objects other than the object to be detected on the water surface.

[0100] The object detection device described herein can be used, for example, to detect driftwood and other floating objects.

[0101] The various aspects of this disclosure are summarized below as an appendix.

[0102] (Note 1) A data acquisition unit that acquires video information of the water surface and temperature information in the area of ​​the video information, An object detection device comprising: an inference unit that infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area shown by the video information, as well as the difference in temperature information between the water surface and an object on the water surface. (Note 2) The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting area including the temperature information is input, outputs an inference result of whether or not the object to be detected exists in the shooting area. The object detection device described in Appendix 1, characterized by the features described herein. (Note 3) The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting region is input, outputs an inference result of whether or not the object to be detected exists in the shooting region. The inference unit has a determination unit, The determination unit determines, based on the temperature information, whether or not the object to be detected exists in the imaging area based on the inference result output from the trained model, and outputs the determination result as the inference result. The object detection device described in Appendix 1, characterized by the features described herein. (Note 4) The temperature information is an image showing the temperature distribution of the shooting area. The trained model is a model that, when an image of the image area with the temperature distribution associated with it is input, outputs an inference result of whether or not the object to be detected exists in the image area. The object detection device described in Appendix 2, characterized by the features described herein. (Note 5) The system includes a notification processing unit that, upon obtaining an inference result indicating the presence of the object to be detected in the imaging area, generates notification information to notify the presence of the object to be detected. An object detection device as described in any one of the appendices 1 to 4, characterized by the features described herein. (Note 6) The system includes a superposition processing unit that generates a notification image by superimposing the image area of ​​the object to be detected onto the image of the shooting area. An object detection device as described in any one of the appendices 1 to 5, characterized by the features described herein. (Note 7) The system includes a storage processing unit that stores the inference results from the inference unit in a storage unit, reads the inference results according to the request from the storage unit, and outputs them to the requester. An object detection device as described in any one of the appendices 1 to 6, characterized by the features described herein. (Note 8) A camera unit that captures the aforementioned video information, The system includes a temperature detection unit for detecting the aforementioned temperature information, The data acquisition unit acquires the video information from the imaging unit and the temperature information from the temperature detection unit. An object detection device as described in any one of the appendices 1 to 7, characterized by the features described herein. (Note 9) An object detection method using an object detection device, The data acquisition unit performs the steps of acquiring video information of the water surface and temperature information in the area of ​​the video information, The inference unit includes the step of inferring whether or not an object to be detected exists in the shooting area, based on the image of the shooting area indicated by the video information, as well as the difference in temperature information between the water surface and an object on the water surface. The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting area including the temperature information is input, outputs an inference result of whether or not the object to be detected exists in the shooting area. A method for detecting an object characterized by the following features. (Note 10) An object detection method using an object detection device, The data acquisition unit performs the steps of acquiring video information of the water surface and temperature information in the area of ​​the video information, The inference unit includes the step of inferring whether or not an object to be detected exists in the shooting area, based on the image of the shooting area indicated by the video information, as well as the difference in temperature information between the water surface and an object on the water surface. The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting region is input, outputs an inference result of whether or not the object to be detected exists in the shooting region. The inference unit has a determination unit, The determination unit determines, based on the temperature information, whether or not the object to be detected exists in the imaging area based on the inference result output from the trained model, and outputs the determination result as the inference result. A method for detecting an object characterized by the following features. (Note 11) A program to cause a computer to function as an object detection device as described in any one of the appendices 1 through 8. (Note 12) An object detection device described in any one of the appendices 1 to 8, A monitoring device that monitors the object to be detected in the shooting area based on the information output from the object detection device, A monitoring system equipped with this system.

[0103] Furthermore, it is possible to combine each embodiment, modify any component of each embodiment, or omit any component in each embodiment. [Explanation of Symbols]

[0104] 1 Monitoring system, 2,2A,2B,2C Object detection device, 3 Imaging device, 4 Temperature sensor, 5 Monitoring device, 21,21A Data acquisition unit, 22,22A Inference unit, 23,23A Memory unit, 24 Notification processing unit, 25 Superposition processing unit, 26 Storage processing unit, 27 Imaging unit, 28 Temperature detection unit, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 Processor, 104 Memory, 221 First inference unit, 222 Judgment unit, 231,231A Trained model, 232 Storage unit.

Claims

1. A data acquisition unit that acquires video information of the water surface and temperature information in the area of ​​the video information, An object detection device comprising: an inference unit that infers whether or not an object to be detected exists in the shooting area based on the image of the shooting area shown by the video information, as well as the difference in temperature information between the water surface and an object on the water surface.

2. The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting area including the temperature information is input, outputs an inference result of whether or not the object to be detected exists in the shooting area. The object detection device according to feature 1.

3. The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting area is input, outputs an inference result of whether or not the object to be detected exists in the shooting area. The inference unit has a determination unit, The determination unit determines, based on the temperature information, whether or not the object to be detected exists in the imaging area based on the inference result output from the trained model, and outputs the determination result as the inference result. The object detection device according to feature 1.

4. The temperature information is an image showing the temperature distribution of the shooting area. The trained model is a model that, when an image of the image area with the temperature distribution associated with it is input, outputs an inference result of whether or not the object to be detected exists in the image area. The object detection device according to feature 2.

5. The system includes a notification processing unit that, upon obtaining an inference result indicating the presence of the object to be detected in the imaging area, generates notification information to notify the presence of the object to be detected. The object detection device according to feature 1.

6. The system includes a superposition processing unit that generates a notification image by superimposing the image area of ​​the object to be detected onto the image of the shooting area. The object detection device according to feature 1.

7. The system includes a storage processing unit that stores the inference results from the inference unit in a storage unit, reads the inference results according to the request from the storage unit, and outputs them to the requester. The object detection device according to feature 1.

8. A camera unit that captures the aforementioned video information, The system includes a temperature detection unit for detecting the aforementioned temperature information, The data acquisition unit acquires the video information from the imaging unit and the temperature information from the temperature detection unit. The object detection device according to feature 1.

9. An object detection method using an object detection device, The data acquisition unit performs the steps of acquiring video information of the water surface and temperature information in the area of ​​the video information, The inference unit includes the step of inferring whether or not an object to be detected exists in the shooting area, based on the image of the shooting area indicated by the video information, as well as the difference in temperature information between the water surface and an object on the water surface. The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting area including the temperature information is input, outputs an inference result of whether or not the object to be detected exists in the shooting area. A method for detecting an object characterized by the following features.

10. An object detection method using an object detection device, The data acquisition unit performs the steps of acquiring video information of the water surface and temperature information in the area of ​​the video information, The inference unit includes the step of inferring whether or not an object to be detected exists in the shooting area, based on the image of the shooting area indicated by the video information, as well as the difference in temperature information between the water surface and an object on the water surface. The inference unit uses the trained model to infer whether or not the object to be detected exists in the imaging area. The trained model is a model that, when an image of the shooting area is input, outputs an inference result of whether or not the object to be detected exists in the shooting area. The inference unit has a determination unit, The determination unit determines, based on the temperature information, whether or not the object to be detected exists in the imaging area based on the inference result output from the trained model, and outputs the determination result as the inference result. A method for detecting an object characterized by the following features.

11. A program for causing a computer to function as an object detection device according to any one of claims 1 to 8.

12. An object detection device according to any one of claims 1 to 8, A monitoring device that monitors the object to be detected in the shooting area based on the information output from the object detection device, A monitoring system equipped with this system.

Images