Monitoring system and object detection method

Abstract

To realize an object detection method that can adapt to changes in brightness within the shooting environment. [Solution] An object detection method is provided for detecting a target object in a captured image using an object detector that detects a target object in the input image and outputs a score value indicating the likelihood that the object detected from the input image is the target object. In this method, brightness information is obtained from the captured image, and a judgment threshold, which is compared with the score value, is changed according to the brightness such that the value decreases as it gets darker. If the score value output from the object detector in response to the input image of the captured image is greater than the adjusted judgment threshold, it is determined that the object detected from the captured image is the target object.

Description

Technical Field

[0001] The present invention relates to a monitoring system and an object detection method.

Background Art

[0002] Monitoring cameras are installed at various locations such as airports, harbors, power plants, railroad crossings, and commercial facilities for safety assurance. Conventionally, a monitor visually checks the video output from the monitoring camera, and when an intruder, obstacle, suspicious object, etc. is confirmed, the monitor takes actions such as going to the site. Recently, intrusion detection systems (see, for example, Patent Documents 1 and 2) assuming use in facilities with a wide monitoring area such as airports, harbors, and power plants, and obstacle detection systems (see, for example, Patent Document 3) for detecting obstacles within a railroad crossing are also being used.

[0003] For example, in the system of Patent Document 1, a laser sensor is used to detect intruders, and the video output from the monitoring camera is used for the monitor to confirm. In the systems of Patent Documents 2 and 3, an intruder or an obstacle is detected by comparing the video taken by the monitoring camera during normal times with the actually taken video. Although any of these systems can detect that something has intruded into the monitoring area, it is necessary for the monitor to view and confirm the video in order to identify what the detected object is. Depending on the type of the detected object and the shooting environment, it may be difficult to confirm in the video, and in that case, it is necessary for the monitor to go to the site to confirm.

[0004] Furthermore, in conventional systems, the type of detected object is not clear, so even if the object is not a target object to be monitored (e.g., a person, animal, vehicle, etc.), the monitor needs to take action to ensure safety, such as going to the site. If objects that are not the target object are frequently detected, the burden on the monitor may actually increase despite the introduction of the system. Therefore, a means of determining whether a detected object is a target object is necessary. Incidentally, although it is not a technology related to monitoring systems, Patent Document 4 shows an example of an object detector for detecting a specific type of object from an input image. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2005-145602 [Patent Document 2] Japanese Patent Publication No. 2008-066864 [Patent Document 3] Japanese Patent Publication No. 2019-147443 [Patent Document 4] Japanese Patent Publication No. 2021-026644 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] An object detector detects specific types of objects in an input image and outputs a score value indicating the likelihood that the detected object is of that specific type. Such object detectors can be constructed using learning methods such as SVM (Support Vector Machine) or DNN (Deep Neural Network). By comparing the score value output by the object detector in response to the image input with a preset threshold, it is possible to determine whether or not an object in the input image is of a specific type.

[0007] Using the object detector described above, a monitoring system can be constructed that issues warnings or other notifications to the monitor when a target object is detected in the captured image. If the accuracy of target object detection improves, the problem of increased burden on the monitor due to the frequent detection of objects that are not the target of monitoring can be solved. Even with conventional methods, it is possible to construct an object detector that shows a sufficiently high score value for target objects in images taken under certain conditions, and it is possible to distinguish between target objects and other objects with high accuracy in images taken under those specific conditions.

[0008] However, even with the same object detector and threshold, it is not always possible to distinguish between target objects and other objects with high accuracy in images taken under different environmental conditions. For example, even if an object detector and threshold that have high detection accuracy in images taken in bright environments are used, they often fail to correctly identify target objects in images taken in dark environments where details and contours of objects become unclear. In surveillance systems, object detection is performed on images taken in environments where the brightness changes moment by moment, so object detection technology that can adapt to such changes in brightness is required.

[0009] To adapt conventional systems to changes in brightness, it is necessary to rebuild the object detection algorithm or re-optimize the parameters that make up the object detection algorithm in response to the change in brightness. However, it is practically difficult to rebuild or re-optimize the parameters of an object detection algorithm after it has been incorporated into a system, and it is even more difficult to continuously modify the object detection algorithm to keep up with the changes in brightness in an environment where the brightness changes moment by moment. In light of these circumstances, there is a need for an object detection technology that can adapt to changes in brightness in a simpler way.

[0010] Therefore, the present invention aims to provide an object detection method that can adapt to changes in brightness within the shooting environment in a simpler manner, and a monitoring system that implements the same. [Means for solving the problem]

[0011] According to a first aspect of the present invention, a monitoring system is provided that has the function of detecting a target object included in a captured image using an object detector that detects a target object (one or more objects of a specific type to be monitored) included in an input image and outputs a score value indicating the likelihood that the object detected from the input image is a target object. This monitoring system includes a threshold adjustment unit that acquires brightness information from a captured image and changes a judgment threshold that is compared with a score value according to the brightness, such that the value decreases as the brightness decreases, and a determination unit that determines that the object detected from the captured image is a target object when the score value output from the object detector in response to the input of a captured image is greater than the adjusted judgment threshold.

[0012] As described above, the score value indicates the likelihood that the detected object is the target object. Images taken in bright environments often have clear details and contours of the object, and therefore tend to have relatively high score values ​​for the target object. On the other hand, images taken in dark environments often have unclear details and contours of the object, and therefore tend to have relatively lower score values ​​for the target object compared to images taken in bright environments.

[0013] Setting a higher detection threshold reduces the risk of mistakenly identifying an object that is not part of the target object. If a misidentification occurs, a monitor who receives a notification of object detection from the monitoring system will go to the site to check, only to find that the target object is not there, resulting in wasted effort. Reducing the risk of misidentification also reduces the burden on such monitors. On the other hand, setting the detection threshold too high increases the risk of a target object being mistakenly identified as an object that is not part of the target object, and the monitoring system will not issue a notification that a target object has been detected even though the target object is actually present (risk of missed detection). Since missed detection can lead to accidents and problems, it is important to avoid missed detections.

[0014] If the detection threshold is set to allow for high-accuracy differentiation between target and non-target objects under bright conditions, detection failures are more likely to occur under dark conditions. On the other hand, if the detection threshold is set to allow for differentiation between target and non-target objects under dark conditions, the risk of mistakenly identifying non-target objects as target objects increases under bright conditions, leading to an increased burden on the monitor. Therefore, if the detection threshold is set as a fixed value, it is difficult to reduce detection failures in dark environments while simultaneously minimizing the chances of non-target objects being mistakenly detected as target objects under bright conditions.

[0015] However, by applying the monitoring system according to the first embodiment described above, the judgment threshold is adjusted so that the value decreases as the brightness decreases. This makes it less likely for detection to be missed in dark environments, and reduces the chance of objects that are not subject to monitoring being mistakenly detected as target objects at any brightness level. As a result, it becomes possible to reduce the burden on the monitor while suppressing detection misses that could lead to accidents and troubles.

[0016] Furthermore, according to a second aspect of the present invention, an object detection method is provided for detecting a target object included in a captured image using an object detector that detects a target object included in an input image and outputs a score value indicating the likelihood that the object detected from the input image is a target object. This method includes the steps of: acquiring brightness information from the captured image and changing a judgment threshold compared with the score value according to the brightness such that the value decreases as it gets darker; and determining that the object detected from the captured image is a target object if the score value output from the object detector in response to the input of the captured image is greater than the adjusted judgment threshold.

[0017] With this configuration, similar to the first embodiment described above, it becomes possible to reduce the burden on the monitor while suppressing detection failures that could lead to accidents or troubles. [Effects of the Invention]

[0018] According to the present invention, there are provided an object detection method adaptable to brightness changes in a shooting environment in a simpler manner and a monitoring system implementing the same.

Brief Description of Drawings

[0019] [Figure 1] It is a block diagram showing a configuration example of the monitoring system according to the present embodiment. [Figure 2] It is a block diagram showing a configuration example of the monitoring system according to a modified example of the present embodiment. [Figure 3A] It is a first explanatory diagram for explaining the object detection method according to the present embodiment. [Figure 3B] It is a second explanatory diagram for explaining the object detection method according to the present embodiment. [Figure 3C] It is a third explanatory diagram for explaining the object detection method according to the present embodiment. [Figure 3D] It is a fourth explanatory diagram for explaining the object detection method according to the present embodiment. [Figure 4A] It is a first explanatory diagram for explaining the method for adjusting the determination threshold according to the present embodiment. [Figure 4B] It is a second explanatory diagram for explaining the method for adjusting the determination threshold according to the present embodiment. [Figure 4C] It is a third explanatory diagram for explaining the method for adjusting the determination threshold according to the present embodiment. [Figure 4D] It is a fourth explanatory diagram for explaining the method for adjusting the determination threshold according to the present embodiment. [Figure 5A] It is a first explanatory diagram for explaining a further method for adjusting the determination threshold according to the present embodiment. [Figure 5B] It is a second explanatory diagram for explaining a further method for adjusting the determination threshold according to the present embodiment. [Figure 5C] It is a third explanatory diagram for explaining a further method for adjusting the determination threshold according to the present embodiment. [Figure 6]This flowchart illustrates the processing flow performed by the monitoring system according to this embodiment. [Figure 7] This is an explanatory diagram illustrating a method for adjusting the timing of threshold change according to one modified example of this embodiment. [Figure 8] This block diagram shows an example of a computer hardware configuration that can implement the functions of the image processing device and monitoring terminal according to this embodiment. [Modes for carrying out the invention]

[0020] Embodiments of the present invention will be described below with reference to the attached drawings. Note that elements having substantially the same function in this specification and the drawings may be denoted by the same reference numerals, thus omitting redundant explanations.

[0021] (Configuration of the monitoring system) The configuration of the monitoring system according to this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the configuration of the monitoring system according to this embodiment. Note that the monitoring system 10 shown in Figure 1 is just one example of the monitoring system according to this embodiment, and some elements may be omitted during implementation, or additional elements not shown may be added.

[0022] As shown in Figure 1, the monitoring system 10 includes an imaging device 11, an image processing device 12, and a monitoring terminal 13.

[0023] Here, the system including the imaging device 11, the image processing device 12, and the monitoring terminal 13 is referred to as a "monitoring system." However, when manufacturing, shipping, and distributing the product, for example, the system combining the imaging device 11 and the image processing device 12, the system combining the image processing device 12 and the monitoring terminal 13, or any system including the functions of the image processing device 12 may be referred to as a "monitoring system." In the following explanation, we will use the example in Figure 1 for ease of explanation, but the form of the monitoring system according to this embodiment is not limited to this.

[0024] For example, in the example in Figure 1, only one imaging device is shown for the sake of brevity, but the number of imaging devices may be two or more. Also, the means for connecting at least two of the elements of the imaging device 11, image processing device 12, and monitoring terminal 13 may be a direct wired connection, an indirect connection via one or more other devices, or a communication connection via a wired and / or wireless communication network. Furthermore, the imaging device 11, image processing device 12, and monitoring terminal 13 may be located in separate locations from each other, or at least two of the elements may be located in the same location.

[0025] Furthermore, although the imaging device 11, image processing device 12, and monitoring terminal 13 are represented as separate blocks in the example in Figure 1, at least two of the imaging device 11, image processing device 12, and monitoring terminal 13 may be configured as a single unit. Also, some of the functions of any of the imaging device 11, image processing device 12, and monitoring terminal 13 may be included in the other elements. For example, some of the functions of the image processing device 12 may be included in the monitoring terminal 13, and some of the functions of the monitoring terminal 13 may be included in the image processing device 12. In addition, some of the functions of the imaging device 11, image processing device 12, and monitoring terminal 13 may be implemented by external devices other than those.

[0026] As described above, the configuration of the monitoring system 10 can be modified in various ways, and it should be noted that such modified forms are also included within the technical scope of this embodiment.

[0027] (Regarding the imaging device 11) The imaging device 11 is a surveillance camera capable of capturing video (moving images) composed of multiple video frames and / or multiple still images (time-lapse images) at a predetermined time interval. In the following description, for the sake of brevity, a single video frame or time-lapse image output from the imaging device 11 will be referred to as a "captured image".

[0028] The imaging device 11 includes an optical system comprising one or more lens groups, an aperture, and an optical filter; a drive and control mechanism for moving the lenses within the optical system for focusing and zooming; an image sensor that converts light incident through the optical system into an electrical signal and outputs it; and a signal processing circuit that processes the electrical signal output from the image sensor to generate image data. The imaging device 11 further includes a communication interface for transmitting image data to the image processing device 12 and the monitoring terminal 13. The imaging device 11 may also further include other communication interfaces for receiving operation signals from the monitoring terminal 13.

[0029] (Regarding the image processing device 12) The image processing device 12 acquires the captured image output from the imaging device 11 and has an object detection function for detecting target objects from the acquired captured image. The target object is an object to be monitored, and the type of object to be the target object is set in advance. Note that the number of target object types may be two or more.

[0030] In cases where the system is used to detect objects entering a railway crossing, the target object types may include people, animals, and vehicles. In cases where the system is used to detect obstacles on a road (falling rocks, soil, snow, fallen trees, etc.), the target object types may include stones, rocks, sand, mud, snow, ice, tree branches, and trunks. Depending on the monitoring purpose, other target object types may include concrete blocks, metal wires, paper, plastic bags, motorcycles, bicycles, wheelchairs, strollers, skateboards, trains, aircraft, drones, and ships. As you can see, a wide variety of objects can be monitored.

[0031] The image processing device 12 includes a score value acquisition unit 121, a determination unit 122, a storage unit 123, and a threshold adjustment unit 124.

[0032] The score acquisition unit 121 acquires the captured image output from the imaging device 11, inputs the acquired image to the object detector 121a, and acquires the score value output from the object detector 121a. The object detector 121a detects objects included in the input image and outputs a score value indicating the likelihood that the detected object is the target object.

[0033] The object detector 121a is either a software module (computer program, AI model, etc.) for implementing an object detection algorithm, or a hardware module that implements an object detection algorithm. If the object detector 121a is a software module, the score acquisition unit 121 operates that software module to realize the functions of the object detector 121a. The applicable object detection algorithm is any currently available object detection algorithm or any object detection algorithm that will become available in the future. For example, the object detection algorithm can be constructed using learning methods such as SVM or DNN.

[0034] The score value mentioned above represents the likelihood that the object detected in the input image is the target object. The range of this value is arbitrary, but for example, a normalized value in the range of 0 to 1 may be used. When the score value is 0, the likelihood that the detected object is the target object is the lowest, and when the score value is 1, the likelihood that the detected object is the target object is the highest. Note that even for the same object in the same input image, the score value output from the object detector 121a may differ if the object detection algorithm used is different. However, the technology of this embodiment is applicable to any object detection algorithm.

[0035] In the example shown in Figure 1, the object detector 121a is held by the score value acquisition unit 121. However, as a modified example, as shown in Figure 2, the image processing device 12 can be configured to acquire the score value using an external object detector.

[0036] In the example shown in Figure 2, the score acquisition unit 121 of the monitoring system 10a does not hold the object detector 121a, but accesses an external object detector 14 via the network and acquires the score value using the object detector 14. For example, the object detector 14 may be a cloud service or server that provides an object detection function based on the object detection algorithm described above. Such modifications are also included within the technical scope of this embodiment. Note that the configuration of the monitoring system 10a shown in Figure 2 is the same as the monitoring system 10 shown in Figure 1, except for the configuration of the score acquisition unit 121 described above, so a detailed explanation is omitted.

[0037] Refer to Figure 1 again. When a captured image is input, the object detector 121a outputs information about each object detected from the captured image (hereinafter referred to as detected object information) and a score value corresponding to each object. The detected object information includes the position of the object in the captured image. The score value acquisition unit 121 outputs the detected object information and score value output from the object detector 121a. The detected object information and score value are input to the determination unit 122.

[0038] The determination unit 122 compares the score value output from the score value acquisition unit 121 with the determination threshold output from the threshold adjustment unit 124 (described later), and determines that the object corresponding to the score value is the target object if the score value is equal to or greater than the determination threshold. The determination result by the determination unit 122 is output to the monitoring terminal 13 as a detection result. For example, the determination unit 122 associates the location of the object determined to be the target object with the score value corresponding to that object and sends it to the monitoring terminal 13 as a detection result. At this time, the determination unit 122 may also send to the monitoring terminal 13 the location of the object whose score value is less than the determination threshold and the score value corresponding to that object as information about objects that are not the target object.

[0039] The threshold adjustment unit 124 acquires brightness information from the captured image and adjusts the judgment threshold so that the value decreases as the brightness indicated by the acquired brightness information decreases. For example, the threshold adjustment unit 124 acquires the average brightness value of the captured image as brightness information.

[0040] The average brightness value of the captured image corresponds to the brightness within the monitoring area. For example, if the monitoring area is outdoors, the average brightness value of the captured image will be high during the day because the monitoring area is bright. On the other hand, at night the monitoring area becomes very dark, so the average brightness value of the captured image will be low. In addition, the brightness within the monitoring area changes depending on various environmental conditions, such as weather conditions like sunny, rainy, cloudy, snowy, and fog, shadows from buildings and trees, and light rays from streetlights and vehicle lights. Indoors, in addition to the on / off / flashing of lights, the brightness changes at night due to light entering through windows. Thus, although there are various environmental factors, the brightness within the monitoring area can be evaluated by referring to the average brightness value of the captured image.

[0041] Here, the average brightness value is shown as an example, but other statistical values ​​related to the brightness value of the captured image may be used as brightness information. For example, the median brightness value of the entire captured image may be used. Alternatively, statistical values ​​of brightness values ​​in a part of the captured image, rather than the entire image, may be used as brightness information. For example, statistical values ​​of brightness values ​​in a part of a circular or rectangular area of ​​a predetermined size based on the center of the image may be used as brightness information, or statistical values ​​of brightness values ​​in a part of an area including individual objects and their surroundings for which score values ​​have been obtained may be used as brightness information. When using score values, the threshold adjustment unit 124 is modified to acquire the output of the object detector 121a from the score value acquisition unit 121.

[0042] The threshold adjustment unit 124, which has acquired the luminance information, refers to the judgment threshold information 123a that is pre-stored in the storage unit 123 and identifies the judgment threshold corresponding to the brightness indicated by the acquired luminance information. The judgment threshold information 123a includes information that associates the brightness value indicated by the luminance information (hereinafter referred to as the luminance value BR) with the judgment threshold (TH).

[0043] The relationship between the luminance value BR and the judgment threshold TH is predetermined such that the judgment threshold TH decreases as the luminance value BR decreases. This relationship may be expressed in a table, a mathematical formula, or any other format. The relationship between the luminance value BR and the judgment threshold TH will be described in detail later with specific examples (see Figures 4A-4D and 5A-5C).

[0044] The threshold adjustment unit 124 outputs a judgment threshold TH corresponding to the brightness value BR to the judgment unit 122. As already mentioned, the judgment unit 122 compares the judgment threshold TH output from the threshold adjustment unit 124 with the score value and outputs information including the judgment result as a detection result to the monitoring terminal 13. As a result, even in environments where the brightness changes moment by moment, the judgment threshold TH is adjusted according to the brightness at that time, making it possible to adapt to changes in brightness.

[0045] (Regarding monitoring terminal 13) The monitoring terminal 13 is a terminal device or system used by users such as monitors or system administrators for monitoring tasks. In the example in Figure 1, the monitoring terminal 13 is represented as a single block, but multiple monitoring terminals may be provided within the monitoring system 10, or the system may separately include one or more operation terminals for operating the imaging device 11 and image processing device 12, and one or more display terminals for displaying captured images output from the imaging device 11. Furthermore, the monitoring terminal 13 may be a portable terminal such as a smartphone or tablet. Thus, various variations in the form of the monitoring terminal 13 are possible, and such variations are naturally included within the technical scope of this embodiment.

[0046] The monitoring terminal 13 includes an output unit 131, a control unit 132, and an input unit 133.

[0047] The output unit 131 has the function of displaying the captured image output from the imaging device 11, and / or outputting voice or alert sounds based on the detection result output from the determination unit 122 of the image processing device 12. For example, the output unit 131 can display the captured image output from the imaging device 11 in real time and superimpose the position of the target object and its corresponding score value onto the captured image based on the detection result. In addition, when the output unit 131 obtains a detection result indicating the detection of a target object, it can display a warning indicating that the target object has entered the monitoring area, and can also output voice and / or alert sounds to indicate the warning.

[0048] The control unit 132 controls the overall operation of the monitoring terminal 13. For example, the control unit 132 receives the detection result output from the determination unit 122 of the image processing device 12 and controls the display output and / or audio output by the output unit 131 described above based on the detection result. The input unit 133 is an input interface for the user to input information. For example, the input unit 133 may be an input interface that supports information input by manual input and / or voice input, such as a keyboard, mouse, touchpad, or microphone. Furthermore, using a touch panel can realize the functions of both the output unit 131 and the input unit 133.

[0049] Furthermore, the control unit 132 may send setting information to the image processing device 12 or operation signals to the imaging device 11 in response to information input from the input unit 133. For example, if the settings of the imaging device 11 (zoom amount, exposure setting, shooting interval, etc.) can be operated from the monitoring terminal 13, operation signals indicating the operation content may be sent to the imaging device 11 in response to user operation. Also, if the setting of the judgment threshold TH can be adjusted by user operation, setting information is sent to the image processing device 12 in response to user operation. The setting adjustment of the judgment threshold TH will be described later (see Figures 5A to 5C).

[0050] In the following explanation, for the sake of clarity, we will use the above-described example configuration of the monitoring system 10, but the scope of application of the technology according to this embodiment is not limited to this example. For example, in Figure 1, the system is divided into blocks by function for the sake of clarity, but the division is arbitrary, and some or all of the above-described functions can be realized by a computer equipped with one or more processors or chips, one or more memories, or a combination thereof. Furthermore, some of the functions of each of the above-described elements may be realized in cooperation with an external system or server, such as a cloud system or a server on a network. Such modified forms may also be included in the technical scope of this embodiment.

[0051] Next, the object detection method according to this embodiment will be explained in more detail with reference to Figures 3A to 3D. Here, for the sake of simplicity, the case where the target object is a "person" will be used as an example, but the same applies to cases where the target object is not a "person".

[0052] First, refer to Figure 3A. Figure 3A schematically shows an image P1 taken under bright conditions (Condition 1: Luminance value BR is 0.8 or higher), and also shows Table T1, which shows examples of score values ​​corresponding to each object in the image P1. Note that the score values ​​in Table T1 are merely examples for illustrative purposes, and different score values ​​may be output depending on the type of object detection algorithm used.

[0053] Objects M1, M2, and M3 in the captured image P1 are the target objects, while objects OBJ1, OBJ2, OBJ3, and OBJ4 in the captured image P1 are not the target objects.

[0054] Condition 1 assumes brightness conditions in environments such as outdoors on a sunny day or indoors with sufficient lighting. Images taken in such bright environments often have clear details and outlines of objects, and tend to output high scores for target objects and low scores for non-target objects.

[0055] As shown in Table T1, in this example, the score values ​​of target objects M1, M2, and M3 are 0.8 or higher, while the score values ​​of non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are less than 0.4. Therefore, by setting the judgment threshold TH to around 0.5 to 0.8, target objects M1, M2, and M3 can be detected without fail. Also, non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are determined to be non-target objects. In other words, target objects and non-target objects can be correctly distinguished.

[0056] It should be noted that the images actually taken may contain various objects, and the background may be complex. For example, there may be non-human objects that resemble "people," or the color of the clothes a "person" is wearing may be similar to the color of the building in the background, causing the "person's" outline to become unclear where they overlap. Also, the direction the "person" is facing and the way the light hits them can make their outline unclear or obscur the details of their features. Therefore, in reality, even under bright conditions, the score value of an object that is a "person" may be low, while the score value of objects other than "people" may be high.

[0057] Considering the above circumstances, when determining the judgment threshold TH, one should prepare many sample images taken under the same brightness conditions and determine the judgment threshold TH to be used as such that the target object and non-target objects are distinguished with a probability greater than or equal to a predetermined probability in those sample images. Of course, one could also construct an AI model to determine a judgment threshold TH that can distinguish between target objects and non-target objects with sufficient accuracy, and use that AI model to determine the judgment threshold TH.

[0058] Next, refer to Figure 3B. Figure 3B schematically shows an image P2 taken under slightly dark conditions (Condition 2: Brightness value BR is 0.4 or higher and less than 0.8), and also shows Table T2, which shows examples of score values ​​corresponding to each object in the image P2. Note that the score values ​​in Table T2 are merely examples for explanation, and different score values ​​may be output depending on the type of object detection algorithm used. Image P2 shows the same scene as image P1 but taken under slightly dark conditions.

[0059] Condition 2 assumes brightness conditions in slightly dark environments, such as during twilight, outdoors in rainy weather, or indoors with little lighting. While details and outlines of objects are clearly visible in images taken in bright environments like Condition 1, the characteristics of individual objects tend to become unclear in slightly darker environments like Condition 2. Therefore, the overall score tends to decrease as the brightness decreases. However, it should be noted that depending on the color and shape of an object, its characteristics may be clearer in slightly darker environments than in images taken in bright environments, so the score of all objects does not necessarily decrease.

[0060] As shown in Table T2, in this example, the score values ​​of target objects M1, M2, and M3 are 0.6 or higher, while the score values ​​of non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are less than 0.45. Therefore, if the judgment threshold TH (around 0.5 to 0.8) suitable for the above condition 1 is used as is, detection failures will occur (if the judgment threshold TH is 0.7, target object M3 will be judged as a non-target object). However, in the monitoring system 10 according to this embodiment, the judgment threshold TH is changed according to changes in brightness conditions.

[0061] For example, in the case shown in Figure 3B, by setting the judgment threshold TH to 0.5, target objects M1, M2, and M3 can be detected without fail. Furthermore, non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are determined to be non-target objects. In other words, target objects and non-target objects can be correctly distinguished.

[0062] When determining the judgment threshold TH for condition 2, as with condition 1, you should prepare many sample images taken under the same brightness conditions and determine the judgment threshold TH to be used as such that the target object and non-target objects are distinguished with a probability greater than or equal to a predetermined probability in those sample images. Of course, as with condition 1, you may also use an AI model to determine the judgment threshold TH.

[0063] Next, refer to Figure 3C. Figure 3C schematically shows an image P3 taken under dark conditions (Condition 3: Luminance value BR is less than 0.4), and also shows Table T3, which shows examples of score values ​​corresponding to each object in the image P3. Note that the score values ​​in Table T3 are merely examples for illustrative purposes, and different score values ​​may be output depending on the type of object detection algorithm used. Image P3 shows the same scene as images P1 and P2, but taken under dark conditions.

[0064] Condition 3 assumes brightness conditions in dark environments such as outdoors at night or indoors without lighting. In dark environments like Condition 3, details and contours of objects become very unclear. In addition, ambient light and shadows can alter the characteristics of objects that can be identified from the captured image. Therefore, under Condition 3, it becomes difficult to accurately evaluate the certainty of the target object. However, there is a tendency for the overall score to decrease with decreasing brightness, and the score is often lower than in the cases of Conditions 1 and 2.

[0065] As shown in Table T3, in this example, the score values ​​of target objects M1 and M2 are 0.4 or higher, and the score value of target object M3, which is located far away, is 0.28. For the non-target object OBJ3, the object detector 121a was unable to detect it as an object, and therefore no score value was output. Also, the score values ​​of the non-target objects OBJ1, OBJ2, and OBJ4 are less than 0.2. Therefore, if the judgment threshold TH (0.5) suitable for the above condition 2 is used as is, detection failures will occur (if the judgment threshold TH is 0.5, target objects M1, M2, and M3 will all be judged as non-target objects). However, in the monitoring system 10 according to this embodiment, the judgment threshold TH is changed according to changes in brightness conditions.

[0066] For example, in the case shown in Figure 3C, by setting the judgment threshold TH to 0.25, target objects M1, M2, and M3 can be detected without fail. Furthermore, non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are determined to be non-target objects. In other words, target objects and non-target objects can be correctly distinguished.

[0067] When determining the judgment threshold TH for condition 3, as with conditions 1 and 2, prepare many sample images taken under the same brightness conditions, and determine the judgment threshold TH to be used as such that the target object and non-target objects are distinguished with a probability greater than or equal to a predetermined probability in those sample images. Of course, as with conditions 1 and 2, you may also use an AI model to determine the judgment threshold TH.

[0068] Under the conditions 1 to 3 described above, the relationship between the judgment threshold TH and the combination of objects detected as target objects is summarized in the table in Figure 3D.

[0069] As shown in Figure 3D, in the cases shown in Figures 3A to 3C, when the judgment threshold TH is set to 0.9, the target object M2 is detected under condition 1, but nothing is detected under conditions 2 and 3. In other words, under condition 1, target objects M1 and M3 are not detected, and under conditions 2 and 3, target objects M1, M2, and M3 are not detected. On the other hand, under all conditions, the non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are not mistakenly detected as target objects.

[0070] When the detection threshold TH is lowered to 0.8, all target objects M1, M2, and M3 are detected under condition 1, target object M2 is detected under condition 2, but nothing is detected under condition 3. In other words, there are no detection errors under condition 1, but target objects M1 and M3 are missed under condition 2, and target objects M1, M2, and M3 are missed under condition 3. On the other hand, under all conditions, the non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are never mistakenly detected as target objects.

[0071] When the detection threshold TH is lowered to 0.7, all target objects M1, M2, and M3 are detected under condition 1, target objects M1 and M2 are detected under condition 2, but nothing is detected under condition 3. The same detection results are obtained even when the detection threshold TH is lowered to 0.6 or 0.5. In other words, no detection errors occur under condition 1, but target object M3 is not detected under condition 2, and target objects M1, M2, and M3 are not detected under condition 3. On the other hand, under all conditions, the non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are not mistakenly detected as target objects.

[0072] Lowering the detection threshold TH to 0.4 results in the detection of all target objects M1, M2, and M3 under conditions 1 and 2, while under condition 3, only target objects M1 and M2 are detected. In other words, no detection errors occur under conditions 1 and 2, but target object M3 is missed under condition 3. On the other hand, under all conditions, the non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are never mistakenly detected as target objects.

[0073] Lowering the judgment threshold TH to 0.3 results in the detection of non-target object OBJ2 in addition to target objects M1, M2, and M3 under conditions 1 and 2. Under condition 3, non-target object OBJ2 is detected in addition to target objects M1 and M2. In other words, under conditions 1 and 2, no detection errors occur, but the non-target object OBJ2 is incorrectly detected as a target object. Under condition 3, target object M3 is not detected, and the non-target object OBJ2 is incorrectly detected as a target object.

[0074] Lowering the detection threshold TH to 0.25 results in the following: Under condition 1, in addition to the target objects M1, M2, and M3, the non-target objects OBJ2 and OBJ4 are detected. Under condition 2, in addition to the target objects M1, M2, and M3, the non-target objects OBJ1 and OBJ2 are detected. Under condition 3, the target objects M1, M2, and M3 are detected. In other words, under condition 1, there are no detection omissions, but the non-target objects OBJ2 and OBJ4 are incorrectly detected as target objects. Under condition 2, there are no detection omissions, but the non-target objects OBJ1 and OBJ2 are incorrectly detected as target objects. Under condition 3, there are no detection omissions, and the non-target objects OBJ1, OBJ2, OBJ3, and OBJ4 are not incorrectly detected as target objects.

[0075] Lowering the judgment threshold TH to 0.2 results in the following detections: Under condition 1, in addition to the target objects M1, M2, and M3, the non-target objects OBJ2 and OBJ4 are detected. Under condition 2, in addition to the target objects M1, M2, and M3, the non-target objects OBJ1 and OBJ2 are detected. Under condition 3, in addition to the target objects M1, M2, and M3, the non-target object OBJ2 is detected. In other words, no detection failures occur in any of conditions 1 to 3, but one of the non-target objects OBJ1, OBJ2, OBJ3, or OBJ4 is mistakenly detected as a target object.

[0076] Referring to the example in Figure 3D, it can be seen that if the judgment threshold TH is fixed to a certain value, either detection failures will occur under one of the conditions, or non-target objects will be mistakenly detected as target objects. However, in the monitoring system 10 according to this embodiment, the judgment threshold TH is changed according to changes in brightness conditions. For example, if the judgment threshold TH is set to 0.7 under condition 1, 0.4 under condition 2, and 0.25 under condition 3, then in the example shown in Figure 3D, no detection failures will occur under any of the conditions, and non-target objects will not be mistakenly detected as target objects. Of course, the judgment threshold TH for each condition is not limited to this example, and can be set to a value appropriate to the embodiment during implementation.

[0077] As explained above, by changing the detection threshold TH in accordance with changes in brightness, it becomes possible to obtain better detection results. If the number of instances inaccurate detection of non-target objects as target objects is reduced, the number of times when a monitor who has received a notification of object detection from the monitoring system goes to the site to check but finds no target object and their efforts are wasted will be reduced, thereby reducing the burden on the monitor. Furthermore, if the number of undetected target objects is reduced, the chances of accidents and troubles will decrease, contributing to improved safety. As a result, by applying the technology according to this embodiment, it becomes possible to improve the efficiency of monitoring operations without sacrificing safety.

[0078] (How to change the judgment threshold) Up to this point, we have given three brightness conditions (Condition 1 to Condition 3) as specific examples and explained in detail how to change the judgment threshold TH according to the changes in these conditions. However, the method for changing the judgment threshold TH according to this embodiment is not limited to the above examples. Below, we will further explain the method for changing the judgment threshold TH according to this embodiment with reference to Figures 4A to 4D.

[0079] First, refer to Figure 4A. Figure 4A shows a graph illustrating the relationship between the luminance value BR and the judgment threshold TH. Here, for the sake of simplicity, the relationship between the luminance value BR and the judgment threshold TH is represented graphically, but the method of representation is not limited. For example, if information showing the relationship between the luminance value BR and the judgment threshold TH is stored as judgment threshold information 123a in the storage unit 123 of the image processing device 12, it may be represented in table format, mathematical formula, or other format.

[0080] In the graph in Figure 4A, the horizontal axis represents the luminance value BR, and the vertical axis represents the judgment threshold TH. In this example, judgment thresholds TH are set for three ranges of luminance value BR corresponding to conditions 1 to 3 above. For the range of luminance value BR corresponding to condition 1 (0 ≤ BR < 0.4), the judgment threshold TH is set to 0.25; for the range of luminance value BR corresponding to condition 2 (0.4 ≤ BR < 0.8), the judgment threshold TH is set to 0.4; and for the range of luminance value BR corresponding to condition 3 (0.8 ≤ BR < 1.0), the judgment threshold TH is set to 0.8.

[0081] In this way, the entire range of the brightness value BR (0 to 1) is divided into three ranges, and a judgment threshold TH is set for each range. This setting is stored as judgment threshold information 123a in the storage unit 123 of the image processing device 12. The threshold adjustment unit 124 refers to the judgment threshold information 123a and identifies the corresponding judgment threshold TH based on the brightness value actually obtained from the captured image. Then, the judgment unit 122 compares the judgment threshold TH identified by the threshold adjustment unit 124 with the score value to obtain the detection result of the target object.

[0082] In the example of FIG. 4A, the entire range of the luminance value BR is divided into three, but it may be divided into four or more ranges, or it may be divided into two ranges. In the example of FIG. 4B, the entire range of the luminance value BR is divided into five ranges. The determination threshold TH corresponding to each range may be determined as follows. For example, prepare a number of sample images taken under brightness conditions corresponding to the reference value (such as the median value, lower limit value, or upper limit value) of the target luminance value range, and in these sample images, the determination threshold TH such that the target object and the object outside the target can be distinguished with a probability equal to or higher than a preset probability may be determined as the determination threshold TH corresponding to that range. Of course, an AI model may be used to determine the determination threshold TH corresponding to each range.

[0083] As another example, as shown in FIGS. 4C and 4D, in at least a part of the range of the luminance value BR, the determination threshold TH may be changed continuously according to the change in the luminance value BR.

[0084] In the example of FIG. 4C, in the range from the first luminance value BR1 corresponding to the first brightness to the second luminance value BR2 (BR1 < BR2) corresponding to the second brightness, the determination threshold TH increases in proportion to the increase in the luminance value BR. Also, in the example of FIG. 4D, in the range from the first luminance value BR1 corresponding to the first brightness to the second luminance value BR2 (BR1 < BR2) corresponding to the second brightness, the determination threshold TH increases smoothly according to the increase in the luminance value BR. For example, it is possible to set the determination threshold TH to increase along a preset linear or non-linear function according to the increase in the luminance value BR.

[0085] In the cases of Figures 4C and 4D, the method for determining the judgment threshold TH corresponding to the first luminance value BR1 and the second luminance value BR2 is the same as the method described for Figures 4A and 4B. Furthermore, the judgment threshold TH between the first luminance value BR1 and the second luminance value BR2 can be calculated based on a preset linear or nonlinear function. Therefore, in the cases of Figures 4C and 4D as well, the threshold adjustment unit 124 can refer to the judgment threshold information 123a and identify the corresponding judgment threshold TH based on the luminance values ​​actually obtained from the captured image. The determination unit 122 can then compare the judgment threshold TH identified by the threshold adjustment unit 124 with the score value to obtain the detection result of the target object.

[0086] (Regarding the setting of prioritizing accuracy / prioritizing detection) Next, with reference to Figures 5A to 5C, a method for further adjusting the judgment threshold TH according to this embodiment will be described. Here, for the sake of brevity, the explanation will be based on the example in Figure 4A corresponding to conditions 1 to 3 described above, but the adjustment method described below can also be applied to the judgment threshold TH settings and their variations shown in Figures 4B to 4D.

[0087] (Example prioritizing accuracy) First, refer to Figure 5A. Figure 5A corresponds to the graph in Figure 4A. However, in the graph of Figure 5A, the judgment threshold TH is set to be larger by the adjustment amount dTH in each range of the brightness value BR compared to the graph of Figure 4A. As can be seen from the detection results exemplified in Figure 3D, the higher the judgment threshold TH, the lower the probability that an object that is not the target object will be mistakenly detected as the target object. Therefore, if you prioritize preventing an object that is not the target object from being mistakenly detected as the target object (hereinafter referred to as "prioritizing accuracy"), you should adjust the judgment threshold TH to be slightly higher, as shown in Figure 5A.

[0088] The adjustment amount dTH can be set arbitrarily. For example, the adjustment amount dTH may be set to a value of 0.1 or less (e.g., 0.05). Also, although the same adjustment amount dTH is used for each range of the luminance value BR in the example in Figure 5A, the adjustment amount may be set differently for each range of the luminance value BR. For example, in the case of prioritizing accuracy (when the adjustment amount is a positive value), the adjustment amount may be set relatively large under bright conditions, relatively small under dark conditions, and 0 under slightly dark conditions. The adjustment values ​​for each range may also be set by the user from the monitoring terminal 13.

[0089] (Example focusing on detection) Next, refer to Figure 5B. Figure 5B corresponds to the graph in Figure 4A. However, in the graph of Figure 5B, the judgment threshold TH is set smaller by the adjustment amount dTH in each range of the brightness value BR compared to the graph of Figure 4A. As can be seen from the detection results exemplified in Figure 3D, the lower the judgment threshold TH, the lower the probability of detection failure. Therefore, if the priority is to reduce the risk of detection failure (hereinafter referred to as detection priority), the judgment threshold TH should be adjusted to be slightly lower, as shown in Figure 5B.

[0090] In this case as well, the adjustment amount dTH can be set arbitrarily. For example, the adjustment amount dTH may be set to a value of 0.1 or less (for example, 0.05). Also, although the same adjustment amount dTH is used for each range of the luminance value BR in the example in Figure 5B, the adjustment amount may be set differently for each range of the luminance value BR. For example, in the case of prioritizing detection (when the adjustment amount is a negative value), the adjustment amount may be set to a relatively small value under bright conditions, a relatively large value under dark conditions, and 0 under slightly dark conditions. The adjustment values ​​for each range may also be set by the user from the monitoring terminal 13.

[0091] (Emphasis on accuracy & emphasis on detection) Next, refer to Figure 5C. Figure 5C corresponds to the graph in Figure 4A. However, compared to the graph in Figure 4A, the graph in Figure 5C has the adjustment amount dTH set larger in the range of luminance values ​​BR corresponding to condition 1 (bright environment), the judgment threshold TH is not adjusted in the range of luminance values ​​BR corresponding to condition 2 (slightly dark environment), and the adjustment amount dTH is set smaller in the range of luminance values ​​BR corresponding to condition 3 (dark environment).

[0092] As already explained, in bright environments, the characteristics of objects are often clearer, and the score value of the target object tends to be higher, making detection failures less likely. Therefore, by setting the judgment threshold TH slightly higher within the range of brightness values ​​BR that corresponds to bright conditions, it is possible to reduce the probability of non-target objects being mistakenly detected as target objects without increasing the risk of detection failures.

[0093] On the other hand, in dark environments, it becomes difficult for observers to notice the presence of objects in the video, making support from the surveillance system crucial. If detection failures occur in the surveillance system, the risk of accidents and problems increases as observers may also fail to notice the presence of the target object in the video. Therefore, in dark environments where it is difficult to see objects in the video, it is sometimes desirable to prioritize avoiding detection failures, even if it slightly increases the probability of non-target objects being mistakenly detected as target objects. In such cases, as shown in Figure 5C, the risk of detection failures can be reduced by setting the judgment threshold TH slightly lower within the range of luminance values ​​BR corresponding to dark conditions.

[0094] In this case as well, the adjustment amount dTH can be set arbitrarily. For example, the adjustment amount dTH may be set to a value of 0.1 or less (for example, 0.05). In addition, the adjustment values ​​for each range may be set by the user from the monitoring terminal 13. Furthermore, in order to simplify the setting operation, modes such as accuracy-focused mode (corresponding to Figure 5A), detection-focused mode (corresponding to Figure 5B), and balanced mode (corresponding to Figure 5C) may be set, and the user may be able to select the operating mode.

[0095] Furthermore, the method for adjusting the judgment threshold TH is not limited to the examples in Figures 5A to 5C. For example, the judgment threshold TH may be adjusted only within the range of luminance values ​​BR corresponding to dark conditions, or only within the range of luminance values ​​BR corresponding to bright conditions.

[0096] Furthermore, the above-described method for adjusting the judgment threshold TH can also be applied to the methods for changing the judgment threshold TH illustrated in Figures 4B to 4D. For example, in the example of Figure 4B, the judgment threshold TH for each range can be adjusted by an adjustment amount dTH. Also, in the examples of Figures 4C and 4D, the judgment threshold TH in the range of first luminance value BR1 or less and second luminance value BR2 or more can be adjusted by an adjustment amount dTH, and the judgment threshold TH in between can be adjusted based on the adjusted judgment threshold TH for the first luminance value BR1 and the second luminance value BR2. Such modified forms are also included within the technical scope of this embodiment.

[0097] (Processing flow) Next, the processing flow performed by the monitoring system according to this embodiment will be described with reference to Figure 6. Figure 6 is a flowchart illustrating the processing flow performed by the monitoring system according to this embodiment.

[0098] (S101) The imaging device 11 continuously or intermittently photographs the monitoring area and outputs the captured images. The image processing device 12 acquires the captured images output from the imaging device 11. There are no particular limitations on the interval between image captures or the timing of image acquisition. For example, the captured image may be a single video frame or a single time-lapse image.

[0099] (S102) The image processing device 12 acquires brightness information from the captured image. For example, the image processing device 12 acquires the average brightness value of the captured image as brightness information. The average brightness value of the captured image corresponds to the brightness within the monitoring area. Note that other statistical values ​​related to the brightness value of the captured image (such as the median) may also be used as brightness information. Furthermore, statistical values ​​of the brightness value for the entire captured image may be used as brightness information, or statistical values ​​of the brightness value for a part of the captured image may be used as brightness information.

[0100] (S103) The image processing device 12 determines whether the brightness conditions have changed based on the brightness information. For example, the image processing device 12 stores brightness information for each captured image, compares the brightness information corresponding to the currently acquired captured image with the brightness information corresponding to the previously acquired captured image, and determines whether the brightness conditions have changed based on the comparison result. If it is determined that the brightness conditions have changed, the process proceeds to S104. On the other hand, if it is determined that the brightness conditions have not changed, the process proceeds to S105.

[0101] For example, in cases where the judgment threshold TH is set for each range of luminance values, as shown in Figures 4A, 4B, and 5A to 5C, the image processing device 12 determines that the brightness condition has changed when the luminance value changes across the boundary of the luminance value range in which the judgment threshold TH changes (0.4 and 0.8 in the example of Figure 4A).

[0102] For example, in the case of Figure 4A, if the brightness value corresponding to the previously acquired image is 0.82 and the brightness value corresponding to the currently acquired image is 0.78, the image processing device 12 determines that the brightness conditions have changed. Also, if the average brightness value corresponding to the previously acquired image is 0.7 and the average brightness value corresponding to the currently acquired image is 0.5, the image processing device 12 determines that the brightness conditions have not changed.

[0103] As another example, in the case shown in Figures 4C and 4D, where the judgment threshold TH is set to change continuously in response to changes in the brightness value BR, the image processing device 12 determines that the brightness condition has changed when the brightness value changes to a preset change threshold (for example, 0.1). Alternatively, instead of setting a change threshold, the process may be modified to skip S103 and proceed to S104.

[0104] (S104) The image processing device 12 adjusts the judgment threshold TH so that the value decreases as the brightness indicated by the acquired brightness information decreases. For example, as shown in Figures 4A to 5C, the image processing device 12 identifies the judgment threshold TH corresponding to the brightness information of the captured image based on the pre-set judgment threshold information 123a, and changes the judgment threshold TH that is compared with the score value of each object in the captured image from the current judgment threshold to the identified judgment threshold.

[0105] (S105, S106) The image processing device 12 inputs the captured image to the object detector 121a, obtains the score value output from the object detector 121a, compares the obtained score value with the judgment threshold TH, and determines whether the score value is equal to or greater than the judgment threshold TH.

[0106] If the captured image contains multiple objects, the image processing device 12 compares the score value corresponding to each object with the judgment threshold TH. If the score value corresponding to an object is equal to or greater than the judgment threshold TH, the image processing device 12 determines that the object is the target object. On the other hand, if the score value corresponding to an object is less than the judgment threshold TH, the image processing device 12 determines that the object is not the target object.

[0107] After performing a judgment on each score value, the image processing device 12 outputs the detection result of the target object. For example, the detection result may include the position of each target object in the captured image. The detection result may also include a score value associated with the position of each target object. Furthermore, the detection result may include the positions of objects other than the target object in the captured image, and may also include a score value corresponding to those objects.

[0108] The detection results output from the image processing device 12 are sent to the monitoring terminal 13. Once the processing in S106 is completed, the process moves to S101 and proceeds to processing the next captured image. When the conditions for terminating the object detection process are met (for example, when the user performs a termination operation), the series of processes shown in Figure 6 are terminated.

[0109] The processing flow shown in Figure 6 is just one example; the execution order of some processes may be changed, or new steps may be added as needed. For example, the acquisition of the score value in S105 may be performed before any of steps S102 to S104 is performed, or in parallel with any of steps S102 to S104. Furthermore, the detection results may be output for each captured image, or the detection results for two or more captured images may be output together. Such variations are also included within the technical scope of this embodiment.

[0110] (Variation: Adjustment of threshold change timing) Incidentally, depending on the environment of the monitoring area, light rays from vehicle headlights may be temporarily reflected, or external light may be reflected in the captured image due to the on / off / flashing of signal lights or level crossing warning lights. As a result, brightness conditions may change significantly in a short period of time. Also, even during the day, on windy days, brightness conditions may change significantly in a short period of time due to the influence of fast-moving clouds. In addition to the examples given here, brightness conditions may change in a short period of time due to reflections of artificial light or changes in the natural environment. In the example in Figure 6 above, the judgment threshold TH is changed when it is determined that the brightness conditions have changed. Therefore, depending on the environment, the judgment threshold TH may be changed repeatedly in a short period of time.

[0111] To avoid repeated changes to the judgment threshold TH in a short period of time, this section describes a method of modifying the brightness condition judgment process and changing the judgment threshold TH only after the brightness conditions have stabilized, with reference to Figure 7.

[0112] In the upper part of FIG. 7, a graph showing the temporal change of the luminance value BR is described, and in the lower part, a graph showing the temporal change of the luminance difference dBR is described. The luminance difference dBR(t) corresponding to the time point t is the difference (BR(t) - BR(t - 1)) between the luminance value BR(t) of the captured image taken at the time point t and the luminance value BR(t - 1) of the captured image taken at the previous time point (t - 1), and represents the amount of change in brightness. When the luminance difference dBR is within the range of the allowable change amount (-δ < dBR < +δ), it is considered that the change in brightness is sufficiently small. δ is the allowable value of the preset brightness change amount.

[0113] The change in brightness accompanying the daily sunlight change is gradual. In this case, the change amount dBR of the luminance value BR between the captured images is small, and the change of the determination threshold value TH is not repeated frequently in a short period. On the other hand, the change in brightness due to light such as vehicle lights and signal lights is often temporary and sudden. Also, there may be a temporary and sudden change in brightness due to natural phenomena such as fast cloud flow. In such a case, the change amount dBR of the luminance value BR between the captured images is large, and the change of the determination threshold value TH is not infrequently repeated in a short period. Therefore, when it is determined that the brightness condition has changed, instead of immediately changing the determination threshold value TH, a method is proposed in which the determination threshold value TH is changed when the luminance difference dBR is within the range of the allowable change amount (-δ < dBR < +δ) for a certain period.

[0114] In the example of FIG. 7, the luminance value BR becomes 0.4 or more at the timings of the symbols a, b, and c. In the case of FIG. 4A, it is determined that the brightness condition has changed at the timing when the luminance value BR becomes 0.4 or more. On the other hand, in the method proposed here, since the luminance difference dBR exceeds the change amount δ at the timing of the symbol a, the threshold adjustment unit 124 of the image processing apparatus 12 does not change the determination threshold value TH. Also, since the luminance difference dBR exceeds the change amount δ at the timing of the symbol b, the threshold adjustment unit 124 of the image processing apparatus 12 does not change the determination threshold value TH.

[0115] On the other hand, at the timing of code c, the luminance difference dBR does not exceed the change amount δ. In this case as well, the threshold adjustment unit 124 of the image processing device 12 does not immediately change the judgment threshold TH, but changes the judgment threshold TH when the period during which the luminance difference dBR does not exceed the change amount δ reaches a predetermined time length L. In the example in Figure 7, the period during which the luminance difference dBR does not exceed the change amount δ has reached a predetermined time length L, so the judgment threshold TH is changed at this timing.

[0116] As described above, even in situations where a change in brightness conditions is detected, the detection threshold TH is not changed when the luminance difference dBR is large, and even when the luminance difference dBR is small, the detection threshold TH is not changed immediately. Instead, it is changed only after confirming that the luminance value BR is stable. This makes it less likely for the detection threshold TH to be changed repeatedly in a short period of time. As a result, frequent fluctuations in the detection threshold TH are suppressed even when there are temporary changes in brightness due to ambient light.

[0117] Furthermore, the above-mentioned change amount δ and time duration L can be set arbitrarily. Therefore, it is possible to easily respond to short-term fluctuations in brightness conditions due to reflections of artificial light, as well as short-term fluctuations in brightness conditions caused by natural phenomena. For example, the above-mentioned change amount δ and time duration L can be set to values ​​suitable for the implementation environment by using a set of sample images taken in the environment in which the monitoring system 10 is applied or in an environment similar thereto. In addition, the user or operator of the monitoring system may be allowed to arbitrarily adjust at least one of the change amount δ and time duration L.

[0118] (Example hardware configuration) Next, an example of the hardware configuration of the image processing device 12 and the monitoring terminal 13 will be described with reference to Figure 8. Figure 8 shows an example of the hardware configuration of a computer 20 that can implement at least some of the functions of the image processing device 12 and the monitoring terminal 13. For example, at least some of the functions of the image processing device 12 and the monitoring terminal 13 can be implemented by controlling the hardware of the computer 20 using a computer program.

[0119] As shown in Figure 8, the computer 20 includes a processor 21, memory 22, connection interface 23, communication interface 24, and display device 26, which are connected to each other via a bus 25. Depending on the embodiment, some of the elements shown in Figure 8 may be omitted, and elements not shown may be added. For example, the display device 26 may be omitted when the computer 20 is used as an image processing device 12.

[0120] The processor 21 may be a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a GPU (Graphic Processing Unit), or a combination of some or all of these. Of course, the specific examples listed here do not limit the scope of application of this embodiment, and the processor 21 may be any other processing device currently available or any processing device that may be available in the future.

[0121] Memory 22 may be ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination of some or all of these. Of course, the specific examples listed here do not limit the scope of application of this embodiment, and memory 22 may be other currently available storage devices or any storage device that may become available in the future.

[0122] The connection IF23 is an interface such as a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface). Input / output devices such as a keyboard, mouse, touch panel, touchpad, microphone, speaker, or headset may be connected to the connection IF23. Removable storage media such as magnetic recording media, optical discs, magneto-optical discs, or semiconductor memory may also be connected to the connection IF23. Of course, the specific examples listed here do not limit the scope of application of this embodiment, and the connection IF23 may be any other type of connection interface currently available, or any connection interface that may become available in the future. Furthermore, the devices connected to the connection IF23 are not limited to the examples above, and devices not listed here may also be connected to the connection IF23.

[0123] The communication IF24 is an interface for connecting to a communication network such as a LAN (Local Area Network), WAN (Wide Area Network), optical communication network, or mobile communication network. The display device 26 is an LCD (Liquid Crystal Display) or OLED (Organic Electro Luminescence Diode) display, etc. Of course, the specific examples listed here do not limit the scope of application of this embodiment, and the communication IF24 may be an interface to other communication standards currently available, or to any communication standard that may become available in the future. Similarly, the display device 26 may be any other type of display device currently available, or to any display device that may become available in the future.

[0124] Of the image processing device 12, the functions of the score value acquisition unit 121, the determination unit 122, and the threshold adjustment unit 124 can be implemented mainly using the processor 21. The function of the storage unit 123 can be implemented using the memory 22. In addition, of the monitoring terminal 13, the function of the output unit 131 can be implemented using the display device 26 or an output device connected to the connection IF 23. The function of the control unit 132 can be implemented mainly using the processor 21. The function of the input unit 133 can be implemented using an input device connected to the connection IF 23. Of course, other elements may be used in combination to implement each individual function.

[0125] Preferred embodiments of the present invention have been described above with reference to the attached drawings, but the present invention is not limited to these examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these also naturally fall within the technical scope of the present invention. [Explanation of Symbols]

[0126] 10, 10a Surveillance camera system 11 Imaging device 12 Image Processing Devices 13 Surveillance terminals 14, 121a Object detector 121 Score Value Acquisition Section 122 Judgment section 123 Storage section 123a Judgment threshold 124 Threshold adjustment section 131 Output section 132 Control Unit 133 Input section P1, P2, P3 images T1, T2, T3 Tables

Claims

1. A surveillance system having a function to detect a target object included in a captured image, using an object detector that detects a target object included in an input image and outputs a score value indicating the likelihood that the object detected from the input image is the target object, A threshold adjustment unit acquires brightness information from the captured image and adjusts the judgment threshold, which is compared with the score value, according to the brightness, such that the value decreases as the brightness decreases. A determination unit determines that the object detected in the captured image is the target object if the score value output from the object detector in response to the input of the captured image is greater than the adjusted determination threshold. A monitoring system, including a surveillance system.

2. The determination unit notifies the monitor of information about the object that has been determined to be the target object. The monitoring system according to claim 1.

3. The imaging device further includes an imaging device that acquires the aforementioned captured image. The monitoring system according to claim 1.

4. The object detector is an object detection algorithm that has learned the characteristics of the target object, or a device equipped with the object detection algorithm. The monitoring system according to claim 1.

5. The aforementioned captured image is one image included in a series of images captured continuously or intermittently. The threshold adjustment unit determines the amount of change in brightness between images for the series of images, and maintains the determination threshold without changing it until the period during which the amount of change in brightness is less than a preset amount reaches a predetermined time length. The monitoring system according to claim 1.

6. The luminance information includes the average luminance value of the captured image. The monitoring system according to claim 1.

7. The monitoring system further includes a storage unit that stores a first determination threshold corresponding to a preset first brightness level. The first determination threshold is set to a value in which the target object is detected and non-target objects are not detected, based on the set of score values ​​output from the object detector using a set of sample images previously captured under the first brightness and the first determination threshold. The monitoring system according to claim 1.

8. The memory unit further stores a second determination threshold corresponding to a second brightness that is brighter than the first brightness. The second determination threshold is set to a value in which the target object is detected and no non-target objects are detected, based on the set of score values ​​output from the object detector using a set of sample images previously captured under the second brightness and the second determination threshold. The monitoring system according to claim 7.

9. The threshold adjustment unit changes the judgment threshold used for comparison with the score value output from the object detector in response to the input of the captured image, based on the first judgment threshold and the second judgment threshold. The monitoring system according to claim 8.

10. An object detection method for detecting a target object in a captured image, using an object detector that detects a target object in an input image and outputs a score value indicating the likelihood that the object detected from the input image is the target object, wherein the object in the captured image is detected by an object detector, The steps include: obtaining brightness information from the captured image, and changing the judgment threshold, which is compared with the score value, according to the brightness, such that the value decreases as the brightness decreases based on the brightness information; The step of determining that the object detected in the captured image is the target object if the score value output from the object detector in response to the input of the captured image is greater than the adjusted determination threshold. An object detection method, including the following.

Images