Information processing systems, information processing methods, and programs

Abstract

The objective is to provide a mechanism for appropriately updating reference images. [Solution] A threshold for updating information related to a reference image is received, and information related to the image to be judged is acquired sequentially. Based on the information related to the image to be judged and the threshold, control is made whether to update the information related to the reference image using the information related to the image to be judged, or to update the information related to the reference image without using the information related to the image to be judged.

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and more particularly to a technique suitable for use in detecting abnormalities.

Background Art

[0002] Conventionally, there has been an information processing system that notifies an abnormality based on a comparison between a reference image and an analysis target image.

[0003] Patent Document 1 discloses a technique for detecting the occurrence of smoke based on a comparison process between a current image captured by a surveillance camera and a reference image previously stored in a storage unit.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, it is described that the reference image is updated after the determination of the occurrence of a luminance change.

[0006] However, when updating the reference image, there may be cases where an image other than the image expected by the user is included. For example, in the case of noise or when some object crosses. Although Patent Document 1 discloses that the reference image is updated after the determination of the occurrence of a luminance change, it does not disclose the case where an image other than the image expected by the user is included when updating the reference image.

[0007] Therefore, an object of the present invention is to provide a mechanism for appropriately updating a reference image. [[ID=4G]]

Means for Solving the Problems

[0008] To achieve the above objectives, the present invention A receiving means for receiving a threshold for updating information related to a reference image, An acquisition means for sequentially acquiring information related to the image to be judged, Update control means that controls whether to update the information relating to the reference image using the information relating to the image to be determined, or to update the information relating to the reference image without using the information relating to the image to be determined, based on the information relating to the image to be determined and the threshold, It is characterized by having the following features. [Effects of the Invention]

[0009] According to the present invention, a mechanism for appropriately updating a reference image can be provided. [Brief explanation of the drawing]

[0010] [Figure 1] This is a system configuration diagram of information processing system 100. [Figure 2] This is a hardware block diagram of the information processing device 104. [Figure 3] This is an example of a block diagram showing the software configuration. [Figure 4] (a) A flowchart of the configuration process and (b) an example of a flowchart of the anomaly detection parameter setting process for the monitoring task. [Figure 5] This is an example of the initial display of the settings screen. [Figure 6] This is an example of an overview of the settings screen. [Figure 7] This is an example of the settings screen (input tab). [Figure 8] This is an example of the settings screen (input tab). [Figure 9] This is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11] This is an example of the settings screen (monitoring processes and monitoring tasks). [Figure 12]This is an example of the display of the settings screen (input tab). [Figure 13] This is an example of the display of the settings screen (input tab). [Figure 14] This is an example of the display of the settings screen (notification tab). [Figure 15] This is an example of the display of the settings screen (notification tab). [Figure 16] This is an example of the display of the settings screen (monitoring task). [Figure 17] This is an example of the display of the adjustment screen for analysis settings. [Figure 18] This is an example of the display of the grid settings screen (grid division). [Figure 19] This is an example of the display of the grid settings screen (grid correction). [Figure 20] This is an example of the display of the analysis settings screen. [Figure 21] This is an example of the flowchart of the anomaly detection process. [Figure 22] This is an example of the display of task registration information. [Figure 23] This is an example of the flowchart of the image difference (anomaly) detection process. [Figure 24] This is an example of the flowchart of each histogram calculation process. [Figure 25] This is an example of the display of grid correction. [Figure 26] This is an example of the display of the notification during grid correction. [Figure 27] This is an example of the flowchart of the anomaly notification process. [Figure 28] This is an example of the display of the settings screen (anomaly notification). [Figure 29] This is an example of the display of the anomaly notification dashboard screen. [Figure 30] This is an example of a diagram showing an overview of the reference image update in the second embodiment.<​​​​​​​This is an example of a flowchart for the reference image update process in the second embodiment. [Figure 34] This is an example of a diagram illustrating the problems in the third embodiment. [Figure 35] This is an example of a flowchart for image difference (anomaly) detection processing in the third embodiment. [Figure 36] This is an example of a flowchart for the reference image update process in the third embodiment. [Figure 37] This is an example of a flowchart for the reference image update process that takes thresholds into account in the third embodiment. [Figure 38] This is an example of a setting screen for the reference image update process that takes thresholds into account in the third embodiment. [Figure 39] This figure illustrates an example of updating a reference image while considering the threshold in the third embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0012] Figure 1 is a system configuration diagram showing an example of the configuration of the information processing system 100 of the present invention.

[0013] Figure 1 shows a configuration in which the camera 102 and the information processing device 104 are connected via an image transfer cable (USB, Ethernet, CameraLink, etc.). However, a configuration of an information processing device equipped with a camera is also acceptable, even without this specific setup.

[0014] The information processing device 104 acquires an image obtained by the camera 102 capturing the target object via an image transfer cable, and performs processing related to the detection of abnormalities in the image.

[0015] Furthermore, the information processing system 100 of this embodiment may also have a configuration in which the camera 102 with the above-described configuration does not exist.

[0016] In that case, processing related to the detection of abnormalities in images is performed on images acquired from other terminals on the network or on images stored in the memory (RAM) of the information processing device 104.

[0017] The following describes the hardware configuration of an information processing device applicable to the information processing device 104 shown in Figure 1, using Figure 2 as an example.

[0018] Figure 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in Figure 1.

[0019] In Figure 2, 201 is the CPU, which comprehensively controls each device and controller connected to the system bus 204. The ROM 202 or external memory 212 stores the BIOS (Basic Input / Output System), which is the control program for the CPU 201, the operating system program (hereinafter referred to as the OS), and various programs necessary to implement the functions executed by each PC, as described later.

[0020] 203 is RAM, which functions as the main memory, work area, etc., of the CPU 201. The CPU 201 loads the necessary programs, etc., from ROM 202 or external memory 212 into RAM 203, and then executes the loaded programs to perform various operations.

[0021] 205 is an input controller that controls input from pointing devices such as the keyboard (KB) 210 and a mouse (not shown).

[0022] 206 is a video controller that controls the display on indicators such as display 211.

[0023] 207 is a memory controller that controls access to external memory 212, such as external storage devices (hard disks (HDs)) that store various types of data, flexible disks (FDs), or CompactFlash® memory connected to a PCMCIA card slot via an adapter.

[0024] 208 is a communication interface controller that controls the reception of image data from the external PC 213 via the network (TCP / IP). 209 is an image I / F controller that controls the reception of image data from camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0025] The various programs described later for realizing the present invention are stored in RAM 203 and executed by CPU 201.

[0026] Furthermore, the image data used when executing the above program is stored in ROM202, external memory212, external PC213, and camera102 depending on the application, and is stored in RAM203 via various controllers when the program is executed.

[0027] Figure 3 is an example of a block diagram showing the software configuration of an embodiment of the present invention.

[0028] The information processing device 104 includes the following functional units.

[0029] The acquisition unit 301 is a functional unit that acquires images captured at the same angle of view.

[0030] The reception unit 302 is a functional unit that receives specifications for the area to be included in the image and the method of displaying that area.

[0031] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on changes between corresponding regions of multiple images acquired by the acquisition unit 301.

[0032] The output unit 304 is a functional unit that outputs the determination result from the determination unit 303.

[0033] The output unit 304 is a functional unit that displays the area specified by the reception unit 302 using the display method specified by the reception unit 302, and controls whether or not to output a judgment result based on the area specified by the reception unit 302.

[0034] The area change unit 305 is a functional unit that changes the area specified by the reception unit 302 based on the area specified by the reception unit 302 and the determination result of that area.

[0035] The splitting unit 306 is a functional unit that splits the image acquired by the acquisition unit 301.

[0036] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on changes between corresponding regions that have been divided by the division means from among multiple images acquired by the acquisition unit 301.

[0037] The division unit 306 is a functional unit that divides an image using two or more division methods.

[0038] The output unit 304 is a functional unit that outputs a determination result based on the region divided by the first division method when the determination unit 303 determines that there has been a change in the region divided by the second division method.

[0039] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on the changes between corresponding regions of multiple images acquired by the acquisition means and multiple types of thresholds.

[0040] The output unit 304 is a functional unit that identifies and outputs which of the multiple types of thresholds was used to determine that a change had occurred.

[0041] The reception unit 302 is a functional unit that accepts specifications for the method of identification and output for each type of threshold.

[0042] The reception unit 302 is a functional unit that controls the system to prevent the same output method specification from being accepted for different threshold types.

[0043] The determination unit 303 is a functional unit that determines whether there is a change that should be notified based on a comparison between the reference image and the image to be analyzed.

[0044] The acquisition unit 301 is a functional unit that notifies the determination result that there has been a change that should be notified.

[0045] The notification unit 307 is a functional unit that acquires a predetermined value related to the suppression of notifications.

[0046] The notification unit 307 is a functional unit that suppresses the notification of a determination result indicating that a change requiring notification has occurred, based on a predetermined value.

[0047] The notification unit 307 is a functional unit that suppresses the notification of a determination result indicating that a change requiring notification has occurred, based on at least one of the number of changes to be notified and a predetermined value.

[0048] The notification unit 307 is a functional unit that suppresses the notification of the determination result that there has been a change that should be notified when the number of changes that should be notified is less than or equal to a predetermined value.

[0049] This concludes the explanation of Figure 3.

[0050] The setup process shown in Figure 4(a) will be explained below.

[0051] In S401, the information processing device 104 determines whether a configuration file for the operating process exists in the external memory 212. If it is not registered, the process proceeds to S402; if it is registered, the process proceeds to S403.

[0052] In S402, the information processing device 104 displays the settings screen 510 (Figure 5), and when it receives a press of the add monitoring process button 511 from the user, it registers the operation process in RAM 203 and displays the settings screen 520.

[0053] In S403, when the information processing device 104 receives a press of the button 521 for selecting the operation process to be edited on the setting screen 502, it displays the setting screen 610 (Figure 6).

[0054] The setting acceptance process performed on setting screen 610 in S403 will be explained below using Figures 6 to 10.

[0055] When the information processing device 104 receives input from the user for the name 611 and description 612 of the operating process, it edits the operating process information in the RAM 203 and displays the settings screen 620.

[0056] When the information processing device 104 receives a press from the user on the settings screen 610, it displays the settings screen 710 (Figure 7).

[0057] When the information processing device 104 receives a selection from the user for "Video file loading (FFmpeg)" (721 in Figure 7) from the list of functions 711, it displays the settings screen 720. At this time, the settings screen 720 edits the operation process information of RAM 203 based on the location 722 and option 723 values ​​of the command entered by the user.

[0058] When the information processing device 104 receives a selection from the user for "Video file loading (OpenCV)" (731 in Figure 7) from the list of functions 711, it displays the settings screen 730 and edits the operation process information of RAM 203.

[0059] When the information processing device 104 receives a selection from the user for "Live video loading (video surveillance server)" (811 in Figure 8) from the list of functions 711, it displays the settings screen 810 (Figure 8). "Live video loading (video surveillance server)" is a function that loads live video captured by a network camera from a video surveillance server (not shown) that can be connected via the network.

[0060] Based on the computer name 812, username 813, and password 814 values ​​entered by the user on the settings screen 810, the operating process information of RAM 203 is edited. Also, if the connection confirmation 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in RAM 203.

[0061] When the information processing device 104 receives a click from the user on the settings screen 610 to select the analysis tab 614 (Figure 6), it displays the settings screen 910 (Figure 9).

[0062] When the information processing device 104 receives a selection of "Detection by video comparison" from the function list 911 from the user, it displays the settings screen 920 and edits the operation process information of RAM 203. At this time, if the license is not authenticated, "No license" is displayed in the license information 922.

[0063] When the information processing device 104 receives a press of the license read button 923 from the user, if a valid license has been authenticated, it displays the settings screen 930, displays the license information 931, and edits the operation process information of the RAM 203.

[0064] When the information processing device 104 receives a notification from the user on the settings screen 610 by clicking the notification tab 615 (Figure 6), it displays the settings screen 1010 (Figure 10).

[0065] When the information processing device 104 receives a selection of a "custom command" (1021 in Figure 10) from the function list 1011 from the user, it displays the settings screen 1020 and edits the operation process information of the RAM 203.

[0066] A "custom command" is a function that executes commands to instruct batches or applications to run when an anomaly is detected. Using this function, it becomes possible to execute batch processing, instruct email sending to mail clients, instruct automated voice calls to be made, and integrate with other applications or external systems.

[0067] When the information processing device 104 receives a selection from the user for "Analysis Result Notification (Video Surveillance Server)" (1031 in Figure 10) from the function list 1011, it displays the settings screen 1030.

[0068] "Analysis Result Notification (Video Surveillance Server)" is a function that notifies the video surveillance server of the analysis results. This function is used when the video surveillance server is configured to notify users of anomalies, or when the video surveillance server analyzes and manages events such as anomalies.

[0069] Based on the values ​​of address 1032 and port number 1033 entered by the user on the settings screen 1030, the operating process information of RAM203 is edited.

[0070] When the information processing device 104 receives a connection confirmation 1034 from the user, it performs a connection test via the communication I / F controller 208 using the information in the RAM 203.

[0071] Returning to the explanation of Figure 4.

[0072] In S404, when the information processing device 104 receives a press of the OK button 631 on the setting screen 630 (Figure 6), it saves the operation process information registered in RAM 203 to external memory 212 as an operation process setting file.

[0073] In S405, the information processing device 104 determines whether a configuration file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406; if it is registered, the process proceeds to S407.

[0074] In S406, the information processing device 104 displays the settings screen 630, and when it receives a press from the user of the Add task button 632 for the monitoring process, it registers the monitoring task in RAM 203 and displays the settings screen 1110 (Figure 11).

[0075] In S407, when the information processing device 104 receives a click on the monitoring task 1111 to be edited on the settings screen 1110, it displays the settings screen 1120.

[0076] The following describes the process for receiving settings for monitoring tasks performed in S407, using Figures 11 to 15.

[0077] When the information processing device 104 receives input from the user on the settings screen 1120, including a name 1121 and a description 1122, and a change in the toggle button 1123 for enabling this task, it edits the operation process information of the RAM 203 and displays the settings screen 1130.

[0078] When the information processing device 104 receives a press of the input tab 1124 from the user on the settings screen 1120, it displays either the settings screen 1200 (Figure 12) or the settings screen 1300 (Figure 13) depending on the settings of the operation process.

[0079] If the input setting for the operation process is "Read video file (FFmpeg)" or "Read video file (OpenCV)", the settings screen 1200 (Figure 12) will be displayed.

[0080] When the information processing device 104 receives a click on a video folder 1201 and a folder selection from the user, input of a frame acquisition interval 1202, and a change in the toggle button for deleting processed videos 1203, it edits the operation process information of RAM 203 and displays the settings screen 1210.

[0081] If the input setting for the operation process is "live video loading," the settings screen 1300 (Figure 13) is displayed. When the information processing device 104 receives a selection of a list item for camera 1301 from the user, it edits the operation process information in RAM 203 and displays the settings screen 1310.

[0082] When the information processing device 104 receives a notification from the user on the settings screen 1120 by clicking the notification tab 1125, it displays either the settings screen 1400 or the settings screen 1500, depending on the settings of the operation process.

[0083] If the notification setting for the operation process is "custom command," the settings screen 1400 (Figure 14) is displayed. When the information processing device 104 receives input of the command path 1401 from the user or a change in the toggle button for asynchronous execution 1402, it edits the operation process information in RAM 203 and displays the settings screen 1410.

[0084] If the notification setting for the operation process is "Analysis Result Notification (Video Surveillance Server)", the settings screen 1500 (Figure 15) is displayed. The information processing device 104 accepts the input of name 1501 and camera ID 1502 from the user and displays screen 1510. At this time, if the test event button 1511 is pressed by the user, a test notification process is executed to check whether the notification with the input content works correctly. At this time, if the setting of the detection frame display toggle button 1512 is enabled, test pseudo detection result information is also added to the notification content.

[0085] Returning to the explanation of Figure 4.

[0086] In S408, when the information processing device 104 receives a press of the OK button 1602 from the user on the setting screen 1600 (Figure 16), it saves the monitoring task information registered in RAM 203 as a monitoring task setting file in external memory 212 and displays the setting screen 1610.

[0087] In S409, when the user clicks the analysis tab 1611 on the settings screen 1610, the settings screen 1700 (Figure 17) is displayed. At this time, when the user clicks the adjustment button 1701, the analysis settings adjustment screen 1710 is displayed.

[0088] Subsequently, the information processing device 104 executes the process shown in Figure 4(b).

[0089] The process for setting anomaly detection parameters for the monitoring task shown in Figure 4(b) will be explained below.

[0090] In S421, when the information processing device 104 receives a press of the basic settings button 1711 from the user on the analysis settings adjustment screen 1710 (Figure 17), it displays the basic settings screen 1720 for analysis settings and proceeds to S422. If the Basic Settings button is not pressed, proceed to S428.

[0091] In S422, when the information processing device 104 receives a request from the user to change the slider bar to 1801 rows (Figure 18) and 1802 columns, it updates the display of the grid (dashed lines) on the screen, which is divided by 1801 rows and 1802 columns, as shown in screen 1800 (Figure 18), and saves the settings to RAM 203.

[0092] Specifically, on screen 1800, the number of rows is set to 10 (1801) and the number of columns to 10 (1802), so the image to be analyzed is displayed, divided into a 10x10 grid.

[0093] While a larger number of grids allows for more precise detection, it also increases the number of processes involved, such as the image difference (anomaly) detection process described later (Figure 23), potentially increasing the processing time.

[0094] In S423, when the information processing device 104 receives a press 1811 of any area divided by a grid on the user's screen, a press of the full mask setting button 1812, or a press of the full mask release button 1813, it displays the presence or absence of a mask (notification suppression area) as shown on screen 1810 and saves the setting in RAM 203. In other words, this step is an example of a process that accepts the specification of an area included in an image.

[0095] If there are many areas to mask, you can easily set the areas to be masked by first masking all areas using the "Set All Masks" button (1812), and then specifying the areas to be unmasked.

[0096] If the masked area is small, you can easily set the area to be masked by first removing the mask from the entire area using the "Remove All Mask" button (1813), and then specifying the area to be masked.

[0097] Furthermore, if the user changes the mask color inversion 1821 toggle button at this time, the mask drawing color is changed as shown in screen 1820. Specifically, in screen 1810, the mask area was represented by transparent black, but in screen 1820, the mask area is represented by transparent white. In other words, this step is an example of a process that accepts a specification for how to display an area included in an image. This step is also an example of a process that accepts a specification for the display method of the area, and displays the area that has been specified using the accepted display method.

[0098] This allows the user to select the color of the mask area according to the color tone and brightness of the image being analyzed. This is achieved by representing the mask area with transparent black when the image is bright and transparent white when the image is dark. As a result, the user can easily identify and understand the appearance of the mask area.

[0099] In this embodiment, the color of the mask area is set to "transparent black" or "transparent white," but this method is not limited to this. Other colors may be used, the user may specify a color, patterns such as diagonal lines may be used to add visual variation, a frame may be used, or other methods may be used to distinguish the monitored area from the mask area.

[0100] In S424, when the information processing device 104 receives a change from the user to the grid correction (shifting the grid) toggle button 1901 (Figure 19) or 1911, it updates the display of the frame 1902 (with frame) or 1912 (without frame) that indicates the detection range and saves the setting to RAM 203.

[0101] In this embodiment, grid correction is set for the monitoring process, but this method is not limited to this one. Grid correction may also be set directly for the network camera or video monitoring server that is the source of the image input.

[0102] In S425, when the information processing device 104 receives a numerical input 1921 for the number of reference image frames from the user, it saves the setting in the RAM 203.

[0103] In S426, when the information processing device 104 receives a press of the back button 1803 from the user while the basic settings screen 1800 (Figure 18) for analysis settings is displayed, it determines whether there are any changes to the basic settings. If there are changes, it proceeds to S427. If there are no changes, it proceeds to S428.

[0104] In S427, the information processing device 104 reflects the contents set in RAM 203 in the monitoring task configuration file in external memory 212. The information processing device 104 also detects changes in the monitoring task configuration file and performs reference image discarding during the analysis process. (In S2306, it determines that updating the base information is necessary.) In S428, when the information processing device 104 receives a change in the slider bar for each item of the threshold 2001 on the analysis setting screen 2000 (Figure 20), it saves the setting to the RAM 203.

[0105] In this embodiment, the user sets each threshold, but this method is not the only way. The user may select a dataset for each threshold, the user may set the objects to be monitored or the anomalies to be detected to automatically set the threshold parameters, or the information processing device 104 may determine the objects to be monitored or the anomalies to be detected and automatically set the threshold parameters.

[0106] This makes it easy to set anomaly detection thresholds.

[0107] In S429, when the information processing device 104 receives a press of the apply button 2002 from the user, it reflects the contents set in RAM 203 into the monitoring task configuration file in external memory 212. The information processing device 104 also detects changes in the monitoring task configuration file and modifies the parameter values ​​used for detection processing. (This modifies the threshold used in S2310.) The anomaly detection process shown in Figure 21 will be explained below.

[0108] In S2101, the information processing device 104 executes anomaly detection processing if the external memory 212 contains a configuration file for the monitoring task (task registration information 2201 (Figure 22)) and the task activation setting toggle 2202 is set to enabled.

[0109] In S2102, the information processing device 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and repeats.

[0110] In S2103, the information processing device 104 acquires input information according to the configured input method (configured on the configuration screen 1200 or configuration screen 1300).

[0111] Specifically, if the input setting for the operation process (settings screen 1200) is "Read video file (FFmpeg)" or "Read video file (OpenCV)", the video files located in the specified video folder are used as input information and moved to the working area on external memory 212.

[0112] Furthermore, if the input setting for the operation process is "Live video loading (video monitoring server)" (settings screen 1300), the process of acquiring video from the specified video acquisition destination is executed, and if a still image is acquired, the process proceeds to S2106.

[0113] In S2104, if the information processing device 104 performed a video file transfer on the external memory 212 in S2103, it proceeds to S2105. If it has performed a video acquisition process and acquired a still image, it proceeds to S2106.

[0114] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into a series of still images for each frame, saves them as files on the external memory 212 or stores them as data on the RAM 203, and proceeds to S2106.

[0115] In S2106, the information processing device 104 executes the process shown in Figure 23.

[0116] The first embodiment will now be described.

[0117] The image difference (anomaly) detection process shown in Figure 23 will be explained.

[0118] In S2301, if the information processing device 104 has been splitting the video file into a series of still images frame by frame (S2105), it repeats the subsequent processing until processing is completed for all the generated still images.

[0119] In S2302, the information processing device 104 reads the still image to be processed and loads it onto the RAM 203. In other words, this step is an example of a process for acquiring images captured at the same angle of view. In other words, this step is an example of a process for sequentially acquiring information related to the image to be judged.

[0120] In S2303, the information processing device 104 calculates the hue, saturation, and lightness information from the image data loaded on the RAM 203.

[0121] In S2304, the information processing device 104 performs each histogram calculation process (Figure 24) to calculate a histogram from the image data.

[0122] Here, we will explain the calculation process for each histogram in Figure 24.

[0123] In S2401, the information processing device 104 proceeds to S2402 if the grid correction set in S424 is enabled among the settings stored in RAM 203. Otherwise, it proceeds to S2403.

[0124] In S2402, the information processing device 104 includes not only the regions demarcated by the grid 2500 (Figure 25) set in S422, but also regions demarcated by grid 2501, which is shifted horizontally and vertically by half the size of one region, as the target regions for histogram calculation. In other words, this step is an example of a process for dividing the acquired image.

[0125] Specifically, as shown in 2510 and 2520, if an anomaly occurs near the 3x4 grid set in S422, the anomaly spans multiple areas. As a result, the threshold is not exceeded in each area, and all areas are judged as normal, potentially leading to low anomaly detection accuracy.

[0126] As in S2402, by determining anomalies not only in the area demarcated by the normal grid 2500 set in S422, but also in the area demarcated by the corrected grid 2501 (shifted grid), it becomes easier to improve the accuracy of anomaly detection even when an anomaly occurs near the normal grid 2500 set in S422.

[0127] In this embodiment, the grid is shifted horizontally and vertically by half the size of one region, but this method is not limited to this. Two new grids may be created by shifting them horizontally and vertically by "1 / 3" and "2 / 3" of the size of one region, or a hexagonal grid may be created instead of a grid (square). In other words, this step is an example of a process in which an image is divided using two or more division methods. Specifically, this is an example in which a region divided by the first division method partially overlaps with at least one of the regions divided by the second division method.

[0128] This makes it easier to improve the accuracy of detecting anomalies, even if an anomaly occurs near grid 2500.

[0129] In S2403, the information processing device 104 calculates a histogram of luminance information from the luminance information acquired in S2303 for each processing target area set in S422 and S2402.

[0130] In S2404, the information processing device 104 calculates a histogram of hue information from the hue and saturation information acquired in S2303 for each processing target area set in S422 and S2402.

[0131] In S2405, the information processing device 104 calculates a histogram of saturation information from the saturation information acquired in S2303 for each processing target area set in S422 and S2402.

[0132] In S2406, the information processing device 104 calculates edge gradient information from the lightness information acquired in S2303 for each processing area set in S422 and S2402, and uses the edge gradient information to calculate an edge gradient intensity histogram and an edge gradient angle histogram.

[0133] In other words, steps S2304 and Figure 24 show an example of a process that compares a reference image and an image to be analyzed based on the image's features.

[0134] Furthermore, S2304 and Figure 24 show an example of a process that compares a reference image and an image to be analyzed based on information relating to at least one of the following: brightness, hue, saturation, edge gradient intensity, and edge gradient angle.

[0135] Let's return to the explanation of Figure 23.

[0136] In S2305, the information processing device 104 calculates the similarity for each processing area set in S422 and S2402 using a histogram of reference information and a histogram intersection, which is one of the histogram comparison methods.

[0137] In this case, the reference information is the histogram calculated in S2308, described later. If reference histogram data is unavailable, the similarity is set to 100%.

[0138] In this embodiment, a histogram intersection was used as the histogram comparison method, but the method is not limited to this, and other methods such as the Bhattacharya coefficient may be used.

[0139] Furthermore, while this embodiment involves detecting anomalies by comparing histograms, it is not limited to this method. Other methods, such as template matching or feature point detection, may also be used to detect anomalies.

[0140] In S2306, if the number of data already set in the reference image is less than the "number of reference image frames" set in S425, the information processing device 104 determines that the reference image needs to be updated and proceeds to S2307. If data that satisfies the "number of reference image frames" has already been set in the reference image, the process proceeds to S2309.

[0141] Furthermore, the information processing device 104 discards the reference image in the following cases and initializes the number of set data items to 0.

[0142] • When the basic settings are changed during user operation of S427 • When the user clicks the "Recreate Reference Image" button (2003) This allows users to easily instruct the system to recreate the reference image.

[0143] In S2307, the information processing device 104 determines whether the data is suitable as a reference image based on the similarity calculated in S2305. If there are no areas with low similarity, the process proceeds to S2308. If there are many areas with low similarity, the device determines that the data is not suitable as a reference image and terminates the process.

[0144] In S2308, the information processing device 104 adds the histogram calculated in S2304 to the histogram of the reference information currently held, recalculates the average value, updates the histogram of the reference information, and terminates the process.

[0145] In S2309, the information processing device 104 compares the similarity of each processing target area set in S422, which was calculated in S2305, with the threshold set in S428, and records the areas that fall below the threshold as abnormal areas on the RAM 203.

[0146] In other words, this step is an example of a process that determines whether a change has occurred in a given region based on changes between corresponding regions of multiple acquired images.

[0147] In other words, this step is an example of a process that determines whether there are any changes that should be notified, based on a comparison between a reference image and the image to be analyzed.

[0148] In other words, this step is an example of a process that determines whether there is a predetermined change based on a comparison between information relating to a reference image and information relating to the image to be judged.

[0149] Specifically, in threshold 2001 (Figure 20), if the brightness threshold is set to 22%, the region to be processed will be judged as abnormal if the brightness similarity of the region is less than 22%.

[0150] In this embodiment, similarity < threshold is used, but this is not the only option; similarity ≤ threshold may also be used.

[0151] Furthermore, while this embodiment uses similarity, it is not limited to similarity; change amount may also be used. In that case, the region will be judged as abnormal if change amount > threshold or change amount ≥ threshold.

[0152] In other words, this step is to show an example of what constitutes a change that should be notified: a change where the similarity between the reference image and the image being analyzed is less than a predetermined threshold, or the similarity is less than or equal to a predetermined threshold, or the amount of change exceeds a predetermined threshold, or the amount of change is greater than or equal to a predetermined threshold.

[0153] Furthermore, this step compares the reference image and the image to be analyzed in corresponding regions and provides an example of when the similarity between the regions is less than a predetermined threshold, or the similarity between the regions is less than or equal to a predetermined threshold, or the amount of change between the regions exceeds a predetermined threshold, or the amount of change between the regions is greater than or equal to a predetermined threshold.

[0154] In S2310, the information processing device 104 compares the similarity of each region (regions separated by a correction grid) that was calculated in S2305 and set as the region to be processed in S2402 with the threshold set in S428, and detects regions that fall below the threshold.

[0155] If an abnormality is detected at this point, proceed to S2311. If no abnormality is detected, proceed to S2312.

[0156] In S2311, the information processing device 104 determines that the area 2600 (Figure 26) where an anomaly was detected is different from the area to be notified to the user as set in S422. Therefore, it considers all areas 2610 (four areas in the case of Figure 26) that overlap with the detected area 2600 as areas where an anomaly was detected (2620) and records them as anomaly locations on the RAM 203.

[0157] As a result, the area that receives notification of an anomaly is the area demarcated by the normal grid 2500 set in S422, regardless of whether grid correction is performed or not. In other words, this step is an example of a process that outputs a determination result based on the area demarcated by the first division method when it is determined that there has been a change in the area demarcated by the second division method.

[0158] In this way, anomalies can always be notified to users and external applications based on areas divided by a fixed grid. In other words, because the notification method is consistent regardless of whether grid correction is applied or not, it is easy for users to understand the notifications, and for external applications, it has the advantage of not requiring complex methods for receiving anomaly notifications, as the receiving logic and output method for anomalies can be kept consistent.

[0159] In S2312, the information processing device 104 compares the abnormal area recorded on the RAM with the mask area set in S423. If there is an abnormal area other than the mask area, the process proceeds to S2313. If there is no abnormal area, or if all of the abnormal areas are mask areas, the process terminates. In other words, this step is an example of a process that controls whether or not to output a judgment result based on the area specified.

[0160] In this embodiment, the histogram calculation process (S2304) is performed even in the masked region, but this method is not limited to this, and the histogram calculation process (S2304) may not be performed in the masked region. By doing so, the number of times the histogram calculation process is performed is reduced when there is a masked region, and the overall processing time can be shortened.

[0161] On the other hand, when performing histogram calculation processing even in masked areas, as in this embodiment, there are advantages such as being able to record anomalies in the masked areas and suggest changes to the masked areas to the user, and being able to stably meet non-functional requirements by always maintaining a constant processing speed.

[0162] In S2313, the information processing device 104 performs an anomaly notification process (Figure 27) and notifies the user and external applications of information about the anomaly area.

[0163] The abnormality notification process shown in Figure 27 will be explained below.

[0164] In S2701, the information processing device 104 determines whether there is an anomaly to be notified based on judgment criteria such as the continuity of anomaly detection. If it determines that there is an anomaly to be notified, it proceeds to S2702. If it determines that there is no anomaly to be notified, it proceeds to S2703.

[0165] Specifically, a criterion is needed to determine whether to notify a system of an anomaly if it is detected in a single image, or if to notify a system of an anomaly if it is detected consecutively in images spanning several seconds.

[0166] For example, the information processing device 104 may accept the user's pre-set criteria for how many consecutive frames of anomaly detection will trigger a notification, or it may determine what the image being analyzed is of (factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and then determine the criteria for notifying an anomaly based on the object and environment.

[0167] In S2702, the information processing device 104 records information about the region where the notification target abnormality has occurred on the RAM 203.

[0168] In S2703, the information processing device 104 deletes the notification-targeted abnormal information recorded on the RAM 203.

[0169] In S2710, the information processing device 104 passes the still image information read in S2302 to the processing unit that draws the settings screen.

[0170] In S2711, the information processing device 104 passes the abnormal information recorded in S2702 to the processing unit that draws the settings screen.

[0171] In S2712, the information processing device 104 determines whether the settings screen is being displayed based on user input. If the settings screen is being displayed, the process proceeds to S2713. If the settings screen is not being displayed, the process terminates.

[0172] In S2713, the information processing device 104 checks the settings 2801 for the detection area drawing color on RAM 203. If there is a specification regarding the drawing method, the process proceeds to S2714. If there is no specification, the process proceeds to S2715.

[0173] In S2714, the information processing device 104 changes the highlighting during the drawing process in S2715, described later, based on the setting 2801 (Figure 28). In other words, this step is an example of a process that identifies and outputs which of several types of thresholds was used to determine that a change had occurred.

[0174] Setting 2801 specifies how to highlight items according to the detection factors for anomaly detection (items that fall below the threshold in S2309). In other words, this step is an example of a process that accepts the specification of how to identify and output each type of threshold.

[0175] Here, to ensure that the highlighting for each threshold is distinguishable, control is implemented to prevent the same highlighting settings from overlapping (e.g., displaying an error message if the same highlighting is selected for different thresholds, preventing the selection of a previously selected highlighting). In other words, this step demonstrates an example of a process that prevents the specification of the same output method for different types of thresholds.

[0176] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0177] In this embodiment, highlighting is done using color, but this method is not limited to this. Other methods may be used, such as using patterns like diagonal lines, setting the highlighting method to surround items with dotted or thick lines, or setting different notification methods such as voice or email. It is also possible to set notification methods using methods other than highlighting.

[0178] In S2715, the information processing device 104 overlays highlighting on the areas containing the still image information acquired in S2710 and the abnormal information acquired in S2711, and then displays the drawing image in the drawing memory on the RAM 203.

[0179] In S2716, the information processing device 104 draws the drawing image created in S2715 on the setting screen 2810 (Figure 28). In other words, this step is an example of a process that outputs the result of determining whether or not there has been a change in the region. In other words, this step is an example of a process that outputs the result of a process based on an image comparison between a reference image and a target image.

[0180] If the drawing method is changed in S2714, the method of highlighting the abnormal region is changed and displayed as shown in abnormal region 2811(1)(2)(4).

[0181] This makes it easy for users to understand what threshold was used to determine that something was abnormal.

[0182] Alternatively, the highlighting of the abnormal area and the display of the masked area may be shown on the same screen (not illustrated).

[0183] Furthermore, although S2312 states that abnormal areas will not be displayed in the masked area, it is also possible to highlight the abnormal detection area (not shown) or highlight it in a different way (not shown) even in the masked area, based on user instructions. Doing so would give the user a reason to decide to remove the mask even in an area designated as a masked area, depending on the nature of the abnormal detection. In other words, it becomes possible to easily configure the abnormal detection desired by the user.

[0184] Furthermore, the information processing device 104 may have a function to notify the user on screens 1810 and 1820 of what kind of abnormality has occurred in the mask area, even without receiving instructions from the user.

[0185] Furthermore, the system may have functions to suggest to the user that they change the mask settings based on the number and nature of anomaly detections, or for the information processing device 104 to automatically change the mask settings. Doing so would enable more accurate anomaly detection. In other words, this step is an example of a process that changes the specified area based on the specified area and the judgment result of that area.

[0186] In S2720, the information processing device 104 notifies the output destination set in S407 (e.g., video monitoring server, mailer, software that can instruct to make calls using automated voice communication) of the abnormal information recorded in S2702.

[0187] In other words, this step is an example of a process that notifies the result of a determination that a change requiring notification has occurred.

[0188] In other words, this step is an example of a process that outputs the results of an image comparison between a reference image and a target image.

[0189] Figure 29 shows an example of the anomaly notification dashboard screen, which displays a list of anomalies detected so far in a table format. Here, the detection time, monitoring task name, and coordinates of the detected area are displayed.

[0190] As a result, users will be able to easily find out what threshold was used to determine that something was abnormal.

[0191] Furthermore, it will be possible to improve the accuracy of detecting anomalies near the grid.

[0192] Furthermore, it becomes easier to determine what criteria were used to detect the anomaly.

[0193] This concludes the description of the first embodiment.

[0194] Next, a second embodiment for updating the reference image will be described.

[0195] Figure 30 is an example of a diagram showing an overview of the reference image update in the second embodiment.

[0196] When using three image frames as reference images, conventionally (Figure 30, upper side) features are extracted from the three reference images (for each grid), the average value is calculated (for each grid), and a histogram of the reference images is created (S2308 in Figure 23). Here, we assume that the average of the features of the three reference images is 4.0. (Figure 30, upper right) The improved second embodiment (bottom of Figure 30) allows for the automatic updating of the reference image's features using the features of images at 10-frame intervals, whereas previously the anomaly detection system could not automatically update the reference image's features at the timing desired by the user. This is achieved by setting a numerical value (10 in this case) to the "interval frame count" setting.

[0197] Specifically, if the feature value of the image used for the update is 7.0, and the average feature value of the three previous reference images was 4.0, then by replacing one of the reference images, the average feature value of the new reference image is updated to 5.0 by (4.0 + 4.0 + 7.0) / 3. (Figure 30, bottom right) This update process will continue automatically at 10-frame intervals until the "interval frame count" is changed.

[0198] In this embodiment, the reference image is used as a feature, but this method is not limited to this one. Alternatively, the reference image may be stored as an image, and an image comparison between the reference image and the image to be analyzed may be performed.

[0199] This concludes the explanation of Figure 30.

[0200] Figure 31 is an example of a flowchart that rewrites the flowchart in Figure 23 of the first embodiment for use in the second embodiment. The same processes as in Figure 23 are omitted from the explanation.

[0201] In S3101, the information processing device 104 performs the initialization process for automatic updating of the reference image (Figure 32).

[0202] In S3102, the information processing device 104 performs the process of automatically updating the reference image (Figure 33).

[0203] This concludes the explanation of Figure 31.

[0204] Figure 32 is an example of a flowchart for the initialization process of updating the reference image in the second embodiment.

[0205] In S3201, the information processing device 104 initializes the variable for the number of interval frames used to determine the timing of the automatic update of the reference image to 0.

[0206] In S3202, the information processing device 104 accepts the setting of the interval frame setting value 3211.

[0207] In other words, this step is an example of a process that accepts a first predetermined setting related to updating information relating to a reference image.

[0208] In other words, the interval frame setting value 3211 is a setting for identifying the information related to the target image to be used for updating the information related to the reference image, from among the information related to the target image to be judged that is acquired sequentially.

[0209] If the interval frame setting value 3211 ≥ 1, the reference image is updated using the analysis target image when the interval frame variable ≥ the interval frame setting value 3211. However, if the interval frame setting value 3211 = 0, the reference image is not automatically updated.

[0210] In S3203, the information processing device 104 accepts the setting of an anomaly detection ratio threshold 3212 for determining whether or not to automatically update the reference image.

[0211] In other words, the anomaly detection rate threshold 3212 is a second predetermined setting related to the information of the reference image.

[0212] In other words, this step is an example of a process for accepting a second predetermined setting related to information relating to a reference image.

[0213] The anomaly detection rate is calculated as the number of regions in which anomalies were detected in the analyzed image used for updating the baseline, divided by the number of regions divided by the grid. It is 100% if anomalies were detected in all regions, and 0% if no anomalies were detected.

[0214] If the anomaly detection rate threshold 3212 = 100, and the interval frame variable > interval frame setting value 3211, the reference image will always be updated using the image being analyzed.

[0215] If the anomaly detection rate threshold 3212 = 60, the reference image will be updated using the analyzed image only when the interval frame variable > interval frame setting value 3211, and the number of grids in the analyzed image used for reference updating that contain anomalies is less than 60%.

[0216] If the anomaly detection threshold 3212 = 0, the reference image will not be updated even if the interval frame variable > interval frame setting value 3211.

[0217] By having this anomaly detection threshold of 3212, if the image used for updating the reference image contains significant noise or other abnormalities (i.e., if the number of grids is judged to be "abnormal" at or above the anomaly detection threshold of 3212), the reference image will not be updated.

[0218] This prevents abnormal images, such as images containing noise above a threshold or images that deviate significantly from the reference image, from being included in the reference image, thereby suppressing excessive anomaly notifications.

[0219] This concludes the explanation of Figure 32.

[0220] Figure 33 is an example of a flowchart for the reference image update process in the second embodiment.

[0221] In S3301, the information processing device 104 determines whether to automatically update the reference image. If the interval frame setting value 3211 = 0 set in S3202, it determines NO, terminates Figure 33, and does not automatically update the reference image. If the interval frame setting value 3211 ≥ 1, it determines YES and proceeds to S3302.

[0222] In S3302, the information processing device 104 adds 1 to the variable for the number of interval frames used to determine the timing of the automatic update of the reference image.

[0223] In S3303, the information processing device 104 determines whether the interval frame number variable has reached the interval frame number setting value 3211 (i.e., whether it is the automatic update timing for the reference image). Specifically, if the interval frame number variable is less than the interval frame number setting value of 3211, the result is determined to be NO, Figure 33 is terminated, and the reference image is not automatically updated at this stage. If the variable for the number of interval frames is greater than or equal to the set value for the number of interval frames (3211), the result is determined to be YES, and the process proceeds to S3304.

[0224] In this embodiment, the determination is made if the variable for the number of interval frames is greater than or equal to the set value for the number of interval frames 3211. However, the determination method is not limited to this, and the variable for the number of interval frames may be greater than or equal to the set value for the number of interval frames 3211.

[0225] In S3304, the information processing device 104 calculates the anomaly detection area ratio (= number of areas where anomalies were detected / number of areas divided by the grid) for the image to be analyzed used for updating the reference.

[0226] In S3305, the information processing device 104 determines whether the anomaly detection area ratio calculated in S3304 is greater than the anomaly detection ratio threshold 3212 set in S3203.

[0227] If the anomaly detection rate threshold 3212 > anomaly detection area percentage, the result is YES, and the process proceeds to S3304. If the anomaly detection rate threshold 3212 ≤ anomaly detection area percentage, the result is NO, and Figure 33 is terminated. Automatic updating of the reference image using this image (which is an abnormal image, such as an image containing large noise above the threshold or an image far removed from the reference image) is not performed.

[0228] In other words, this step is an example of a process that, based on a comparison between information relating to a reference image and information relating to the image to be judged, determines that there is a change greater than or equal to, or less than or equal to, the threshold, and if such a change is determined to exist, the information relating to the image to be judged is not used to update the information relating to the reference image.

[0229] In this embodiment, the reference image is updated when the anomaly detection rate threshold 3212 > anomaly detection rate. However, this method is not limited to this, and other conditions may be used, such as the anomaly detection rate threshold 3212 ≥ anomaly detection rate, or the average of the anomaly detection rate threshold 3212 > anomaly detection rate.

[0230] In S3306, the information processing device 104 adds the histogram of the reference information currently stored (of the analyzed image used for updating the reference) calculated in S2304 to the histogram of the reference information currently stored, recalculates the average value, and updates the histogram of the reference information.

[0231] In other words, this step is an example of a process that updates information relating to the reference image based on information relating to the first predetermined setting relating to the target image for determination.

[0232] In other words, this step is an example of a process that updates information relating to the reference image based on information relating to the sequentially acquired images to be judged and information acquired based on the first predetermined setting.

[0233] In other words, this step is an example of a process that updates the information relating to the reference image using the information relating to the image to be judged according to the first predetermined setting, from among the information relating to the image to be judged that has been acquired sequentially.

[0234] In other words, this step is an example of a process in which information relating to the target image for determination related to the first predetermined setting is mixed with information relating to the reference image at a predetermined ratio and updated.

[0235] In other words, this step is an example of a process that updates the information related to the reference image based on the information related to the image to be judged when it is determined that there has been a change in the second predetermined setting, based on a comparison between the information related to the reference image and the information related to the image to be judged.

[0236] In S3307, the information processing device 104 clears the variable for the number of interval frames used to determine the timing of the automatic update of the reference image to 0.

[0237] This concludes the explanation of Figure 33.

[0238] As a result, the reference image can be easily updated automatically at the timing and interval desired by the user (=interval frame setting value 3211).

[0239] This enables proper anomaly detection even in rapidly changing images, such as in environments where proper anomaly detection was not possible without updating the reference image (e.g., outdoors).

[0240] Specifically, in outdoor settings, using the same reference image during the day and night makes it difficult to detect anomalies correctly because the overall brightness of the image differs. By automatically updating the reference image periodically (for example, every minute), it becomes possible to appropriately detect anomalies even in images where the brightness changes rapidly.

[0241] Furthermore, when the image being analyzed includes the sky, the movement of clouds is sometimes mistakenly identified as an anomaly, making it difficult to correctly detect anomalies using the same reference image. However, by automatically updating the reference image periodically (for example, every 10 seconds), it becomes possible to appropriately detect anomalies even in images where clouds are constantly moving.

[0242] In this embodiment, the reference image is updated based on the setting of the interval frame setting value 3211. However, this is not limited to the setting of the interval frame setting value 3211. For example, the reference image may be automatically updated based on the setting of the type of object to be analyzed (e.g., water, smoke, fire, sky, etc.), the shooting environment (e.g., indoor, outdoor, underwater, telephoto shooting, magnified shooting, etc.), and by selecting a template suitable for the object to be analyzed and the shooting environment and the setting value of that template.

[0243] As a result, even with objects to be analyzed or shooting environments that were previously unsuitable for detection, false positives will be reduced, and abnormalities will be detected appropriately.

[0244] Based on the above, a mechanism for easily updating the reference image can be provided.

[0245] This concludes the explanation of the second embodiment.

[0246] Next, a third embodiment will be described. In the third embodiment, the process of updating the reference image using a set similarity value (threshold) will be described.

[0247] In the first and second embodiments, when updating the reference image (or its features) in S2306-S2307, if there are many areas with low similarity (i.e., if the number of grids with an anomaly detection threshold of 3212 or higher is judged to be "abnormal"), the update process (S2307) is not performed. However, if there are few areas with low similarity, the update will be performed.

[0248] In other words, not only images that show no differences (normal images) but also images that show significant differences in parts will be subject to updating.

[0249] Specifically, if the image has only minor issues, such as severe noise in a part of the image or something crossing a part of the image, it will be subject to updating.

[0250] As already explained, when updating the reference image, the updated feature vector 3403 is obtained by calculating the feature vector 3401 for each grid of the image, as shown in Figure 34(a), and then calculating the average value 3402 for all images. In the example in Figure 34(a), the feature vector for the top-left grid is "1" for each of the three images, so the average value is "1.0".

[0251] However, as described above, if there is a difference in some areas, the feature of that grid will be larger than the feature of the other grids. In the example in Figure 34(b), the feature 3410 of some grids is "30", while the feature of the other grids is in the range of "1" to "3". Therefore, for the grid with the feature "30", the average value is "11.0", which is larger than the updated feature 3411 compared to the others.

[0252] In this case, a deviation from the ideal reference image occurs, leading to, for example, the detection of a grid that should not be abnormal as abnormal. Conversely, it is also possible that an abnormal grid may be treated as normal, resulting in an inappropriate detection of anomalies. In other words, the creation of an unintended reference image can lead to false detections.

[0253] Therefore, in the third embodiment, in order to reduce such discrepancies, the reference image is updated using a similarity setting value (threshold). This will be explained in detail using Figures 35 to 38.

[0254] Figure 35 is an example of a flowchart obtained by rewriting the flowchart of Figure 31 of the second embodiment for use in the third embodiment. The same processes as in Figure 23 or Figure 31 will not be explained.

[0255] In S3501, the information processing device 104 receives the setting value for the similarity to be applied to the reference image and stores it in the RAM 203.

[0256] Figure 38 shows an example of a screen that accepts setting values. The basic settings screen 3810 is an example of a screen that appears after the basic settings button 1711 has been pressed on the analysis settings adjustment screen 1710 (Figure 17). In other words, it is a different screen example from the basic settings screen 1720. The basic settings screen 3810 accepts the setting value for the similarity 3811 to be applied. This similarity is accepted as a value from 0 to 100, for example, and a similarity of 100 indicates a perfect match. Although a slider-type setting screen is used as an example, other methods are also acceptable as long as the screen configuration can accept setting values ​​(for example, a configuration that accepts value input).

[0257] That is, this step shows an example of a process of receiving a threshold value related to updating information regarding a reference image.

[0258] In S3502, the information processing apparatus 104 updates the histogram value of the reference image. Specifically, it will be described with reference to FIG. 37.

[0259] FIG. 36 is an example of a flowchart obtained by rewriting the flowchart of FIG. 33 of the second embodiment for the third embodiment. Description of the same processes as in FIG. 33 is omitted.

[0260] In S3501, the information processing apparatus 104 updates the histogram value of the reference image. Specifically, it will be described with reference to FIG. 37. That is, also in the automatic reference image update described in the second embodiment, the update method of the third embodiment is applicable.

[0261] FIG. 37 is an example of a flowchart showing a process of updating a reference image using a similarity setting value (threshold value). The processing of each step is executed by the CPU 201 of the information processing apparatus 104.

[0262] In S3701, the information processing apparatus 104 acquires the number of grids set in the RAM 203 and repeats until all grids are updated. That is, it acquires the number of grids of the image that is the update target of the reference image and repeats the processing for all grids.

[0263] That is, this step shows an example of a process of sequentially acquiring information regarding the determination target image.

[0264] That is, this step shows an example of a process of acquiring information regarding the determination target image for each region obtained by dividing the determination target image into a predetermined region.

[0265] In S3702, the information processing device 104 obtains the applicable similarity set in S3501 from RAM 203 and determines whether the edge among the similarities calculated in S2305 is equal to or greater than the applicable similarity. If it is equal to or greater than the applicable similarity, it processes S3703.

[0266] Specifically, if the similarity threshold is greater than or equal to the applicable threshold, S3703 will reflect the histogram calculated in S2406. On the other hand, if the similarity threshold is less than the applicable threshold, it will not reflect the histogram.

[0267] For example, let's assume that the feature quantities for each grid shown in Figure 39 are edge features (also called histograms). Also, let's assume that the two images on the left, 3900-3910, are the current reference information, and the image on the right, 3920, is the image to be updated (the image determined to be usable in S2307). We will also explain an example where the applied similarity is set to "70". The specific method for calculating the similarity is as shown in S2305, so this is a simplified example where we compare the provisionally calculated similarity value with the applied similarity. Note that the image used as reference information is an image acquired in the past than the image to be updated.

[0268] If the similarity of grid 3930 in the upper left is 100, then compared to the applicable similarity of "70", it is greater than or equal to the similarity, so proceed to S3703 and reflect the histogram calculated in S2406.

[0269] If the similarity of grid 3931 in the lower left is less than "70", then the value "30" of the grid for the image to be updated will not be used for the update because it is less than the applicable similarity of "70". Specifically, instead of the feature "30", the feature of the previous image will be reused to calculate the average. The previous image refers to image 3910, which is the image acquired most recently from images 3900 to 3910 used as reference information, prior to image 3920 which is the image to be updated. In other words, instead of the feature "30" of image 3920, the feature "1" of image 3910 will be used to calculate the average. In the example in Figure 39, the grid 3931 in the lower left of images 3900 to 3910 used as reference information is "2" and "1", respectively. Grid 3931 for image 3920 is "30", but since the feature "1" of image 3910 will be used instead, the average will be calculated from the features "2", "1", and "1".

[0270] Furthermore, any method is acceptable as long as it corrects the grid features where large differences occur in a part of the image according to the reference information. For example, in addition to using the features of the previous image, features of an image randomly selected from each of the features of the image used as reference information may be used. For example, the mean or median of the features of the image used as reference information may be used.

[0271] If we keep the value at "30" and calculate the average, we get "(2+1+30) / 3 = 11.0". As already explained, this average is higher than the values ​​of other grids. Therefore, if we reuse the features from the previous image instead of "30" and calculate, we get "(2+1+1) / 3 = 1.3". With this value, the average is not higher than the values ​​of other grids.

[0272] In this way, by using the applied similarity metric, it is possible to correct for each grid to prevent the calculation of extremely large values ​​compared to other grids. In other words, it is possible to create a suitable reference image that is not affected by noise or objects crossing the frame, which contributes to reducing false detections.

[0273] For the sake of explanation, examples of calculations for the top-left grid 3910 and the bottom-left grid 3911 have been explained in this step, but the actual process is as shown in the flowchart. That is, after performing processes S3702 to S3709 on the top-left grid 3910, the next grid is obtained (for example, the grid below it). Similarly, after performing processes S3702 to S3709 on the grid below grid 3910, the next grid is obtained. In this flowchart, these processes are performed for all grids.

[0274] In other words, this step is an example of a process that controls whether to update the information relating to the reference image using the information relating to the image to be judged, or to update the information relating to the reference image without using the information relating to the image to be judged, based on the information relating to the image to be judged and the threshold.

[0275] In other words, this step is an example of a process in which, if the information relating to the image to be judged is equal to or greater than the threshold, the information relating to the reference image is updated using the information relating to the image to be judged, and if the information relating to the image to be judged is less than or equal to the threshold, the information relating to the reference image is updated without using the information relating to the image to be judged.

[0276] In other words, this step is an example of a process that calculates a value based on information relating to the target image acquired by the acquisition means and information relating to a target image that is older than the current target image or a target image that was acquired in the past by the acquisition means.

[0277] In other words, this step is an example of a process that corrects the information relating to the image to be judged, acquired by the acquisition means, if the information relating to the image to be judged is less than or equal to the threshold.

[0278] In other words, this step is an example of a process that calculates a value based on the information relating to the corrected image to be judged and the information relating to an image to be judged that is older than the current image to be judged or an image to be judged that was acquired in the past by the acquisition means.

[0279] In other words, this step is an example of a process that controls whether to update the information relating to the relevant region of the reference image using the information relating to the relevant region of the target image, or to update the information relating to the relevant region of the reference image without using the information relating to the relevant region of the target image, based on the information relating to the relevant region of the target image and the threshold.

[0280] In S3703, the information processing device 104 reflects the histogram related to edges, calculated in S2304, onto the reference image.

[0281] In S3704, the information processing device 104 obtains the applicable similarity set in S3501 from RAM 203 and determines whether the hue among the similarities calculated in S2305 is equal to or greater than the applicable similarity. If it is equal to or greater than the applicable similarity, it processes S3705.

[0282] In S3705, the information processing device 104 reflects the histogram related to hue, calculated in S2304, onto the reference image. The processing in S3704 and S3705 is the same as the processing in S3702 and S3703. That is, only the processing target has been changed from edges to hue, but the specific processing method is the same.

[0283] In S3706, the information processing device 104 obtains the applicable similarity set in S3501 from RAM 203 and determines whether the saturation of the similarity calculated in S2305 is equal to or greater than the applicable similarity. If it is equal to or greater than the applicable similarity, it processes S3707.

[0284] In S3707, the information processing apparatus 104 reflects the histogram regarding the saturation among the histograms calculated in S2304 in the reference image. The processing of S3706 and S3707 is the same as the processing of S3702 and S3703. That is, only the processing target is changed from the edge to the saturation, and the specific processing method is the same.

[0285] In S3708, the information processing apparatus 104 acquires the applicable similarity set in S3501 from the RAM 203, and determines whether the luminance among the similarities calculated in S2305 is greater than or equal to the applicable similarity. If it is greater than or equal to the applicable similarity, the process of S3709 is performed.

[0286] In S3709, the information processing apparatus 104 reflects the histogram regarding the luminance among the histograms calculated in S2304 in the reference image. The processing of S3708 and S3709 is the same as the processing of S3702 and S3703. That is, only the processing target is changed from the edge to the luminance, and the specific processing method is the same.

[0287] The information processing apparatus 104 repeats the above processing until the update of all grids is completed.

[0288] As described above, according to the present embodiment, a mechanism for appropriately updating the reference image can be provided.

[0289] In the present embodiment, the feature amount is updated when it is greater than or equal to the applicable similarity, but the present invention is not limited to this, and the feature amount may be updated when it exceeds the applicable similarity.

[0290] Also, in the present embodiment, the similarity is used, but the present invention is not limited to the similarity, and other numerical values such as the amount of change may be used. In that case, the feature amount is updated when the amount of change from the reference image is less than or equal to or less than the "applicable amount of change".

[0291] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium containing a program that realizes the functions of the embodiments described above to a system or device, and by having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0292] In this case, the program read from the recording medium itself realizes the novel function of the present invention, and the recording medium on which that program is recorded constitutes the present invention.

[0293] For recording media used to supply programs, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, and the like.

[0294] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0295] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0296] Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device. It goes without saying that the present invention can also be applied when the results are achieved by supplying a program to a system or device. In this case, by reading a recording medium containing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0297] The above program may consist of object code, program code executed by an interpreter, script data supplied to the OS (operating system), and other forms.

[0298] Furthermore, by downloading and reading the program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. It should be noted that all configurations combining the above-described embodiments and their modified forms are also included in the present invention. [Explanation of Symbols]

[0299] 100 Information Processing Systems 102 Cameras 104 Information Processing Device

Claims

1. An acquisition means for sequentially acquiring information related to the image to be judged, A receiving means for receiving a threshold for updating information related to a reference image, A determination means for determining whether or not to use the image to be determined for updating the reference image, based on the similarity between the information relating to the image to be determined and the information relating to the reference image, Based on the information relating to the image to be determined, which has been determined by the determination means to be used for updating, and the threshold, Whether to update the information relating to the reference image using the information relating to the image subject to judgment, An update control means controls whether to update the information relating to the reference image using information relating to the image to be judged, which has been corrected based on the information relating to the image used to update the information relating to the reference image, An information processing system characterized by comprising the following features.

2. The update control means is If the information relating to the image to be judged is equal to or greater than the threshold, the information relating to the reference image is updated using the information relating to the image to be judged. If the information relating to the image to be judged is less than or equal to the threshold, the information relating to the reference image is updated using the corrected information relating to the image to be judged. The information processing system according to claim 1, characterized by the following:

3. If the information relating to the image to be determined, which has been determined by the determination means to be used for updating, is less than or equal to the threshold, the correction means corrects the information relating to the image to be determined based on the information relating to the image used to update the information relating to the reference image, A calculation means that calculates a value based on the information relating to the image to be judged corrected by the correction means and the information relating to the image used to update the information relating to the reference image, The update control means updates the information relating to the reference image using the value calculated by the calculation means. The information processing system according to claim 1, characterized by the following:

4. The acquisition means acquires information relating to the image to be determined for each region obtained by dividing the image to be determined into predetermined regions. The update control means, based on the information relating to the region of the image to be determined and the threshold, Whether to update the information pertaining to that region of the reference image using the information pertaining to that region of the image subject to determination, Control whether to update the information relating to that region of the reference image using the information relating to the target image that has been corrected based on the information relating to the image used to update the information relating to the reference image. The information processing system according to claim 1, characterized by the following:

5. The information relating to the aforementioned reference image is information relating to the feature quantities of the aforementioned reference image, The information relating to the image to be judged is the information relating to the feature quantities of the said image to be judged. The information processing system according to claim 1, characterized by the following:

6. The aforementioned feature quantity is a feature quantity related to one of the following: edge, hue, saturation, or brightness. The information processing system according to claim 5, characterized by the following:

7. The aforementioned reference image is one or more images used as the reference image, When multiple images are used as the reference image, the information relating to the reference image is information relating to the average of the feature quantities of the multiple images. The information processing system according to claim 1, characterized by the following:

8. The update control means controls the information relating to the reference image not to be updated if the determination means determines that it is not to be used for updating. The information processing system according to claim 1, characterized by the following:

9. The system further includes an output means that outputs a message indicating that an abnormality has been detected in the target image when it is determined that a predetermined change has occurred based on a comparison between the information relating to the reference image updated by the update control means and the information relating to the target image. An information processing system according to any one of claims 1 to 8, characterized by the above.

10. A method for controlling an information processing system, The acquisition means of the information processing system includes an acquisition step of sequentially acquiring information related to the image to be judged, The receiving means of the information processing system includes a receiving step of receiving a threshold related to the updating of information relating to a reference image, The determination means of the information processing system determines, based on the degree of similarity between the information relating to the image to be determined and the information relating to the reference image, whether or not to use the image to be determined for updating the reference image. The update control means of the information processing system, based on the information relating to the target image that has been determined to be used for updating in the determination step and the threshold, Whether to update the information relating to the reference image using the information relating to the image subject to judgment, An update control step that controls whether to update the information relating to the reference image using information relating to the image to be judged, which has been corrected based on the information relating to the image used to update the information relating to the reference image, A control method for an information processing system, characterized by comprising the following:

11. A program for causing at least one computer to function as one of the means of the information processing system described in any one of claims 1 to 8.

Images