Information processing system, information processing method and program

Abstract

A mechanism is provided that allows for suitable setting of thresholds for anomaly detection. [Solution] An information processing system characterized by comprising a setting display means for displaying settings related to determining whether an image has changed, an image display means for distinguishably displaying the image and areas in the image where it has been determined that a change has occurred, and a graph display means for displaying information related to the areas of the image as a graph.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and in particular to a technique suitable for use in detecting an abnormality. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there exists a method for detecting abnormalities in an imaged object based on changes in the images by processing a plurality of images obtained by taking images using a camera or the like.

[0003] Patent Document 1 discloses a technique for detecting localized different regions between images using image feature amounts and displaying the different regions in a distinguishable manner. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO2012 / 108088 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0005] Patent Document 1 describes displaying a graph of the degree of difference between videos along with the time axis of the videos. However, it is not possible to adjust the reference value of the degree of difference, and there is a problem that it is desirable to enable the user to easily adjust the reference value while checking the difference area.

[0006] Therefore, an object of the present invention is to provide a mechanism that allows a threshold value for anomaly detection to be suitably set. [Means for solving the problem]

[0007] The present invention is characterized by comprising a setting display means for displaying settings related to a determination that an image has changed, an image display means for distinguishably displaying the image and an area in which it has been determined that a change has occurred in the image, and a graph display means for displaying information related to the area of ​​the image as a graph. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a mechanism that allows a threshold value for anomaly detection to be suitably set. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a system configuration diagram of an information processing system 100. [Figure 2] FIG. 1 is a hardware block diagram of an information processing device 104. [Figure 3] FIG. 2 is a block diagram illustrating an example of a software configuration. [Figure 4] 10A is a flowchart of a setting process; FIG. 10B is a flowchart of an anomaly detection parameter setting process for a monitoring task; [Figure 5] 10 is an example of an initial display of a setting screen. [Figure 6] 10 is a display example of the overall image of a setting screen. [Figure 7] This is an example of the settings screen (input tab). [Figure 8] This is an example of the settings screen (input tab). [Figure 9] This is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11] 10 is a display example of a setting screen (monitoring process and monitoring task). [Figure 12] This is an example of the settings screen (input tab). [Figure 13] This is an example of the settings screen (input tab). [Figure 14] This is an example of the settings screen (notification tab). [Figure 15] This is an example of the settings screen (notification tab). [Figure 16] 10 is a display example of a setting screen (monitoring task). [Figure 17] 10 is a display example of an analysis setting adjustment screen. [Figure 18] 10 is a display example of a grid setting screen (grid division). [Figure 19] 10 is a display example of a grid setting screen (grid correction). [Figure 20] 10 is a display example of an analysis setting screen. [Figure 21] 10 is an example of a flowchart of an abnormality detection process. [Figure 22] 10 is a display example of task registration information. [Figure 23] 10 is an example of a flowchart of an image difference (abnormality) detection process. [Figure 24] 10 is an example of a flowchart of a histogram calculation process. [Figure 25] 10 is a display example of grid correction. [Figure 26] 10 is a display example of a notification when grid correction is performed. [Figure 27] 10 is an example of a flowchart of an abnormality notification process. [Figure 28] 10 is a display example of a setting screen (abnormality notification). [Figure 29] 10 is a display example of an abnormality notification dashboard screen. [Figure 30] 10 is an example of a flowchart of an image difference (abnormality) confirmation process. [Figure 31] 10 is an example of a flowchart of a graph display process of a selected grid. [Figure 32] 10 shows an example of an analysis setting adjustment screen and display of each histogram. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] FIG. 1 is a system configuration diagram showing an example of the configuration of an information processing system 100 according to the present invention.

[0012] 1 shows a configuration in which a camera 102 and an information processing device 104 are connected via an image transfer cable (USB, Ethernet, Camera Link, etc.). However, instead of having such a configuration, an information processing device having a camera may also be used.

[0013] The information processing device 104 acquires an image obtained by photographing an object with the camera 102 via an image transfer cable, and executes processing related to identifying abnormalities in the image.

[0014] Furthermore, the information processing system 100 of this embodiment may be configured without the camera 102 configured as described above.

[0015] In this case, the image processing device 104 executes processing related to determining whether there is an abnormality in an image, for an image acquired from another terminal on the network or an image stored in the memory (RAM) of the information processing device 104.

[0016] Hereinafter, the hardware configuration of an information processing device applicable to the information processing device 104 shown in FIG. 1 will be described with reference to FIG.

[0017] FIG. 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in FIG.

[0018] 2, reference numeral 201 denotes a CPU, which performs overall control of each device and controller connected to a system bus 204. In addition, a ROM 202 or an external memory 212 stores a BIOS (Basic Input / Output System), which is a control program for the CPU 201, an operating system program (hereinafter referred to as OS), and various programs (described later) required to realize the functions executed by each PC.

[0019] A RAM 203 functions as the main memory, work area, etc. of the CPU 201. The CPU 201 loads programs and the like required for executing processing from the ROM 202 or the external memory 212 into the RAM 203 and executes the loaded programs to realize various operations.

[0020] An input controller 205 controls input from a keyboard (KB) 210 and a pointing device such as a mouse (not shown).

[0021] A video controller 206 controls the display on a display device such as a display 211 .

[0022] 207 is a memory controller, which connects to an external storage device (hard disk (HD)) that stores various data, a flexible disk (FD), or a PCMCIA card slot. It controls access to an external memory 212 such as a CompactFlash (registered trademark) memory connected via an adapter.

[0023] A communication I / F controller 208 controls the reception of image data from an external PC 213 via a network (TCP / IP). An image I / F controller 209 controls the reception of image data from the camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0024] Various programs described below for implementing the present invention are recorded in the RAM 203 and executed by the CPU 201.

[0025] Furthermore, image data used when the above programs are executed is stored in the ROM 202, external memory 212, external PC 213, and camera 102 depending on the application, and is stored in the RAM 203 via various controllers when the programs are executed. FIG. 3 is a block diagram showing an example of a software configuration according to an embodiment of the present invention.

[0026] The information processing device 104 includes the following functional units.

[0027] The acquisition unit 301 is a functional unit that acquires images captured at the same angle of view.

[0028] The receiving unit 302 is a functional unit that receives designation of an area included in an image and a display method for that area.

[0029] The determination unit 303 is a functional unit that determines, based on a change between corresponding areas of the multiple images acquired by the acquisition unit 301, that a change has occurred in the corresponding area.

[0030] The output unit 304 is a functional unit that outputs the determination result from the determination unit 303 .

[0031] The output unit 304 is a functional unit that displays the area specified by the receiving unit 302 in the display method specified by the receiving unit 302, and controls whether or not to output the judgment result based on the area specified by the receiving unit 302.

[0032] The area change unit 305 is a functional unit that changes the area whose designation has been accepted by the acceptance unit 302, based on the area whose designation has been accepted by the acceptance unit 302 and the determination result of the area.

[0033] The dividing unit 306 is a functional unit that divides the image acquired by the acquiring unit 301 .

[0034] The determining unit 303 is a functional unit that determines, based on a change between corresponding areas divided by the dividing means among the multiple images acquired by the acquiring unit 301, that a change has occurred in the area.

[0035] The division unit 306 is a functional unit that divides an image using two or more division methods.

[0036] The output unit 304 is a functional unit that outputs a determination result based on the area divided by the first division method when the determination unit 303 determines that there has been a change in the area divided by the second division method.

[0037] The determination unit 303 is a functional unit that determines whether a change has occurred in a corresponding region of a plurality of images acquired by the acquisition unit, based on the change between corresponding regions of the plurality of images and a plurality of thresholds.

[0038] The output unit 304 is a functional unit that identifies which of a plurality of thresholds was used to determine that a change occurred, and outputs the result.

[0039] The receiving unit 302 is a functional unit that receives a designation of a method for identifying and outputting each type of threshold.

[0040] The receiving unit 302 is a functional unit that controls so as not to receive the same output method designation for different types of thresholds.

[0041] This concludes the explanation of FIG. The setting process in FIG. 4(a) will be described.

[0042] In S401, the information processing apparatus 104 determines whether or not a setting file for the operating process exists in the external memory 212. If it is not registered, the process proceeds to S402, and if it is registered, the process proceeds to S403.

[0043] In S402, the information processing apparatus 104 displays the setting screen 510 (FIG. 5), and upon receiving a press of the add monitoring process button 511 from the user, registers the operating process in the RAM 203 and displays the setting screen 520.

[0044] In S403, when the information processing apparatus 104 receives a press of the selection button 521 of the operation process to be edited on the setting screen 502, it displays the setting screen 610 (FIG. 6). The setting reception process on the setting screen 610 performed in S403 will be described below with reference to FIGS.

[0045] When the information processing device 104 receives input of the name 611 and description 612 of the operating process from the user, it edits the operating process information in the RAM 203 and displays the setting screen 620 .

[0046] When the information processing apparatus 104 receives a user's press of the input tab 613 on the setting screen 610, it displays a setting screen 710 (FIG. 7).

[0047] When the information processing device 104 receives a selection of "Load video file (FFmpeg)" (721 in FIG. 7) from the user in the function list 711, it displays a setting screen 720. At this time, the information processing device 104 edits the operating process information in the RAM 203 based on the values ​​of the command location 722 and option 723 input by the user on the setting screen 720.

[0048] When the information processing device 104 receives a selection of "Read video file (OpenCV)" (731 in FIG. 7) from the user in the function list 711, it displays a setting screen 730 and edits the operating process information of the RAM 203.

[0049] When the information processing device 104 receives a selection of "Load live video (video monitoring server)" (811 in FIG. 8) from the user in the function list 711, it displays a setting screen 810 (FIG. 8). "Load live video (video monitoring server)" is a function that loads live video captured by a network camera from a video monitoring server (not shown) that can be connected via a network.

[0050] The operating process information in RAM 203 is edited based on the values ​​of computer name 812, user name 813, and password 814 input by the user on setting screen 810. Furthermore, when connection confirmation 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in RAM 203.

[0051] When the information processing device 104 receives a user's press of the analysis tab 614 (FIG. 6) on the setting screen 610, it displays a setting screen 910 (FIG. 9).

[0052] When the information processing device 104 receives a selection of "detection by video comparison" from the user in the function list 911, it displays a setting screen 920 and edits the operating process information in the RAM 203. At this time, if the license has not been authenticated, the license information 922 displays "No license."

[0053] When the information processing apparatus 104 receives a license read 923 press from the user, if a valid license has been authenticated, it displays a setting screen 930, displays license information 931, and edits the operating process information in the RAM 203.

[0054] When the information processing device 104 receives a press of the notification tab 615 (FIG. 6) from the user on the setting screen 610, it displays the setting screen 1010 (FIG. 10).

[0055] When the information processing device 104 receives a selection of "custom command" (1021 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays the setting screen 1020 and edits the operating process information in the RAM 203.

[0056] "Custom Command" is a function that executes commands to execute batches or applications when an abnormality is detected. By using this function, it is possible to execute batch processing, instruct emails to be sent to an email client, instruct calls using an automated voice call, and link with other applications or external systems.

[0057] When the information processing device 104 receives a selection of "Analysis result notification (video monitoring server)" (1031 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays a setting screen 1030.

[0058] "Analysis Result Notification (Video Surveillance Server)" is a function that notifies the video surveillance server of the analysis results. This function is used when the video surveillance server is configured to notify users of abnormalities, or when analyzing and managing events such as abnormalities on the video surveillance server.

[0059] Based on the values ​​of the address 1032 and port number 1033 input by the user on the setting screen 1030, the operating process information of the RAM 203 is edited.

[0060] When the information processing device 104 receives a press of the connection confirmation button 1034 from the user, it performs a connection test via the communication I / F controller 208 using the information in the RAM 203 . Returning to the explanation of Figure 4.

[0061] In S404, when the information processing apparatus 104 receives a press of the OK button 631 on the setting screen 630 (FIG. 6), it saves the operating process information registered in the RAM 203 in the external memory 212 as an operating process setting file.

[0062] In S405, the information processing apparatus 104 determines whether a setting file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406, and if it is registered, the process proceeds to S407.

[0063] In S406, the information processing apparatus 104 displays the setting screen 630, and upon receiving a press of the add button 632 for the monitoring process task from the user, registers the monitoring task in the RAM 203 and displays the setting screen 1110 (FIG. 11).

[0064] In S407 , when the information processing apparatus 104 receives a press on the monitoring task 1111 to be edited on the setting screen 1110 , it displays the setting screen 1120 . The monitoring task setting acceptance process performed in S407 will be described below with reference to FIGS.

[0065] When the information processing device 104 receives input of a name 1121 and a description 1122 from the user on a setting screen 1120 and a change to a toggle button for enabling this task 1123, it edits the operating process information in the RAM 203 and displays a setting screen 1130.

[0066] When the information processing device 104 receives a user's press of the input tab 1124 on the setting screen 1120, it displays the setting screen 1200 (FIG. 12) or the setting screen 1300 (FIG. 13) depending on the setting of the operation process.

[0067] If the input setting for the operating process is "Read video file (FFmpeg)" or "Read video file (OpenCV)", the setting screen 1200 (FIG. 12) is displayed.

[0068] When the information processing device 104 receives a user's click on the video folder 1201, selection of the folder, input of the frame acquisition interval 1202, or change of the toggle button for deleting processed video 1203, it edits the operating process information in the RAM 203 and displays the setting screen 1210.

[0069] When the input setting for the operation process is "load live video," the setting screen 1300 (FIG. 13) is displayed. When the information processing device 104 receives a selection of a list item for the camera 1301 from the user, it edits the operation process information in the RAM 203 and displays the setting screen 1310.

[0070] When the information processing device 104 receives a press of the notification tab 1125 from the user on the setting screen 1120, it displays the setting screen 1400 or the setting screen 1500 depending on the setting of the operation process.

[0071] If the notification setting for the operating process is "custom command," a setting screen 1400 (FIG. 14) is displayed. When the information processing device 104 receives input of a command path 1401 from the user and a change to the toggle button for asynchronous execution 1402, it edits the operating process information in the RAM 203 and displays a setting screen 1410.

[0072] When the notification setting for the operating process is "Analysis result notification (video monitoring server)," a setting screen 1500 (FIG. 15) is displayed. The information processing device 104 accepts input of a name 1501 and a camera ID 1502 from the user, and displays a screen 1510. At this time, if the user presses a test event button 1511, a test notification process is executed to check whether the notification works correctly with the input content. At this time, if the setting for the detection frame display toggle button 1512 is enabled, pseudo detection result information for testing purposes is also added to the notification content. Returning to the explanation of Figure 4.

[0073] In S408, when the information processing device 104 receives a user pressing the OK button 1602 on the setting screen 1600 (FIG. 16), it saves the monitoring task information registered in the RAM 203 as a monitoring task setting file in the external memory 212 and displays the setting screen 1610.

[0074] In S409, when the user presses the analysis tab 1611 on the setting screen 1610, the setting screen 1700 (FIG. 17) is displayed. At this time, when the user presses the adjustment button 1701, the analysis setting adjustment screen 1710 is displayed.

[0075] Thereafter, the information processing device 104 executes the process shown in FIG. The abnormality detection parameter setting process for the monitoring task in FIG. 4(b) will be described.

[0076] In S421, when the information processing apparatus 104 receives a user's press of the basic settings button 1711 on the analysis settings adjustment screen 1710 (FIG. 17), it displays the basic analysis settings screen 1720 and proceeds to S422. If the basic setting button has not been pressed, the process proceeds to S428.

[0077] In S422, when the information processing device 104 receives a change to the slide bars for the number of rows 1801 (FIG. 18) and the number of columns 1802 from the user, it updates the display of the grid (dashed lines) on the screen divided by the number of rows 1801 and the number of columns 1802, as shown in screen 1800 (FIG. 18), and saves the setting contents in RAM 203.

[0078] Specifically, on the screen 1800, the number of rows 1801 is set to 10 and the number of columns 1802 is set to 10, so the image to be analyzed is displayed divided into a grid of 10 rows and 10 columns.

[0079] The more grids there are, the more pinpoint detection becomes possible, but this increases the number of processes, such as the image difference (abnormality) detection process (Figure 23) described below, which may increase the time required for processing.

[0080] In S423, when the information processing device 104 receives a user's press 1811 on any area separated by a grid on the screen, a press of the Set All Masks button 1812, or a press of the Cancel All Masks button 1813, the information processing device 104 displays the presence or absence of a mask (notification suppression area) as shown on a screen 1810, and saves the setting contents in the RAM 203. That is, this step is a step showing an example of a process for receiving a designation of an area included in an image.

[0081] If there are many mask areas, the mask target area can be easily set by first masking all areas with the mask all setting button 1812 and then specifying the area to be unmasked.

[0082] If the masked area is small, the mask target area can be easily set by releasing the mask for all areas with the release all mask button 1813 and then specifying the area to be masked.

[0083] Furthermore, at this time, if a change in the mask color inversion 1821 toggle button by the user is accepted, the mask drawing color is changed as shown on screen 1820. Specifically, the mask area is represented in transparent black on screen 1810, but is represented in transparent white on screen 1820. That is, this step is a step showing an example of a process for accepting a designation of a display method for an area included in an image. Also, this step is a step showing an example of a process for displaying the area for which designation has been accepted in the display method for which designation has been accepted.

[0084] This allows the user to select the color of the mask area depending on the color tone and brightness of the image to be analyzed, by representing the mask area in transparent black when the image to be analyzed is bright, and in transparent white when the image to be analyzed is dark. This makes it easy to identify the mask area and also allows the user to understand the appearance of the mask area.

[0085] In this embodiment, the color of the mask area is "transparent black" or "transparent white," but this is not limited to this method. Other colors may be used, the user may specify the color, patterns such as diagonal lines may be used to vary the display, the area may be surrounded by a frame, or other methods may be used to distinguish between the monitored area and the mask area.

[0086] In S424, when the information processing device 104 receives a change from the user to the grid correction (shift grid) toggle button 1901 (Figure 19) or 1911, it updates the display of the frame 1902 (with frame) or 1912 (without frame) indicating the detection range and saves the setting contents in RAM 203.

[0087] In this embodiment, grid correction is set for the monitoring process, but this is not limited to this method, and grid correction may be set directly for the network camera or video monitoring server that is the image input source.

[0088] In S425 , when the information processing apparatus 104 receives a numerical input 1921 of the number of reference image frames from the user, the information processing apparatus 104 saves the setting content in the RAM 203 .

[0089] In S426, when the information processing device 104 receives a press of the back button 1803 from the user while the basic setting screen 1800 (FIG. 18) for analysis settings is displayed, it determines whether or not the basic settings have been changed. If there have been changes, the process proceeds to S427. If there have been no changes, the process proceeds to S428.

[0090] In S427, the information processing device 104 reflects the contents set in the RAM 203 in the monitoring task setting file in the external memory 212. The information processing device 104 also detects any changes to the monitoring task setting file and discards the reference image in the analysis process (in S2306, it is determined that the base information needs to be updated). In S428, when the information processing apparatus 104 receives a change to the slide bar for each item of the threshold 2001 on the analysis setting screen 2000 (FIG. 20), it saves the setting content in the RAM 203.

[0091] In this embodiment, the user sets each threshold value, but this method is not limited to this. The user may select a data set for each threshold value, or the user may set the object to be monitored or the abnormality to be detected so that the threshold parameters are automatically set, or the information processing device 104 may determine the object to be monitored or the abnormality to be detected and automatically set the threshold parameters, etc.

[0092] This makes it possible to easily set a threshold value for detecting an abnormality.

[0093] In S429, when the information processing device 104 receives a press of the Apply button 2002 from the user, it reflects the contents set in the RAM 203 in the monitoring task setting file in the external memory 212. The information processing device 104 also detects changes to the monitoring task setting file and changes the parameter values ​​used in the detection process (changing the threshold values ​​used in S2310). The abnormality detection process in FIG. 21 will be described.

[0094] In S2101, the information processing apparatus 104 executes an abnormality detection process if the external memory 212 contains a setting file for the monitoring task (task registration information 2201 (FIG. 22)) and the toggle 2202 for task validation setting is set to valid.

[0095] In S2102, the information processing apparatus 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and repeats the process.

[0096] In S2103, the information processing apparatus 104 acquires input information according to the set input method (set on the setting screen 1200 or the setting screen 1300).

[0097] Specifically, if the input setting for the operation process (setting screen 1200) is "Load video file (FFmpeg)" or "Load video file (OpenCV)", the video file placed in the specified video folder is moved to the working area on the external memory 212 as input information.

[0098] Also, if the input setting for the operation process is "Read live video (video monitoring server)" (setting screen 1300), video acquisition processing is executed for the specified video acquisition destination, and if a still image is acquired, the process proceeds to S2106.

[0099] In S2104, if the information processing apparatus 104 has moved the video file on the external memory 212 in S2103, the process proceeds to S2105. If the information processing apparatus 104 has executed the video acquisition process and acquired a still image, the process proceeds to S2106.

[0100] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into consecutive still images for each frame, and files it on the external memory 212 or stores it as data on the RAM 203, and then proceeds to S2106.

[0101] In S2106, the information processing device 104 executes the process shown in FIG. The image difference (abnormality) detection process of FIG. 23 will be described.

[0102] In S2301, if the information processing apparatus 104 has divided the moving image file into successive still images for each frame (S2105), it repeats the subsequent processes until the processes for all the generated still images are completed.

[0103] In S2302, the information processing device 104 reads a still image to be processed and develops it on the RAM 203. That is, this step is a step showing an example of processing for acquiring images captured at the same angle of view.

[0104] In S2303, the information processing apparatus 104 calculates information on hue, saturation, and lightness from the image data expanded on the RAM 203 by calculation.

[0105] In S2304, the information processing device 104 performs each histogram calculation process (FIG. 24) to calculate a histogram from the image data. Here, the histogram calculation process of FIG. 24 will be described.

[0106] In S2401, the information processing apparatus 104 proceeds to S2402 if the grid correction set in S424 among the settings saved in the RAM 203 is valid, or proceeds to S2403 if it is invalid.

[0107] In S2402, the information processing device 104 sets, as target areas for histogram calculation processing, not only the areas divided by the grid 2500 (FIG. 25) set in S422, but also areas divided by grids 2501 shifted by half the size of one area in the horizontal and vertical directions. That is, this step is a step showing an example of processing for dividing the acquired image.

[0108] Specifically, as shown in 2510 and 2520, if an abnormality occurs near the 3-row x 4-column grid set in S422, since the abnormality spans multiple areas, the threshold value will not be exceeded in each area, and all areas will be determined to be normal, which may result in low accuracy in abnormality detection.

[0109] As in S2402, by determining whether an abnormality exists not only in the area separated by the normal grid 2500 set in S422 but also in the area separated by the corrected grid 2501 (shifted grid), it becomes possible to easily improve the accuracy of detecting an abnormality even when an abnormality occurs near the normal grid 2500 set in S422.

[0110] In this embodiment, the grids are shifted horizontally and vertically by half the size of one region. However, this is not limited to this method. It is also possible to create two new grids that are shifted horizontally and vertically by 1 / 3 and 2 / 3 of the size of one region, or to create a new hexagonal grid instead of a lattice (square) grid. This step is an example of a process for dividing an image using two or more division methods. This is an example in which an area divided by a first division method partially overlaps with at least one of the areas divided by a second division method.

[0111] This makes it possible to easily improve the accuracy of detecting an abnormality even when an abnormality occurs near the grid 2500.

[0112] In S2403, the information processing apparatus 104 calculates a histogram of luminance information from the luminance (Lightness) information acquired in S2303 for each processing target region set in S422 and S2402.

[0113] In S2404, the information processing apparatus 104 calculates a histogram of hue information from the hue and saturation information acquired in S2303 for each processing target region set in S422 and S2402.

[0114] In S2405, the information processing apparatus 104 calculates a histogram of saturation information from the saturation information acquired in S2303 for each processing target region set in S422 and S2402.

[0115] In S2406, the information processing device 104 calculates edge gradient information from the luminance (Lightness) information acquired in S2303 for each processing target area set in S422 and S2402, and calculates an edge gradient intensity histogram and an edge gradient angle histogram using the edge gradient information. Returning to the explanation of FIG.

[0116] In S2305, the information processing apparatus 104 calculates the degree of similarity for each processing target area set in S422 and S2402 using the histogram of the reference information and histogram intersection, which is one of the methods for comparing histograms.

[0117] In this case, the reference information is a histogram calculated in S2308, which will be described later. If there is no reference histogram data, the similarity is set to 100%.

[0118] In this embodiment, histogram intersection is used as the method for comparing histograms, but the method is not limited to this, and other methods such as Bhattacharyya coefficients may also be used.

[0119] Furthermore, in this embodiment, anomalies are detected by comparing histograms, but this is not limited to this method, and anomalies may be detected using other methods, such as template matching or feature point detection.

[0120] In S2306, if the number of data items already set as the reference image does not satisfy the "number of reference image frames" set in S425, the information processing device 104 determines that the reference image needs to be updated and proceeds to S2307. If data items that satisfy the "number of reference image frames" have already been set as the reference image, the process proceeds to S2309.

[0121] The information processing device 104 discards the reference image and initializes the number of set data items to 0 in the following cases:

[0122] When the basic settings are changed by the user of the S427 When the user clicks the Recreate Reference Image button 2003 This allows the user to easily instruct the re-creation of the reference image.

[0123] In S2307, the information processing device 104 determines whether the data is suitable as a reference image based on the similarity calculated in S2305. If there are no areas with low similarity, the process proceeds to S2308. If there are many areas with low similarity, the data is determined to be unsuitable as a reference image and the process ends.

[0124] In S2308, the information processing apparatus 104 adds the histogram calculated in S2304 to the histogram of the reference information currently held, recalculates the average value, updates the histogram of the reference information, and ends the process.

[0125] In S2309, the information processing device 104 compares the similarity for each processing target region set in S422, calculated in S2305, with the threshold set in S428, and records any region below the threshold as an abnormal region in the RAM 203. That is, this step is a step illustrating an example of processing for determining that a change has occurred in a region based on changes between corresponding regions of multiple images among the multiple acquired images.

[0126] Specifically, in the threshold 2001 (FIG. 20), when the brightness threshold is set to 22%, if the brightness similarity of the area to be processed is less than 22%, the area is determined to be abnormal.

[0127] In S2310, the information processing device 104 compares the similarity for each area (area separated by a correction grid) set as the area to be processed in S2402, calculated in S2305, with the threshold set in S428, and detects areas that fall below the threshold.

[0128] At this time, if an abnormality is detected, the process proceeds to S2311. If no abnormality is detected, the process proceeds to S2312.

[0129] In S2311, the information processing device 104 determines (2620) that all areas 2610 (four areas in the case of FIG. 26) that overlap with the detection area 2600 have detected an abnormality, since the area 2600 (FIG. 26) where the abnormality was detected is different from the area to be notified to the user that was set in S422, and records the detected abnormality in RAM 203.

[0130] As a result, the area for which an abnormality is notified is the area divided by the normal grid 2500 set in S422, regardless of whether grid correction is performed or not. In other words, this step is an example of a process for outputting a determination result based on the area divided by the first division method when it is determined that a change has occurred in the area divided by the second division method.

[0131] In this way, it is possible to notify users and external applications of abnormalities based on areas that are always separated by a constant grid. In other words, because the notification method is constant regardless of whether grid correction is performed or not, it has the advantage that the notification is easy for users to understand, and for external applications, the method for receiving abnormality notifications does not need to be complicated, so the reception logic and method for outputting abnormalities can be constant.

[0132] In S2312, the information processing device 104 compares the abnormal area recorded on the RAM with the mask area set in S423, and if there is an abnormal area other than the mask area, proceeds to S2313. If there is no abnormal area or if all of the abnormal areas are mask areas, the process ends. In other words, this step is a step that shows an example of a process of controlling whether or not to output the determination result based on the area whose designation has been accepted.

[0133] In this embodiment, the histogram calculation process (S2304) is performed even in a masked area, but this is not limited to this method, and the histogram calculation process (S2304) may not be performed for a masked area. By doing so, the number of times each histogram calculation process is performed when there is a masked area is reduced, thereby shortening the overall processing time.

[0134] On the other hand, when performing histogram calculation processing even in a masked area, as in this embodiment, there are advantages such as being able to record abnormalities in the masked area and suggest changes to the masked area to the user, and being able to stably meet non-functional requirements by always keeping the processing speed constant.

[0135] In S2313, the information processing device 104 performs an abnormality notification process (FIG. 27) and notifies the user and an external application of information about the abnormal area. The abnormality notification process in FIG. 27 will be described.

[0136] In S2701, the information processing device 104 determines whether there is an abnormality to be notified based on criteria such as the continuity of abnormality detection. If it is determined that there is an abnormality to be notified, the process proceeds to S2702. If it is determined that there is no abnormality to be notified, the process proceeds to S2703.

[0137] Specifically, criteria are needed to determine whether an abnormality should be reported if an abnormality is detected in only one image frame, or if an abnormality is detected in several consecutive images over a period of several seconds.

[0138] For example, the information processing device 104 may accept the user's setting of criteria for notifying an abnormality in advance, such as how many consecutive frames an abnormality must be detected before notifying an abnormality, or may determine what the image being analyzed is (a factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and based on the object or environment, the information processing device 104 may determine the criteria for notifying an abnormality.

[0139] In step S2702, the information processing apparatus 104 records in the RAM 203 area information in which the abnormality to be notified has occurred.

[0140] In S2703, the information processing apparatus 104 deletes the abnormality information to be notified that is recorded on the RAM 203.

[0141] In S2710, the information processing apparatus 104 passes the still image information read in S2302 to the processing unit that draws the setting screen.

[0142] In S2711, the information processing apparatus 104 passes the abnormality information recorded in S2702 to the processing unit that draws the setting screen.

[0143] In S2712, the information processing apparatus 104 determines whether the setting screen is being drawn by a user operation. If the setting screen is being drawn, the process proceeds to S2713. If the setting screen is not being drawn, the process ends.

[0144] In S2713, the information processing apparatus 104 checks the detection area drawing color setting content 2801 in the RAM 203. If there is a designation regarding the drawing method, the process proceeds to S2714. If there is no designation, the process proceeds to S2715.

[0145] In S2714, the information processing device 104 changes the highlighting during the drawing process in S2715 (described later) based on the setting contents 2801 (FIG. 28). That is, this step is an example of a process of identifying and outputting which of multiple thresholds was used to determine that a change occurred.

[0146] The setting content 2801 is a setting for how to highlight the display depending on the detection factor of the abnormality detection (the item that fell below the threshold in S2309). That is, this step is an example of a process for accepting a designation of a method for identifying and outputting for each type of threshold.

[0147] Here, control is performed to prevent overlapping of the same highlighting settings so that the highlighting for each threshold can be distinguished (for example, an error message is displayed when the same highlighting is selected for different thresholds, or a highlighting that has already been selected cannot be newly selected, etc.). In other words, this step is an example of a process for controlling so that the same output method specification is not accepted for different threshold types.

[0148] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0149] In this embodiment, highlighting is performed using color, but the present invention is not limited to this method. The highlighting method can be set to use a pattern such as diagonal lines, to surround the information with a frame such as dotted or thick lines, or to set different notification methods such as voice or email, or to set the notification method using methods other than highlighting in addition to the type of highlighting.

[0150] In S2715, the information processing apparatus 104 develops an image for drawing in the drawing memory on the RAM 203 by highlighting the still image information acquired in S2710 and the area containing the abnormality information acquired in S2711.

[0151] In S2716, the information processing device 104 draws the drawing image created in S2715 on the setting screen 2810 (FIG. 28). That is, this step is a step showing an example of a process of outputting a determination result as to whether or not there has been a change in the area.

[0152] At this time, if the rendering method is changed in S2714, the highlighting method for the abnormal area is changed and displayed as shown in abnormal area 2811(1), (2), and (4).

[0153] This allows the user to easily know which threshold was used to determine the abnormality.

[0154] The highlighting of the abnormal region and the display of the masked region may be performed on the same screen (not shown).

[0155] Furthermore, although it was stated in S2312 that the anomaly region is not displayed in the masked region, it is also possible to highlight the anomaly detection region (not shown) or highlight it differently (not shown) in the masked region in response to an instruction from the user. This is because, even in a region designated as a masked region, this can trigger the user to decide that the mask should be removed depending on the content of the anomaly detection. In other words, it becomes possible for the user to easily set the anomaly detection desired.

[0156] Furthermore, the information processing device 104 may have a function of informing the user on the screen 1810 or 1820 of the type of abnormality occurring in the masked area without receiving an instruction from the user.

[0157] Furthermore, there may be a function that suggests to the user to change the mask settings based on the number and content of anomaly detections, or that the information processing device 104 automatically changes the mask settings. This enables more accurate anomaly detection. That is, this step is a step that shows an example of a process of changing the area whose designation has been accepted based on the area whose designation has been accepted and the determination result of that area.

[0158] In S2720, the information processing device 104 notifies the abnormality information recorded in S2702 to the output destination set in S407 (for example, a video monitoring server, a mailer, software capable of instructing a call using an automated voice call, etc.).

[0159] Figure 29 shows an example of the anomaly notification dashboard screen, which displays a table listing all the anomalies that have been detected so far. The screen displays the detection time, monitoring task name, and coordinates of the detected area.

[0160] 30 is an example of a flowchart of an image difference (abnormality) confirmation process. In S3001, the information processing device 104 accepts an abnormality determination threshold setting for each comparison element in order to determine an abnormal location based on image difference. That is, this step is an example of a process of accepting a similarity threshold setting for multiple images.

[0161] In S3002, the information processing device 104 reads a reference image and further reads a still image to be compared and develops it on the RAM 203 in order to display the location determined to be abnormal.

[0162] The still image to be compared may be any captured image, such as the most recent captured image or an image in which an abnormality has been detected (a change from the reference image). Alternatively, if the video file has been divided into a series of frame-by-frame still images (S2105) as in S2301, the generated still images may also be used.

[0163] In S3003, the information processing apparatus 104 acquires information on hue, saturation, and lightness, which are comparison elements, from the image data expanded on the RAM 203.

[0164] In S3004, the information processing device 104 performs the above-described histogram calculation process (FIG. 24) to calculate a histogram from the image data.

[0165] In S3005, the information processing apparatus 104 calculates the degree of similarity with the histogram of the reference information for each processing target area set in S422 and S2402 using histogram intersection, which is one of the histogram comparison methods.

[0166] In S3006, the information processing device 104 compares the similarity for each processing target region set in S422, calculated in S3005, with the threshold set in S3001, and records any region below the threshold as an abnormal region in the RAM 203. That is, this step is a step showing an example of processing for determining that a change has occurred in a region based on the similarity of information related to a plurality of images.

[0167] In S3007, the information processing device 104 displays the processing results of S3006, so if the grid to be displayed is set to "normal grid + correction grid," the process proceeds to S3010. If it is set to "normal grid only," the process proceeds to S3008. If it is set to "correction grid only," the process proceeds to S3009. Here, the "correction grid" is a setting that can be enabled in S2401, and refers to a grid that is shifted horizontally and vertically by half the size of one area.

[0168] That is, this is a step showing an example of a process in which the area to be specified is at least one of a first area obtained by dividing an image into a plurality of rectangles and a second area obtained by dividing the image using a division method different from that of the first area. Note that the change in display of the normal grid and correction grid will be described later in 3202 of FIG. 32.

[0169] In S3008, the information processing device 104 compares the similarity for each normal grid area 2500 (Figure 25) among the areas set as the areas to be processed in S2402 calculated in S3005 with the threshold set in S3001, detects areas below the threshold, and sets them as abnormality detection areas.

[0170] In S3009, the information processing device 104 compares the similarity for each area 2501 (Figure 25) of the correction grid among the areas set as the areas to be processed in S2402 calculated in S3005 with the threshold set in S3001, detects areas below the threshold, and sets them as abnormality detection areas.

[0171] In S3010, the information processing device 104 compares the similarity for each normal grid area 2500 (Figure 25) among the areas set as the areas to be processed in S2402 calculated in S3005 with the threshold set in S3001, detects areas below the threshold, and sets them as abnormality detection areas.

[0172] In S3011, the information processing device 104 compares the similarity for each correction grid area 2501 (area partitioned by the correction grid) among the areas set as processing target areas in S2402, calculated in S3005, with the threshold set in S3001, and detects areas below the threshold, thereby determining whether an abnormality has been detected. If an abnormality is detected, the process proceeds to S3012, and if no abnormality is detected, the process proceeds to S3013.

[0173] In S3012, the information processing device 104 regards all areas that overlap with the detection area in which an abnormality was detected in S3011 (in the case of FIG. 26, the four areas 2610 that overlap with the area 2600 in which an abnormality was detected) as areas in which an abnormality was detected (2620), and records these as abnormal locations in the RAM 203. That is, the areas of the normal grid that overlap the correction grid are regarded as abnormality detection areas. At this time, the correction grid in which an abnormality was detected and the normal grid that overlaps with it are displayed in different colors. Note that in this embodiment, all areas of the normal grid that overlap the correction grid are regarded as abnormal areas, but the present invention is not limited to this, and it is also possible to set only some of the normal grids that overlap with the correction grid as abnormality detection areas.

[0174] The display of the grid will be described using 3205(a) to (c) in Fig. 32. In this embodiment, when a grid is selected, it is displayed surrounded by a thick frame as in 3205(a), and when it is not selected, it is displayed without a thick frame as in 3205(b). By surrounding the edge of the selected grid with a thick line, the selected grid status is displayed in an easy-to-see manner. When selecting a correction grid, by selecting "Correction grid only" on the grid setting screen (3202), it becomes possible to select the correction grid in the same way as a normal grid.

[0175] Next, a method for displaying an area when an abnormality is detected will be described according to the display grid selected in S3007. If "correction grid only" or "normal grid only" is selected, the area where an abnormality is detected is displayed so that it can be identified in the area separated by the selected grid, as described above in S3008 and S3009.

[0176] 3205(a) and (b) show the state when "normal grid + correction grid" is selected, and the normal grid area (in S3012, the normal grid that overlaps the correction grid where an abnormality has been detected is also considered to have detected an abnormality, so the normal grid may or may not have detected an abnormality) and the correction grid area where an abnormality has been detected are displayed in different colors so that they can be distinguished.

[0177] 3205(c) is the display when "normal grid + correction grid" is selected and no abnormality is detected in the correction grid (the judgment result in S3011 is No), but an abnormality is detected only in the normal grid. Since no abnormality is detected in the correction grid, only the normal grid is displayed in a solid color.

[0178] As described above, when "normal grid + correction grid" is selected, the correction grid and the normal grid are displayed in different colors, allowing the user to see at a glance whether an abnormality has been detected in the correction grid.

[0179] In S3013, the information processing device 104 determines whether the display image setting is set to display only the current image (e.g., the most recently captured still image), only the reference image, or the reference image and the current image superimposed on each other. If the display image setting is set to "current image," the process proceeds to S3014. If the display image setting is set to "reference image," the process proceeds to S3015. If the display image setting is set to "reference image + current image," the process proceeds to S3016. Note that in this embodiment, the "reference image" and "current image" are displayed, but an image that is determined to have changed from the reference image may also be acquired and set. That is, this step illustrates an example of a process in which the image for which the first accepting means accepts the designation of an area is at least one of the reference image, the image determined to have changed, and the current image among the multiple images. This step also illustrates an example of a process in which the image is a superimposed image of at least two of the reference image, the image determined to have changed, and the current image among the multiple images.

[0180] In S3014, the information processing device 104 superimposes highlighting on the still image information acquired in S3002 (in this embodiment, the current image or the image determined to have changed) and the area containing the abnormality information acquired in S3008, S3009, S3010, or S3012, and develops a drawing image in the drawing memory on the RAM 203. That is, this step is a step showing an example of processing for identifiably displaying an area determined to have changed on the image based on the determination.

[0181] In S3015, the information processing device 104 expands a drawing image in the drawing memory on RAM 203 by superimposing highlighting on one still image information (in this embodiment, the reference image) among the reference information updated in S2308 and on the area containing the abnormality information acquired in S3008, S3009, S3010, or S3012.

[0182] In S3016, the information processing device 104 combines one piece of still image information from the reference information updated in S2308 with the still image information acquired in S3002 (in this embodiment, the current image or an image determined to have changed), and expands the combined still image information into a drawing memory on RAM 203 as a drawing image.

[0183] In S3017, the information processing device 104 develops a drawing image in the drawing memory on the RAM 203 by superimposing highlighting on the still image information synthesized in S3016 and the area containing the abnormality information acquired in S3008, S3009, S3010, or S3012.

[0184] In S3018, the information processing device 104 displays on the screen the drawing image expanded in the RAM 203 in S3014, S3015, and S3017. That is, the abnormality determination area displayed based on the type of grid selected in S3007 is displayed on the image selected in S3013. That is, this is a step showing an example of processing for accepting a selection of an image division method and, in accordance with the accepted selection, identifiably displaying on the image an area determined to have changed based on the determination of the determination means.

[0185] In S3019, the information processing device 104 accepts the selection of the grid and displays the calculated histogram as a graph (FIG. 31).

[0186] In S3020, the information processing device 104 accepts an operation to end the image difference (abnormality) confirmation process, and if the operation to end the image difference (abnormality) confirmation process is accepted (for example, pressing the X button in the upper right corner of 3200 in Fig. 32), the information processing device 104 ends the process in Fig. 30. If the operation to end the image difference (abnormality) confirmation process is not accepted, the process returns to S3001.

[0187] Fig. 31 is an example of a flowchart of a graph display process for a selected grid. In S3101, the information processing device 104 checks whether a normal grid or a correction grid (areas 2500 and 2501 (Fig. 25)) has been selected, and if not, ends the process in Fig. 30. If selected, proceed to S3102. That is, this step is an example of a process for accepting designation of an area included in a captured image.

[0188] In S3102, the information processing device 104 acquires the positions of the selected grids (areas 2500 and 2501 (FIG. 25)).

[0189] In S3103, the information processing device 104 graphs and displays the histogram of the reference information updated in S2308 and the histogram calculated in S3004 according to the position of the selected grid acquired in S3102. That is, this step is a step illustrating an example of processing for displaying information related to multiple images in a specified area as a graph that enables identification of which image among the multiple images the information relates to. Note that, although the multiple images are two images in this embodiment, information for each of three or more images may be displayed. The display of each histogram will be described later with reference to FIG. 32.

[0190] FIG. 32 shows an example of the analysis setting adjustment screen and the display of each histogram. The analysis setting adjustment screen (3200) displays, in the center of the screen, a reference image showing an area where an abnormality was detected, the current image, an image determined to have changed (a still image to be compared with the reference image acquired in S3002), or an image in which these are superimposed. The right side of the screen displays a histogram calculated for each comparison element. Furthermore, the left side of the screen displays an area for accepting an abnormality determination threshold setting for each comparison element. That is, this screen shows an example of a process for displaying settings related to determining whether an image has changed, a process for distinguishably displaying the image and the area determined to have changed in the image, and a process for displaying information related to the image area as a graph.

[0191] In this embodiment, the screen configuration is as described above, but this is not limited to this, and the central part of the screen, the histogram section for each comparison element, and the abnormality judgment threshold setting section may be displayed in a different layout, or each may be displayed on a separate screen.

[0192] In the anomaly determination threshold setting area (3201), the threshold value for each element can be changed by operating the slider. If the toggle button for the AND detection item is turned on, it can be set to detect an anomaly only when all of the set thresholds are exceeded. In other words, it is possible to set it so that an anomaly is less likely to be detected than when AND detection is not enabled. When the threshold is changed by operating the slider, the area detected as an anomaly changes accordingly, and the anomaly detection area displayed on the image in the center of the screen changes accordingly. In other words, this is a screen showing an example of processing in which the setting display means displays the threshold setting for the image, and the image display means identifiably displays the area determined to have changed based on the change in the threshold set by the setting display means.

[0193] In this embodiment, the threshold can be changed by operating a slider. However, for example, the similarity value and scale of the selected grid may be displayed on the slider. That is, this is a screen showing an example of a process in which the setting and display means displays the similarity of information related to an image region near the display of the threshold for judgment. This allows the user to set the threshold while comparing the threshold and the similarity on the slider, making it easy to adjust the threshold. As described above, the threshold for anomaly detection can be set appropriately.

[0194] As described above in the explanation of the analysis setting screen 2000 (FIG. 20), in this embodiment, the user sets each threshold value, but this is not limited to this method. The user may select a data set for each threshold value, or the user may set the object to be monitored or the abnormality to be detected, thereby automatically setting the threshold parameters, or the information processing device 104 may determine the object to be monitored or the abnormality to be detected, and automatically set the threshold parameters, etc.

[0195] The grid setting screen (3202) allows the user to select whether to display only the normal grid, only the correction grid, or both the normal grid and the correction grid on the image (S3007 in FIG. 30). If the normal grid is selected, dotted lines indicating the boundaries of the normal grid are displayed. If the correction grid is selected, dotted lines indicating the boundaries of the correction grid are displayed. If displaying both the normal grid and the correction grid is selected, dotted lines indicating the boundaries of the normal grid are displayed. Note that the boundaries of both the normal grid and the correction grid may be displayed, or the boundaries of the normal grid and the correction grid may be displayed using lines of different shapes or colors so that they can be distinguished from each other. That is, these are steps illustrating an example of a process of accepting selection of at least one of a first division method for dividing an image into multiple rectangles and a second division method that is different from the first division method, and displaying the boundaries of the areas divided by the accepted division method on the image for which the first acceptance unit accepts the designation of the areas.

[0196] On the image display setting screen (3203), it is possible to select whether to display only the current image, only the reference image, or the current image and the reference image superimposed on each other, with regard to the image displayed in the center (S3013 in FIG. 30). This allows the user to easily compare the current image and the reference image and check the results of anomaly detection.

[0197] The histograms calculated for each comparison element will be described using histogram examples 3204(a) and 3204(b). The numbers displayed on the histograms represent the similarity between the reference image and the current image of the selected grid. In other words, this is an example of processing in which the graph display means displays the similarity of information related to an image region. In this embodiment, if the selected grid has been detected as abnormal, the similarity numbers displayed on the histograms are displayed in red to make them stand out. As long as the user can identify whether the selected grid has been detected as abnormal, other methods such as highlighting in bold or with a frame, or displaying a pop-up may also be used.

[0198] The calculated histograms show the distribution of each comparison element for the reference image and the current image. In this embodiment, the histogram for the reference image is displayed in green, the histogram for the current image in yellow, and the overlapping portions of the histograms for the reference image and the current image in blue. That is, this screen shows an example of a process for distinguishably displaying graph elements of information relating to each of multiple images and graph elements where information from multiple images overlap. Note that in this embodiment, the information for each image is displayed in a different color so that it can be distinguished from the information for that image; however, as long as the user can distinguish which image the information relates to, other methods such as distinguishing the information by pattern or adding a legend may also be used.

[0199] As described above, by displaying information for each comparison element of the reference image and the current image in a distinguishable manner using colors, patterns, etc., the user can easily compare the information for the reference image and the current image, making it easier to decide whether to loosen or tighten the threshold. For example, if the user checks the histogram and sees that the brightness of a selected grid is higher in the current image than in the reference image, and then checks the image and determines that the reason for the higher brightness is not an abnormality (e.g., due to frequent exposure to direct sunlight), the user can make changes such as loosening the threshold. As a result, the threshold for anomaly detection can be set appropriately.

[0200] An edge gradient histogram and its calculation method will be described using an example histogram 3204(a). In the embodiment of FIG. 32, the items of edge gradient intensity and edge gradient angle in FIG. 20 are displayed as a single histogram. First, an arbitrary point (e.g., one pixel) in the grid is set as the center and the brightness difference between the diagonally opposite points is calculated. The brightness difference is plotted on an XY Cartesian coordinate system to set virtual coordinates, and the angle (argument) of the virtual coordinates obtained using an inverse trigonometric function and the intensity, which is the length of the line between the virtual coordinates and the origin, are calculated. The calculated angle and intensity are each averaged and classified into ranges divided by a certain number (eight in this embodiment) based on the angle, and an intensity histogram is created within each classified range.

[0201] In other words, it shows the edge gradient strength for each of the eight divided virtual edge gradient angles, allowing the user to see at a glance the similarity between the reference image and the current image at each edge gradient angle.

[0202] The histograms for hue, saturation, and brightness will be described using an example histogram 3204(b). For the hue histogram, the RGB values ​​(numerical indices of the intensity of the primary colors red, green, and blue) of any point in the grid are converted into a hue wheel, and a histogram is created with a horizontal axis of 360 degrees. In other words, the hue histogram shows the number of points in the grid for each color on the 360-degree hue wheel. This allows the user to check the hue distribution of the reference image and the current image and set a threshold. However, in this embodiment, if the brightness is below a certain level and the color cannot be distinguished, it may be displayed as not being measured (hue similarity 100%), as shown in the hue diagram of the example histogram 3204(a).

[0203] For the saturation histogram, the saturation is calculated from the RGB values ​​of any point in the grid, and a histogram is created based on this value. The horizontal axis of the histogram is saturation, and the vertical axis is the number of points in the grid.

[0204] For the brightness histogram, brightness is calculated from the RGB values ​​of any point in the grid, and a histogram is created based on this value. The horizontal axis of the histogram represents brightness, and the vertical axis represents the number of any point in the grid. In other words, this is a step that shows an example of processing in which image-related information includes at least one of edge gradient, hue, saturation, and brightness.

[0205] As a result, the user can visually know which threshold was used to determine an anomaly, or how similar the image is to the reference image, thereby enabling the threshold for anomaly detection to be set appropriately.

[0206] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0207] In this case, the program itself read from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0208] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon disks.

[0209] Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing a program it has read, but also cases are included in which an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0210] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0211] Furthermore, the present invention may be applied to a system consisting of multiple devices, or to a device consisting of a single device. It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or device. In this case, the system or device can enjoy the effects of the present invention by reading a recording medium containing a program for achieving the present invention into the system or device.

[0212] The program may be in the form of object code, program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

[0213] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modifications. [Explanation of symbols]

[0214] 100 Information Processing Systems 102 Camera 104 Information processing equipment

Claims

1. a setting display means for displaying settings relating to whether or not a change has occurred in an image; an image display means for displaying the image and an area determined to have changed in the image in a distinguishable manner; the setting display means accepts changes to thresholds set for each type of information related to the image; In response to the setting display means accepting the change of the threshold value, the image display means displays the image and the area where it is determined that a change has occurred in the image on the same screen as the screen accepting the change of the threshold value, in a manner that allows the image and the area where it is determined that a change has occurred in the image to be distinguishable based on the similarity between the changed threshold value and the information. An information processing system characterized by:

2. Further comprising an analysis result display means for displaying the analysis results of the information.

2. The information processing device according to claim 1,

3. The settings displayed by the setting display means, the image and the area displayed by the image display means, and the analysis results displayed by the analysis result display means are displayed on one screen.

3. The information processing device according to claim 2, wherein:

4. The area is at least one of a first area obtained by dividing the image into a plurality of rectangles and a second area obtained by dividing the image into a plurality of rectangles using a division method different from that of the first area.

2. The information processing system according to claim 1, wherein:

5. 2. The information processing system according to claim 1, wherein the information includes at least one of edge gradient, hue, saturation, and luminance.

6. The analysis result display means controls the display of the analysis result as a graph in which information relating to each of the plurality of images and information that overlaps among the plurality of images can be distinguished.

3. The information processing system according to claim 2, wherein:

7. The analysis result display means displays the similarity of information related to the region of the image.

3. The information processing system according to claim 2, wherein:

8. further comprising a receiving means for receiving a designation of an area included in the image displayed by the image display means; The analysis result display means displays information relating to the area of ​​the plurality of images corresponding to the area accepted by the accepting means as a graph that enables identification of which image among the plurality of images the information relates to.

3. The information processing system according to claim 2, wherein:

9. The image is at least one of an image determined to have changed and the current image.

2. The information processing system according to claim 1, wherein:

10. a setting display means for displaying settings relating to whether or not a change has occurred in an image; an image display means for displaying the image and an area determined to have changed in the image in a distinguishable manner; the image display means, in response to the setting display means accepting a change in the threshold value for the entire image, displays the image and an area determined to have changed in the image on the same screen as the screen accepting the change in the threshold value, in a manner that allows the image and an area determined to have changed in the image to be distinguished based on the changed threshold value; The setting display means displays a similarity of information relating to the region of the image near the display of the threshold value relating to the determination. An information processing system characterized by:

11. a setting display means for displaying settings relating to whether or not a change has occurred in an image; an image display means for displaying the image and a region determined to have changed in the image in a distinguishable manner; an analysis result display means for displaying an analysis result of information relating to the region of the image; the setting display means accepts a change of a threshold value for the entire image that is set for each type of information; The image display means displays, in a distinguishable manner, an area determined to have changed based on the threshold value set by the setting display means and the similarity of information relating to the area of ​​the image. An information processing system characterized by:

12. a setting display step in which a setting display means of the information processing system displays a setting related to a determination that a change has occurred in the image; an image display step in which an image display means of the information processing system displays the image and an area determined to have changed in the image in a distinguishable manner; the setting display step accepts a change of a threshold value set for each type of information related to the image, In response to the setting and displaying step receiving the change of the threshold value, the image displaying step displays the image and a region determined to have changed in the image on the same screen as the screen receiving the change of the threshold value, in a manner that allows the image and a region determined to have changed in the image to be distinguished based on the similarity between the changed threshold value and the information. A method for controlling an information processing system, comprising:

13. a setting display step in which a setting display means of the information processing system displays a setting related to a determination that a change has occurred in the image; an image display step in which an image display means of the information processing system displays the image and an area determined to have changed in the image in a distinguishable manner; the image display step, in response to the setting display step accepting a change in the threshold value for the entire image, displays the image and an area determined to have changed in the image on the same screen as the screen accepting the change in the threshold value, in a manner that allows the image and an area determined to have changed in the image to be distinguished based on the changed threshold value; The setting display step displays a similarity of information related to the region of the image near the display of the threshold value related to the determination. A method for controlling an information processing system, comprising:

14. a setting display step in which a setting display means of the information processing system displays a setting related to a determination that a change has occurred in the image; an image display step in which an image display means of the information processing system displays the image and a region determined to have changed in the image in a distinguishable manner; an analysis result display step in which analysis result display means of the information processing system displays the analysis result of the information related to the region of the image; the setting display step accepts a change of a threshold value related to the entire image that is set for each type of information, The image display step displays an area determined to have changed based on the threshold value set in the setting and display step and the similarity of information related to the area of ​​the image in a distinguishable manner. A method for controlling an information processing system, comprising:

15. 12. A program for causing at least one computer to function as each of the means of the information processing system according to any one of claims 1 to 11.

Images