Information processing system, information processing method, and program

Abstract

To provide a mechanism for improving the accuracy of detecting abnormalities in the vicinity of a grid.SOLUTION: An information processing system is configured to acquire images taken at the same angle of view, divide the acquired images, and determine that there has been a change in the area on the basis of the changes in the corresponding divided areas among the acquired multiple images and output the determined result. The information processing system is configured to divide the images using two or more division methods.SELECTED DRAWING: Figure 25

Description

[Technical field]

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and in particular to a technique suitable for use in detecting an abnormality. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there exists a method for detecting an abnormality in an imaged object based on a change in the image by processing a plurality of images obtained by photographing with a camera or the like.

[0003] Patent Document 1 discloses a technology that clearly displays the changed parts from the perspective of a worker who investigates, inspects, checks for deterioration, and repairs the changed parts of the subject, for example from the perspective of the work position, diagnoses the condition of the changed parts, and investigates whether there are any abnormalities. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2001-250117 A DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0005] Patent Document 1 describes a method of dividing an image into a plurality of regions (blocks) by a grid in the range to be photographed, and judging an abnormality for each region. However, if an abnormality occurs in a portion that straddles the regions (near the grid), the threshold is not exceeded in each region, and all the regions are judged to be normal, resulting in a problem of low accuracy in detecting an abnormality.

[0006] Therefore, an object of the present invention is to provide a mechanism for improving the accuracy of detecting anomalies near a grid. [Means for solving the problem]

[0007] An information processing system comprising: an acquisition means for acquiring images captured at the same angle of view; a division means for dividing the images acquired by the acquisition means; a determination means for determining that a change has occurred in a corresponding area divided by the division means among a plurality of images acquired by the acquisition means, based on a change between the corresponding areas; and an output means for outputting a result of the determination by the determination means, wherein the division means divides the images using two or more division methods. Effect of the Invention

[0008] According to the present invention, it is possible to provide a mechanism for improving the accuracy of detecting anomalies near a grid. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a system configuration diagram of an information processing system 100. [Diagram 2] FIG. 1 is a hardware block diagram of an information processing device 104. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a software configuration. [Figure 4] 13A is a flowchart of a setting process; FIG. 13B is a flowchart of an anomaly detection parameter setting process for a monitoring task; [Diagram 5] 13 is an example of an initial display of a setting screen. [Figure 6] 13 is a display example of an overall image of a setting screen. [Figure 7] 13 is an example of the setting screen (input tab). [Figure 8] 13 is an example of the settings screen (input tab). [Figure 9] 13 is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11] 13 is a display example of a setting screen (monitoring process and monitoring task). [Figure 12] 13 is an example of the settings screen (input tab). [Figure 13]13 is an example of the setting screen (input tab). [Figure 14] This is an example of the settings screen (notification tab). [Figure 15] This is an example of the settings screen (notification tab). [Figure 16] 13 is an example of a setting screen (monitoring task). [Figure 17] 13 is a display example of an analysis setting adjustment screen. [Figure 18] 13 is a display example of a grid setting screen (grid division). [Figure 19] 13 is a display example of a grid setting screen (grid correction). [Figure 20] 13 is a display example of an analysis setting screen. [Figure 21] 13 is an example of a flowchart of an abnormality detection process. [Figure 22] 13 is a display example of task registration information. [Diagram 23] 13 is an example of a flowchart of an image difference (abnormality) detection process. [Figure 24] 11 is an example of a flowchart of each histogram calculation process. [Diagram 25] 13 is a display example of grid correction. [Figure 26] 13 is a display example of a notification during grid correction. [Figure 27] 13 is an example of a flowchart of an abnormality notification process. [Figure 28] 13 is a display example of a setting screen (abnormality notification). [Figure 29] 13 is a display example of an abnormality notification dashboard screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0011] FIG. 1 is a system configuration diagram showing an example of the configuration of an information processing system 100 according to the present invention.

[0012] 1 shows a configuration in which a camera 102 and an information processing device 104 are connected via an image transfer cable (USB, Ethernet, Camera Link, etc.). Note that, instead of providing such a configuration, an information processing device including a camera may also be used.

[0013] The information processing device 104 acquires an image obtained by photographing an object with the camera 102 via an image transfer cable, and executes processing related to discrimination of abnormalities in the image.

[0014] Furthermore, the information processing system 100 of this embodiment may have a configuration in which the camera 102 configured as described above is not present.

[0015] In this case, processing related to discrimination of abnormalities in images is executed for images acquired from other terminals on the network or images stored in the memory (RAM) of the information processing device 104 .

[0016] Hereinafter, a hardware configuration of an information processing device applicable to the information processing device 104 shown in FIG. 1 will be described with reference to FIG.

[0017] FIG. 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in FIG.

[0018] 2, reference numeral 201 denotes a CPU, which performs overall control of each device and controller connected to a system bus 204. In addition, a ROM 202 or an external memory 212 stores a BIOS (Basic Input / Output System), which is a control program for the CPU 201, an operating system program (hereinafter, OS), and various programs (to be described later) required to realize the functions executed by each PC.

[0019] A RAM 203 functions as a main memory, a work area, etc., for the CPU 201. The CPU 201 loads programs and the like required for executing processing from the ROM 202 or the external memory 212 into the RAM 203 and executes the loaded programs to realize various operations.

[0020] An input controller 205 controls input from a keyboard (KB) 210 and a pointing device such as a mouse (not shown).

[0021] Reference numeral 206 denotes a video controller that controls display on a display device such as a display 211 .

[0022] 207 is a memory controller, which connects to an external storage device (hard disk (HD)) that stores various data, a flexible disk (FD), or a PCMCIA card slot. The controller 214 controls access to an external memory 212 such as a CompactFlash memory connected via an adapter.

[0023] Reference numeral 208 denotes a communication I / F controller, which controls reception of image data from an external PC 213 via a network (TCP / IP). An image I / F controller 209 controls the reception of image data from the camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0024] Various programs for implementing the present invention, which will be described later, are recorded in the RAM 203 and executed by the CPU 201 .

[0025] Furthermore, image data used when the above programs are executed is stored in the ROM 202, the external memory 212, the external PC 213, and the camera 102 depending on the application, and is stored in the RAM 203 via various controllers when the programs are executed.

[0026] FIG. 3 is a block diagram showing an example of a software configuration according to the embodiment of the present invention. The information processing device 104 includes the following functional units.

[0027] The acquisition unit 301 is a functional unit that acquires images captured at the same angle of view.

[0028] The receiving unit 302 is a functional unit that receives designation of an area included in an image and a display method for that area.

[0029] The determination unit 303 is a functional unit that determines, based on a change between corresponding areas of the multiple images acquired by the acquisition unit 301, that a change has occurred in the corresponding area.

[0030] The output unit 304 is a functional unit that outputs the result of the determination by the determination unit 303 .

[0031] The output unit 304 is a functional unit that displays the area specified by the receiving unit 302 in the display method specified by the receiving unit 302, and controls whether or not to output the judgment result based on the area specified by the receiving unit 302.

[0032] The area changing unit 305 is a functional unit that changes the area designated by the receiving unit 302 based on the area designated by the receiving unit 302 and the determination result of the area.

[0033] The division unit 306 is a functional unit that divides the image acquired by the acquisition unit 301 .

[0034] The determination unit 303 is a functional unit that determines, based on a change between corresponding areas divided by the division means among the multiple images acquired by the acquisition unit 301, that a change has occurred in the corresponding area.

[0035] The division unit 306 is a functional unit that divides an image using two or more division methods.

[0036] The output unit 304 is a functional unit that outputs a determination result based on the area divided by the first division method when the determination unit 303 determines that a change has occurred in the area divided by the second division method.

[0037] The determination unit 303 is a functional unit that determines whether a change has occurred in a corresponding region of a plurality of images acquired by the acquisition means, based on a change between corresponding regions of the plurality of images and a plurality of types of threshold values.

[0038] The output unit 304 is a functional unit that identifies which of a plurality of thresholds it has been determined that a change has occurred and outputs the result.

[0039] The receiving unit 302 is a functional unit that receives a designation of a method for identifying and outputting each type of threshold.

[0040] The receiving unit 302 is a functional unit that performs control so as not to receive the same designation of the output method for different types of thresholds. This concludes the explanation of FIG.

[0041] The setting process in FIG. 4(a) will now be described.

[0042] In S401, the information processing device 104 judges whether or not a setting file for an operating process exists in the external memory 212. If it is not registered, the process proceeds to S402, and if it is registered, the process proceeds to S403.

[0043] In S402, the information processing apparatus 104 displays a setting screen 510 (FIG. 5), and upon receiving a press of an add monitoring process button 511 from the user, registers the operating process in the RAM 203 and displays a setting screen 520.

[0044] In S403, when the information processing apparatus 104 accepts pressing of the selection button 521 of the operation process to be edited on the setting screen 502, the information processing apparatus 104 displays the setting screen 610 (FIG. 6).

[0045] The setting reception process on the setting screen 610 performed in S403 will be described below with reference to FIGS. When the information processing device 104 receives input of the name 611 and description 612 of the running process from the user, it edits the running process information in the RAM 203 and displays a setting screen 620 .

[0046] When the information processing apparatus 104 receives a press of the input tab 613 on the setting screen 610 from the user, it displays a setting screen 710 (FIG. 7).

[0047] When the information processing device 104 receives a selection of “Load video file (FFmpeg)” (721 in FIG. 7) from the user in the function list 711, it displays a setting screen 720. At this time, based on the values ​​of command location 722 and option 723 input by the user on the setting screen 720, the information processing device 104 edits the operating process information of the RAM 203.

[0048] When the information processing device 104 receives a selection of “Load video file (OpenCV)” (731 in FIG. 7) from the user in the function list 711, the information processing device 104 displays a setting screen 730 and edits the operating process information of the RAM 203.

[0049] When the information processing device 104 receives a selection of "Live video loading (video monitoring server)" (811 in FIG. 8) from the user in the function list 711, it displays a setting screen 810 (FIG. 8). "Live video loading (video monitoring server)" is a function for loading live video captured by a network camera from a video monitoring server (not shown) connectable via a network.

[0050] The operating process information in the RAM 203 is edited based on the values ​​of the computer name 812, the user name 813, and the password 814 input by the user on the setting screen 810. Furthermore, when a connection check 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in the RAM 203.

[0051] When the information processing device 104 receives a press of the analysis tab 614 (FIG. 6) from the user on the setting screen 610, it displays a setting screen 910 (FIG. 9).

[0052] When the information processing device 104 receives a selection of "detection by video comparison" from the user in the function list 911, it displays a setting screen 920 and edits the operating process information in the RAM 203. At this time, if the license has not been authenticated, "No license" is displayed in the license information 922.

[0053] When the information processing apparatus 104 receives a license read 923 press from the user, if a valid license has been authenticated, it displays a setting screen 930, displays license information 931, and edits the operating process information in the RAM 203.

[0054] When the information processing device 104 receives a press of the notification tab 615 (FIG. 6) from the user on the setting screen 610, it displays the setting screen 1010 (FIG. 10).

[0055] When the information processing device 104 receives a selection of “custom command” (1021 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays the setting screen 1020 and edits the operating process information in the RAM 203.

[0056] "Custom Command" is a function that executes commands to execute batches or applications when an abnormality is detected. By using this function, it is possible to execute batch processing, instruct a mailer to send an email, instruct a call using an automated voice call, link with other applications or external systems, etc.

[0057] When the information processing device 104 receives a selection of “Analysis result notification (video monitoring server)” (1031 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays a setting screen 1030.

[0058] "Analysis result notification (video surveillance server)" is a function that notifies the video surveillance server of the analysis result. This function is used when the video surveillance server is configured to notify the user of an abnormality, or when the video surveillance server analyzes and manages events such as abnormalities.

[0059] Based on the values ​​of the address 1032 and the port number 1033 input by the user on the setting screen 1030, the operating process information of the RAM 203 is edited.

[0060] When the information processing device 104 receives a press of a connection confirmation button 1034 from the user, the information processing device 104 performs a connection test using the information in the RAM 203 via the communication I / F controller 208 .

[0061] Returning to the explanation of Figure 4. In S404, when the information processing apparatus 104 accepts pressing of the OK button 631 on the setting screen 630 (FIG. 6), it saves the operating process information registered in the RAM 203 in the external memory 212 as a setting file of the operating process.

[0062] In S405, the information processing apparatus 104 judges whether or not a setting file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406, and if it is registered, the process proceeds to S407.

[0063] In S406, the information processing apparatus 104 displays the setting screen 630, and upon receiving a press of the add button 632 for the task of the monitoring process from the user, registers the monitoring task in the RAM 203 and displays the setting screen 1110 (FIG. 11).

[0064] In S407 , when the information processing apparatus 104 accepts pressing of the monitoring task 1111 to be edited on the setting screen 1110 , it displays the setting screen 1120 . The monitoring task setting acceptance process performed in S407 will be described below with reference to FIGS.

[0065] When the information processing device 104 receives input of a name 1121 and a description 1122 from the user on a setting screen 1120 and a change to a toggle button to enable this task 1123 , it edits the operating process information in the RAM 203 and displays a setting screen 1130 .

[0066] When the information processing device 104 receives a user's press of an input tab 1124 on the setting screen 1120, it displays the setting screen 1200 (FIG. 12) or the setting screen 1300 (FIG. 13) depending on the setting of the operation process.

[0067] When the input setting for the operating process is "Load video file (FFmpeg)" or "Load video file (OpenCV)", a setting screen 1200 (FIG. 12) is displayed.

[0068] When the information processing device 104 receives a user input such as clicking on a video folder 1201 to select the folder, inputting a frame acquisition interval 1202, and changing a toggle button for deleting processed videos 1203, the information processing device 104 edits the operating process information in the RAM 203 and displays a setting screen 1210.

[0069] When the input setting for the operation process is "read live video", a setting screen 1300 (FIG. 13) is displayed. When the information processing device 104 accepts a selection of a list item of the camera 1301 from the user, the information processing device 104 edits the operation process information in the RAM 203 and displays a setting screen 1310.

[0070] When the information processing device 104 receives a press of a notification tab 1125 from the user on the setting screen 1120, the information processing device 104 displays the setting screen 1400 or the setting screen 1500 depending on the setting of the operation process.

[0071] When the notification setting for the operating process is “custom command”, a setting screen 1400 ( FIG. 14 ) is displayed. When the information processing device 104 receives input of a command path 1401 from the user and a change to the toggle button for asynchronous execution 1402, it edits the operating process information in the RAM 203 and displays a setting screen 1410.

[0072] When the notification setting for the operating process is "Analysis result notification (video monitoring server)", a setting screen 1500 (FIG. 15) is displayed. The information processing device 104 accepts input of name 1501 and camera ID 1502 from the user, and displays screen 1510. At this time, if test event button 1511 is accepted by the user, a test notification process is executed to check whether the notification works normally with the input contents. At this time, if the setting for detection frame display toggle button 1512 is enabled, pseudo detection result information for testing is also added to the notification contents.

[0073] Returning to the explanation of Figure 4. In S408, when the information processing apparatus 104 receives a press of an OK button 1602 from the user on the setting screen 1600 (FIG. 16), it saves the monitoring task information registered in the RAM 203 as a monitoring task setting file in the external memory 212 and displays a setting screen 1610.

[0074] In S409, when the user presses the analysis tab 1611 on the setting screen 1610, the setting screen 1700 (FIG. 17) is displayed. At this time, when the user presses the adjustment button 1701, an adjustment screen 1710 for analysis settings is displayed. After that, the information processing device 104 executes the process shown in FIG.

[0075] The abnormality detection parameter setting process of the monitoring task in FIG. 4(b) will be described.

[0076] In S421, when the information processing apparatus 104 receives a press of the basic settings button 1711 on the analysis settings adjustment screen 1710 (FIG. 17) from the user, it displays the basic settings screen 1720 for analysis settings and proceeds to S422. If the basic setting button has not been pressed, the process proceeds to S428.

[0077] In S422, when the information processing device 104 receives a change to the slide bars for the number of rows 1801 (FIG. 18) and the number of columns 1802 from the user, it updates the display of the grid (dashed lines) on the screen divided by the number of rows 1801 and the number of columns 1802, as shown on screen 1800 (FIG. 18), and saves the setting contents in RAM 203.

[0078] Specifically, on the screen 1800, the number of rows 1801=10 and the number of columns 1802=10 are set, so that the image to be analyzed is displayed divided into a grid of 10 rows and 10 columns.

[0079] It should be noted that the more grids there are, the more pinpoint detection becomes possible; however, this increases the number of processes, such as the image difference (anomaly) detection process (FIG. 23) described below, which may result in increased processing time.

[0080] In S423, when the information processing device 104 receives a user's press 1811 of any area divided by a grid on the screen, a press of a set all mask button 1812, or a press of a release all mask button 1813, the information processing device 104 displays the presence or absence of a mask (notification suppression area) as shown on a screen 1810, and saves the setting contents in the RAM 203. That is, this step is a step showing an example of a process for receiving a designation of an area included in an image.

[0081] When there are many masked regions, the region to be masked can be easily set by first masking all the regions with the mask all setting button 1812 and then specifying the region to be unmasked.

[0082] When the masked area is small, the area to be masked can be easily set by releasing the mask for all areas with the all mask release button 1813 and then specifying the area to be masked.

[0083] Furthermore, at this time, when a change in the toggle button of mask color inversion 1821 by the user is accepted, the mask drawing color is changed as shown on screen 1820. Specifically, the mask area is represented in transparent black on screen 1810, but is represented in transparent white on screen 1820. That is, this step is a step showing an example of a process for accepting a designation of a display method for an area included in an image. Also, this step is a step showing an example of a process for displaying the area, the designation of which has been accepted, in the display method, the designation of which has been accepted.

[0084] This allows the user to select the color of the mask area according to the color tone and brightness of the image to be analyzed by representing the mask area in a transparent black color when the image to be analyzed is bright, and in a transparent white color when the image to be analyzed is dark. This makes it easy to identify the mask area while also understanding its appearance.

[0085] In this embodiment, the color of the mask area is "transparent black" or "transparent white," but this is not limited to this method. Other colors may be used, the user may specify a color, the display may be varied using patterns such as diagonal lines, the area may be surrounded by a frame, or other methods may be used to distinguish between the monitored area and the mask area.

[0086] In S424, when the information processing device 104 receives a change from the user to the grid correction (shifting the grid) toggle button 1901 (Figure 19) or 1911, it updates the display of the frame 1902 (with frame) or 1912 (without frame) indicating the detection range and saves the setting contents in the RAM 203.

[0087] In this embodiment, grid correction is set for the monitoring process, but this is not limited to this method. Grid correction may be set directly for the network camera or video surveillance server that is the image input source.

[0088] In S425 , when the information processing apparatus 104 accepts a numerical input 1921 of the reference number of image frames from the user, the information processing apparatus 104 stores the setting content in the RAM 203 .

[0089] In S426, when the information processing device 104 receives a press of the back button 1803 from the user while displaying the basic setting screen 1800 (FIG. 18) for analysis settings, it determines whether or not the basic settings have been changed. If there has been a change, the process proceeds to S427. If there has not been a change, the process proceeds to S428.

[0090] In S427, the information processing device 104 reflects the contents set in the RAM 203 in the setting file of the monitoring task in the external memory 212. In addition, the information processing device 104 detects a change in the setting file of the monitoring task, and discards the reference image in the analysis process. (In S2306, it is determined that the base information needs to be updated.) In S428, when the information processing apparatus 104 accepts a change to the slide bar of each item of the threshold 2001 on the analysis setting screen 2000 (FIG. 20), the information processing apparatus 104 saves the setting contents in the RAM 203.

[0091] In this embodiment, the user sets each threshold value, but this method is not limited to this. The user may select a data set for each threshold value, or the user may set the object to be monitored or the abnormality to be detected and automatically set the threshold parameters, or the information processing device 104 may determine the object to be monitored or the abnormality to be detected and automatically set the threshold parameters, etc.

[0092] This makes it possible to easily set the threshold value for detecting an abnormality.

[0093] In S429, when the information processing device 104 receives a press of the apply button 2002 from the user, it reflects the contents set in the RAM 203 in the setting file of the monitoring task in the external memory 212. In addition, the information processing device 104 detects a change in the setting file of the monitoring task, and changes the parameter values ​​used in the detection process (changes the threshold value used in S2310).

[0094] The abnormality detection process in FIG. 21 will be described. In S2101, if there is a setting file for a monitoring task in the external memory 212 (task registration information 2201 (FIG. 22)) and the toggle 2202 for the task activation setting is set to active, the information processing apparatus 104 executes anomaly detection processing.

[0095] In S2102, the information processing device 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and the process is repeated.

[0096] In S2103, the information processing apparatus 104 acquires input information according to the set input method (set on the setting screen 1200 or the setting screen 1300).

[0097] Specifically, when the input setting for the operation process (setting screen 1200) is “Load video file (FFmpeg)” or “Load video file (OpenCV)”, the video file placed in the specified video folder is moved to the working area on the external memory 212 as input information.

[0098] If the input setting for the operation process is "Live video loading (video monitoring server)" (setting screen 1300), video acquisition processing is executed for the designated video acquisition destination, and if a still image is acquired, the process proceeds to S2106.

[0099] In S2104, if the information processing apparatus 104 has moved the moving image file on the external memory 212 in S2103, the process proceeds to S2105. If the information processing apparatus 104 has executed a video acquisition process and acquired a still image, the process proceeds to S2106.

[0100] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into consecutive still images for each frame, and stores it as a file on the external memory 212 or as data on the RAM 203, and proceeds to S2106.

[0101] In S2106, the information processing device 104 executes the process shown in FIG.

[0102] The image difference (abnormality) detection process of FIG. 23 will be described. In S2301, if the information processing device 104 has divided the moving image file into successive still images for each frame (S2105), the information processing device 104 repeats the subsequent processes until the processes are completed for all the generated still images.

[0103] In S2302, the information processing device 104 reads a still image to be processed and develops it on the RAM 203. That is, this step is a step showing an example of a process of acquiring images captured at the same angle of view.

[0104] In S2303, the information processing apparatus 104 calculates information on hue, saturation, and lightness from the image data expanded on the RAM 203 by calculation.

[0105] In S2304, the information processing device 104 performs each histogram calculation process (FIG. 24) to calculate a histogram from the image data.

[0106] Here, the histogram calculation process in FIG. 24 will be described. In S2401, the information processing apparatus 104 proceeds to S2402 if the grid correction set in S424 among the settings stored in the RAM 203 is valid, or proceeds to S2403 if it is invalid.

[0107] In S2402, the information processing device 104 sets, in addition to the area divided by the grid 2500 (FIG. 25) set in S422, each area divided by a grid 2501 shifted by half the size of one area in the horizontal and vertical directions as a target area for the histogram calculation process. That is, this step is an example of a process of dividing an acquired image.

[0108] Specifically, as shown in 2510 and 2520, if an abnormality occurs near the 3 rows x 4 columns grid set in S422, since the abnormality spans multiple regions, none of the regions will exceed the threshold value, and all of the regions will be determined to be normal, which may result in low accuracy in detecting an abnormality.

[0109] As in S2402, by determining an abnormality not only in the area defined by the normal grid 2500 set in S422 but also in the area defined by the corrected grid 2501 (shifted grid), it becomes possible to easily improve the accuracy of detecting an abnormality even when an abnormality occurs near the normal grid 2500 set in S422.

[0110] In this embodiment, the grid is shifted horizontally and vertically by half the size of one region, but the present invention is not limited to this method. Two new grids may be created that are shifted horizontally and vertically by 1 / 3 and 2 / 3 of the size of one region, or a new hexagonal grid may be created instead of a lattice (square) shape. That is, this step is a step showing an example of a process of dividing an image using two or more division methods. That is, this is an example in which an area divided by a first division method partially overlaps at least one of the areas divided by a second division method.

[0111] This makes it possible to easily improve the accuracy of detecting anomalies even when the anomalies occur near the grid 2500.

[0112] In S2403, the information processing apparatus 104 calculates a histogram of luminance information from the luminance (Lightness) information acquired in S2303 for each of the processing target areas set in S422 and S2402.

[0113] In S2404, the information processing apparatus 104 calculates a histogram of hue information from the hue and saturation information acquired in S2303 for each processing target area set in S422 and S2402.

[0114] In S2405, the information processing apparatus 104 calculates a histogram of saturation information from the saturation information acquired in S2303 for each of the processing target areas set in S422 and S2402.

[0115] In S2406, the information processing device 104 calculates edge gradient information from the luminance (Lightness) information acquired in S2303 for each processing target area set in S422 and S2402, and calculates an edge gradient intensity histogram and an edge gradient angle histogram using the edge gradient information.

[0116] Returning to the explanation of Figure 23. In S2305, the information processing apparatus 104 calculates a similarity degree for each processing target area set in S422 and S2402, using a histogram of the reference information and a histogram intersection, which is one of the methods for comparing histograms.

[0117] At this time, the reference information is a histogram calculated in S2308, which will be described later. If there is no reference histogram data, the similarity is set to 100%.

[0118] In this embodiment, the histogram comparison method uses histogram intersection, but the method is not limited to this, and other methods such as Bhattacharyya coefficient may be used.

[0119] In addition, in this embodiment, anomalies are detected by comparing histograms, but this is not limited to this method, and anomalies may be detected using other methods, such as template matching or feature point detection.

[0120] In S2306, if the number of data items already set as the reference image does not satisfy the "reference image frame count" set in S425, the information processing device 104 determines that the reference image needs to be updated and proceeds to S2307. If the number of data items already set as the reference image satisfies the "reference image frame count", the information processing device 104 proceeds to S2309.

[0121] The information processing device 104 discards the reference image and initializes the number of data items that has already been set to 0 in the following cases.

[0122] When the basic settings are changed by the user of the S427 When the user clicks the Recreate Reference Image button 2003 This allows the user to easily instruct the re-creation of the reference image.

[0123] In S2307, the information processing device 104 determines whether the data is suitable as a reference image based on the similarity calculated in S2305. If there are no areas with low similarity, the process proceeds to S2308. If there are many areas with low similarity, the data is determined to be unsuitable as a reference image and the process ends.

[0124] In S2308, the information processing device 104 adds the histogram calculated in S2304 to the histogram of the reference information currently held, recalculates the average value, updates the histogram of the reference information, and ends the process.

[0125] In S2309, the information processing device 104 compares the similarity for each processing target region set in S422 calculated in S2305 with the threshold set in S428, and records a region below the threshold as an abnormal region on the RAM 203. That is, this step is a step showing an example of a process for determining that a change has occurred in a region based on a change between corresponding regions of multiple images among the multiple acquired images.

[0126] Specifically, in the threshold 2001 (FIG. 20), when the brightness threshold is set to 22%, if the brightness similarity of the processing target area is less than 22%, the area is determined to be abnormal.

[0127] In S2310, the information processing device 104 compares the similarity for each area (area separated by the correction grid) set as the area to be processed in S2402 calculated in S2305 with the threshold set in S428, and detects areas below the threshold.

[0128] At this time, if an abnormality is detected, the process proceeds to S2311. If no abnormality is detected, the process proceeds to S2312.

[0129] In S2311, since the area 2600 (FIG. 26) in which the abnormality was detected is different from the area to be notified to the user set in S422, the information processing device 104 determines (2620) that all areas 2610 (four areas in the case of FIG. 26) that overlap with the detection area 2600 as having detected an abnormality, and records these as abnormal locations on the RAM 203.

[0130] As a result, the area for which an abnormality is notified is the area divided by the normal grid 2500 set in S422, regardless of whether grid correction is performed or not. That is, this step is an example of a process for outputting a determination result based on the area divided by the first division method when it is determined that a change has occurred in the area divided by the second division method.

[0131] In this way, it is possible to notify users and external applications of abnormalities based on areas that are always separated by a constant grid. In other words, because the notification method is constant regardless of whether grid correction is performed or not, it has the advantage that the notification is easy for users to understand, and for external applications, the method of receiving abnormality notifications does not need to be complicated, so the reception logic and method of outputting abnormalities can be constant.

[0132] In S2312, the information processing device 104 compares the abnormal area recorded on the RAM with the mask area set in S423, and if there is an abnormal area other than the mask area, proceeds to S2313. If there is no abnormal area or if all of the abnormal areas are mask areas, the process ends. That is, this step is an example of a process for controlling whether or not to output the determination result based on the area for which designation has been accepted.

[0133] In this embodiment, the histogram calculation process (S2304) is performed even in a masked area, but this is not limited to this method, and the histogram calculation process (S2304) may not be performed for a masked area. By doing so, the number of times each histogram calculation process is performed is reduced when there is a masked area, and the overall processing time can be shortened.

[0134] On the other hand, in the case of this embodiment, when each histogram calculation process is performed even in a masked area, there are advantages such as being able to record abnormalities in the masked area and suggest changes to the masked area to the user, and being able to stably meet non-functional requirements by always keeping the processing speed constant.

[0135] In S2313, the information processing device 104 performs an abnormality notification process (FIG. 27) and notifies the user and an external application of information about the abnormal area.

[0136] The abnormality notification process in FIG. 27 will be described. In S2701, the information processing device 104 determines whether there is an abnormality to be notified based on criteria such as the continuity of abnormality detection. If it is determined that there is an abnormality to be notified, the process proceeds to S2702. If it is determined that there is no abnormality to be notified, the process proceeds to S2703.

[0137] Specifically, a criterion is needed to determine whether an abnormality should be notified if an abnormality is detected in only one image frame, or if an abnormality is detected continuously in images spanning several seconds.

[0138] For example, the information processing device 104 may accept the user's setting of criteria for notifying an abnormality in advance, such as the number of consecutive frames an abnormality must be detected before notifying the user, or may determine what the image to be analyzed is (a factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and based on the object or environment, determine the criteria for notifying an abnormality.

[0139] In S2702, the information processing apparatus 104 records in the RAM 203 area information in which the abnormality to be notified occurs.

[0140] In S2703, the information processing device 104 deletes the abnormality information to be notified that is recorded on the RAM 203.

[0141] In S2710, the information processing device 104 passes the still image information read in S2302 to a processing unit that draws the setting screen.

[0142] In S2711, the information processing apparatus 104 passes the abnormality information recorded in S2702 to the processing unit that renders the setting screen.

[0143] In S2712, the information processing apparatus 104 determines whether or not the setting screen is being drawn by a user operation. If the setting screen is being drawn, the process proceeds to S2713. If the setting screen is not being drawn, the process ends.

[0144] In S2713, the information processing apparatus 104 checks the setting contents 2801 of the detection area drawing color in the RAM 203. If there is a designation regarding the drawing method, the process proceeds to S2714. If there is no designation, the process proceeds to S2715.

[0145] In S2714, the information processing device 104 changes the highlighting during the drawing process in S2715, which will be described later, based on the setting contents 2801 (FIG. 28). That is, this step is an example of a process of identifying and outputting which of a plurality of thresholds is used to determine that a change has occurred.

[0146] The setting content 2801 is a setting of how to highlight depending on the detection factor of the anomaly detection (item that fell below the threshold in S2309). That is, this step is an example of a process of accepting a designation of a method of identifying and outputting for each type of threshold.

[0147] Here, control is performed so that the same highlighting settings do not overlap, so that the highlighting for each threshold can be distinguished (for example, an error message is displayed when the same highlighting is selected for different thresholds, a highlighting that has already been selected cannot be newly selected, etc.). In other words, this step is an example of a process of controlling so as not to accept the same output method specification for different threshold types.

[0148] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0149] In this embodiment, the highlighting is done using color, but the present invention is not limited to this method. The highlighting method can be set to use a pattern such as diagonal lines, to surround the content with a frame such as dotted or thick lines, or to set different notification methods such as voice or email. Alternatively, the notification method can be set not only by type of highlighting but also by methods other than highlighting.

[0150] In S2715, the information processing apparatus 104 develops an image for drawing in the drawing memory on the RAM 203 by superimposing highlighting on the still image information acquired in S2710 and the area containing the abnormality information acquired in S2711.

[0151] In S2716, the information processing device 104 draws the drawing image created in S2715 on the setting screen 2810 (FIG. 28). That is, this step is a step showing an example of a process of outputting a determination result as to whether or not there has been a change in the area.

[0152] At this time, if the rendering method has been changed in S2714, the method of highlighting the abnormal area is changed and displayed as shown by abnormal areas 2811(1), (2), and (4).

[0153] This allows the user to easily know which threshold value was used to determine the abnormality.

[0154] The highlighting of the abnormal region and the display of the masked region may be performed on the same screen (not shown).

[0155] Also, although in S2312 it was stated that the anomaly region is not displayed in the masked region, it is also possible to highlight the anomaly detection region (not shown) / highlight it differently (not shown) in the masked region upon receiving an instruction from the user, etc. By doing so, even if the region has been designated as a masked region, this can trigger the user to decide that the mask should be removed depending on the content of the anomaly detection. In other words, it becomes possible for the user to easily set the anomaly detection desired.

[0156] Furthermore, the information processing device 104 may have a function of notifying on the screen 1810 or the screen 1820 what kind of abnormality has occurred in the masked region, without receiving an instruction from the user.

[0157] Also, there may be a function to suggest to the user to change the mask setting based on the number of times or contents of anomaly detection, or to have the information processing device 104 automatically change the mask setting. This makes it possible to perform anomaly detection with higher accuracy. That is, this step is a step showing an example of a process of changing the area whose specification has been accepted based on the area whose specification has been accepted and the determination result of the area.

[0158] In S2720, the information processing device 104 notifies the abnormality information recorded in S2702 to the output destination set in S407 (for example, a video monitoring server, a mailer, software capable of instructing a call using an automated voice call, etc.).

[0159] Figure 29 shows an example of the anomaly notification dashboard screen, which displays a table listing all anomalies that have been detected so far. Here, the detection time, monitoring task name, and coordinates of the detected area are displayed.

[0160] In this way, the user can easily know which threshold value was used to determine whether an abnormality occurred. In addition, the accuracy of detecting anomalies near the grid can be improved. It also makes it easy to know what the basis for detecting an abnormality was.

[0161] As described above, it goes without saying that the object of the present invention can be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0162] In this case, the program itself read out from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0163] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, etc.

[0164] Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by the computer executing a program it has read, but also includes cases where an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0165] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0166] The present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. Needless to say, the present invention can also be applied to a case where the object is achieved by supplying a program to a system or apparatus. In this case, the effect of the present invention can be enjoyed by the system or apparatus by reading a recording medium storing a program for achieving the present invention into the system or apparatus.

[0167] The program may be in the form of object code, program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

[0168] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0169] 100 Information Processing Systems 102 Camera 104 Information processing equipment

Claims

1. An acquisition means for acquiring a captured image; A first segmentation means for segmenting the image acquired by the acquisition means into a plurality of first segmentation regions by a first segmentation method; A second segmentation means for segmenting the image acquired by the acquisition means into a plurality of second segmentation regions by a second segmentation method; A first output means for identifying and outputting a first segmentation region determined to have a predetermined change among the plurality of first segmentation regions obtained by dividing the plurality of images acquired by the acquisition means by the first segmentation means; A second output means for identifying and outputting a first segmentation region corresponding to a second segmentation region determined to have a predetermined change among the plurality of second segmentation regions obtained by dividing the plurality of images acquired by the acquisition means by the second segmentation means; An information processing system comprising the above.

2. The first segmentation region corresponding to the second segmentation region is A first segmentation region in which at least a part of the second segmentation region overlaps. The information processing system according to claim 1, characterized by the above.

3. Further comprising a second setting acquisition means for acquiring a setting for whether or not to perform output by the second output means, When the setting acquired by the second setting acquisition means is a setting for performing output by the second output means, perform output by the second output means, When the setting acquired by the second setting acquisition means is a setting for not performing output by the second output means, do not perform output by the second output means. The information processing system according to claim 1, characterized by the above.

4. The second segmentation region is each region obtained by dividing the image by a segmentation method different from the first segmentation method. The information processing system according to claim 1, characterized by the above.

5. The first segmentation region is each region obtained by dividing the image into a plurality of rectangles, The second segmentation region is each region obtained by dividing the image into a plurality of rectangles by a segmentation method different from the first segmentation method. The information processing system according to claim 1, characterized by the above.

6. The predetermined change is The similarity of the information related to the image of the corresponding region is less than a predetermined threshold. The information processing system according to claim 1, characterized by the above.

7. Further comprising a threshold setting reception means for receiving a setting of the threshold. The information processing system according to claim 6, characterized by the above.

8. The threshold is a value related to at least one of edge gradient, hue, saturation, and luminance. The information processing system according to claim 6, characterized in that

9. An acquisition step of acquiring a captured image; A first division step of dividing the image acquired in the acquisition step into a plurality of first division regions by a first division method; A second division step of dividing the image acquired in the acquisition step into a plurality of second division regions by a second division method; A first output step of identifying and outputting a first division region determined to have a predetermined change among the plurality of first division regions obtained by dividing the plurality of images acquired in the acquisition step by the first division step; A second output step of identifying and outputting the first division region corresponding to the second division region determined to have a predetermined change among the plurality of second division regions obtained by dividing the plurality of images acquired in the acquisition step by the second division step A control method for an information processing system, characterized by comprising the above.

10. At least one computer A program for causing the computer to function as each means of the information processing system according to any one of claims 1 to 8.

Images