Information processing system, information processing method and program

Abstract

To provide a mechanism for setting an area to be masked appropriately.SOLUTION: An information processing system is configured to obtain images taken at the same angle of view, accept specifications for an area contained in the images and a display method for the area, determine that there has been a change in the area on the basis of the change in the corresponding area of multiple images among the acquired multiple images and output the determination result, display the area for which the specification has been accepted using the display method for which the specification has been accepted, and control whether to output the determination results on the basis of the area for which the specification has been accepted.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and in particular to a technique suitable for use in detecting an abnormality. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there exists a method for detecting abnormalities in an imaged object based on changes in the images by processing a plurality of images obtained by taking images using a camera or the like.

[0003] Patent Document 1 discloses a technology that displays the changed parts in an easy-to-understand manner from the viewpoint of a worker who investigates, inspects, checks for deterioration, and repairs the changed parts of the subject, for example, from the viewpoint of the work position, diagnoses the condition of the changed parts, and investigates whether there are any abnormalities. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-10630 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0005] Patent Document 1 describes a method of specifying an image area within a subject range where changes in the image are originally expected to occur using a mask pattern to exclude it. However, when specifying a mask pattern, there is a problem in that it is necessary to clearly show to the user which area has been specified as a mask.

[0006] Therefore, an object of the present invention is to provide a mechanism that allows an area to be masked to be suitably set. [Means for solving the problem]

[0007] an acquisition means for acquiring images captured at the same angle of view; a receiving means for receiving designation of an area included in the image and a display method for the area; a determination means for determining, based on a change between corresponding regions of the plurality of images acquired by the acquisition means, that a change has occurred in the corresponding region; an output means for outputting the determination result by the determination means; Equipped with The output means displays the area designated by the reception means in a display method designated by the reception means, and controls whether or not to output the determination result based on the area designated by the reception means. An information processing system characterized by: [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a mechanism that allows an area to be masked to be suitably set. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a system configuration diagram of an information processing system 100. [Figure 2] FIG. 1 is a hardware block diagram of an information processing device 104. [Figure 3] FIG. 2 is a block diagram illustrating an example of a software configuration. [Figure 4] 10A is a flowchart of a setting process; FIG. 10B is a flowchart of an anomaly detection parameter setting process for a monitoring task; [Figure 5] 10 is an example of an initial display of a setting screen. [Figure 6] 10 is a display example of the overall image of a setting screen. [Figure 7] This is an example of the settings screen (input tab). [Figure 8] This is an example of the settings screen (input tab). [Figure 9] This is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11]10 is a display example of a setting screen (monitoring process and monitoring task). [Figure 12] This is an example of the settings screen (input tab). [Figure 13] This is an example of the settings screen (input tab). [Figure 14] This is an example of the settings screen (notification tab). [Figure 15] This is an example of the settings screen (notification tab). [Figure 16] 10 is a display example of a setting screen (monitoring task). [Figure 17] 10 is a display example of an analysis setting adjustment screen. [Figure 18] 10 is a display example of a grid setting screen (grid division). [Figure 19] 10 is a display example of a grid setting screen (grid correction). [Figure 20] 10 is a display example of an analysis setting screen. [Figure 21] 10 is an example of a flowchart of an abnormality detection process. [Figure 22] 10 is a display example of task registration information. [Figure 23] 10 is an example of a flowchart of an image difference (abnormality) detection process. [Figure 24] 10 is an example of a flowchart of a histogram calculation process. [Figure 25] 10 is a display example of grid correction. [Figure 26] 10 is a display example of a notification when grid correction is performed. [Figure 27] 10 is an example of a flowchart of an abnormality notification process. [Figure 28] 10 is a display example of a setting screen (abnormality notification). [Figure 29] 10 is a display example of an abnormality notification dashboard screen. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] FIG. 1 is a system configuration diagram showing an example of the configuration of an information processing system 100 according to the present invention.

[0012] 1 shows a configuration in which a camera 102 and an information processing device 104 are connected via an image transfer cable (USB, Ethernet, Camera Link, etc.). However, instead of having such a configuration, an information processing device having a camera may also be used.

[0013] The information processing device 104 acquires an image obtained by photographing an object with the camera 102 via an image transfer cable, and executes processing related to identifying abnormalities in the image.

[0014] Furthermore, the information processing system 100 of this embodiment may be configured without the camera 102 configured as described above.

[0015] In this case, the image processing device 104 executes processing related to determining whether there is an abnormality in an image, for an image acquired from another terminal on the network or an image stored in the memory (RAM) of the information processing device 104.

[0016] Hereinafter, the hardware configuration of an information processing device applicable to the information processing device 104 shown in FIG. 1 will be described with reference to FIG.

[0017] FIG. 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in FIG.

[0018] 2, reference numeral 201 denotes a CPU, which performs overall control of each device and controller connected to a system bus 204. In addition, a ROM 202 or an external memory 212 stores a BIOS (Basic Input / Output System), which is a control program for the CPU 201, an operating system program (hereinafter referred to as OS), and various programs (described later) required to realize the functions executed by each PC.

[0019] A RAM 203 functions as the main memory, work area, etc. of the CPU 201. The CPU 201 loads programs and the like required for executing processing from the ROM 202 or the external memory 212 into the RAM 203 and executes the loaded programs to realize various operations.

[0020] An input controller 205 controls input from a keyboard (KB) 210 and a pointing device such as a mouse (not shown).

[0021] A video controller 206 controls the display on a display device such as a display 211 .

[0022] 207 is a memory controller, which connects to an external storage device (hard disk (HD)) that stores various data, a flexible disk (FD), or a PCMCIA card slot. It controls access to an external memory 212 such as a CompactFlash (registered trademark) memory connected via an adapter.

[0023] A communication I / F controller 208 controls the reception of image data from an external PC 213 via a network (TCP / IP). An image I / F controller 209 controls the reception of image data from the camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0024] Various programs described below for implementing the present invention are recorded in the RAM 203 and executed by the CPU 201.

[0025] Furthermore, image data used when the above programs are executed is stored in the ROM 202, external memory 212, external PC 213, and camera 102 depending on the application, and is stored in the RAM 203 via various controllers when the programs are executed.

[0026] FIG. 3 is a block diagram showing an example of a software configuration according to an embodiment of the present invention. The information processing device 104 includes the following functional units.

[0027] The acquisition unit 301 is a functional unit that acquires images captured at the same angle of view.

[0028] The receiving unit 302 is a functional unit that receives designation of an area included in an image and a display method for that area.

[0029] The determination unit 303 is a functional unit that determines, based on a change between corresponding areas of the multiple images acquired by the acquisition unit 301, that a change has occurred in the corresponding area.

[0030] The output unit 304 is a functional unit that outputs the determination result from the determination unit 303 .

[0031] The output unit 304 is a functional unit that displays the area specified by the receiving unit 302 in the display method specified by the receiving unit 302, and controls whether or not to output the judgment result based on the area specified by the receiving unit 302.

[0032] The area change unit 305 is a functional unit that changes the area whose designation has been accepted by the acceptance unit 302, based on the area whose designation has been accepted by the acceptance unit 302 and the determination result of the area.

[0033] The dividing unit 306 is a functional unit that divides the image acquired by the acquiring unit 301 .

[0034] The determining unit 303 is a functional unit that determines, based on a change between corresponding areas divided by the dividing means among the multiple images acquired by the acquiring unit 301, that a change has occurred in the area.

[0035] The division unit 306 is a functional unit that divides an image using two or more division methods.

[0036] The output unit 304 is a functional unit that outputs a determination result based on the area divided by the first division method when the determination unit 303 determines that there has been a change in the area divided by the second division method.

[0037] The determination unit 303 is a functional unit that determines whether a change has occurred in a corresponding region of a plurality of images acquired by the acquisition unit, based on the change between corresponding regions of the plurality of images and a plurality of thresholds.

[0038] The output unit 304 is a functional unit that identifies which of a plurality of thresholds was used to determine that a change occurred, and outputs the result.

[0039] The receiving unit 302 is a functional unit that receives a designation of a method for identifying and outputting each type of threshold.

[0040] The receiving unit 302 is a functional unit that controls so as not to receive the same output method designation for different types of thresholds. This concludes the explanation of FIG.

[0041] The setting process in FIG. 4(a) will be described.

[0042] In S401, the information processing apparatus 104 determines whether or not a setting file for the operating process exists in the external memory 212. If it is not registered, the process proceeds to S402, and if it is registered, the process proceeds to S403.

[0043] In S402, the information processing apparatus 104 displays the setting screen 510 (FIG. 5), and upon receiving a press of the add monitoring process button 511 from the user, registers the operating process in the RAM 203 and displays the setting screen 520.

[0044] In S403, when the information processing apparatus 104 receives a press of the selection button 521 of the operation process to be edited on the setting screen 502, it displays the setting screen 610 (FIG. 6).

[0045] The setting reception process on the setting screen 610 performed in S403 will be described below with reference to FIGS. When the information processing device 104 receives input of the name 611 and description 612 of the operating process from the user, it edits the operating process information in the RAM 203 and displays the setting screen 620 .

[0046] When the information processing apparatus 104 receives a user's press of the input tab 613 on the setting screen 610, it displays a setting screen 710 (FIG. 7).

[0047] When the information processing device 104 receives a selection of "Load video file (FFmpeg)" (721 in FIG. 7) from the user in the function list 711, it displays a setting screen 720. At this time, the information processing device 104 edits the operating process information in the RAM 203 based on the values ​​of the command location 722 and option 723 input by the user on the setting screen 720.

[0048] When the information processing device 104 receives a selection of "Read video file (OpenCV)" (731 in FIG. 7) from the user in the function list 711, it displays a setting screen 730 and edits the operating process information of the RAM 203.

[0049] When the information processing device 104 receives a selection of "Load live video (video monitoring server)" (811 in FIG. 8) from the user in the function list 711, it displays a setting screen 810 (FIG. 8). "Load live video (video monitoring server)" is a function that loads live video captured by a network camera from a video monitoring server (not shown) that can be connected via a network.

[0050] The operating process information in RAM 203 is edited based on the values ​​of computer name 812, user name 813, and password 814 input by the user on setting screen 810. Furthermore, when connection confirmation 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in RAM 203.

[0051] When the information processing device 104 receives a user's press of the analysis tab 614 (FIG. 6) on the setting screen 610, it displays a setting screen 910 (FIG. 9).

[0052] When the information processing device 104 receives a selection of "detection by video comparison" from the user in the function list 911, it displays a setting screen 920 and edits the operating process information in the RAM 203. At this time, if the license has not been authenticated, the license information 922 displays "No license."

[0053] When the information processing apparatus 104 receives a license read 923 press from the user, if a valid license has been authenticated, it displays a setting screen 930, displays license information 931, and edits the operating process information in the RAM 203.

[0054] When the information processing device 104 receives a press of the notification tab 615 (FIG. 6) from the user on the setting screen 610, it displays the setting screen 1010 (FIG. 10).

[0055] When the information processing device 104 receives a selection of "custom command" (1021 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays the setting screen 1020 and edits the operating process information in the RAM 203.

[0056] "Custom Command" is a function that executes commands to execute batches or applications when an abnormality is detected. By using this function, it is possible to execute batch processing, instruct emails to be sent to an email client, instruct calls using an automated voice call, and link with other applications or external systems.

[0057] When the information processing device 104 receives a selection of "Analysis result notification (video monitoring server)" (1031 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays a setting screen 1030.

[0058] "Analysis Result Notification (Video Surveillance Server)" is a function that notifies the video surveillance server of the analysis results. This function is used when the video surveillance server is configured to notify users of abnormalities, or when analyzing and managing events such as abnormalities on the video surveillance server.

[0059] Based on the values ​​of the address 1032 and port number 1033 input by the user on the setting screen 1030, the operating process information of the RAM 203 is edited.

[0060] When the information processing device 104 receives a press of the connection confirmation button 1034 from the user, it performs a connection test via the communication I / F controller 208 using the information in the RAM 203 .

[0061] Returning to the explanation of Figure 4. In S404, when the information processing apparatus 104 receives a press of the OK button 631 on the setting screen 630 (FIG. 6), it saves the operating process information registered in the RAM 203 in the external memory 212 as an operating process setting file.

[0062] In S405, the information processing apparatus 104 determines whether a setting file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406, and if it is registered, the process proceeds to S407.

[0063] In S406, the information processing apparatus 104 displays the setting screen 630, and upon receiving a press of the add button 632 for the monitoring process task from the user, registers the monitoring task in the RAM 203 and displays the setting screen 1110 (FIG. 11).

[0064] In S407 , when the information processing apparatus 104 receives a press on the monitoring task 1111 to be edited on the setting screen 1110 , it displays the setting screen 1120 . The monitoring task setting acceptance process performed in S407 will be described below with reference to FIGS.

[0065] When the information processing device 104 receives input of a name 1121 and a description 1122 from the user on a setting screen 1120 and a change to a toggle button for enabling this task 1123, it edits the operating process information in the RAM 203 and displays a setting screen 1130.

[0066] When the information processing device 104 receives a user's press of the input tab 1124 on the setting screen 1120, it displays the setting screen 1200 (FIG. 12) or the setting screen 1300 (FIG. 13) depending on the setting of the operation process.

[0067] If the input setting for the operating process is "Read video file (FFmpeg)" or "Read video file (OpenCV)", the setting screen 1200 (FIG. 12) is displayed.

[0068] When the information processing device 104 receives a user's click on the video folder 1201, selection of the folder, input of the frame acquisition interval 1202, or change of the toggle button for deleting processed video 1203, it edits the operating process information in the RAM 203 and displays the setting screen 1210.

[0069] When the input setting for the operation process is "load live video," the setting screen 1300 (FIG. 13) is displayed. When the information processing device 104 receives a selection of a list item for the camera 1301 from the user, it edits the operation process information in the RAM 203 and displays the setting screen 1310.

[0070] When the information processing device 104 receives a press of the notification tab 1125 from the user on the setting screen 1120, it displays the setting screen 1400 or the setting screen 1500 depending on the setting of the operation process.

[0071] If the notification setting for the operating process is "custom command," a setting screen 1400 (FIG. 14) is displayed. When the information processing device 104 receives input of a command path 1401 from the user and a change to the toggle button for asynchronous execution 1402, it edits the operating process information in the RAM 203 and displays a setting screen 1410.

[0072] When the notification setting for the operating process is "Analysis result notification (video monitoring server)," a setting screen 1500 (FIG. 15) is displayed. The information processing device 104 accepts input of a name 1501 and a camera ID 1502 from the user, and displays a screen 1510. At this time, if the user presses a test event button 1511, a test notification process is executed to check whether the notification works correctly with the input content. At this time, if the setting for the detection frame display toggle button 1512 is enabled, pseudo detection result information for testing purposes is also added to the notification content.

[0073] Returning to the explanation of Figure 4. In S408, when the information processing device 104 receives a user pressing the OK button 1602 on the setting screen 1600 (FIG. 16), it saves the monitoring task information registered in the RAM 203 as a monitoring task setting file in the external memory 212 and displays the setting screen 1610.

[0074] In S409, when the user presses the analysis tab 1611 on the setting screen 1610, the setting screen 1700 (FIG. 17) is displayed. At this time, when the user presses the adjustment button 1701, the analysis setting adjustment screen 1710 is displayed. Thereafter, the information processing device 104 executes the process shown in FIG.

[0075] The abnormality detection parameter setting process for the monitoring task in FIG. 4(b) will be described.

[0076] In S421, when the information processing apparatus 104 receives a user's press of the basic settings button 1711 on the analysis settings adjustment screen 1710 (FIG. 17), it displays the basic analysis settings screen 1720 and proceeds to S422. If the basic setting button has not been pressed, the process proceeds to S428.

[0077] In S422, when the information processing device 104 receives a change to the slide bars for the number of rows 1801 (FIG. 18) and the number of columns 1802 from the user, it updates the display of the grid (dashed lines) on the screen divided by the number of rows 1801 and the number of columns 1802, as shown in screen 1800 (FIG. 18), and saves the setting contents in RAM 203.

[0078] Specifically, on the screen 1800, the number of rows 1801 is set to 10 and the number of columns 1802 is set to 10, so the image to be analyzed is displayed divided into a grid of 10 rows and 10 columns.

[0079] The more grids there are, the more pinpoint detection becomes possible, but this increases the number of processes, such as the image difference (abnormality) detection process (Figure 23) described below, which may increase the time required for processing.

[0080] In S423, when the information processing device 104 receives a user's press 1811 on any area separated by a grid on the screen, a press of the Set All Masks button 1812, or a press of the Cancel All Masks button 1813, the information processing device 104 displays the presence or absence of a mask (notification suppression area) as shown on a screen 1810, and saves the setting contents in the RAM 203. That is, this step is a step showing an example of a process for receiving a designation of an area included in an image.

[0081] If there are many mask areas, the mask target area can be easily set by first masking all areas with the mask all setting button 1812 and then specifying the area to be unmasked.

[0082] If the masked area is small, the mask target area can be easily set by releasing the mask for all areas with the release all mask button 1813 and then specifying the area to be masked.

[0083] Furthermore, at this time, if a change in the mask color inversion 1821 toggle button by the user is accepted, the mask drawing color is changed as shown on screen 1820. Specifically, the mask area is represented in transparent black on screen 1810, but is represented in transparent white on screen 1820. That is, this step is a step showing an example of a process for accepting a designation of a display method for an area included in an image. Also, this step is a step showing an example of a process for displaying the area for which designation has been accepted in the display method for which designation has been accepted.

[0084] This allows the user to select the color of the mask area depending on the color tone and brightness of the image to be analyzed, by representing the mask area in transparent black when the image to be analyzed is bright, and in transparent white when the image to be analyzed is dark. This makes it easy to identify the mask area and also allows the user to understand the appearance of the mask area.

[0085] In this embodiment, the color of the mask area is "transparent black" or "transparent white," but this is not limited to this method. Other colors may be used, the user may specify the color, patterns such as diagonal lines may be used to vary the display, the area may be surrounded by a frame, or other methods may be used to distinguish between the monitored area and the mask area.

[0086] In S424, when the information processing device 104 receives a change from the user to the grid correction (shift grid) toggle button 1901 (Figure 19) or 1911, it updates the display of the frame 1902 (with frame) or 1912 (without frame) indicating the detection range and saves the setting contents in RAM 203.

[0087] In this embodiment, grid correction is set for the monitoring process, but this is not limited to this method, and grid correction may be set directly for the network camera or video monitoring server that is the image input source.

[0088] In S425 , when the information processing apparatus 104 receives a numerical input 1921 of the number of reference image frames from the user, the information processing apparatus 104 saves the setting content in the RAM 203 .

[0089] In S426, when the information processing device 104 receives a press of the back button 1803 from the user while the basic setting screen 1800 (FIG. 18) for analysis settings is displayed, it determines whether or not the basic settings have been changed. If there have been changes, the process proceeds to S427. If there have been no changes, the process proceeds to S428.

[0090] In S427, the information processing device 104 reflects the contents set in the RAM 203 in the monitoring task setting file in the external memory 212. The information processing device 104 also detects any changes to the monitoring task setting file and discards the reference image in the analysis process (in S2306, it is determined that the base information needs to be updated). In S428, when the information processing apparatus 104 receives a change to the slide bar for each item of the threshold 2001 on the analysis setting screen 2000 (FIG. 20), it saves the setting content in the RAM 203.

[0091] In this embodiment, the user sets each threshold value, but this method is not limited to this. The user may select a data set for each threshold value, or the user may set the object to be monitored or the abnormality to be detected so that the threshold parameters are automatically set, or the information processing device 104 may determine the object to be monitored or the abnormality to be detected and automatically set the threshold parameters, etc.

[0092] This makes it possible to easily set a threshold value for detecting an abnormality.

[0093] In S429, when the information processing device 104 receives a press of the Apply button 2002 from the user, it reflects the contents set in the RAM 203 in the monitoring task setting file in the external memory 212. The information processing device 104 also detects changes to the monitoring task setting file and changes the parameter values ​​used in the detection process (changing the threshold values ​​used in S2310).

[0094] The abnormality detection process in FIG. 21 will be described. In S2101, the information processing apparatus 104 executes an abnormality detection process if the external memory 212 contains a setting file for the monitoring task (task registration information 2201 (FIG. 22)) and the toggle 2202 for task validation setting is set to valid.

[0095] In S2102, the information processing apparatus 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and repeats the process.

[0096] In S2103, the information processing apparatus 104 acquires input information according to the set input method (set on the setting screen 1200 or the setting screen 1300).

[0097] Specifically, if the input setting for the operation process (setting screen 1200) is "Load video file (FFmpeg)" or "Load video file (OpenCV)", the video file placed in the specified video folder is moved to the working area on the external memory 212 as input information.

[0098] Also, if the input setting for the operation process is "Read live video (video monitoring server)" (setting screen 1300), video acquisition processing is executed for the specified video acquisition destination, and if a still image is acquired, the process proceeds to S2106.

[0099] In S2104, if the information processing apparatus 104 has moved the video file on the external memory 212 in S2103, the process proceeds to S2105. If the information processing apparatus 104 has executed the video acquisition process and acquired a still image, the process proceeds to S2106.

[0100] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into consecutive still images for each frame, and files it on the external memory 212 or stores it as data on the RAM 203, and then proceeds to S2106.

[0101] In S2106, the information processing device 104 executes the process shown in FIG.

[0102] The image difference (abnormality) detection process of FIG. 23 will be described. In S2301, if the information processing apparatus 104 has divided the moving image file into successive still images for each frame (S2105), it repeats the subsequent processes until the processes for all the generated still images are completed.

[0103] In S2302, the information processing device 104 reads a still image to be processed and develops it on the RAM 203. That is, this step is a step showing an example of processing for acquiring images captured at the same angle of view.

[0104] In S2303, the information processing apparatus 104 calculates information on hue, saturation, and lightness from the image data expanded on the RAM 203 by calculation.

[0105] In S2304, the information processing device 104 performs each histogram calculation process (FIG. 24) to calculate a histogram from the image data.

[0106] Here, the histogram calculation process of FIG. 24 will be described. In S2401, the information processing apparatus 104 proceeds to S2402 if the grid correction set in S424 among the settings saved in the RAM 203 is valid, or proceeds to S2403 if it is invalid.

[0107] In S2402, the information processing device 104 sets, as target areas for histogram calculation processing, not only the areas divided by the grid 2500 (FIG. 25) set in S422, but also areas divided by grids 2501 shifted by half the size of one area in the horizontal and vertical directions. That is, this step is a step showing an example of processing for dividing the acquired image.

[0108] Specifically, as shown in 2510 and 2520, if an abnormality occurs near the 3-row x 4-column grid set in S422, since the abnormality spans multiple areas, the threshold value will not be exceeded in any of the areas, and all areas will be determined to be normal, which may result in low accuracy in abnormality detection.

[0109] As in S2402, by determining whether an abnormality exists not only in the area separated by the normal grid 2500 set in S422 but also in the area separated by the corrected grid 2501 (shifted grid), it becomes possible to easily improve the accuracy of detecting an abnormality even when an abnormality occurs near the normal grid 2500 set in S422.

[0110] In this embodiment, the grids are shifted horizontally and vertically by half the size of one region. However, this is not limited to this method. It is also possible to create two new grids that are shifted horizontally and vertically by 1 / 3 and 2 / 3 of the size of one region, or to create a new hexagonal grid instead of a lattice (square) grid. This step is an example of a process for dividing an image using two or more division methods. This is an example in which an area divided by a first division method partially overlaps with at least one of the areas divided by a second division method.

[0111] This makes it possible to easily improve the accuracy of detecting an abnormality even when an abnormality occurs near the grid 2500.

[0112] In S2403, the information processing apparatus 104 calculates a histogram of luminance information from the luminance (Lightness) information acquired in S2303 for each processing target region set in S422 and S2402.

[0113] In S2404, the information processing apparatus 104 calculates a histogram of hue information from the hue and saturation information acquired in S2303 for each processing target region set in S422 and S2402.

[0114] In S2405, the information processing apparatus 104 calculates a histogram of saturation information from the saturation information acquired in S2303 for each processing target region set in S422 and S2402.

[0115] In S2406, the information processing device 104 calculates edge gradient information from the luminance (Lightness) information acquired in S2303 for each processing target area set in S422 and S2402, and calculates an edge gradient intensity histogram and an edge gradient angle histogram using the edge gradient information.

[0116] Returning to the explanation of FIG. In S2305, the information processing apparatus 104 calculates the degree of similarity for each processing target area set in S422 and S2402 using the histogram of the reference information and histogram intersection, which is one of the methods for comparing histograms.

[0117] In this case, the reference information is a histogram calculated in S2308, which will be described later. If there is no reference histogram data, the similarity is set to 100%.

[0118] In this embodiment, histogram intersection is used as the method for comparing histograms, but the method is not limited to this, and other methods such as Bhattacharyya coefficients may also be used.

[0119] Furthermore, in this embodiment, anomalies are detected by comparing histograms, but this is not limited to this method, and anomalies may be detected using other methods, such as template matching or feature point detection.

[0120] In S2306, if the number of data items already set as the reference image does not satisfy the "number of reference image frames" set in S425, the information processing device 104 determines that the reference image needs to be updated and proceeds to S2307. If data items that satisfy the "number of reference image frames" have already been set as the reference image, the process proceeds to S2309.

[0121] The information processing device 104 discards the reference image and initializes the number of set data items to 0 in the following cases:

[0122] When the basic settings are changed by the user of the S427 When the user clicks the Recreate Reference Image button 2003 This allows the user to easily instruct the re-creation of the reference image.

[0123] In S2307, the information processing device 104 determines whether the data is suitable as a reference image based on the similarity calculated in S2305. If there are no areas with low similarity, the process proceeds to S2308. If there are many areas with low similarity, the data is determined to be unsuitable as a reference image and the process ends.

[0124] In S2308, the information processing apparatus 104 adds the histogram calculated in S2304 to the histogram of the reference information currently held, recalculates the average value, updates the histogram of the reference information, and ends the process.

[0125] In S2309, the information processing device 104 compares the similarity for each processing target region set in S422, calculated in S2305, with the threshold set in S428, and records any region below the threshold as an abnormal region in the RAM 203. That is, this step is a step illustrating an example of processing for determining that a change has occurred in a region based on changes between corresponding regions of multiple images among the multiple acquired images.

[0126] Specifically, in the threshold 2001 (FIG. 20), when the brightness threshold is set to 22%, if the brightness similarity of the area to be processed is less than 22%, the area is determined to be abnormal.

[0127] In S2310, the information processing device 104 compares the similarity for each area (area separated by a correction grid) set as the area to be processed in S2402, calculated in S2305, with the threshold set in S428, and detects areas that fall below the threshold.

[0128] At this time, if an abnormality is detected, the process proceeds to S2311. If no abnormality is detected, the process proceeds to S2312.

[0129] In S2311, the information processing device 104 determines (2620) that all areas 2610 (four areas in the case of FIG. 26) that overlap with the detection area 2600 have detected an abnormality, since the area 2600 (FIG. 26) where the abnormality was detected is different from the area to be notified to the user that was set in S422, and records the detected abnormality in RAM 203.

[0130] As a result, the area for which an abnormality is notified is the area divided by the normal grid 2500 set in S422, regardless of whether grid correction is performed or not. In other words, this step is an example of a process for outputting a determination result based on the area divided by the first division method when it is determined that a change has occurred in the area divided by the second division method.

[0131] In this way, it is possible to notify users and external applications of abnormalities based on areas that are always separated by a constant grid. In other words, because the notification method is constant regardless of whether grid correction is performed or not, it has the advantage that the notification is easy for users to understand, and for external applications, the method for receiving abnormality notifications does not need to be complicated, so the reception logic and method for outputting abnormalities can be constant.

[0132] In S2312, the information processing device 104 compares the abnormal area recorded on the RAM with the mask area set in S423, and if there is an abnormal area other than the mask area, proceeds to S2313. If there is no abnormal area or if all of the abnormal areas are mask areas, the process ends. In other words, this step is a step that shows an example of a process of controlling whether or not to output the determination result based on the area whose designation has been accepted.

[0133] In this embodiment, the histogram calculation process (S2304) is performed even in a masked area, but this is not limited to this method, and the histogram calculation process (S2304) may not be performed for a masked area. By doing so, the number of times each histogram calculation process is performed when there is a masked area is reduced, thereby shortening the overall processing time.

[0134] On the other hand, when performing histogram calculation processing even in a masked area, as in this embodiment, there are advantages such as being able to record abnormalities in the masked area and suggest changes to the masked area to the user, and being able to stably meet non-functional requirements by always keeping the processing speed constant.

[0135] In S2313, the information processing device 104 performs an abnormality notification process (FIG. 27) and notifies the user and an external application of information about the abnormal area.

[0136] The abnormality notification process in FIG. 27 will be described. In S2701, the information processing device 104 determines whether there is an abnormality to be notified based on criteria such as the continuity of abnormality detection. If it is determined that there is an abnormality to be notified, the process proceeds to S2702. If it is determined that there is no abnormality to be notified, the process proceeds to S2703.

[0137] Specifically, criteria are needed to determine whether an abnormality should be reported if an abnormality is detected in only one image frame, or if an abnormality is detected in several consecutive images over a period of several seconds.

[0138] For example, the information processing device 104 may accept the user's setting of criteria for notifying an abnormality in advance, such as how many consecutive frames an abnormality must be detected before notifying an abnormality, or may determine what the image being analyzed is (a factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and based on the object or environment, the information processing device 104 may determine the criteria for notifying an abnormality.

[0139] In step S2702, the information processing apparatus 104 records in the RAM 203 area information in which the abnormality to be notified has occurred.

[0140] In S2703, the information processing apparatus 104 deletes the abnormality information to be notified that is recorded on the RAM 203.

[0141] In S2710, the information processing apparatus 104 passes the still image information read in S2302 to the processing unit that draws the setting screen.

[0142] In S2711, the information processing apparatus 104 passes the abnormality information recorded in S2702 to the processing unit that draws the setting screen.

[0143] In S2712, the information processing apparatus 104 determines whether the setting screen is being drawn by a user operation. If the setting screen is being drawn, the process proceeds to S2713. If the setting screen is not being drawn, the process ends.

[0144] In S2713, the information processing apparatus 104 checks the detection area drawing color setting content 2801 in the RAM 203. If there is a designation regarding the drawing method, the process proceeds to S2714. If there is no designation, the process proceeds to S2715.

[0145] In S2714, the information processing device 104 changes the highlighting during the drawing process in S2715 (described later) based on the setting contents 2801 (FIG. 28). That is, this step is an example of a process of identifying and outputting which of multiple thresholds was used to determine that a change occurred.

[0146] The setting content 2801 is a setting for how to highlight the display depending on the detection factor of the abnormality detection (the item that fell below the threshold in S2309). That is, this step is an example of a process for accepting a designation of a method for identifying and outputting for each type of threshold.

[0147] Here, control is performed to prevent overlapping of the same highlighting settings so that the highlighting for each threshold can be distinguished (for example, an error message is displayed when the same highlighting is selected for different thresholds, or a highlighting that has already been selected cannot be newly selected, etc.). In other words, this step is an example of a process for controlling so that the same output method specification is not accepted for different threshold types.

[0148] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0149] In this embodiment, highlighting is performed using color, but the present invention is not limited to this method. The highlighting method can be set to use a pattern such as diagonal lines, to surround the information with a frame such as dotted or thick lines, or to set different notification methods such as voice or email, or to set the notification method using methods other than highlighting in addition to the type of highlighting.

[0150] In S2715, the information processing apparatus 104 develops an image for drawing in the drawing memory on the RAM 203 by highlighting the still image information acquired in S2710 and the area containing the abnormality information acquired in S2711.

[0151] In S2716, the information processing device 104 draws the drawing image created in S2715 on the setting screen 2810 (FIG. 28). That is, this step is a step showing an example of a process of outputting a determination result as to whether or not there has been a change in the area.

[0152] At this time, if the rendering method is changed in S2714, the highlighting method for the abnormal area is changed and displayed as shown in abnormal area 2811(1), (2), and (4).

[0153] This allows the user to easily know which threshold was used to determine the abnormality.

[0154] The highlighting of the abnormal region and the display of the masked region may be performed on the same screen (not shown).

[0155] Furthermore, although it was stated in S2312 that the anomaly region is not displayed in the masked region, it is also possible to highlight the anomaly detection region (not shown) or highlight it differently (not shown) in the masked region in response to an instruction from the user. This is because, even in a region designated as a masked region, this can trigger the user to decide that the mask should be removed depending on the content of the anomaly detection. In other words, it becomes possible for the user to easily set the anomaly detection desired.

[0156] Furthermore, the information processing device 104 may have a function of informing the user on the screen 1810 or 1820 of the type of abnormality occurring in the masked area without receiving an instruction from the user.

[0157] Furthermore, there may be a function that suggests to the user to change the mask settings based on the number and content of anomaly detections, or that the information processing device 104 automatically changes the mask settings. This enables more accurate anomaly detection. That is, this step is a step that shows an example of a process of changing the area whose designation has been accepted based on the area whose designation has been accepted and the determination result of that area.

[0158] In S2720, the information processing device 104 notifies the abnormality information recorded in S2702 to the output destination set in S407 (for example, a video monitoring server, a mailer, software capable of instructing a call using an automated voice call, etc.).

[0159] Figure 29 shows an example of the anomaly notification dashboard screen, which displays a table listing all the anomalies that have been detected so far. The screen displays the detection time, monitoring task name, and coordinates of the detected area.

[0160] As a result, the user can easily know which threshold value was used to determine the abnormality. Furthermore, it is possible to improve the accuracy of detecting abnormalities near the grid. It also becomes possible to easily know what the basis for detecting an abnormality was.

[0161] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0162] In this case, the program itself read from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0163] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon disks.

[0164] Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing a program it has read, but also cases are included in which an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0165] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0166] Furthermore, the present invention may be applied to a system consisting of multiple devices, or to a device consisting of a single device. It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or device. In this case, the system or device can enjoy the effects of the present invention by reading a recording medium containing a program for achieving the present invention into the system or device.

[0167] The program may be in the form of object code, program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

[0168] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modifications. [Explanation of symbols]

[0169] 100 Information Processing Systems 102 Camera 104 Information processing equipment

Claims

1. an acquisition means for acquiring a captured image; a dividing means for dividing the image into a plurality of regions; a receiving means for receiving a designation of a predetermined area from among the plurality of areas divided by the dividing means; a determination means for determining whether a change has occurred in a corresponding region of a plurality of images acquired by the acquisition means, based on a change between the corresponding regions of the plurality of images; an output control means for controlling whether or not to output the result of the determination relating to the predetermined area whose designation has been accepted by the accepting means; an area change means for changing the predetermined area based on the result of the determination regarding the predetermined area whose designation has been accepted by the accepting means; An information processing system comprising:

2. the receiving means receives a designation of a method for identifying the predetermined area; The output control means controls the predetermined area to be identified and output by the method accepted by the accepting means.

2. The information processing system according to claim 1, wherein:

3. The method of identification is at least one of a method of displaying the predetermined area darker or brighter than other areas, a method of surrounding the predetermined area with a frame, or a method of superimposing a predetermined texture pattern on the predetermined area.

3. The information processing system according to claim 2, wherein:

4. The output control means does not output the result of the determination related to the predetermined area whose designation has been accepted by the accepting means.

2. The information processing system according to claim 1, wherein:

5. The output control means outputs the result of the determination relating to the predetermined area whose designation has been accepted by the accepting means.

2. The information processing system according to claim 1, wherein:

6. The output control means outputs the result of the determination relating to the predetermined area, the designation of which is accepted by the accepting means, based on the number of times the determination is made or the content of the determination.

2. The information processing system according to claim 1, wherein:

7. The determining means determines whether or not there has been a change in the predetermined area whose designation has been accepted by the accepting means.

2. The information processing system according to claim 1, wherein:

8. The determining means does not determine whether or not there has been a change in the predetermined area designated by the accepting means.

2. The information processing system according to claim 1, wherein:

9. The image acquired by the acquisition means is either a still image or a video image.

2. The information processing system according to claim 1, wherein:

10. The receiving means receives an operation relating to a setting of whether or not to output the result of the determination, or whether or not to determine whether or not there has been a change, for all areas divided by the dividing means.

2. The information processing system according to claim 1, wherein:

11. The division by the division means includes dividing the image into a plurality of regions; The receiving means receives designation of the predetermined area based on an operation on the partitioned area.

2. The information processing system according to claim 1, wherein:

12. the dividing means divides the image into a plurality of regions using a grid; The receiving means receives designation of the predetermined area based on an operation on an area partitioned using the grid.

2. The information processing system according to claim 1, wherein:

13. an acquisition step of acquiring a captured image; a segmentation step of segmenting the image into a plurality of regions; a receiving step of receiving designation of a predetermined area from among the plurality of areas divided by the dividing step; a determination step of determining that a change has occurred in a corresponding region based on a change between the corresponding regions of the plurality of images among the plurality of images acquired by the acquisition step; an output control step of controlling whether or not to output the result of the determination relating to the predetermined area whose designation has been accepted in the accepting step; Based on the result of the determination regarding the predetermined area whose designation is accepted in the accepting step, an area changing step of changing the predetermined area; 1. A method for controlling an information processing system, comprising:

14. A program for causing at least one computer to function as each of the means of the information processing system according to any one of claims 1 to 12.

Images