Information processing systems, information processing methods, and programs

Abstract

This system provides a mechanism that can detect anomalies using videos provided by third parties. [Solution] An information processing system characterized by comprising: a receiving means for receiving identification information that identifies a video from a video streaming service; and an output means for outputting a statement that an anomaly has been detected in the video corresponding to the identification information received by the receiving means.

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and more particularly to a technique suitable for use in detecting abnormalities.

Background Art

[0002] Conventionally, there is a method of detecting an abnormality of an imaging object based on a change in an image by processing an image obtained by photographing with a camera or the like.

[0003] In Patent Document 1, when an abnormality is detected based on image information from a video camera, a still image camera is operated in conjunction with this abnormality detection to capture a still image, thereby reliably obtaining image information regarding an intruder or the like. A technique is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, a still image camera is operated in conjunction with the detection of an abnormality to capture a still image. However, when verifying for improving the accuracy of abnormality detection, it is necessary to verify videos taken in various environments, and for this purpose, there is a problem that it is desired to verify using videos provided by a third party.

[0006] Therefore, an object of the present invention is to provide a mechanism for detecting an abnormality using a video provided by a third party.

Means for Solving the Problems

[0007] An information processing system characterized by comprising: a receiving means for receiving identification information that identifies a video of a video streaming service; and an output means for outputting a statement that an anomaly has been detected in the video corresponding to the identification information received by the receiving means. [Effects of the Invention]

[0008] According to the present invention, a mechanism can be provided that detects anomalies using videos provided by a third party. [Brief explanation of the drawing]

[0009] [Figure 1] This is a system configuration diagram of information processing system 100. [Figure 2] This is a hardware block diagram of the information processing device 104. [Figure 3] This is an example of a block diagram showing the software configuration. [Figure 4] (a) A flowchart of the configuration process and (b) an example of a flowchart of the anomaly detection parameter setting process for the monitoring task. [Figure 5] This is an example of the initial display of the settings screen. [Figure 6] This is an example of an overall view of the settings screen. [Figure 7] This is an example of the settings screen (input tab). [Figure 8] This is an example of the settings screen (input tab). [Figure 9] This is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11] This is an example of the settings screen (monitoring processes and monitoring tasks). [Figure 12] This is an example of the settings screen (input tab). [Figure 13] This is an example of the settings screen (input tab). [Figure 14] This is an example of the settings screen (notification tab). [Figure 15] This is an example of the settings screen (notification tab). [Figure 16] This is an example of the display of the setting screen (monitoring task). [Figure 17] This is an example of the display of the adjustment screen for analysis settings. [Figure 18] This is an example of the display of the grid setting screen (grid division). [Figure 19] This is an example of the display of the grid setting screen (grid correction). [Figure 20] This is an example of the display of the analysis setting screen. [Figure 21] This is an example of the flowchart of the abnormality detection process. [Figure 22] This is an example of the display of task registration information. [Figure 23] This is an example of the flowchart of the image difference (abnormality) detection process. [Figure 24] This is an example of the flowchart of each histogram calculation process. [Figure 25] This is an example of the display of grid correction. [Figure 26] This is an example of the display of the notification during grid correction. [Figure 27] This is an example of the flowchart of the abnormality notification process. [Figure 28] This is an example of the display of the setting screen (abnormality notification). [Figure 29] This is an example of the display of the abnormality notification dashboard screen. [Figure 30] This is an example of the flowchart of the video acquisition process for online video. [Figure 31] This is an example of the display of the video acquisition screen for online video.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] FIG. 1 is a system configuration diagram showing an example of the configuration of the information processing system 100 of the present invention.

[0012] Figure 1 shows a configuration in which the camera 102 and the information processing device 104 are connected via an image transfer cable (USB, Ethernet, CameraLink, etc.). However, a configuration of an information processing device equipped with a camera is also acceptable, even without this specific setup.

[0013] The information processing device 104 acquires an image obtained by the camera 102 capturing the target object via an image transfer cable, and performs processing related to the detection of abnormalities in the image.

[0014] Furthermore, the information processing system 100 of this embodiment may also have a configuration in which the camera 102 with the above-described configuration does not exist.

[0015] In that case, processing related to the detection of abnormalities in images is performed on images acquired from other terminals on the network or on images stored in the memory (RAM) of the information processing device 104.

[0016] The following describes the hardware configuration of an information processing device applicable to the information processing device 104 shown in Figure 1, using Figure 2 as an example.

[0017] Figure 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in Figure 1.

[0018] In Figure 2, 201 is the CPU, which comprehensively controls each device and controller connected to the system bus 204. The ROM 202 or external memory 212 stores the BIOS (Basic Input / Output System), which is the control program for the CPU 201, the operating system program (hereinafter referred to as the OS), and various programs necessary to implement the functions executed by each PC, as described later.

[0019] 203 is RAM, which functions as the main memory, work area, etc., of the CPU 201. The CPU 201 loads the necessary programs, etc., from ROM 202 or external memory 212 into RAM 203, and then executes the loaded programs to perform various operations.

[0020] 205 is an input controller that controls input from pointing devices such as the keyboard (KB) 210 and a mouse (not shown).

[0021] 206 is a video controller that controls the display on indicators such as display 211.

[0022] 207 is the memory controller, which stores various data on external storage devices (hard disks (HDs)), flexible disks (FDs), or PCMCIA card slots. It controls access to external memory 212, such as CompactFlash® memory, which is connected via an adapter.

[0023] 208 is a communication interface controller that controls the reception of image data from the external PC 213 via the network (TCP / IP). 209 is an image I / F controller that controls the reception of image data from camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0024] The various programs described later for realizing the present invention are stored in RAM 203 and executed by CPU 201.

[0025] Furthermore, the image data used when executing the above program is stored in ROM202, external memory212, external PC213, and camera102 depending on the application, and is stored in RAM203 via various controllers when the program is executed. Figure 3 is an example of a block diagram showing the software configuration of an embodiment of the present invention.

[0026] The information processing device 104 includes the following functional units.

[0027] The acquisition unit 301 is a functional unit that acquires images captured at the same angle of view.

[0028] The reception unit 302 is a functional unit that receives specifications for the area to be included in the image and the method of displaying that area.

[0029] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on changes between corresponding regions of multiple images acquired by the acquisition unit 301.

[0030] The output unit 304 is a functional unit that outputs the determination result from the determination unit 303.

[0031] The output unit 304 is a functional unit that displays the area specified by the reception unit 302 using the display method specified by the reception unit 302, and controls whether or not to output a judgment result based on the area specified by the reception unit 302.

[0032] The area modification unit 305 is a functional unit that modifies the area specified by the reception unit 302 based on the area specified by the reception unit 302 and the determination result of that area.

[0033] The splitting unit 306 is a functional unit that splits the image acquired by the acquisition unit 301.

[0034] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on changes between corresponding regions that have been divided by the division means from among multiple images acquired by the acquisition unit 301.

[0035] The division unit 306 is a functional unit that divides an image using two or more division methods.

[0036] The output unit 304 is a functional unit that outputs a determination result based on the region divided by the first division method when the determination unit 303 determines that there has been a change in the region divided by the second division method.

[0037] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on the changes between corresponding regions of multiple images acquired by the acquisition means and multiple types of thresholds.

[0038] The output unit 304 is a functional unit that identifies and outputs which of the multiple types of thresholds was used to determine that a change had occurred.

[0039] The reception unit 302 is a functional unit that accepts specifications for the method of identification and output for each type of threshold.

[0040] The reception unit 302 is a functional unit that controls the system to prevent the same output method specification from being accepted for different threshold types.

[0041] This concludes the explanation of Figure 3. The setup process shown in Figure 4(a) will be explained below.

[0042] In S401, the information processing device 104 determines whether a configuration file for the operating process exists in the external memory 212. If it is not registered, the process proceeds to S402; if it is registered, the process proceeds to S403.

[0043] In S402, the information processing device 104 displays the settings screen 510 (Figure 5), and when it receives a press of the add monitoring process button 511 from the user, it registers the operation process in RAM 203 and displays the settings screen 520.

[0044] In S403, when the information processing device 104 receives a press of the button 521 for selecting the operation process to be edited on the setting screen 502, it displays the setting screen 610 (Figure 6). The setting acceptance process performed on setting screen 610 in S403 will be explained below using Figures 6 to 10.

[0045] When the information processing device 104 receives input from the user for the name 611 and description 612 of the operating process, it edits the operating process information in the RAM 203 and displays the settings screen 620.

[0046] When the information processing device 104 receives a press from the user on the settings screen 610, it displays the settings screen 710 (Figure 7).

[0047] When the information processing device 104 receives a selection from the user for "Video file loading (FFmpeg)" (721 in Figure 7) from the list of functions 711, it displays the settings screen 720. At this time, the settings screen 720 edits the operation process information of RAM 203 based on the location 722 and option 723 values ​​of the command entered by the user.

[0048] When the information processing device 104 receives a selection from the user for "Video file loading (OpenCV)" (731 in Figure 7) from the list of functions 711, it displays the settings screen 730 and edits the operation process information of RAM 203.

[0049] When the information processing device 104 receives a selection from the user for "Live video loading (video surveillance server)" (811 in Figure 8) from the list of functions 711, it displays the settings screen 810 (Figure 8). "Live video loading (video surveillance server)" is a function that loads live video captured by a network camera from a video surveillance server (not shown) that can be connected via the network.

[0050] Based on the computer name 812, username 813, and password 814 values ​​entered by the user on the settings screen 810, the operating process information of RAM 203 is edited. Also, if the connection confirmation 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in RAM 203.

[0051] When the information processing device 104 receives a click from the user on the settings screen 610 to select the analysis tab 614 (Figure 6), it displays the settings screen 910 (Figure 9).

[0052] When the information processing device 104 receives a selection of "Detection by video comparison" from the function list 911 from the user, it displays the settings screen 920 and edits the operation process information of RAM 203. At this time, if the license is not authenticated, "No license" is displayed in the license information 922.

[0053] When the information processing device 104 receives a press of the license read button 923 from the user, if a valid license has been authenticated, it displays the settings screen 930, displays the license information 931, and edits the operation process information of the RAM 203.

[0054] When the information processing device 104 receives a notification from the user on the settings screen 610 by clicking the notification tab 615 (Figure 6), it displays the settings screen 1010 (Figure 10).

[0055] When the information processing device 104 receives a selection of a "custom command" (1021 in Figure 10) from the function list 1011 from the user, it displays the settings screen 1020 and edits the operation process information of the RAM 203.

[0056] A "custom command" is a function that executes commands to instruct batches or applications to run when an anomaly is detected. Using this function, it becomes possible to execute batch processing, instruct email sending to mail clients, instruct automated voice calls to be made, and integrate with other applications or external systems.

[0057] When the information processing device 104 receives a selection from the user for "Analysis Result Notification (Video Surveillance Server)" (1031 in Figure 10) from the function list 1011, it displays the settings screen 1030.

[0058] "Analysis Result Notification (Video Surveillance Server)" is a function that notifies the video surveillance server of the analysis results. This function is used when the video surveillance server is configured to notify users of anomalies, or when the video surveillance server analyzes and manages events such as anomalies.

[0059] Based on the values ​​of address 1032 and port number 1033 entered by the user on the settings screen 1030, the operating process information of RAM203 is edited.

[0060] When the information processing device 104 receives a connection confirmation 1034 from the user, it performs a connection test via the communication I / F controller 208 using the information in the RAM 203. Returning to the explanation of Figure 4.

[0061] In S404, when the information processing device 104 receives a press of the OK button 631 on the setting screen 630 (Figure 6), it saves the operation process information registered in RAM 203 to external memory 212 as an operation process setting file.

[0062] In S405, the information processing device 104 determines whether a configuration file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406; if it is registered, the process proceeds to S407.

[0063] In S406, the information processing device 104 displays the settings screen 630, and when it receives a press from the user of the Add task button 632 for the monitoring process, it registers the monitoring task in RAM 203 and displays the settings screen 1110 (Figure 11).

[0064] In S407, when the information processing device 104 receives a click on the monitoring task 1111 to be edited on the settings screen 1110, it displays the settings screen 1120. The following describes the process for receiving settings for monitoring tasks performed in S407, using Figures 11 to 15.

[0065] When the information processing device 104 receives input from the user on the settings screen 1120, including a name 1121 and a description 1122, and a change in the toggle button 1123 for enabling this task, it edits the operation process information of the RAM 203 and displays the settings screen 1130.

[0066] When the information processing device 104 receives a press of the input tab 1124 from the user on the settings screen 1120, it displays either the settings screen 1200 (Figure 12) or the settings screen 1300 (Figure 13) depending on the settings of the operation process.

[0067] If the input setting for the operation process is "Load video file (FFmpeg)" or "Load video file (OpenCV)", the settings screen 1200 (Figure 12) will be displayed.

[0068] When the information processing device 104 receives a click on a video folder 1201 and a folder selection from the user, input of a frame acquisition interval 1202, and a change in the toggle button for deleting processed videos 1203, it edits the operation process information of RAM 203 and displays the settings screen 1210.

[0069] If the input setting for the operation process is "live video loading," the settings screen 1300 (Figure 13) is displayed. When the information processing device 104 receives a selection of a list item for camera 1301 from the user, it edits the operation process information in RAM 203 and displays the settings screen 1310.

[0070] When the information processing device 104 receives a notification from the user on the settings screen 1120 by clicking the notification tab 1125, it displays either the settings screen 1400 or the settings screen 1500, depending on the settings of the operation process.

[0071] If the notification setting for the operation process is "custom command," the settings screen 1400 (Figure 14) is displayed. When the information processing device 104 receives input of the command path 1401 from the user or a change in the toggle button for asynchronous execution 1402, it edits the operation process information in RAM 203 and displays the settings screen 1410.

[0072] If the notification setting for the operation process is "Analysis Result Notification (Video Surveillance Server)", the settings screen 1500 (Figure 15) is displayed. The information processing device 104 accepts the input of name 1501 and camera ID 1502 from the user and displays screen 1510. At this time, if the test event button 1511 is pressed by the user, a test notification process is executed to check whether the notification with the input content works correctly. At this time, if the setting of the detection frame display toggle button 1512 is enabled, test pseudo detection result information is also added to the notification content. Returning to the explanation of Figure 4.

[0073] In S408, when the information processing device 104 receives a press of the OK button 1602 from the user on the setting screen 1600 (Figure 16), it saves the monitoring task information registered in RAM 203 as a monitoring task setting file in external memory 212 and displays the setting screen 1610.

[0074] In S409, when the user clicks the analysis tab 1611 on the settings screen 1610, the settings screen 1700 (Figure 17) is displayed. At this time, when the user clicks the adjustment button 1701, the analysis settings adjustment screen 1710 is displayed.

[0075] Subsequently, the information processing device 104 executes the process shown in Figure 4(b). The process for setting anomaly detection parameters for the monitoring task shown in Figure 4(b) will be explained below.

[0076] In S421, when the information processing device 104 receives a press of the basic settings button 1711 from the user on the analysis settings adjustment screen 1710 (Figure 17), it displays the basic settings screen 1720 for analysis settings and proceeds to S422. If the Basic Settings button is not pressed, proceed to S428.

[0077] In S422, when the information processing device 104 receives a request from the user to change the slider bar to 1801 rows (Figure 18) and 1802 columns, it updates the display of the grid (dashed lines) on the screen, which is divided by 1801 rows and 1802 columns, as shown in screen 1800 (Figure 18), and saves the settings to RAM 203.

[0078] Specifically, on screen 1800, the number of rows is set to 10 (1801) and the number of columns to 10 (1802), so the image to be analyzed is displayed, divided into a 10x10 grid.

[0079] While a larger number of grids allows for more precise detection, it also increases the number of processes involved, such as the image difference (anomaly) detection process described later (Figure 23), potentially increasing the processing time.

[0080] In S423, when the information processing device 104 receives a press 1811 of any area divided by a grid on the user's screen, a press of the full mask setting button 1812, or a press of the full mask release button 1813, it displays the presence or absence of a mask (notification suppression area) as shown on screen 1810 and saves the setting in RAM 203. In other words, this step is an example of a process that accepts the specification of an area included in an image.

[0081] If there are many areas to mask, you can easily set the areas to be masked by first masking all areas using the "Set All Masks" button (1812), and then specifying the areas to be unmasked.

[0082] If the masked area is small, you can easily set the area to be masked by first removing the mask from the entire area using the "Remove All Mask" button (1813), and then specifying the area to be masked.

[0083] Furthermore, if the user changes the mask color inversion 1821 toggle button at this time, the mask drawing color is changed as shown in screen 1820. Specifically, in screen 1810, the mask area was represented by transparent black, but in screen 1820, the mask area is represented by transparent white. In other words, this step is an example of a process that accepts a specification for how to display an area included in an image. This step is also an example of a process that accepts a specification for the display method of the area, and displays the area that has been specified using the accepted display method.

[0084] This allows the user to select the color of the mask area according to the color tone and brightness of the image being analyzed. This is achieved by representing the mask area with transparent black when the image is bright and transparent white when the image is dark. As a result, the user can easily identify and understand the appearance of the mask area.

[0085] In this embodiment, the color of the mask area is set to "transparent black" or "transparent white," but this method is not limited to this. Other colors may be used, the user may specify a color, patterns such as diagonal lines may be used to add visual variation, a frame may be used, or other methods may be used to distinguish the monitored area from the mask area.

[0086] In S424, when the information processing device 104 receives a change from the user to the grid correction (shifting the grid) toggle button 1901 (Figure 19) or 1911, it updates the display of the frame 1902 (with frame) or 1912 (without frame) that indicates the detection range and saves the setting to RAM 203.

[0087] In this embodiment, grid correction is set for the monitoring process, but this method is not limited to this one. Grid correction may also be set directly for the network camera or video monitoring server that is the source of the image input.

[0088] In S425, when the information processing device 104 receives a numerical input 1921 for the number of reference image frames from the user, it saves the setting in the RAM 203.

[0089] In S426, when the information processing device 104 receives a press of the back button 1803 from the user while the basic settings screen 1800 (Figure 18) for analysis settings is displayed, it determines whether there are any changes to the basic settings. If there are changes, it proceeds to S427. If there are no changes, it proceeds to S428.

[0090] In S427, the information processing device 104 reflects the contents set in RAM 203 in the monitoring task configuration file in external memory 212. The information processing device 104 also detects changes in the monitoring task configuration file and performs reference image discarding during the analysis process. (In S2306, it determines that updating the base information is necessary.) In S428, when the information processing device 104 receives a change in the slider bar for each item of the threshold 2001 on the analysis setting screen 2000 (Figure 20), it saves the setting to the RAM 203.

[0091] In this embodiment, the user sets each threshold, but this method is not the only way. The user may select a dataset for each threshold, the user may set the objects to be monitored or the anomalies to be detected to automatically set the threshold parameters, or the information processing device 104 may determine the objects to be monitored or the anomalies to be detected and automatically set the threshold parameters.

[0092] This makes it easy to set anomaly detection thresholds.

[0093] In S429, when the information processing device 104 receives a press of the apply button 2002 from the user, it reflects the contents set in RAM 203 into the monitoring task configuration file in external memory 212. The information processing device 104 also detects changes in the monitoring task configuration file and modifies the parameter values ​​used for detection processing. (This modifies the threshold used in S2310.) The anomaly detection process shown in Figure 21 will be explained below.

[0094] In S2101, the information processing device 104 executes anomaly detection processing if the external memory 212 contains a configuration file for the monitoring task (task registration information 2201 (Figure 22)) and the task activation setting toggle 2202 is set to enabled.

[0095] In S2102, the information processing device 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and repeats.

[0096] In S2103, the information processing device 104 acquires input information according to the configured input method (configured on the configuration screen 1200 or configuration screen 1300).

[0097] Specifically, if the input setting for the operation process (settings screen 1200) is "Read video file (FFmpeg)" or "Read video file (OpenCV)", the video files located in the specified video folder are used as input information and moved to the working area on external memory 212.

[0098] Furthermore, if the input setting for the operation process is "Live video loading (video monitoring server)" (settings screen 1300), the process of acquiring video from the specified video acquisition destination is executed, and if a still image is acquired, the process proceeds to S2106.

[0099] In S2104, if the information processing device 104 performed a video file transfer on the external memory 212 in S2103, it proceeds to S2105. If it is acquiring video from an online video on the web, it proceeds to step S2107. Details of step S2107 will be described later in Figure 30. If the video acquisition process has been executed and still images have been acquired, it proceeds to S2106.

[0100] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into a series of still images for each frame, saves them as files on the external memory 212 or stores them as data on the RAM 203, and proceeds to S2106.

[0101] In S2107, the information processing device 104 executes the process shown in Figure 30.

[0102] In S2106, the information processing device 104 executes the process shown in Figure 23. The image difference (anomaly) detection process shown in Figure 23 will be explained.

[0103] In S2301, if the information processing device 104 has been splitting the video file into a series of still images frame by frame (S2105), it repeats the subsequent processing until processing is completed for all the generated still images.

[0104] In S2302, the information processing device 104 reads the still image to be processed and loads it onto the RAM 203.

[0105] In S2302, the information processing device 104 reads the still image to be processed and loads it onto RAM 203. If online video is specified in S2104, in S3007, the still image stored in memory is read and loaded onto RAM 203.

[0106] In other words, this step demonstrates an example of a process for acquiring images captured at the same field of view.

[0107] In S2303, the information processing device 104 calculates the hue, saturation, and lightness information from the image data loaded on the RAM 203.

[0108] In S2304, the information processing device 104 performs each histogram calculation process (Figure 24) to calculate a histogram from the image data. Here, we will explain the calculation process for each histogram in Figure 24.

[0109] In S2401, the information processing device 104 proceeds to S2402 if the grid correction set in S424 is enabled among the settings stored in RAM 203. Otherwise, it proceeds to S2403.

[0110] In S2402, the information processing device 104 includes not only the regions demarcated by the grid 2500 (Figure 25) set in S422, but also regions demarcated by grid 2501, which is shifted horizontally and vertically by half the size of one region, as the target regions for histogram calculation. In other words, this step is an example of a process for dividing the acquired image.

[0111] Specifically, as shown in 2510 and 2520, if an anomaly occurs near the 3x4 grid set in S422, the anomaly spans multiple areas. As a result, the threshold is not exceeded in each area, and all areas are judged as normal, potentially leading to low anomaly detection accuracy.

[0112] As in S2402, by determining anomalies not only in the area demarcated by the normal grid 2500 set in S422, but also in the area demarcated by the corrected grid 2501 (shifted grid), it becomes easier to improve the accuracy of anomaly detection even when an anomaly occurs near the normal grid 2500 set in S422.

[0113] In this embodiment, the grid is shifted horizontally and vertically by half the size of one region, but this method is not limited to this. Two new grids may be created by shifting them horizontally and vertically by "1 / 3" and "2 / 3" of the size of one region, or a hexagonal grid may be created instead of a grid (square). In other words, this step is an example of a process in which an image is divided using two or more division methods. Specifically, this is an example in which a region divided by the first division method partially overlaps with at least one of the regions divided by the second division method.

[0114] This makes it easier to improve the accuracy of detecting anomalies, even if an anomaly occurs near grid 2500.

[0115] In S2403, the information processing device 104 calculates a histogram of luminance information from the luminance information acquired in S2303 for each processing target area set in S422 and S2402.

[0116] In S2404, the information processing device 104 calculates a histogram of hue information from the hue and saturation information acquired in S2303 for each processing target area set in S422 and S2402.

[0117] In S2405, the information processing device 104 calculates a histogram of saturation information from the saturation information acquired in S2303 for each processing target area set in S422 and S2402.

[0118] In S2406, the information processing device 104 calculates edge gradient information from the lightness information acquired in S2303 for each processing area set in S422 and S2402, and uses the edge gradient information to calculate an edge gradient intensity histogram and an edge gradient angle histogram. Let's return to the explanation of Figure 23.

[0119] In S2305, the information processing device 104 calculates the similarity for each processing area set in S422 and S2402 using a histogram of reference information and a histogram intersection, which is one of the histogram comparison methods.

[0120] In this case, the reference information is the histogram calculated in S2308, described later. If reference histogram data is unavailable, the similarity is set to 100%.

[0121] In this embodiment, a histogram intersection was used as the histogram comparison method, but the method is not limited to this, and other methods such as the Bhattacharya coefficient may be used.

[0122] Furthermore, while this embodiment involves detecting anomalies by comparing histograms, it is not limited to this method. Other methods, such as template matching or feature point detection, may also be used to detect anomalies.

[0123] In S2306, if the number of data already set in the reference image is less than the "number of reference image frames" set in S425, the information processing device 104 determines that the reference image needs to be updated and proceeds to S2307. If data that satisfies the "number of reference image frames" has already been set in the reference image, the process proceeds to S2309.

[0124] Furthermore, the information processing device 104 discards the reference image in the following cases and initializes the number of set data items to 0.

[0125] • When the basic settings are changed during user operation of S427 • When the user clicks the "Recreate Reference Image" button (2003) This allows users to easily instruct the system to recreate the reference image.

[0126] In S2307, the information processing device 104 determines whether the data is suitable as a reference image based on the similarity calculated in S2305. If there are no areas with low similarity, the process proceeds to S2308. If there are many areas with low similarity, the device determines that the data is not suitable as a reference image and terminates the process.

[0127] In S2308, the information processing device 104 adds the histogram calculated in S2304 to the histogram of the reference information currently held, recalculates the average value, updates the histogram of the reference information, and terminates the process.

[0128] In S2309, the information processing device 104 compares the similarity of each processing target area set in S422, calculated in S2305, with the threshold set in S428, and records the areas that fall below the threshold as abnormal areas on the RAM 203. In other words, this step is an example of a process that determines whether a change has occurred in a given area based on changes between corresponding areas of multiple acquired images.

[0129] Specifically, in threshold 2001 (Figure 20), if the brightness threshold is set to 22%, the region to be processed will be judged as abnormal if the brightness similarity of the region is less than 22%.

[0130] In S2310, the information processing device 104 compares the similarity of each region (regions separated by a correction grid) that was calculated in S2305 and set as the region to be processed in S2402 with the threshold set in S428, and detects regions that fall below the threshold.

[0131] If an abnormality is detected at this point, proceed to S2311. If no abnormality is detected, proceed to S2312.

[0132] In S2311, the information processing device 104 determines that the area 2600 (Figure 26) where an anomaly was detected is different from the area to be notified to the user as set in S422. Therefore, it considers all areas 2610 (four areas in the case of Figure 26) that overlap with the detected area 2600 as areas where an anomaly was detected (2620) and records them as anomaly locations on the RAM 203.

[0133] As a result, the area that receives notification of an anomaly is the area demarcated by the normal grid 2500 set in S422, regardless of whether grid correction is performed or not. In other words, this step is an example of a process that outputs a determination result based on the area demarcated by the first division method when it is determined that there has been a change in the area demarcated by the second division method.

[0134] In this way, anomalies can always be notified to users and external applications based on areas divided by a fixed grid. In other words, because the notification method is consistent regardless of whether grid correction is applied or not, it is easy for users to understand the notifications, and for external applications, it has the advantage of not requiring complex methods for receiving anomaly notifications, as the receiving logic and output method for anomalies can be kept consistent.

[0135] In S2312, the information processing device 104 compares the abnormal area recorded on the RAM with the mask area set in S423. If there is an abnormal area other than the mask area, the process proceeds to S2313. If there is no abnormal area, or if all of the abnormal areas are mask areas, the process terminates. In other words, this step is an example of a process that controls whether or not to output a judgment result based on the area specified.

[0136] In this embodiment, the histogram calculation process (S2304) is performed even in the masked region, but this method is not limited to this, and the histogram calculation process (S2304) may not be performed in the masked region. By doing so, the number of times the histogram calculation process is performed is reduced when there is a masked region, and the overall processing time can be shortened.

[0137] On the other hand, when performing histogram calculation processing even in masked areas, as in this embodiment, there are advantages such as being able to record anomalies in the masked areas and suggest changes to the masked areas to the user, and being able to stably meet non-functional requirements by always maintaining a constant processing speed.

[0138] In S2313, the information processing device 104 performs an anomaly notification process (Figure 27) and notifies the user and external applications of information about the anomaly area. The abnormality notification process shown in Figure 27 will be explained below.

[0139] In S2701, the information processing device 104 determines whether there is an anomaly to be notified based on judgment criteria such as the continuity of anomaly detection. If it determines that there is an anomaly to be notified, it proceeds to S2702. If it determines that there is no anomaly to be notified, it proceeds to S2703.

[0140] Specifically, a criterion is needed to determine whether to notify a system of an anomaly if it is detected in a single image, or if to notify a system of an anomaly if it is detected consecutively in images spanning several seconds.

[0141] For example, the information processing device 104 may accept the user's pre-set criteria for how many consecutive frames of anomaly detection will trigger a notification, or it may determine what the image being analyzed is of (factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and then determine the criteria for notifying an anomaly based on the object and environment.

[0142] In S2702, the information processing device 104 records information about the region where the notification target abnormality has occurred on the RAM 203.

[0143] In S2703, the information processing device 104 deletes the notification-targeted abnormal information recorded on the RAM 203.

[0144] In S2710, the information processing device 104 passes the still image information read in S2302 to the processing unit that draws the settings screen.

[0145] In S2711, the information processing device 104 passes the abnormal information recorded in S2702 to the processing unit that draws the settings screen.

[0146] In S2712, the information processing device 104 determines whether the settings screen is being displayed based on user input. If the settings screen is being displayed, the process proceeds to S2713. If the settings screen is not being displayed, the process terminates.

[0147] In S2713, the information processing device 104 checks the settings 2801 for the detection area drawing color on RAM 203. If there is a specification regarding the drawing method, the process proceeds to S2714. If there is no specification, the process proceeds to S2715.

[0148] In S2714, the information processing device 104 changes the highlighting during the drawing process in S2715, described later, based on the setting 2801 (Figure 28). In other words, this step is an example of a process that identifies and outputs which of several types of thresholds was used to determine that a change had occurred.

[0149] Setting 2801 specifies how to highlight items according to the detection factors for anomaly detection (items that fall below the threshold in S2309). In other words, this step is an example of a process that accepts the specification of how to identify and output each type of threshold.

[0150] Here, to ensure that the highlighting for each threshold is distinguishable, control is implemented to prevent the same highlighting settings from overlapping (e.g., displaying an error message if the same highlighting is selected for different thresholds, preventing the selection of a previously selected highlighting). In other words, this step demonstrates an example of a process that prevents the specification of the same output method for different types of thresholds.

[0151] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0152] In this embodiment, highlighting is done using color, but this method is not limited to this. Other methods may be used, such as using patterns like diagonal lines, setting the highlighting method to surround items with dotted or thick lines, or setting different notification methods such as voice or email. It is also possible to set notification methods using methods other than highlighting.

[0153] In S2715, the information processing device 104 overlays highlighting on the areas containing the still image information acquired in S2710 and the abnormal information acquired in S2711, and then displays the drawing image in the drawing memory on the RAM 203.

[0154] In S2716, the information processing device 104 draws the drawing image created in S2715 onto the setting screen 2810 (Figure 28). In other words, this step is an example of a process that outputs a result of determining whether or not there has been a change in the region. In other words, this step is an example of a process that displays the region where an anomaly was detected in an identifiable manner based on the similarity of multiple images obtained from the video. If the drawing method is changed in S2714, the method of highlighting the abnormal region is changed and displayed as shown in abnormal region 2811(1)(2)(4).

[0155] This makes it easy for users to understand what threshold was used to determine that something was abnormal.

[0156] Alternatively, the highlighting of the abnormal area and the display of the masked area may be shown on the same screen (not illustrated).

[0157] Furthermore, although S2312 states that abnormal areas will not be displayed in the masked area, it is also possible to highlight the abnormal detection area (not shown) or highlight it in a different way (not shown) even in the masked area, based on user instructions. Doing so would give the user a reason to decide to remove the mask even in an area designated as a masked area, depending on the nature of the abnormal detection. In other words, it becomes possible to easily configure the abnormal detection desired by the user.

[0158] Furthermore, the information processing device 104 may have a function to notify the user on screens 1810 and 1820 of what kind of abnormality has occurred in the mask area, even without receiving instructions from the user.

[0159] Furthermore, the system may have functions to suggest to the user that they change the mask settings based on the number and nature of anomaly detections, or for the information processing device 104 to automatically change the mask settings. Doing so would enable more accurate anomaly detection. In other words, this step is an example of a process that changes the specified area based on the specified area and the judgment result of that area.

[0160] In S2720, the information processing device 104 notifies the output destination set in S407 (e.g., video monitoring server, mailer, software that can instruct to make calls using automated voice communication) of the abnormal information recorded in S2702.

[0161] Figure 29 shows an example of the anomaly notification dashboard screen, which displays a list of anomalies detected so far in a table format. Here, the detection time, monitoring task name, and coordinates of the detected area are displayed.

[0162] Figure 30 is an example of a flowchart for the process of acquiring video from an online video source.

[0163] In S3001, the information processing device 104 accepts an online video connection destination from the user. For example, an online video connection destination could be a video streaming service that provides videos, but it could also be another destination such as an external video sharing site. In this embodiment, for example, an external streaming service such as YouTube (YouTube is a registered trademark) is assumed. The online video connection destination is accepted by entering the video ID of the video to be connected to (in the case of YouTube, the string that identifies the video written after "v=" in the URL) into the input form 3108 on screen 3107 in Figure 31.

[0164] In addition to the video ID, a URL may be used, or any other information that can identify the destination of the online video. Furthermore, the online video may be a live stream. In other words, this is a step illustrating an example of a process for receiving identification information that identifies a video from a video streaming service.

[0165] In S3002, the information processing device 104 connects to an online video according to the online video connection destination received in S3002. For example, in the case of YouTube, the connection destination is identified by the video ID entered in S3002, and the device connects to the specified destination.

[0166] In S3003, the information processing device 104 displays the online video stream received in S3003 on the screen. The screen displayed in S3004 is shown as screen 3109 in Figure 31. Details of the function will be described later in Figure 31. In other words, this step is an example of a process that displays a video corresponding to identification information.

[0167] In S3004, the information processing device 104 receives the capture time interval for capturing the online video displayed in S3003. Specifically, the user can set the time interval for acquiring still images using the button 3110 on the screen 3109 in Figure 31, and in S3004, the time set therein is obtained and set as the time interval. In other words, this step is an example of a process that receives a setting related to the time for acquiring images from a video.

[0168] In S3005, the information processing device 104 determines whether the time interval for capturing, as received in S3004, has elapsed since the screenshot was saved to the shared memory in the previous S3007. If it determines that the time interval has elapsed, it proceeds to S3006. If it determines that the time interval has not elapsed, it proceeds to S3008. If there are no images saved to the shared memory in S3007, it may determine whether the set time interval has elapsed since the video was displayed in S3003, or whether the set time interval has elapsed since the capture interval was received in S3004.

[0169] In S3006, the information processing device 104 acquires a still image, which is a screenshot of the video displayed on screen 3109 in S3003. In other words, this step is an example of a process that acquires a still image from a video corresponding to the received identification information.

[0170] In S3007, the information processing device 104 saves the screenshot acquired in S3006 to the shared memory. In other words, this step is an example of a process that accepts an instruction to store an image in a storage device.

[0171] In S3008, the information processing device 104 determines whether the process of acquiring data from online video has finished. If it continues, it returns to the process in S3003. If it determines that it has finished (the close button in the upper right corner of screen 3109 is pressed and the window is closed), it terminates the process of acquiring video data from online video.

[0172] Still images acquired from online video and stored in shared memory undergo image difference (anomaly) detection processing as shown in Figure 23 at S2106 in Figure 21. Furthermore, in the anomaly notification process shown in Figure 27, anomaly detection is performed using the still images acquired from online video, and the result is displayed on screen 3115. In other words, this is a step that shows an example of a process that outputs a message indicating that an anomaly has been detected in the video corresponding to the identification information received by the reception means. Also, this is a step that shows an example of a process that outputs a message indicating that an anomaly has been detected based on the similarity of the acquired still images.

[0173] The process described above makes it possible to perform anomaly detection on online videos. This means, for example, that anomaly detection can be performed using video from live cameras installed outdoors that have been uploaded to video streaming services, increasing opportunities for verifying anomaly detection outdoors and helping to improve accuracy. Furthermore, it reduces the effort required to obtain permission to install cameras in order to obtain videos for verification, and allows for the efficient use of videos that are suitable for anomaly detection verification from already uploaded videos. It also makes it possible to verify videos that capture dangerous and rare situations, such as the moment a disaster occurs. In summary, it is possible to detect anomalies using videos provided by third parties.

[0174] Figure 31 shows an example of the display of the online video acquisition screen.

[0175] Screen 3101 is an example of a screen in step S2103 of Figure 21 where the information processing device 104 acquires input information according to the configured input method. In step S2104, if you want to acquire video from an online video on the web, you select "Video Streaming Load (Service A)" using the pull-down button 3102, which takes you to step S2107, and the process of acquiring the online video begins. In this embodiment, only one connection destination is registered as Service A, but other video acquisition destinations can also be registered and selected according to the acquisition destination.

[0176] Select "Load Video Streaming" using the pull-down button 3102 and confirm with button 3103. This will bring up a screen like screen 3104, displaying button 3105 with the name of the target video streaming service. Pressing button 3105 will display screen 3109 for viewing videos from the video streaming site.

[0177] If a connection destination is not specified beforehand, screen 3109 will be displayed, but the video will not be shown. By specifying a monitoring task with button 3106, you will be taken to the task registration information setting screen 3107, where you can edit the monitoring task. Multiple monitoring tasks can be set up, and the online video connection destination can be set for each monitoring task. On the task registration information setting screen 3107, you configure the online video connection destination for S3002. For example, to connect to an external streaming service such as YouTube, you can specify the online video connection destination by entering the aforementioned video ID or URL in input form 3108.

[0178] Screen 3109 is a screen that displays a video from a video streaming site, which is shown when button 3105 is pressed. This screen is displayed in S3003 and displays the online video specified in S3002. The frame acquisition interval can be set using button 3110 on screen 3109. In S3005, it is determined whether the frame acquisition interval set here has elapsed, and a still image is acquired. By making the frame acquisition interval configurable, the transmission interval can be changed according to the user's PC specifications and test environment, preventing the system from becoming slow.

[0179] Button 3111 accepts the size specification for screen 3109. By pressing button 3111, the user can freely select the screen size from a list of size options. This makes it possible to adjust the image resolution. Regardless of the video quality of the acquired online video, increasing the screen size with button 3111 increases the number of pixels in the width and height of the image, and also increases the absolute resolution. By making the image resolution adjustable through resizing, it becomes possible to verify a group of images with the same resolution and to verify the accuracy at each resolution. In other words, this screen shows an example of a process that accepts settings related to the size of the display screen for the video to be shown or settings related to the size of the image acquired from the video.

[0180] The toggle button 3112 is a button that allows you to turn capture ON / OFF. If you want to temporarily stop capturing images or examine a specific scene in a video, you can use this button to start / stop the still image acquisition process. In other words, this screen shows an example of a process that accepts settings for whether or not to acquire images from a video.

[0181] Alternatively, instead of setting an ON / OFF switch for capture, a capture button could be provided that allows for manual acquisition of still images when button 3113 is pressed, and these images can be saved to shared memory. In this case, the user can acquire the necessary still images by pressing the button at any time they choose. This screen shows an example of a process that accepts settings related to a video or images acquired from a video corresponding to identification information, and detects abnormalities in the video based on the accepted settings.

[0182] By using buttons 3110 to 3113 on screen 3109 to set the conditions for acquiring still images from online videos, users can freely change the acquisition conditions to match the conditions they wish to verify. Furthermore, by operating buttons 3110 to 3113 while viewing the video, users can acquire still images under their desired verification conditions. In this embodiment, the screen configuration is as described above, but it is not limited to this configuration; buttons 3110 to 3113 may be displayed in a different layout, or their respective settings may be displayed on separate screens.

[0183] Screen 3115 is a screen displayed when still images acquired from online video and stored in shared memory undergo image difference (anomaly) detection processing as shown in Figure 23 in S2106 of Figure 21, and further anomaly detection is performed using the still images acquired from online video in the anomaly notification processing as shown in Figure 27. In other words, it is a screen that shows an example of a process that displays the location where an anomaly has been detected in an identifiable manner.

[0184] On screen 3115, differences in still images acquired from video 3114 are detected and displayed as anomaly detection areas 3118. Button 3116 is identical to the basic settings button 1711, and when pressed, it displays the basic settings screen 1720 for analysis settings. Button 3117 is identical to the reference image re-creation button 2003, and pressing it updates the reference image.

[0185] Therefore, it is possible to detect anomalies using videos provided by third parties.

[0186] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium containing a program that realizes the functions of the embodiments described above to a system or device, and by having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0187] In this case, the program read from the recording medium itself realizes the novel function of the present invention, and the recording medium on which that program is recorded constitutes the present invention.

[0188] For recording media used to supply programs, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, and the like.

[0189] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0190] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0191] Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device. It goes without saying that the present invention can also be applied when the results are achieved by supplying a program to a system or device. In this case, by reading a recording medium containing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0192] The above program may consist of object code, program code executed by an interpreter, script data supplied to the OS (operating system), and the like.

[0193] Furthermore, by downloading and reading the program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. It should be noted that all configurations combining the above-described embodiments and their modified forms are also included in the present invention. [Explanation of symbols]

[0194] 100 Information Processing Systems 102 Cameras 104 Information Processing Device

Claims

1. A receiving means for receiving identification information that identifies videos from a video streaming service, An output means that outputs a message indicating that an anomaly has been detected in the video corresponding to the identification information received by the aforementioned receiving means, An information processing system characterized by comprising the following features.

2. The output means displays the location where the abnormality was detected in an identifiable manner. The information processing system according to claim 1, characterized by the following:

3. The output means displays the region where an anomaly was detected based on the similarity of multiple images obtained from the video in an identifiable manner. The information processing system according to claim 1, characterized by the following:

4. The system further includes an acquisition means for acquiring still images from a video corresponding to the identification information received by the aforementioned reception means. The output means outputs a message indicating that an anomaly has been detected based on the similarity of the still images acquired by the acquisition means. The information processing system according to claim 1, characterized by the following:

5. The system further includes a setting reception means for receiving settings related to a video corresponding to the identification information received by the aforementioned reception means, The output means detects abnormalities in the video based on the settings received by the setting reception means. The information processing system according to claim 1, characterized by the following:

6. The system further includes a setting reception means for receiving settings related to images obtained from a video corresponding to the identification information received by the aforementioned reception means, The output means detects abnormalities in the video based on the settings received by the setting reception means. The information processing system according to claim 1, characterized by the following:

7. The system further includes a display means for displaying a video corresponding to the identification information received by the aforementioned reception means. The information processing apparatus according to claim 1, characterized by the following:

8. A setting reception means that receives settings related to a video corresponding to the identification information received by the aforementioned reception means, The setting receiving means receives settings relating to the size of the display screen for the video displayed by the display means. The information processing system according to claim 7, characterized by the following:

9. The setting reception means accepts settings related to the size of the image to be obtained from the video. The information processing system according to claim 6, characterized by the following:

10. The setting acceptance means accepts a setting relating to the time for acquiring images from the video. The information processing system according to claim 6, characterized by the following:

11. The setting receiving means accepts a setting of whether or not to acquire images from the video. The information processing system according to claim 6, characterized by the following:

12. The system further includes an instruction receiving means for receiving instructions to store images obtained from the aforementioned video into a storage device. The information processing system according to claim 1, characterized by the following:

13. The system further includes a receiving means for receiving the specification of a region included in the image obtained from the aforementioned video, The output means controls whether or not to display the location where the abnormality was detected in an identifiable manner, based on the area specified by the receiving means. The information processing system according to claim 1, characterized by the following:

14. The information processing system's receiving means includes a receiving step in which it receives identification information that identifies a video from a video streaming service, An output step in which the output means of the information processing system outputs a statement indicating that an anomaly has been detected in the video corresponding to the identification information received in the reception step, An information processing system characterized by comprising the following features.

15. A program for causing at least one computer to function as one of the means of an information processing system described in any one of claims 1 to 13.

Images