Information processing systems, information processing methods, and programs

Abstract

To provide a system that can easily integrate with various devices to detect anomalies. [Solution] An information processing system characterized by comprising: an acquisition means for acquiring a program to be executed; a receiving means for receiving an image acquired or output by the execution of the program acquired by the acquisition means as an input image; and an output control means for controlling the system to output a message indicating that the input image contains an abnormality.

Description

Technical Field

[0005]

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and more particularly to a technique suitable for use in detecting abnormalities.

Background Art

[0002] Conventionally, there has been an information processing system that notifies an abnormality based on a comparison between a reference image and an analysis target image.

[0003] Patent Document 1 discloses a technique for extracting changes in an image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0008] According to the present invention, it is possible to provide a mechanism for detecting abnormalities by easily coordinating with various devices. [Brief explanation of the drawing]

[0009] [Figure 1] This is a system configuration diagram of information processing system 100. [Figure 2] This is a hardware block diagram of the information processing device 104. [Figure 3] This is an example of a block diagram showing the software configuration. [Figure 4] (a) A flowchart of the configuration process and (b) an example of a flowchart of the anomaly detection parameter setting process for the monitoring task. [Figure 5] This is an example of the initial display of the settings screen. [Figure 6] This is an example of an overview of the settings screen. [Figure 7] This is an example of the settings screen (input tab). [Figure 8] This is an example of the settings screen (input tab). [Figure 9] This is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11] This is an example of the settings screen (monitoring processes and monitoring tasks). [Figure 12] This is an example of the settings screen (input tab). [Figure 13] This is an example of the settings screen (input tab). [Figure 14] This is an example of the settings screen (notification tab). [Figure 15] This is an example of the display of the settings screen (notification tab). [Figure 16] This is an example of the display of the settings screen (monitoring task). [Figure 17] This is an example of the display of the adjustment screen for analysis settings. [Figure 18] This is an example of the display of the grid settings screen (grid division). [Figure 19] This is an example of the display of the grid settings screen (grid correction). [Figure 20] This is an example of the display of the analysis settings screen. [Figure 21] This is an example of a flowchart of the anomaly detection process. [Figure 22] This is an example of the display of task registration information. [Figure 23] This is an example of a flowchart of the image difference (anomaly) detection process. [Figure 24] This is an example of a flowchart of each histogram calculation process. [Figure 25] This is an example of the display of grid correction. [Figure 26] This is an example of the display of notifications during grid correction. [Figure 27] This is an example of a flowchart of the anomaly notification process. [Figure 28] This is an example of the display of the settings screen (anomaly notification). [Figure 29] This is an example of the display of the anomaly notification dashboard screen. [Figure 30] This is an example of a diagram showing an overview of reference image update in the second embodiment. [Figure 31] This is an example of a flowchart of the image difference (anomaly) detection process in the second embodiment. [Figure 32] This is an example of a flowchart of the initialization process of reference image update in the second embodiment.<000011​​​​​​​This is an example of a flowchart for the setup process in the third embodiment. [Figure 36] This is an example of the display of the settings screen (analysis tab) in the third embodiment. [Figure 37] This is an example of a script for using the pre-trained model in the third embodiment. [Figure 38] This is an example of a flowchart for the parameter setting process of a trained model in the third embodiment. [Figure 39] This is an example of a flowchart for the anomaly detection process in the third embodiment. [Figure 40] This is an example of a flowchart for the detection process using the trained model in the third embodiment. [Figure 41] This is an example of a flowchart for the process of having the trained model execute the detection process in the third embodiment. [Figure 42] This is an example of the display of the analysis settings screen in the third embodiment. [Figure 43] This is a system configuration diagram of the information processing system 100 in the fourth embodiment. [Figure 44] This is an example of a flowchart for the anomaly detection process in the fourth embodiment. [Figure 45] This is an example of a flowchart for image difference (anomaly) detection processing in the fourth embodiment. [Figure 46] This is an example of a flowchart for the custom loading process in the fourth embodiment. [Figure 47] This is an example of the display of the settings screen (input tab) in the fourth embodiment. [Figure 48] This is an example of a script for using a network camera in the fourth embodiment. [Figure 49] This is an example of a script for using the LiDAR sensor in the fourth embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0011] Figure 1 is a system configuration diagram showing an example of the configuration of the information processing system 100 of the present invention.

[0012] Figure 1 shows a configuration in which the camera 102 and the information processing device 104 are connected via an image transfer cable (USB, Ethernet, CameraLink, etc.). However, a configuration of an information processing device equipped with a camera is also acceptable, even without this specific setup.

[0013] The information processing device 104 acquires an image obtained by the camera 102 capturing the target object via an image transfer cable, and performs processing related to the detection of abnormalities in the image.

[0014] Furthermore, the information processing system 100 of this embodiment may also have a configuration in which the camera 102 with the above-described configuration does not exist.

[0015] In that case, processing related to the detection of abnormalities in images is performed on images acquired from other terminals on the network or on images stored in the memory (RAM) of the information processing device 104.

[0016] The following describes the hardware configuration of an information processing device applicable to the information processing device 104 shown in Figure 1, using Figure 2 as an example.

[0017] Figure 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in Figure 1.

[0018] In Figure 2, 201 is the CPU, which comprehensively controls each device and controller connected to the system bus 204. The ROM 202 or external memory 212 stores the BIOS (Basic Input / Output System), which is the control program for the CPU 201, the operating system program (hereinafter referred to as the OS), and various programs necessary to implement the functions executed by each PC, as described later.

[0019] 203 is RAM, which functions as the main memory, work area, etc., of the CPU 201. The CPU 201 loads the necessary programs, etc., from ROM 202 or external memory 212 into RAM 203, and then executes the loaded programs to perform various operations.

[0020] 205 is an input controller that controls input from pointing devices such as the keyboard (KB) 210 and a mouse (not shown).

[0021] 206 is a video controller that controls the display on indicators such as display 211.

[0022] 207 is a memory controller that controls access to external memory 212, such as external storage devices (hard disks (HDs)) that store various types of data, flexible disks (FDs), or CompactFlash® memory connected to a PCMCIA card slot via an adapter.

[0023] 208 is a communication interface controller that controls the reception of image data from the external PC 213 via the network (TCP / IP). 209 is an image I / F controller that controls the reception of image data from camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0024] The various programs described later for realizing the present invention are stored in RAM 203 and executed by CPU 201.

[0025] Furthermore, the image data used when executing the above program is stored in ROM202, external memory212, external PC213, and camera102 depending on the application, and is stored in RAM203 via various controllers when the program is executed. Figure 3 is an example of a block diagram showing the software configuration of an embodiment of the present invention.

[0026] The information processing device 104 includes the following functional units.

[0027] The acquisition unit 301 is a functional unit that acquires images captured at the same angle of view.

[0028] The reception unit 302 is a functional unit that receives specifications for the area to be included in the image and the method of displaying that area.

[0029] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on changes between corresponding regions of multiple images acquired by the acquisition unit 301.

[0030] The output unit 304 is a functional unit that outputs the determination result from the determination unit 303.

[0031] The output unit 304 is a functional unit that displays the area specified by the reception unit 302 using the display method specified by the reception unit 302, and controls whether or not to output a judgment result based on the area specified by the reception unit 302.

[0032] The area modification unit 305 is a functional unit that modifies the area specified by the reception unit 302 based on the area specified by the reception unit 302 and the determination result of that area.

[0033] The splitting unit 306 is a functional unit that splits the image acquired by the acquisition unit 301.

[0034] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on changes between corresponding regions that have been divided by the division means from among multiple images acquired by the acquisition unit 301.

[0035] The division unit 306 is a functional unit that divides an image using two or more division methods.

[0036] The output unit 304 is a functional unit that outputs a determination result based on the region divided by the first division method when the determination unit 303 determines that there has been a change in the region divided by the second division method.

[0037] The determination unit 303 is a functional unit that determines whether a change has occurred in a given region based on the changes between corresponding regions of multiple images acquired by the acquisition means and multiple types of thresholds.

[0038] The output unit 304 is a functional unit that identifies and outputs which of the multiple types of thresholds was used to determine that a change had occurred.

[0039] The reception unit 302 is a functional unit that accepts specifications for the method of identification and output for each type of threshold.

[0040] The reception unit 302 is a functional unit that controls the system to prevent the same output method specification from being accepted for different threshold types. The determination unit 303 is a functional unit that determines whether there is a change that should be notified based on a comparison between the reference image and the image to be analyzed.

[0041] The acquisition unit 301 is a functional unit that notifies the determination result that there has been a change that should be notified.

[0042] The notification unit 307 is a functional unit that acquires a predetermined value related to the suppression of notifications.

[0043] The notification unit 307 is a functional unit that suppresses the notification of a determination result indicating that a change requiring notification has occurred, based on a predetermined value.

[0044] The notification unit 307 is a functional unit that suppresses the notification of a determination result indicating that a change requiring notification has occurred, based on at least one of the number of changes to be notified and a predetermined value.

[0045] The notification unit 307 is a functional unit that suppresses the notification of the determination result that there has been a change that should be notified when the number of changes that should be notified is less than or equal to a predetermined value.

[0046] This concludes the explanation of Figure 3. The setup process shown in Figure 4(a) will be explained below.

[0047] In S401, the information processing device 104 determines whether a configuration file for the operating process exists in the external memory 212. If it is not registered, the process proceeds to S402; if it is registered, the process proceeds to S403.

[0048] In S402, the information processing device 104 displays the settings screen 510 (Figure 5), and when it receives a press of the add monitoring process button 511 from the user, it registers the operation process in RAM 203 and displays the settings screen 520.

[0049] In S403, when the information processing device 104 receives a press of the button 521 for selecting the operation process to be edited on the setting screen 502, it displays the setting screen 610 (Figure 6). The setting acceptance process performed on setting screen 610 in S403 will be explained below using Figures 6 to 10.

[0050] When the information processing device 104 receives input from the user for the name 611 and description 612 of the operating process, it edits the operating process information in the RAM 203 and displays the settings screen 620.

[0051] When the information processing device 104 receives a press from the user on the settings screen 610, it displays the settings screen 710 (Figure 7).

[0052] When the information processing device 104 receives a selection from the user for "Video file loading (FFmpeg)" (721 in Figure 7) from the list of functions 711, it displays the settings screen 720. At this time, the settings screen 720 edits the operation process information of RAM 203 based on the location 722 and option 723 values ​​of the command entered by the user.

[0053] When the information processing device 104 receives a selection from the user for "Video file loading (OpenCV)" (731 in Figure 7) from the list of functions 711, it displays the settings screen 730 and edits the operation process information of RAM 203.

[0054] When the information processing device 104 receives a selection from the user for "Live video loading (video surveillance server)" (811 in Figure 8) from the list of functions 711, it displays the settings screen 810 (Figure 8). "Live video loading (video surveillance server)" is a function that loads live video captured by a network camera from a video surveillance server (not shown) that can be connected via the network.

[0055] Based on the computer name 812, username 813, and password 814 values ​​entered by the user on the settings screen 810, the operating process information of RAM 203 is edited. Also, if the connection confirmation 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in RAM 203.

[0056] When the information processing device 104 receives a click from the user on the settings screen 610 to select the analysis tab 614 (Figure 6), it displays the settings screen 910 (Figure 9).

[0057] When the information processing device 104 receives a selection of "Detection by video comparison" from the function list 911 from the user, it displays the settings screen 920 and edits the operation process information of RAM 203. At this time, if the license is not authenticated, "No license" is displayed in the license information 922.

[0058] When the information processing device 104 receives a press of license read button 923 from the user, if a valid license has been authenticated, it displays the settings screen 930, displays the license information 931, and edits the operation process information of RAM 203.

[0059] When the information processing device 104 receives a notification from the user on the settings screen 610 by clicking the notification tab 615 (Figure 6), it displays the settings screen 1010 (Figure 10).

[0060] When the information processing device 104 receives a selection of a "custom command" (1021 in Figure 10) from the function list 1011 from the user, it displays the settings screen 1020 and edits the operation process information of the RAM 203.

[0061] A "custom command" is a function that executes commands to instruct batches or applications to run when an anomaly is detected. Using this function, it becomes possible to execute batch processing, instruct email sending to mail clients, instruct automated voice calls to be made, and integrate with other applications or external systems.

[0062] When the information processing device 104 receives a selection from the user for "Analysis Result Notification (Video Surveillance Server)" (1031 in Figure 10) from the function list 1011, it displays the settings screen 1030.

[0063] "Analysis Result Notification (Video Surveillance Server)" is a function that notifies the video surveillance server of the analysis results. This function is used when the video surveillance server is configured to notify users of anomalies, or when the video surveillance server analyzes and manages events such as anomalies.

[0064] Based on the values ​​of address 1032 and port number 1033 entered by the user on the settings screen 1030, the operating process information of RAM203 is edited.

[0065] When the information processing device 104 receives a connection confirmation 1034 from the user, it performs a connection test via the communication I / F controller 208 using the information in the RAM 203. Returning to the explanation of Figure 4.

[0066] In S404, when the information processing device 104 receives a press of the OK button 631 on the setting screen 630 (Figure 6), it saves the operation process information registered in RAM 203 to external memory 212 as an operation process setting file.

[0067] In S405, the information processing device 104 determines whether a configuration file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406; if it is registered, the process proceeds to S407.

[0068] In S406, the information processing device 104 displays the settings screen 630, and when it receives a press from the user of the Add task button 632 for the monitoring process, it registers the monitoring task in RAM 203 and displays the settings screen 1110 (Figure 11).

[0069] In S407, when the information processing device 104 receives a click on the monitoring task 1111 to be edited on the settings screen 1110, it displays the settings screen 1120. The following describes the process for receiving settings for monitoring tasks performed in S407, using Figures 11 to 15.

[0070] When the information processing device 104 receives input from the user on the settings screen 1120, including a name 1121 and a description 1122, and a change in the toggle button 1123 for enabling this task, it edits the operation process information of the RAM 203 and displays the settings screen 1130.

[0071] When the information processing device 104 receives a press of the input tab 1124 from the user on the settings screen 1120, it displays either the settings screen 1200 (Figure 12) or the settings screen 1300 (Figure 13) depending on the settings of the operation process.

[0072] If the input setting for the operation process is "Load video file (FFmpeg)" or "Load video file (OpenCV)", the settings screen 1200 (Figure 12) will be displayed.

[0073] When the information processing device 104 receives a click on a video folder 1201 and a folder selection from the user, input of a frame acquisition interval 1202, and a change in the toggle button for deleting processed videos 1203, it edits the operation process information of RAM 203 and displays the settings screen 1210.

[0074] If the input setting for the operation process is "live video loading," the settings screen 1300 (Figure 13) is displayed. When the information processing device 104 receives a selection of a list item for camera 1301 from the user, it edits the operation process information in RAM 203 and displays the settings screen 1310.

[0075] When the information processing device 104 receives a notification from the user on the settings screen 1120 by clicking the notification tab 1125, it displays either the settings screen 1400 or the settings screen 1500, depending on the settings of the operation process.

[0076] If the notification setting for the operation process is "custom command," the settings screen 1400 (Figure 14) is displayed. When the information processing device 104 receives input of the command path 1401 from the user or a change in the toggle button for asynchronous execution 1402, it edits the operation process information in RAM 203 and displays the settings screen 1410.

[0077] If the notification setting for the operation process is "Analysis Result Notification (Video Surveillance Server)", the settings screen 1500 (Figure 15) is displayed. The information processing device 104 accepts the input of name 1501 and camera ID 1502 from the user and displays screen 1510. At this time, if the test event button 1511 is pressed by the user, a test notification process is executed to confirm whether the notification with the input content works correctly. At this time, if the setting of the detection frame display toggle button 1512 is enabled, test pseudo detection result information is also added to the notification content. Returning to the explanation of Figure 4.

[0078] In S408, when the information processing device 104 receives a press of the OK button 1602 from the user on the setting screen 1600 (Figure 16), it saves the monitoring task information registered in RAM 203 as a monitoring task setting file in external memory 212 and displays the setting screen 1610.

[0079] In S409, when the user clicks the analysis tab 1611 on the settings screen 1610, the settings screen 1700 (Figure 17) is displayed. At this time, when the user clicks the adjustment button 1701, the analysis settings adjustment screen 1710 is displayed.

[0080] Subsequently, the information processing device 104 executes the process shown in Figure 4(b). The process for setting anomaly detection parameters for the monitoring task shown in Figure 4(b) will be explained below.

[0081] In S421, when the information processing device 104 receives a press of the basic settings button 1711 from the user on the analysis settings adjustment screen 1710 (Figure 17), it displays the basic settings screen 1720 for analysis settings and proceeds to S422. If the Basic Settings button is not pressed, proceed to S428.

[0082] In S422, when the information processing device 104 receives a request from the user to change the slider bar to 1801 rows (Figure 18) and 1802 columns, it updates the display of the grid (dashed lines) on the screen, which is divided by 1801 rows and 1802 columns, as shown in screen 1800 (Figure 18), and saves the settings to RAM 203.

[0083] Specifically, on screen 1800, the number of rows is set to 10 (1801) and the number of columns to 10 (1802), so the image to be analyzed is displayed, divided into a 10x10 grid.

[0084] While a larger number of grids allows for more precise detection, it also increases the number of processes involved, such as the image difference (anomaly) detection process described later (Figure 23), potentially increasing the processing time.

[0085] In S423, when the information processing device 104 receives a press 1811 of any area divided by a grid on the user's screen, a press of the full mask setting button 1812, or a press of the full mask release button 1813, it displays the presence or absence of a mask (notification suppression area) as shown on screen 1810 and saves the setting in RAM 203. In other words, this step is an example of a process that accepts the specification of an area included in an image.

[0086] If there are many areas to mask, you can easily set the areas to be masked by first masking all areas using the "Set All Masks" button (1812), and then specifying the areas to be unmasked.

[0087] If the masked area is small, you can easily set the area to be masked by first removing the mask from the entire area using the "Remove All Mask" button (1813), and then specifying the area to be masked.

[0088] Furthermore, if the user changes the mask color inversion 1821 toggle button at this time, the mask drawing color is changed as shown in screen 1820. Specifically, in screen 1810, the mask area was represented by transparent black, but in screen 1820, the mask area is represented by transparent white. In other words, this step is an example of a process that accepts a specification for how to display an area included in an image. This step is also an example of a process that accepts a specification for the display method of the area, and displays the area that has been specified using the accepted display method.

[0089] This allows the user to select the color of the mask area according to the color tone and brightness of the image being analyzed. This is achieved by representing the mask area with transparent black when the image is bright and transparent white when the image is dark. As a result, the user can easily identify and understand the appearance of the mask area.

[0090] In this embodiment, the color of the mask area is set to "transparent black" or "transparent white," but this method is not limited to this. Other colors may be used, the user may specify a color, patterns such as diagonal lines may be used to add visual variation, a frame may be used, or other methods may be used to distinguish the monitored area from the mask area.

[0091] In S424, when the information processing device 104 receives a change from the user to the grid correction (shifting the grid) toggle button 1901 (Figure 19) or 1911, it updates the display of the frame 1902 (with frame) or 1912 (without frame) that indicates the detection range and saves the setting to RAM 203.

[0092] In this embodiment, grid correction is set for the monitoring process, but this method is not limited to this one. Grid correction may also be set directly for the network camera or video monitoring server that is the source of the image input.

[0093] In S425, when the information processing device 104 receives a numerical input 1921 for the number of reference image frames from the user, it saves the setting in the RAM 203.

[0094] In S426, when the information processing device 104 receives a press of the back button 1803 from the user while the basic settings screen 1800 (Figure 18) for analysis settings is displayed, it determines whether there are any changes to the basic settings. If there are changes, it proceeds to S427. If there are no changes, it proceeds to S428.

[0095] In S427, the information processing device 104 reflects the contents set in RAM 203 in the monitoring task configuration file in external memory 212. The information processing device 104 also detects changes in the monitoring task configuration file and performs reference image discarding during the analysis process. (In S2306, it determines that updating the base information is necessary.) In S428, when the information processing device 104 receives a change in the slider bar for each item of the threshold 2001 on the analysis setting screen 2000 (Figure 20), it saves the setting to the RAM 203. In other words, step S428 is a step that shows an example of a process that accepts the setting of a threshold related to similarity. In other words, this step is a step that shows an example of a process that accepts the setting of a threshold for similarity related to at least one piece of information from luminance, hue, saturation, and edge gradient.

[0096] In this embodiment, the user sets each threshold, but this method is not the only way. The user may select a dataset for each threshold, the user may set the objects to be monitored or the anomalies to be detected to automatically set the threshold parameters, or the information processing device 104 may determine the objects to be monitored or the anomalies to be detected and automatically set the threshold parameters.

[0097] This makes it easy to set anomaly detection thresholds.

[0098] In S429, when the information processing device 104 receives a press of the apply button 2002 from the user, it reflects the contents set in RAM 203 in the monitoring task configuration file in external memory 212. The information processing device 104 also detects changes in the monitoring task configuration file and modifies the parameter values ​​used for detection processing. (This modifies the threshold used in S2310.) The anomaly detection process shown in Figure 21 will be explained below.

[0099] In S2101, the information processing device 104 executes anomaly detection processing if the external memory 212 contains a configuration file for the monitoring task (task registration information 2201 (Figure 22)) and the task activation setting toggle 2202 is set to enabled.

[0100] In S2102, the information processing device 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and repeats.

[0101] In S2103, the information processing device 104 acquires input information according to the configured input method (configured on the configuration screen 1200 or configuration screen 1300).

[0102] Specifically, if the input setting for the operation process (settings screen 1200) is "Load video file (FFmpeg)" or "Load video file (OpenCV)", the video files located in the specified video folder are used as input information and moved to the working area on external memory 212.

[0103] Furthermore, if the input setting for the operation process is "Live video loading (video monitoring server)" (settings screen 1300), the process of acquiring video from the specified video acquisition destination is executed, and if a still image is acquired, the process proceeds to S2106.

[0104] In S2104, if the information processing device 104 performed a video file transfer on the external memory 212 in S2103, it proceeds to S2105. If it has performed a video acquisition process and acquired a still image, it proceeds to S2106.

[0105] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into a series of still images for each frame, saves them as files on the external memory 212 or stores them as data on the RAM 203, and proceeds to S2106.

[0106] In S2106, the information processing device 104 executes the process shown in Figure 23. The first embodiment will now be described.

[0107] The image difference (anomaly) detection process shown in Figure 23 will be explained.

[0108] In S2301, if the information processing device 104 has been splitting the video file into a series of still images frame by frame (S2105), it repeats the subsequent processing until processing is completed for all the generated still images.

[0109] In S2302, the information processing device 104 reads the still image to be processed and loads it onto the RAM 203. In other words, this step is an example of a process for acquiring images captured at the same angle of view. In other words, this step is an example of a process for sequentially acquiring information related to the image to be judged.

[0110] In S2303, the information processing device 104 calculates the hue, saturation, and lightness information from the image data loaded on the RAM 203.

[0111] In S2304, the information processing device 104 performs each histogram calculation process (Figure 24) to calculate a histogram from the image data. Here, we will explain the calculation process for each histogram in Figure 24.

[0112] In S2401, the information processing device 104 proceeds to S2402 if the grid correction set in S424 is enabled among the settings stored in RAM 203. Otherwise, it proceeds to S2403.

[0113] In S2402, the information processing device 104 includes not only the regions demarcated by the grid 2500 (Figure 25) set in S422, but also regions demarcated by grid 2501, which is shifted horizontally and vertically by half the size of one region, as the target regions for histogram calculation. In other words, this step is an example of a process for dividing the acquired image.

[0114] Specifically, as shown in 2510 and 2520, if an anomaly occurs near the 3x4 grid set in S422, the anomaly spans multiple areas. As a result, the threshold is not exceeded in each area, and all areas are judged as normal, potentially leading to low anomaly detection accuracy.

[0115] As in S2402, by determining anomalies not only in the area demarcated by the normal grid 2500 set in S422, but also in the area demarcated by the corrected grid 2501 (shifted grid), it becomes easier to improve the accuracy of anomaly detection even when an anomaly occurs near the normal grid 2500 set in S422.

[0116] In this embodiment, the grid is shifted horizontally and vertically by half the size of one region, but this method is not limited to this. Two new grids may be created by shifting them horizontally and vertically by "1 / 3" and "2 / 3" of the size of one region, or a hexagonal grid may be created instead of a grid (square). In other words, this step is an example of a process in which an image is divided using two or more division methods. Specifically, this is an example in which a region divided by the first division method partially overlaps with at least one of the regions divided by the second division method.

[0117] This makes it easier to improve the accuracy of detecting anomalies, even if an anomaly occurs near grid 2500.

[0118] In S2403, the information processing device 104 calculates a histogram of luminance information from the luminance information acquired in S2303 for each processing target area set in S422 and S2402.

[0119] In S2404, the information processing device 104 calculates a histogram of hue information from the hue and saturation information acquired in S2303 for each processing target area set in S422 and S2402.

[0120] In S2405, the information processing device 104 calculates a histogram of saturation information from the saturation information acquired in S2303 for each processing target area set in S422 and S2402.

[0121] In S2406, the information processing device 104 calculates edge gradient information from the lightness information acquired in S2303 for each processing area set in S422 and S2402, and uses the edge gradient information to calculate an edge gradient intensity histogram and an edge gradient angle histogram.

[0122] In other words, steps S2304 and Figure 24 show an example of a process that compares a reference image and an image to be analyzed based on the image's features.

[0123] Furthermore, S2304 and Figure 24 show an example of a process that compares a reference image and an image to be analyzed based on information relating to at least one of the following: brightness, hue, saturation, edge gradient intensity, and edge gradient angle.

[0124] Let's return to the explanation of Figure 23.

[0125] In S2305, the information processing device 104 calculates the similarity for each processing area set in S422 and S2402 using a histogram of reference information and a histogram intersection, which is one of the histogram comparison methods.

[0126] In this case, the reference information is the histogram calculated in S2308, described later. If reference histogram data is unavailable, the similarity is set to 100%.

[0127] In this embodiment, a histogram intersection was used as the histogram comparison method, but the method is not limited to this, and other methods such as the Bhattacharya coefficient may be used.

[0128] Furthermore, while this embodiment involves detecting anomalies by comparing histograms, it is not limited to this method. Other methods, such as template matching or feature point detection, may also be used to detect anomalies.

[0129] In S2306, if the number of data already set in the reference image is less than the "number of reference image frames" set in S425, the information processing device 104 determines that the reference image needs to be updated and proceeds to S2307. If data that satisfies the "number of reference image frames" has already been set in the reference image, the process proceeds to S2309.

[0130] Furthermore, the information processing device 104 discards the reference image in the following cases and initializes the number of set data items to 0.

[0131] • When the basic settings are changed during user operation of S427 • When the user clicks the "Recreate Reference Image" button (2003) This allows users to easily instruct the system to recreate the reference image.

[0132] In S2307, the information processing device 104 determines whether the data is suitable as a reference image based on the similarity calculated in S2305. If there are no areas with low similarity, the process proceeds to S2308. If there are many areas with low similarity, the device determines that the data is not suitable as a reference image and terminates the process.

[0133] In S2308, the information processing device 104 adds the histogram calculated in S2304 to the histogram of the reference information currently held, recalculates the average value, updates the histogram of the reference information, and terminates the process.

[0134] In S2309, the information processing device 104 compares the similarity of each processing target area set in S422, which was calculated in S2305, with the threshold set in S428, and records the areas that fall below the threshold as abnormal areas on the RAM 203.

[0135] In other words, this step is an example of a process that determines whether a change has occurred in a given region based on changes between corresponding regions of multiple acquired images.

[0136] In other words, this step is an example of a process that determines whether there are any changes that should be notified, based on a comparison between a reference image and the image to be analyzed.

[0137] In other words, this step is an example of a process that determines whether a predetermined change has occurred based on a comparison between information relating to a reference image and information relating to the image to be judged. In other words, this step is an example of a process that determines whether a change has occurred based on the similarity of information relating to multiple input images. In other words, this step is an example of a process that determines whether a change that should be output has occurred based on the similarity between information relating to a reference image and information relating to the image to be analyzed. In other words, this step is an example of a process that determines whether a change that should be output has occurred based on the similarity between information relating to a region included in the reference image and information relating to that region included in the image to be analyzed.

[0138] Specifically, in threshold 2001 (Figure 20), if the brightness threshold is set to 22%, the region to be processed will be judged as abnormal if the brightness similarity is less than 22%.

[0139] In this embodiment, similarity < threshold is used, but this is not the only option; similarity ≤ threshold may also be used.

[0140] Furthermore, while this embodiment uses similarity, it is not limited to similarity; change amount may also be used. In that case, the region will be judged as abnormal if change amount > threshold or change amount ≥ threshold.

[0141] In other words, this step is to show an example of what constitutes a change that should be notified: a change where the similarity between the reference image and the image being analyzed is less than a predetermined threshold, or the similarity is less than or equal to a predetermined threshold, or the amount of change exceeds a predetermined threshold, or the amount of change is greater than or equal to a predetermined threshold.

[0142] Furthermore, this step compares the reference image and the image to be analyzed in corresponding regions and provides an example of when the similarity between the regions is less than a predetermined threshold, or the similarity between the regions is less than or equal to a predetermined threshold, or the amount of change between the regions exceeds a predetermined threshold, or the amount of change between the regions is greater than or equal to a predetermined threshold.

[0143] In S2310, the information processing device 104 compares the similarity of each region (regions separated by a correction grid) that was calculated in S2305 and set as the region to be processed in S2402 with the threshold set in S428, and detects regions that fall below the threshold.

[0144] If an abnormality is detected at this point, proceed to S2311. If no abnormality is detected, proceed to S2312.

[0145] In S2311, the information processing device 104 determines that the area 2600 (Figure 26) where an anomaly was detected is different from the area to be notified to the user as set in S422. Therefore, it considers all areas 2610 (four areas in the case of Figure 26) that overlap with the detected area 2600 as areas where an anomaly was detected (2620) and records them as anomaly locations on the RAM 203.

[0146] As a result, the area that receives notification of an anomaly is the area demarcated by the normal grid 2500 set in S422, regardless of whether grid correction is performed or not. In other words, this step is an example of a process that outputs a determination result based on the area demarcated by the first division method when it is determined that there has been a change in the area demarcated by the second division method.

[0147] In this way, anomalies can always be notified to users and external applications based on areas divided by a fixed grid. In other words, because the notification method is consistent regardless of whether grid correction is applied or not, it is easy for users to understand the notifications, and for external applications, it has the advantage of not requiring complex methods for receiving anomaly notifications, as the receiving logic and output method for anomalies can be kept consistent.

[0148] In S2312, the information processing device 104 compares the abnormal area recorded on the RAM with the mask area set in S423. If there is an abnormal area other than the mask area, the process proceeds to S2313. If there is no abnormal area, or if all of the abnormal areas are mask areas, the process terminates. In other words, this step is an example of a process that controls whether or not to output a judgment result based on the area specified.

[0149] In this embodiment, the histogram calculation process (S2304) is performed even in the masked region, but this method is not limited to this, and the histogram calculation process (S2304) may not be performed in the masked region. By doing so, the number of times the histogram calculation process is performed is reduced when there is a masked region, and the overall processing time can be shortened.

[0150] On the other hand, when performing histogram calculation processing even in masked areas, as in this embodiment, there are advantages such as being able to record anomalies in the masked areas and suggest changes to the masked areas to the user, and being able to stably meet non-functional requirements by always maintaining a constant processing speed.

[0151] In S2313, the information processing device 104 performs an anomaly notification process (Figure 27) and notifies the user and external applications of information about the anomaly area. The abnormality notification process shown in Figure 27 will be explained below.

[0152] In S2701, the information processing device 104 determines whether there is an anomaly to be notified based on judgment criteria such as the continuity of anomaly detection. If it determines that there is an anomaly to be notified, it proceeds to S2702. If it determines that there is no anomaly to be notified, it proceeds to S2703.

[0153] Specifically, a criterion is needed to determine whether to notify a system of an anomaly if it is detected in a single image, or if to notify a system of an anomaly if it is detected consecutively in images spanning several seconds.

[0154] For example, the information processing device 104 may accept the user's pre-set criteria for how many consecutive frames of anomaly detection will trigger a notification, or it may determine what the image being analyzed is of (factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and then determine the criteria for notifying an anomaly based on the object and environment.

[0155] In S2702, the information processing device 104 records information about the region where the notification target abnormality has occurred on the RAM 203.

[0156] In S2703, the information processing device 104 deletes the notification-targeted abnormal information recorded on the RAM 203.

[0157] In S2710, the information processing device 104 passes the still image information read in S2302 to the processing unit that draws the settings screen.

[0158] In S2711, the information processing device 104 passes the abnormal information recorded in S2702 to the processing unit that draws the settings screen.

[0159] In S2712, the information processing device 104 determines whether the settings screen is being displayed based on user input. If the settings screen is being displayed, the process proceeds to S2713. If the settings screen is not being displayed, the process terminates.

[0160] In S2713, the information processing device 104 checks the settings 2801 for the detection area drawing color on RAM 203. If there is a specification regarding the drawing method, the process proceeds to S2714. If there is no specification, the process proceeds to S2715.

[0161] In S2714, the information processing device 104 changes the highlighting during the drawing process in S2715, described later, based on the setting 2801 (Figure 28). In other words, this step is an example of a process that identifies and outputs which of several types of thresholds was used to determine that a change had occurred.

[0162] Setting 2801 specifies how to highlight items according to the detection factors for anomaly detection (items that fall below the threshold in S2309). In other words, this step is an example of a process that accepts the specification of how to identify and output each type of threshold.

[0163] Here, to ensure that the highlighting for each threshold is distinguishable, control is implemented to prevent the same highlighting settings from overlapping (e.g., displaying an error message if the same highlighting is selected for different thresholds, preventing the selection of a previously selected highlighting). In other words, this step demonstrates an example of a process that prevents the specification of the same output method for different types of thresholds.

[0164] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0165] In this embodiment, highlighting is done using color, but this method is not limited to this. Other methods may be used, such as using patterns like diagonal lines, setting the highlighting method to surround items with dotted or thick lines, or setting different notification methods such as voice or email. It is also possible to set notification methods using methods other than highlighting.

[0166] In S2715, the information processing device 104 overlays highlighting on the areas containing the still image information acquired in S2710 and the anomaly information acquired in S2711, and then expands the drawing image in the drawing memory on the RAM 203. In other words, this is a step that shows an example of a process that identifies and displays the areas containing anomalies in the input image. In other words, this is a step that shows an example of a process that controls the system to identify and display the areas containing anomalies in the image to be analyzed.

[0167] In S2716, the information processing device 104 draws the drawing image created in S2715 on the setting screen 2810 (Figure 28). In other words, this step is an example of a process that outputs the result of determining whether there has been a change in the region. In other words, this step is an example of a process that outputs the result of a process based on an image comparison between the reference image and the target image. In other words, this step is an example of a process that controls the system to output that an abnormality is contained in the input image. In other words, this step is an example of a process that identifies and displays the region containing the abnormality in the input image based on the execution result of the analysis program. In other words, this step is an example of a process that controls the system to output that an abnormality is contained in the image to be analyzed based on the similarity between the information related to the reference image and the information related to the image to be analyzed.

[0168] If the drawing method is changed in S2714, the method of highlighting the abnormal region is changed and displayed as shown in abnormal region 2811(1)(2)(4).

[0169] This makes it easy for users to understand what threshold was used to determine that something was abnormal.

[0170] Alternatively, the highlighting of the abnormal area and the display of the masked area may be shown on the same screen (not illustrated).

[0171] Furthermore, although S2312 states that abnormal areas are not displayed in the masked area, it is also possible to highlight the abnormal detection area (not shown) or highlight it in a different way (not shown) even in the masked area, based on user instructions. This is because it can prompt the user to decide whether to remove the mask, even in an area designated as a masked area, depending on the nature of the abnormal detection. In other words, it becomes possible to easily configure the abnormal detection desired by the user.

[0172] Furthermore, the information processing device 104 may have a function to notify the user on screens 1810 and 1820 of what kind of abnormality has occurred in the mask area, even without receiving instructions from the user.

[0173] Furthermore, the system may have functions to suggest to the user that they change the mask settings based on the number and nature of anomaly detections, or for the information processing device 104 to automatically change the mask settings. Doing so would enable more accurate anomaly detection. In other words, this step is an example of a process that changes the specified area based on the specified area and the judgment result of that area.

[0174] In S2720, the information processing device 104 notifies the output destination set in S407 (e.g., video monitoring server, mailer, software that can instruct to make calls using automated voice calls, etc.) of the abnormal information recorded in S2702.

[0175] In other words, this step is an example of a process that notifies the result of a determination that a change requiring notification has occurred.

[0176] In other words, this step is an example of a process that outputs the results of an image comparison between a reference image and a target image.

[0177] Figure 29 shows an example of the anomaly notification dashboard screen, which displays a list of anomalies detected so far in a table format. Here, the detection time, monitoring task name, and coordinates of the detected area are displayed. As a result, users will be able to easily find out what threshold was used to determine that something was abnormal.

[0178] Furthermore, it will be possible to improve the accuracy of detecting anomalies near the grid.

[0179] Furthermore, it becomes easier to determine what criteria were used to detect the anomaly.

[0180] This concludes the description of the first embodiment. Next, a second embodiment for updating the reference image will be described.

[0181] Figure 30 is an example of a diagram showing an overview of the reference image update in the second embodiment.

[0182] When using three image frames as reference images, conventionally (Figure 30, upper side) features are extracted from the three reference images (for each grid), the average value is calculated (for each grid), and a histogram of the reference images is created (S2308 in Figure 23). Here, we assume that the average of the features of the three reference images is 4.0. (Figure 30, upper right) The improved second embodiment (bottom of Figure 30) allows for the automatic updating of the reference image's features using the features of images at 10-frame intervals, whereas previously the anomaly detection system could not automatically update the reference image's features at the timing desired by the user. By setting a numerical value (10 in this case) to the "interval frame count" setting, the reference image's features can now be automatically updated.

[0183] Specifically, if the feature value of the image used for the update is 7.0, and the average feature value of the three previous reference images was 4.0, then by replacing one of the reference images, the average feature value of the new reference image is updated to 5.0 by (4.0 + 4.0 + 7.0) / 3. (Figure 30, bottom right) This update process will continue automatically at 10-frame intervals until the "interval frame count" is changed.

[0184] In this embodiment, the reference image is used as a feature, but this method is not limited to this one. Alternatively, the reference image may be stored as an image, and an image comparison between the reference image and the image to be analyzed may be performed.

[0185] This concludes the explanation of Figure 30. Figure 31 is an example of a flowchart that rewrites the flowchart in Figure 23 of the first embodiment for use in the second embodiment. The same processes as in Figure 23 will not be explained.

[0186] In S3101, the information processing device 104 performs the initialization process for automatic updating of the reference image (Figure 32).

[0187] In S3102, the information processing device 104 performs the process of automatically updating the reference image (Figure 33).

[0188] This concludes the explanation of Figure 31. Figure 32 is an example of a flowchart for the initialization process of updating the reference image in the second embodiment.

[0189] In S3201, the information processing device 104 initializes the variable for the number of interval frames used to determine the timing of the automatic update of the reference image to 0.

[0190] In S3202, the information processing device 104 accepts the setting of the interval frame setting value 3211.

[0191] In other words, this step is an example of a process that accepts a first predetermined setting related to updating information relating to a reference image.

[0192] In other words, the interval frame setting value 3211 is a setting to identify the information related to the target image used to update the information related to the reference image, from among the information related to the target image acquired sequentially.

[0193] If the interval frame setting value 3211 ≥ 1, the reference image is updated using the analysis target image when the interval frame variable ≥ the interval frame setting value 3211. However, if the interval frame setting value 3211 = 0, the reference image is not automatically updated.

[0194] In S3203, the information processing device 104 accepts the setting of an anomaly detection ratio threshold 3212 for determining whether or not to automatically update the reference image.

[0195] In other words, the anomaly detection rate threshold 3212 is a second predetermined setting related to the information of the reference image.

[0196] In other words, this step is an example of a process for accepting a second predetermined setting related to information relating to a reference image.

[0197] The anomaly detection rate is calculated as the number of regions in which anomalies were detected in the analyzed image used for updating the baseline, divided by the number of regions divided by the grid. It is 100% if anomalies were detected in all regions, and 0% if no anomalies were detected.

[0198] If the anomaly detection rate threshold 3212 = 100, and the interval frame variable > interval frame setting value 3211, the reference image will always be updated using the image being analyzed.

[0199] If the anomaly detection rate threshold 3212 = 60, the reference image will be updated using the analyzed image only when the interval frame variable > interval frame setting value 3211, and the number of grids in the analyzed image used for reference updating that contain anomalies is less than 60%.

[0200] If the anomaly detection threshold 3212 = 0, the reference image will not be updated even if the interval frame variable > interval frame setting value 3211.

[0201] By having this anomaly detection threshold of 3212, if the image used for updating the reference image contains significant noise or other abnormalities (i.e., if the number of grids is judged to be "abnormal" at or above the anomaly detection threshold of 3212), the reference image will not be updated.

[0202] This prevents abnormal images, such as images containing noise above a threshold or images that deviate significantly from the reference image, from being included in the reference image, thereby suppressing excessive anomaly notifications.

[0203] This concludes the explanation of Figure 32. Figure 33 is an example of a flowchart for the reference image update process in the second embodiment.

[0204] In S3301, the information processing device 104 determines whether to automatically update the reference image. If the interval frame setting value 3211 = 0 set in S3202, it determines NO, terminates Figure 33, and does not automatically update the reference image. If the interval frame setting value 3211 ≥ 1, it determines YES and proceeds to S3302.

[0205] In S3302, the information processing device 104 adds 1 to the variable for the number of interval frames used to determine the timing of the automatic update of the reference image.

[0206] In S3303, the information processing device 104 determines whether the interval frame number variable has reached the interval frame number setting value 3211 (i.e., whether it is the automatic update timing for the reference image). Specifically, if the interval frame number variable is less than the interval frame number setting value of 3211, the result is determined to be NO, Figure 33 is terminated, and the reference image is not automatically updated at this stage. If the variable for the number of interval frames is greater than or equal to the set value for the number of interval frames (3211), the result is determined to be YES, and the process proceeds to S3304.

[0207] In this embodiment, the determination is made if the variable for the number of interval frames is greater than or equal to the set value for the number of interval frames 3211. However, the determination method is not limited to this, and the variable for the number of interval frames may be greater than or equal to the set value for the number of interval frames 3211.

[0208] In S3304, the information processing device 104 calculates the anomaly detection area ratio (= number of areas where anomalies were detected / number of areas divided by the grid) for the image to be analyzed for reference updating.

[0209] In S3305, the information processing device 104 determines whether the anomaly detection area ratio calculated in S3304 is greater than the anomaly detection ratio threshold 3212 set in S3203.

[0210] If the anomaly detection rate threshold 3212 > anomaly detection area percentage, the result is YES, and the process proceeds to S3304. If the anomaly detection rate threshold 3212 ≤ anomaly detection area percentage, the result is NO, and Figure 33 is terminated. Automatic updating of the reference image using this image (which is an abnormal image, such as an image containing large noise above the threshold or an image far removed from the reference image) is not performed.

[0211] In other words, this step is an example of a process that, based on a comparison between the information related to the reference image and the information related to the image to be judged, determines that there is a change above or below a threshold, or below or less than a threshold, and then prevents the use of the information related to the image to be judged to update the information related to the reference image.

[0212] In this embodiment, the reference image is updated when the anomaly detection rate threshold 3212 > anomaly detection rate. However, this method is not limited to this, and other conditions may be used, such as the anomaly detection rate threshold 3212 ≥ anomaly detection rate, or the average of the anomaly detection rate threshold 3212 > anomaly detection rate.

[0213] In S3306, the information processing device 104 adds the histogram of the reference information currently stored (of the analyzed image used for updating the reference) calculated in S2304 to the histogram of the reference information currently stored, recalculates the average value, and updates the histogram of the reference information.

[0214] In other words, this step is an example of a process that updates information relating to a reference image based on information relating to the first predetermined setting of the target image.

[0215] In other words, this step is an example of a process that updates information related to a reference image based on information related to sequentially acquired images to be judged and information acquired based on a first predetermined setting.

[0216] In other words, this step is an example of a process that updates the information relating to the reference image using the information relating to the first predetermined setting of the sequentially acquired information relating to the images to be judged.

[0217] In other words, this step is an example of a process in which information relating to the target image for determination related to the first predetermined setting is mixed with information relating to the reference image at a predetermined ratio and updated.

[0218] In other words, this step is an example of a process in which, based on a comparison between information relating to the reference image and information relating to the image to be judged, if it is determined that there is a change in the second predetermined setting, the information relating to the reference image is updated based on the information relating to the image to be judged. In other words, this step is an example of a process in which information relating to the reference image to be used as a standard is determined based on information relating to the image to be analyzed.

[0219] In S3307, the information processing device 104 clears the variable for the number of interval frames used to determine the timing of the automatic update of the reference image to 0.

[0220] This concludes the explanation of Figure 33.

[0221] As a result, the reference image can be easily updated automatically at the timing and interval desired by the user (=interval frame setting value 3211).

[0222] This will enable proper anomaly detection even in rapidly changing images, such as in environments where proper anomaly detection was not possible without updating the reference image (e.g., outdoors).

[0223] Specifically, in outdoor settings, using the same reference image during the day and night makes it difficult to detect anomalies correctly because the overall brightness of the image differs. By automatically updating the reference image periodically (for example, every minute), it becomes possible to appropriately detect anomalies even in images where the brightness changes rapidly.

[0224] Furthermore, when the image being analyzed includes the sky, the movement of clouds is sometimes mistakenly identified as an anomaly, making it difficult to correctly detect anomalies using the same reference image. However, by automatically updating the reference image periodically (for example, every 10 seconds), it becomes possible to appropriately detect anomalies even in images where clouds are constantly moving.

[0225] In this embodiment, the reference image is updated based on the setting of the interval frame setting value 3211. However, this is not limited to the setting of the interval frame setting value 3211. For example, the reference image may be automatically updated based on the setting of the type of object to be analyzed (e.g., water, smoke, fire, sky, etc.), the shooting environment (e.g., indoor, outdoor, underwater, telephoto shooting, magnified shooting, etc.), and by selecting a template suitable for the object to be analyzed and the shooting environment and the setting value of that template.

[0226] As a result, even with objects to be analyzed or shooting environments that were previously unsuitable for detection, false positives will be reduced, and abnormalities will be detected appropriately.

[0227] Based on the above, a mechanism for easily updating the reference image can be provided.

[0228] This concludes the explanation of the second embodiment. Figure 34 is an example of a diagram illustrating the overview of anomaly detection using a trained model in the third embodiment.

[0229] The improved third embodiment (lower part of Figure 34) is an embodiment that enables object detection and anomaly detection using a pre-trained model by specifying the program to be executed (Figure 37), whereas conventional anomaly detection systems (upper part of Figure 34) could only perform anomaly detection by image comparison.

[0230] This concludes the explanation of Figure 34. Figure 35 is an example of a flowchart that rewrites the flowchart in Figure 4(a) of the first embodiment for use in the third embodiment. The same processes as in Figure 4(a) will not be explained.

[0231] In S3501, the information processing device 104 accepts a selection of a value for the monitoring process analysis method setting 3611 (Figure 36) from the user. In other words, this step is an example of a process for obtaining an analysis program to perform image analysis.

[0232] If the value of analysis method setting 3611 is "detection by image comparison", proceed to S409; if it is "object recognition (using a trained model by program execution)", proceed to S3502.

[0233] Here, let me explain the analysis method setting screen 3630 (Figure 36).

[0234] If you select "Object Recognition (Use of a pre-trained model via program execution)" as the value for the analysis method setting 3611, the system will accept the selection of the command 3631 for executing the program and the program's script path 3632.

[0235] You can select one of these options, or two or more. If you select two or more options, you can run the monitoring process simultaneously using the two or more selected analysis methods.

[0236] In this embodiment, object detection processing is performed using a trained model, but this is not limited to object detection processing. Other processing using a trained model may also be performed, such as anomaly detection processing using a trained model, correctness (normal product / defective product, etc.) processing using a trained model, prediction processing using a trained model, and evaluation processing using a trained model.

[0237] In this embodiment, the program is written in Python®, but it is not limited to Python®; other programming languages ​​such as C, C++, and Java® may also be used.

[0238] In this embodiment, we specify and execute a program script, but this method is not limited to that. You may also specify identification information for an application, API, web service, etc., and execute them.

[0239] In this embodiment, the analysis method is selected from the options of "detection by image comparison" and "object recognition (using a pre-trained model by program execution)." However, it is not limited to these two options. Alternatively, an analysis execution terminal 3640 (with a specified analysis method) may be selected, or a combination of an analysis execution terminal and an analysis method 3650 may be selected.

[0240] The analysis execution terminal may include the information processing device 104, or it may consist only of other terminals that do not include the information processing device 104.

[0241] In other words, this step is an example of a process that accepts the selection of an information processing device and an analysis process for analyzing a target image.

[0242] In other words, this step is an example of a process that accepts at least one selection from a set of options, including options related to image comparison and options related to the trained model.

[0243] That is, this step shows an example of a process of alternatively accepting a selection from options including an option related to image comparison and an option related to a learned model.

[0244] That is, this step shows an example of a process of accepting selections of two or more options from options including an option related to image comparison and an option related to a learned model.

[0245] As described above, if the user can freely select the analysis execution terminal 3640 or the combination 3650 of the analysis execution terminal and the analysis method, a highly loaded monitoring process can be executed on other devices outside the anomaly monitoring system, so that anomaly monitoring by load distribution can be performed.

[0246] In addition, since analysis can be performed using different devices or different analysis methods for each monitoring process, flexible monitoring settings can be made together with load distribution.

[0247] In addition, it becomes possible to flexibly set the monitoring of one monitoring target using different analysis methods on multiple devices.

[0248] In addition, since programs / libraries / learned models, etc. that are not available in the devices of the anomaly monitoring system can be flexibly used, a flexible system design can be achieved. In some cases, a configuration such as grid computing can also be achieved.

[0249] In addition, in the case of the form of selecting the combination 3650 of the analysis execution terminal and the analysis method, if the program / library / learned model, etc. of the analysis method does not exist in the analysis execution terminal, the anomaly monitoring system may send the program / library / learned model, etc. of the analysis method to the analysis execution terminal for execution, or the analysis execution terminal may download the program / library / learned model, etc. of the analysis method from other devices, or the analysis execution terminal may use the program / library / learned model, etc. existing in other devices.

[0250] Furthermore, when the script template save button 3633 is pressed, the script template of the program stored in the information processing device 104 (or external device) can be saved (downloaded) to any location.

[0251] By modifying the script template saved in this way, it becomes possible to use any pre-trained model (3711 in Figure 37) to detect an object desired by the user (3712 in Figure 37, in this case a "bear") and receive the detection results.

[0252] This allows anomaly detection systems, which previously could only perform anomaly detection through image comparison, to detect various objects using any pre-trained model by executing any program.

[0253] In this embodiment, while the detection is described as detecting people and bears, it is not limited to mere objects such as people and bears. Detection may also include the state or parts of living beings or objects, such as a person walking, a person running, a person acting violently, a person who has fallen, a human child, a man in his 30s wearing a red shirt, a woman in her 20s wearing a hat, a teenage boy with his hand raised, a right hand, a knee, a left foot, a standing bear, a bear cub, a swimming bear, etc.

[0254] In this embodiment, we have stated that people and bears are detected by using a trained model obtained by program execution, but this is not limited to people and bears. It may also include living things such as birds, cats, dogs, horses, and cows; vegetables such as carrots, cabbage, lettuce, potatoes, and broccoli; fruits such as apples, oranges, grapes, and melons; objects and buildings such as automobiles, bicycles, buses, motorcycles, trucks, traffic lights, stop lines, pedestrian crossings, buildings, schools, and hospitals; home appliances such as mobile phones, smartphones, televisions, radios, microwave ovens, toasters, ovens, and refrigerators; stationery and tools such as clocks, pencils, ballpoint pens, notebooks, books, scissors, toothbrushes, combs, sprays, cans, bottles, balls, bats, gloves, and rackets; clothing such as ties, trousers, skirts, shirts, and sweaters; works of art, copyrighted works, and products such as paintings, sculptures, photographs, and images; designs and trademarks such as goods and logos; and various other objects, living things, plants, crystals, atoms, molecules, elements, viruses, bacteria, and other objects that can be identified by images.

[0255] In S3502, when the user clicks the analysis tab 1611 on the settings screen 1610 (Figure 16), the settings screen 1700 (Figure 17) is displayed. At this time, when the user clicks the adjustment button 1701, the analysis settings adjustment screen 4210 (Figure 42) is displayed. Subsequently, the information processing device 104 executes the detection parameter setting process for the trained model of the monitoring task shown in Figure 38.

[0256] This concludes the explanation of Figure 35. Figure 38 is an example of a flowchart for the parameter setting process of a trained model in the third embodiment.

[0257] In S3801, when the information processing device 104 receives a press of the judgment area button 4211 on the analysis setting adjustment screen 4210 (Figure 42) from the user, it displays the analysis setting judgment area screen 4220 (Figure 42).

[0258] In S3802, the information processing device 104 accepts user input such as drawing a rectangle 4221 around an area on the detection and judgment area setting screen 4220 (Figure 42) of the analysis settings using a mouse or touch, and accepts the setting of the detection and judgment area 4221 of the trained model. Multiple detection and judgment areas 4221 can be set.

[0259] In this embodiment, the detection and determination area setting screen 4220 accepts the setting of the area near the door as the detection and determination area 4221.

[0260] In this embodiment, the setting of the detection / determination area 4221 is accepted, but this method is not limited to this, and a method of accepting the setting of the detection / non-determination area (mask area) may also be used.

[0261] In other words, this step is an example of a process that accepts the specification of a target region from among the regions corresponding to the shape of the target image, where the result will be output.

[0262] In other words, this step is an example of a process that accepts the specification of a target region among the regions corresponding to the shape of the target image in which the output of the result should be suppressed.

[0263] In S3803, the information processing device 104 stores the settings for the detection and determination area 4221 in the RAM 203.

[0264] In S3804, the information processing device 104 accepts the setting of the detection score threshold 4212 from the user on the analysis setting adjustment screen 4210. If there are multiple types of objects that can be detected by AI, the threshold 4212 can be set for each object type.

[0265] Here, we assume that a threshold of 4212 = 55.0 has been set for the object type person.

[0266] In S3805, the information processing device 104 stores the threshold value 4212 in the RAM 203.

[0267] The description of FIG. 38 is hereby completed. FIG. 39 is an example of a flowchart obtained by rewriting the flowchart of FIG. 21 of the first embodiment for the third embodiment. The description of the same processing as in FIG. 21 is omitted.

[0268] In S3901, the information processing apparatus 104 acquires the value of the analysis method setting 3611 (FIG. 36) for the monitoring process from the operation process information in the RAM 203.

[0269] If the value of the analysis method setting 3611 is "detection by image comparison", the process proceeds to S2106. If it is "object recognition (using a learned model by program execution)", the process proceeds to S3902.

[0270] In S3902, the information processing apparatus 104 executes a detection process (FIG. 40) using the learned model.

[0271] The description of FIG. 39 is hereby completed. FIG. 40 is an example of a flowchart of the detection process using the learned model in the third embodiment.

[0272] In S4001, if the information processing apparatus 104 has executed the process of dividing the video file into consecutive still images for each frame (S2105), the subsequent processes are repeated until the processing of all the generated still images is completed.

[0273] In S4002, the information processing apparatus 104 reads the still image to be processed and expands it on the RAM 203. That is, this step shows an example of the process of acquiring an image captured at the same shooting angle.

[0274] In S4003, the information processing apparatus 104 performs a process of causing the object detection process to be executed in the detection process using the learned model (FIG. 41).

[0275] In S4004, the information processing device 104 obtains a detection score for each detected object received from the trained model.

[0276] In S4005, if the detection score for each detected object acquired in S4004 exceeds the threshold 4212 set in S3805, the information processing device 104 records the detected object on the RAM 203 and proceeds to S4006. If the threshold is not exceeded, the process proceeds to the next still image.

[0277] In this embodiment, the determination is made as follows: detection score > threshold 4212. However, the determination is not limited to this, and other determinations such as detection score ≥ threshold 4212, or the average of the detection scores of all detected objects > threshold 4212, may also be used.

[0278] In S4006, the information processing device 104 proceeds to S4007 if the area where the detected object recorded in RAM is located is within the determination area set in S3802. If the detected object recorded in RAM is outside the determination area, the process proceeds to the next still image processing. In other words, this step is an example of a process that controls whether or not to output a determination result based on the area specified.

[0279] In S4007, the information processing device 104 performs an anomaly notification process (Figure 27) and notifies the user and / or external applications of information about the anomaly area.

[0280] In other words, this step is an example of a process in which a target image is input into a trained model and the results of processing using the trained model are output.

[0281] For example, if a "person" with a detection score of 72.1 is detected within the judgment area set in S3802, the score exceeds the threshold 4212 = 55.0, so a notification is sent indicating that a "person" has been detected.

[0282] This concludes the explanation of Figure 40.

[0283] As a result, it will now be possible to set masks for object detection using pre-trained models, using the same GUI as the mask region setting previously used for anomaly detection by image comparison.

[0284] In this embodiment, the anomaly detection system sets the detection judgment area 4221 and threshold 4212, and determines whether or not to notify based on these settings. However, this method is not the only one available, and other methods may be used, such as sending the detection judgment area 4221 and threshold 4212 set by the anomaly detection system to a trained model, where the trained model determines whether or not to detect or notify.

[0285] Furthermore, in this embodiment, the anomaly detection system sets the detection judgment area 4221 and threshold 4212, and determines whether or not to notify based on these settings. However, this is not the only method, and other methods may be used, such as specifying the detection judgment area 4221 and threshold 4212 in a program script, passing these settings as arguments to the trained model, and having the trained model determine whether or not to detect or notify.

[0286] Figure 41 is an example of a flowchart for the process of having the trained model execute the detection process in the third embodiment.

[0287] In S4101, the information processing device 104 transmits the still image of the object to be analyzed, which was loaded onto the RAM 203 in S4002, to the information processing device 106, which is equipped with a trained model.

[0288] In S4102, the information processing device 106 receives the still image to be analyzed transmitted by the information processing device 104.

[0289] In S4103, the information processing device 106 performs object detection inference on the received still image using a trained model.

[0290] In S4104, the information processing device 106 acquires the detection results performed in S4103.

[0291] In S4105, the information processing device 106 transmits the detection result acquired in S4104 to the information processing device 104.

[0292] In S4106, the information processing device 104 receives the AI ​​detection result transmitted from the information processing device 106 in S4105 and loads it onto the RAM 203.

[0293] This concludes the explanation of Figure 41.

[0294] As a result, in anomaly detection systems that previously could only perform anomaly detection through image comparison, it is now possible to perform object detection and anomaly detection using a pre-trained model by specifying the program to be executed.

[0295] Furthermore, in anomaly detection systems that previously could only detect anomalies through image comparison, it will become possible to detect various objects and anomalies by executing any program on any device.

[0296] A fourth embodiment will be described using Figures 43 to 50.

[0297] Figure 43 is a system configuration diagram showing an example of the configuration of the information processing system 100 in the fourth embodiment.

[0298] Figure 43 shows a configuration in which the camera 102, information processing device 104, and video acquisition device 106 are connected via an image transfer cable (USB, Ethernet, CameraLink, etc.). Note that a configuration as a video acquisition device with a camera is also acceptable, even without this specific setup.

[0299] The information processing device 104 acquires an image obtained by the camera 102 capturing the target object via an image transfer cable and performs processing related to the detection of abnormalities in the image. The camera 102 may be, for example, a thermal imaging camera.

[0300] Furthermore, the information processing device 104 acquires images acquired by the video acquisition device 106 via an image transfer cable and performs processing related to the detection of abnormalities in the images. The video acquisition device 106 may be, for example, a video management system (VMS) or a cloud, which is a device capable of acquiring and managing camera images and recordings. In addition, it may be, for example, a thermographic camera, a measuring device such as a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) sensor, or a device or library that acquires measurement results from a measuring device.

[0301] Furthermore, the information processing system 100 of this embodiment may also be configured without the camera 102 having the configuration described above. In that case, processing related to the detection of abnormalities in images is performed on images acquired from the video acquisition device 106 on the network or on images stored in the memory (RAM) of the information processing device 104.

[0302] Figure 44 is an example of a flowchart of the anomaly detection process in the fourth embodiment. The process that is the same as in Figure 21 will not be explained.

[0303] In S2104, if the information processing device 104 performed a video file transfer on the external memory 212 in S2103, it proceeds to S2105. If it has performed a video acquisition process and acquired a still image, it proceeds to S4404. If it acquires video using custom loading, it proceeds to step S4401.

[0304] In S4401, the information processing device 104 holds a custom read determination flag on RAM 203 and accepts custom read settings (selection of execution script). The screen for accepting custom read settings will be described later in Figure 47. In other words, this step is an example of a process for obtaining settings related to the time at which a freeze is determined.

[0305] In this embodiment, the script to be executed is a script written in Python®, but it is not limited to Python®. Other scripts such as JavaScript, or source code (programs) written in programming languages ​​such as C, C++, or Java® may also be used.

[0306] In S4402, the information processing device 104 retrieves the script selected in S4401. In other words, step S4401 is a step that shows an example of the process of retrieving a program to be executed.

[0307] In S4403, the information processing device 104 executes the script acquired in S4402. Specifically, the execution of the script begins when the OK button 4717 in Figure 47 is pressed.

[0308] In S4404, the information processing device 104 executes the image difference (anomaly) detection process shown in Figure 45.

[0309] Figure 45 is an example of a flowchart for image difference (anomaly) detection processing in the fourth embodiment. The same processing as in Figure 23 will not be explained.

[0310] In step S4501, the information processing device 104 determines whether a custom read determination flag is set on the RAM 203. If it determines that a custom read flag exists, it proceeds to step S4502; otherwise, it proceeds to step S2302. Details of step S4502 will be described later in Figure 46.

[0311] Figure 46 is an example of a flowchart of the custom loading process in the fourth embodiment.

[0312] In S4601, the information processing device 104 sends a video reading request to the video acquisition device 106. The video acquisition device 106 to which the reading request is sent is, for example, the camera specified in connection destination specification 4801 in the script (program) shown in Figure 48. In other words, by executing the script, video can be acquired from the video acquisition device 106 specified in the script.

[0313] In step S4608, the information processing device 104 monitors the execution status of the script in parallel with the processing in S4602 to S4607, and determines whether or not a freeze has occurred based on the response time after sending a request to the script and the freeze monitoring time. If it is determined that a freeze has occurred, it proceeds to step S4609; if it is determined that no freeze has occurred, it returns to step S4601. In other words, step S4608 is a step that shows an example of the process of monitoring the execution status of the acquired program. In other words, this step is a step that shows an example of the process of determining whether or not the program has frozen based on the time from sending a request to the program to receiving a response and the set time.

[0314] In S4609, the information processing device 104 re-executes the frozen script.

[0315] In S4602, the video acquisition device 106 receives a video reading request transmitted from the information processing device 104 in S4601.

[0316] In S4603, the video acquisition device 106 acquires video from the video acquisition target device. The video acquisition target device may be, for example, a network camera compatible with the ONVIF protocol, an infrared camera such as a thermographic camera, a USB-connected webcam, an online video management system on the internet, a measuring device such as a LiDAR sensor, or a library that manages images created using measurement results from a sensor measuring device. In other words, this step indicates that the image acquisition destination specified by a predetermined program is an imaging device or a management system that manages captured images.

[0317] By utilizing the Python interpreter language as a means of acquiring video from these target devices, it becomes possible to easily acquire images generated based on the measurement results from the measuring device.

[0318] In S4604, the video acquisition device 106 transmits the video acquired in S4603 to the information processing device 104.

[0319] In S4605, the information processing device 104 receives the video transmitted from the video acquisition device 105. In this embodiment, the information processing device 104 receives the video transmitted from the video acquisition device 105, but this method is not limited to this method. Other methods are also available for acquiring the video acquired or output by the video acquisition device 105, such as placing the video acquired or output by the video acquisition device 105 into a predetermined device or directory and then acquiring that video on the information processing device 104. In other words, step S4605 is a step that shows an example of a process for accepting an image acquired or output by the execution of a program as an input image. In other words, step S4605 is a step that shows an example of a process for acquiring an image generated based on the measurement results by a measuring device as an image to be analyzed. In other words, this step is a step that shows an example of a process for acquiring an image output by the execution of a program as an image to be analyzed.

[0320] In step S4606, the information processing device 104 acquires the execution log of the script. The execution log can be viewed in the console display window 4720 shown in Figure 47, which will be described later. In other words, step S4606 is a step that shows an example of a process that acquires the log output by the execution of a program and displays the log.

[0321] In S4607, the information processing device 104 reads the still image (picture) to be processed and loads it onto the RAM 203.

[0322] Figure 47 shows an example of the display of the settings screen (input tab) in the fourth embodiment. In the settings screen 4700 of the input tab, the function "Custom Load (Python)" 4701 can be selected. Selecting "Custom Load (Python)" 4701 allows you to configure the settings for executing a script in the custom load settings screen 4710.

[0323] The command location 4711 is the item that specifies the Python installation location.

[0324] Script path 4712 is an item that specifies the file (script) to be executed. In this embodiment, the method is to select the file to be executed, but it is also possible to select the folder where the desired program file is located. Furthermore, the program may be a predetermined program, or any other method is acceptable as long as the program to be executed can be obtained. In other words, this is an example of a process that accepts the selection of a program and obtains the accepted program. Also, it is an example of a process that obtains information about the location where the program is saved.

[0325] Alternatively, instead of specifying a file, you could specify an application, API, or web service, or use any other method that executes a specific program, application, API, or web service.

[0326] Option 4713 allows you to specify additional actions. You can enter a string; for example, the name of the monitoring process, which allows for identification when the same script is executed. If not needed, you do not need to specify this option.

[0327] The freeze monitoring time 4714 is an item that sets the time to determine whether or not a script has frozen during execution. Whether or not a freeze has occurred is determined by whether the time from the request sent from the information processing device 104 to the response is within the set time. For example, a freeze may occur if the Python program enters an infinite loop or if the device that acquires images malfunctions and becomes unresponsive. If a freeze is detected, the script execution is forcibly terminated and the script is restarted. By setting the freeze monitoring time to match the processing time of normal operation, it is possible to wait for processing even if it takes a long time under normal operation, without detecting a freeze.

[0328] The script template save button 4715 is a button that allows users to download a sample script pre-configured on the information processing device 104. By providing a sample script, users can easily use the functions by rewriting only a portion of the sample items.

[0329] The console display button 4716 is the button that displays the console display window 4720.

[0330] When the OK button 4717 is pressed, the custom loading settings are completed and the script execution begins.

[0331] The console display window 4720 is a screen that displays output results based on console output instructions included in the script when the script is executed. By checking this screen, the user can confirm whether or not any problems have occurred.

[0332] The script editing button 4722, when pressed, displays a separate screen (such as the Notepad application screen) and allows editing of the file specified by the script path 4712.

[0333] The restart button 4723 will re-execute the script when pressed.

[0334] Figure 48 shows an example of a script for using a network camera in the fourth embodiment. The camera to connect to can be specified in connection destination specification 4801. In the script shown in Figure 48, an ONVIF camera is specified as the camera to connect to. This indicates that the program includes identification information to identify the source of the images to be analyzed.

[0335] In the script shown in Figure 48, by specifying a thermographic camera as the camera to connect to in connection destination specification 4801, it is possible to acquire images from the thermographic camera. In other words, the measuring device is a device capable of measuring temperature, and this is an example of a process that acquires an image generated based on temperature-related information as the image to be analyzed.

[0336] Even when acquiring images from a thermographic camera, anomaly detection can be performed on the images captured by the thermographic camera, just as it can be done when acquiring images from a regular camera. Generally, thermographic cameras detect infrared radiation emitted from objects. Since the amount of infrared radiation emitted by an object is proportional to the temperature of the object, the amount of infrared radiation is measured and converted into temperature, and then displayed in different colors as a thermal image (thermal distribution image) based on that temperature. Images captured by thermographic cameras generally use multiple colors to represent high and low temperatures.

[0337] Furthermore, when detecting anomalies in images from a thermographic camera, in addition to hue, saturation, brightness, and edge gradient, color temperature information may also be used as information obtained from the image. Color temperature refers to the temperature indicated by the color displayed as a thermal image based on the amount of infrared radiation. In other words, anomaly detection may be performed by converting the image acquired by the thermographic camera into temperature information based on a color palette, etc., and detecting the difference in temperature information. This is an example of a process that accepts the setting of a threshold for color temperature similarity. Also, this is an example of a process that accepts the setting of a threshold related to the similarity of temperature information when the image to be analyzed is an image generated based on temperature information.

[0338] Furthermore, to facilitate the detection of anomalies in images acquired by a thermal imaging camera, default thresholds may be prepared for each type of monitored object and connected device, and these thresholds may be selected or modified. In other words, this is an example of a process that changes the threshold settings according to the type of external measuring device.

[0339] When using a thermal imaging camera, it is possible to detect changes in temperature, making it possible to detect anomalies that cannot be observed visually (such as overheating even though no smoke is present). Furthermore, it is possible to detect anomalies in darkness. Moreover, it is possible to detect anomalies without being affected by lighting or sunlight (such as the overall brightness being too high, or false detections due to changes in lighting over time). In addition, since it is possible to detect anomalies in a state where people cannot be identified, it is possible to detect anomalies in places where privacy must be ensured (for example, toilets and bathrooms) with less resistance from the user. In other words, the analyzed image is an example of an image in which people cannot be identified.

[0340] Figure 49 shows an example of a script for using LiDAR in the fourth embodiment. The sensor to be connected can be specified in connection destination specification 4901. In the script shown in Figure 49, a LiDAR sensor is specified as the sensor to be connected, and a distance image is acquired via the LiDAR library. In other words, the measuring device is a device capable of measuring the distance to an object, and this is an example of a process that acquires an image generated based on distance-related information as the image to be analyzed.

[0341] Even when acquiring images generated based on measurement results from a LiDAR sensor, anomaly detection can be performed based on the acquired images, just as when acquiring images from a regular camera.

[0342] When using LiDAR, for example, the image acquisition device 106 is connected to a LiDAR sensor or a library that manages two-dimensional images created using the LiDAR sensor. For example, a LiDAR sensor measures the distance and direction to an object by emitting laser light and measuring the time it takes for the light to hit the object and reflect. The LiDAR library can generate a virtual image based on two-dimensional distance data from the distance acquired by the sensor through data conversion. Images generated based on LiDAR measurement results are generally represented in shades of black and white (for example, objects at close distances are black, objects at far distances are light gray, etc.).

[0343] When detecting anomalies in images generated based on LiDAR measurement results, information such as hue and saturation may not be sufficient for detecting anomalies. Therefore, for example, when using LiDAR images, it may be possible to set only the edge gradient threshold, or to set the edge gradient threshold to be high (strict). Alternatively, to make it easier to detect anomalies in LiDAR images, default thresholds may be prepared for each type of monitored object and connected device, and these thresholds may be selected or modified. In other words, if the image to be analyzed is an image generated based on distance information, this is an example of a process that determines whether there has been a change that should be output based on the similarity of the edge gradient between the information related to the reference image and the information related to the image to be analyzed.

[0344] When using images generated based on LiDAR measurement results, anomaly detection becomes possible even in darkness. Furthermore, anomalies can be detected without being affected by lighting or sunlight (such as the overall image being too bright, or false detections due to changes in lighting over time). In addition, since anomalies can be detected even when people cannot be identified, it is possible to detect anomalies in places where privacy must be ensured (for example, toilets and bathrooms) with less resistance from the user. In other words, the analyzed image is an example of an image in which people cannot be identified.

[0345] In this embodiment, anomaly detection of images generated from measurement results of LiDAR and a thermographic camera has been described, but images obtained from measurement results of other sensors may also be used. For example, images showing airflow movement visualized by a near-infrared camera, or images generated by a vibration visualization system that visualizes vibrations using an infrared camera may also be used. Any other method is acceptable as long as it acquires images generated based on measurement results obtained from equipment such as sensors or special cameras.

[0346] As mentioned above, by enabling image acquisition through program execution, it becomes possible to provide a system that can easily cooperate with various devices to detect anomalies.

[0347] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium containing a program that realizes the functions of the embodiments described above to a system or device, and by having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0348] In this case, the program read from the recording medium itself realizes the novel function of the present invention, and the recording medium on which that program is recorded constitutes the present invention.

[0349] For recording media used to supply programs, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, and the like.

[0350] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0351] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0352] Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device. It goes without saying that the present invention can also be applied when the results are achieved by supplying a program to a system or device. In this case, by reading a recording medium containing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0353] The above program may consist of object code, program code executed by an interpreter, script data supplied to the OS (operating system), and other forms.

[0354] Furthermore, by downloading and reading the program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. It should be noted that all configurations combining the above-described embodiments and their modified forms are also included in the present invention.

Claims

1. A program acquisition means for acquiring a program that acquires or generates an image that is the subject of anomaly detection, based on the measurement results of a measuring device, Image acquisition means for acquiring an image acquired or generated by the execution of a program acquired by the program acquisition means, The system includes an output control means that controls the output to indicate that the image contains an abnormality, The program acquisition means acquires a program that specifies the destination for acquiring images based on the measurement results or the destination for acquiring the measurement results, based on the user's instructions, The image acquisition means acquires an image based on the measurement result obtained from a source specified in the program acquired by the program acquisition means, or an image generated based on the measurement result obtained from a source specified in the program acquired by the program acquisition means. An information processing system characterized by the following.

2. The output control means controls the system to identify and display the region in the image that contains an anomaly. The information processing system according to claim 1, characterized by the following:

3. The system further includes a determination means that determines whether a change has occurred based on the similarity of information relating to multiple images acquired by the aforementioned image acquisition means. The output control means controls the system to identify and display the region of the image that contains an abnormality, based on the determination result of the determination means. The information processing system according to claim 1, characterized by the following:

4. The system further includes a determination means that determines whether a change has occurred based on a predetermined threshold and the similarity of information relating to multiple images acquired by the image acquisition means. The output control means controls the system to identify and display the region of the image that contains an abnormality, based on the determination result of the determination means. The information processing system according to claim 1, characterized by the following:

5. The system further includes a second program acquisition means for acquiring an analysis program that performs image analysis, The output control means controls the system to identify and display the region in the image that contains anomalies, based on the execution result of the analysis program acquired by the second program acquisition means. The information processing system according to claim 1, characterized by the following:

6. The image acquisition means acquires the log output by the execution of the program, The output control means further controls the display of the log. The information processing system according to claim 1, characterized by the following:

7. The system further includes monitoring means for monitoring the execution status of the program acquired by the program acquisition means. The information processing system according to claim 1, characterized by the following:

8. The program acquisition means acquires the setting related to the time at which it is determined that the system has frozen, The monitoring means determines whether the program has frozen based on the time it takes for the program to respond and the time specified in the setting. The information processing system according to claim 1, characterized by the following:

9. The program acquisition means accepts the selection of a program and acquires the accepted program. The information processing system according to claim 1, characterized by the following:

10. The program acquisition means acquires information about the location where the program is stored and acquires the program related to that storage location information. The information processing system according to claim 1, characterized by the following:

11. The image acquisition destination specified by the aforementioned program is an imaging device or a management system for managing captured images. The information processing system according to claim 1, characterized by the following:

12. The system further includes a selection receiving means that accepts the selection of a method for acquiring an image acquired or generated by the execution of a program from among two or more image acquisition methods. The information processing system according to claim 1, characterized by the following:

13. The aforementioned image acquisition method includes at least one of the following: a method for acquiring an image from a still image file, a method for acquiring an image from a video file, and a method for acquiring an image acquired or generated by the execution of a program. The information processing system according to claim 12, characterized by the above.

14. The program acquisition means of the information processing system includes a program acquisition step of acquiring a program that acquires or generates an image that is the target of anomaly detection and is based on the measurement results of a measuring device, An image acquisition step in which the image acquisition means of the information processing system acquires an image acquired or generated by the execution of a program acquired in the program acquisition step, The information processing system includes an output control step that controls the output control means to output a message indicating that the image contains an abnormality, The program acquisition step involves acquiring a program that specifies the destination for acquiring images based on the measurement results or the destination for acquiring the measurement results, based on the user's instructions, The image acquisition step involves acquiring an image based on the measurement results obtained from a source specified in the program acquired in the program acquisition step, or an image generated based on the measurement results obtained from a source specified in the program acquired in the program acquisition step. A control method for an information processing system characterized by the following.

15. A program for causing at least one computer to function as one of the means of an information processing system described in any one of claims 1 to 13.

Images