Information processing system, information processing method, and program

Abstract

To provide an information processor for performing image analysis and a mechanism for easily determining analysis processing.SOLUTION: An information processing system comprising: a reception unit configured to receive selection of a combination of an information processing apparatus for analyzing a target image and analysis processing; a control unit configured to control the information processing apparatus of the combination whose selection is received by the reception unit to execute the analysis processing of the combination; a reception unit configured to receive a result of the analysis processing from the information processing apparatus executed by the control unit; and an output unit configured to output the result received by the reception unit.SELECTED DRAWING: Figure 34

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and in particular to a technique suitable for use in detecting an abnormality. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there exists an information processing system that notifies an abnormality based on a comparison between a reference image and an image to be analyzed.

[0003] Patent Document 1 discloses a technique for extracting changes in an image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-175381 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0005] Patent Document 1 describes that a feature extraction method is determined based on a change definition parameter.

[0006] However, since it is not possible to determine a combination of information processing devices and analytical processes based on parameters, it is not easy to set which information processing devices are to execute which analytical processes.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a mechanism for easily determining an information processing device that performs image analysis and an analysis process. [Means for solving the problem]

[0008] An information processing system comprising: a receiving means for receiving a selection of a combination of an information processing device and an analysis process for analyzing a target image; a control means for controlling the information processing device of the combination selected by the receiving means to execute the analysis process of the combination; a receiving means for receiving the results of the analysis process from the information processing device executed by the control means; and an output means for outputting the results received by the receiving means. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a mechanism for easily determining an information processing device that performs image analysis and an analysis process. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a system configuration diagram of an information processing system 100. [Figure 2] FIG. 1 is a hardware block diagram of an information processing device 104. [Figure 3] FIG. 2 is a block diagram illustrating an example of a software configuration. [Figure 4] 10A is a flowchart of a setting process; FIG. 10B is a flowchart of an anomaly detection parameter setting process for a monitoring task; [Figure 5] 10 is an example of an initial display of a setting screen. [Figure 6] 10 is a display example of the overall image of a setting screen. [Figure 7] This is an example of the settings screen (input tab). [Figure 8] This is an example of the settings screen (input tab). [Figure 9] This is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11] 10 is a display example of a setting screen (monitoring process and monitoring task). [Figure 12] This is an example of the settings screen (input tab). [Figure 13]This is an example of the settings screen (input tab). [Figure 14] This is an example of the settings screen (notification tab). [Figure 15] This is an example of the settings screen (notification tab). [Figure 16] 10 is a display example of a setting screen (monitoring task). [Figure 17] 10 is a display example of an analysis setting adjustment screen. [Figure 18] 10 is a display example of a grid setting screen (grid division). [Figure 19] 10 is a display example of a grid setting screen (grid correction). [Figure 20] 10 is a display example of an analysis setting screen. [Figure 21] 10 is an example of a flowchart of an abnormality detection process. [Figure 22] 10 is a display example of task registration information. [Figure 23] 10 is an example of a flowchart of an image difference (abnormality) detection process. [Figure 24] 10 is an example of a flowchart of a histogram calculation process. [Figure 25] 10 is a display example of grid correction. [Figure 26] 10 is a display example of a notification when grid correction is performed. [Figure 27] 10 is an example of a flowchart of an abnormality notification process. [Figure 28] 10 is a display example of a setting screen (abnormality notification). [Figure 29] 10 is a display example of an abnormality notification dashboard screen. [Figure 30] FIG. 10 is a diagram illustrating an example of an outline of updating a reference image in the second embodiment. [Figure 31] 10 is an example of a flowchart of an image difference (abnormality) detection process in the second embodiment. [Figure 32] 10 is an example of a flowchart of an initialization process for updating a reference image in the second embodiment. [Figure 33] 10 is an example of a flowchart of a reference image update process in the second embodiment. [Figure 34] FIG. 11 is a diagram illustrating an example of an overview of anomaly detection using a trained model according to the third embodiment. [Figure 35] 10 is an example of a flowchart of a setting process in the third embodiment. [Figure 36] 13 is a display example of a setting screen (analysis tab) in the third embodiment. [Figure 37] 10 is an example of a script for using a trained model in the third embodiment. [Figure 38] 13 is an example of a flowchart of a parameter setting process for a trained model in the third embodiment. [Figure 39] 13 is an example of a flowchart of an abnormality detection process in the third embodiment. [Figure 40] 10 is an example of a flowchart of a detection process using a trained model in the third embodiment. [Figure 41] 10 is an example of a flowchart of a process for causing a trained model to execute a detection process in the third embodiment. [Figure 42] 13 is a display example of an analysis setting screen in the third embodiment. [Figure 43] FIG. 10 is a block diagram illustrating an example of a software configuration according to the second and third embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0012] FIG. 1 is a system configuration diagram showing an example of the configuration of an information processing system 100 according to the present invention.

[0013] 1 shows a configuration in which a camera 102 and an information processing device 104 are connected via an image transfer cable (USB, Ethernet, Camera Link, etc.). However, instead of having such a configuration, an information processing device having a camera may also be used.

[0014] The information processing device 104 acquires an image obtained by photographing an object with the camera 102 via an image transfer cable, and executes processing related to identifying abnormalities in the image.

[0015] Furthermore, the information processing system 100 of this embodiment may be configured without the camera 102 configured as described above.

[0016] In this case, the image processing device 104 executes processing related to determining whether there is an abnormality in an image, for an image acquired from another terminal on the network or an image stored in the memory (RAM) of the information processing device 104.

[0017] Hereinafter, the hardware configuration of an information processing device applicable to the information processing device 104 shown in FIG. 1 will be described with reference to FIG.

[0018] FIG. 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in FIG.

[0019] 2, reference numeral 201 denotes a CPU, which performs overall control of each device and controller connected to a system bus 204. In addition, a ROM 202 or an external memory 212 stores a BIOS (Basic Input / Output System), which is a control program for the CPU 201, an operating system program (hereinafter referred to as OS), and various programs (described later) required to realize the functions executed by each PC.

[0020] A RAM 203 functions as the main memory, work area, etc. of the CPU 201. The CPU 201 loads programs and the like required for executing processing from the ROM 202 or the external memory 212 into the RAM 203 and executes the loaded programs to realize various operations.

[0021] An input controller 205 controls input from a keyboard (KB) 210 and a pointing device such as a mouse (not shown).

[0022] A video controller 206 controls the display on a display device such as a display 211 .

[0023] Reference numeral 207 denotes a memory controller, which controls access to an external memory 212 such as an external storage device (hard disk (HD)) for storing various data, a flexible disk (FD), or a CompactFlash (registered trademark) memory connected to a PCMCIA card slot via an adapter.

[0024] A communication I / F controller 208 controls the reception of image data from an external PC 213 via a network (TCP / IP). An image I / F controller 209 controls the reception of image data from the camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0025] Various programs described below for implementing the present invention are recorded in the RAM 203 and executed by the CPU 201.

[0026] Furthermore, image data used when the above programs are executed is stored in the ROM 202, external memory 212, external PC 213, and camera 102 depending on the application, and is stored in the RAM 203 via various controllers when the programs are executed. FIG. 3 is a block diagram showing an example of a software configuration according to an embodiment of the present invention.

[0027] The information processing device 104 includes the following functional units.

[0028] The acquisition unit 301 is a functional unit that acquires images captured at the same angle of view.

[0029] The receiving unit 302 is a functional unit that receives designation of an area included in an image and a display method for that area.

[0030] The determination unit 303 is a functional unit that determines, based on a change between corresponding areas of the multiple images acquired by the acquisition unit 301, that a change has occurred in the corresponding area.

[0031] The output unit 304 is a functional unit that outputs the determination result from the determination unit 303 .

[0032] The output unit 304 is a functional unit that displays the area specified by the receiving unit 302 in the display method specified by the receiving unit 302, and controls whether or not to output the judgment result based on the area specified by the receiving unit 302.

[0033] The area change unit 305 is a functional unit that changes the area whose designation has been accepted by the acceptance unit 302, based on the area whose designation has been accepted by the acceptance unit 302 and the determination result of the area.

[0034] The dividing unit 306 is a functional unit that divides the image acquired by the acquiring unit 301 .

[0035] The determining unit 303 is a functional unit that determines, based on a change between corresponding areas divided by the dividing means among the multiple images acquired by the acquiring unit 301, that a change has occurred in the area.

[0036] The division unit 306 is a functional unit that divides an image using two or more division methods.

[0037] The output unit 304 is a functional unit that outputs a determination result based on the area divided by the first division method when the determination unit 303 determines that there has been a change in the area divided by the second division method.

[0038] The determination unit 303 is a functional unit that determines whether a change has occurred in a corresponding region of a plurality of images acquired by the acquisition unit, based on the change between corresponding regions of the plurality of images and a plurality of thresholds.

[0039] The output unit 304 is a functional unit that identifies which of a plurality of thresholds was used to determine that a change occurred, and outputs the result.

[0040] The receiving unit 302 is a functional unit that receives a designation of a method for identifying and outputting each type of threshold.

[0041] The receiving unit 302 is a functional unit that controls so as not to receive the same output method designation for different types of thresholds. The determination unit 303 is a functional unit that determines whether there is a change that should be notified based on a comparison between the reference image and the image to be analyzed.

[0042] The acquisition unit 301 is a functional unit that notifies the determination result that there has been a change that should be notified.

[0043] The notification unit 307 is a functional unit that acquires a predetermined value related to suppression of notification.

[0044] The notification unit 307 is a functional unit that, based on a predetermined value, suppresses notification of a determination result indicating that a change that should be notified has occurred.

[0045] The notification unit 307 is a functional unit that suppresses notification of the determination result that there has been a change that should be notified, based on at least one of the number of changes that should be notified and a predetermined value.

[0046] The notification unit 307 is a functional unit that, when the number of changes to be notified is equal to or less than a predetermined value, suppresses notification of the determination result that there is a change to be notified.

[0047] This concludes the explanation of FIG. The setting process in FIG. 4(a) will be described.

[0048] In S401, the information processing apparatus 104 determines whether or not a setting file for the operating process exists in the external memory 212. If it is not registered, the process proceeds to S402, and if it is registered, the process proceeds to S403.

[0049] In S402, the information processing apparatus 104 displays the setting screen 510 (FIG. 5), and upon receiving a press of the add monitoring process button 511 from the user, registers the operating process in the RAM 203 and displays the setting screen 520.

[0050] In S403, when the information processing apparatus 104 receives a press of the selection button 521 of the operation process to be edited on the setting screen 502, it displays the setting screen 610 (FIG. 6). The setting reception process on the setting screen 610 performed in S403 will be described below with reference to FIGS.

[0051] When the information processing device 104 receives input of the name 611 and description 612 of the operating process from the user, it edits the operating process information in the RAM 203 and displays the setting screen 620 .

[0052] When the information processing apparatus 104 receives a user's press of the input tab 613 on the setting screen 610, it displays a setting screen 710 (FIG. 7).

[0053] When the information processing device 104 receives a selection of "Load video file (FFmpeg)" (721 in FIG. 7) from the user in the function list 711, it displays a setting screen 720. At this time, the information processing device 104 edits the operating process information in the RAM 203 based on the values ​​of the command location 722 and option 723 input by the user on the setting screen 720.

[0054] When the information processing device 104 receives a selection of "Read video file (OpenCV)" (731 in FIG. 7) from the user in the function list 711, it displays a setting screen 730 and edits the operating process information of the RAM 203.

[0055] When the information processing device 104 receives a selection of "Load live video (video monitoring server)" (811 in FIG. 8) from the user in the function list 711, it displays a setting screen 810 (FIG. 8). "Load live video (video monitoring server)" is a function that loads live video captured by a network camera from a video monitoring server (not shown) that can be connected via a network.

[0056] The operating process information in RAM 203 is edited based on the values ​​of computer name 812, user name 813, and password 814 input by the user on setting screen 810. Furthermore, when connection confirmation 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in RAM 203.

[0057] When the information processing device 104 receives a user's press of the analysis tab 614 (FIG. 6) on the setting screen 610, it displays a setting screen 910 (FIG. 9).

[0058] When the information processing device 104 receives a selection of "detection by video comparison" from the user in the function list 911, it displays a setting screen 920 and edits the operating process information in the RAM 203. At this time, if the license has not been authenticated, the license information 922 displays "No license."

[0059] When the information processing apparatus 104 receives a license read 923 press from the user, if a valid license has been authenticated, it displays a setting screen 930, displays license information 931, and edits the operating process information in the RAM 203.

[0060] When the information processing device 104 receives a press of the notification tab 615 (FIG. 6) from the user on the setting screen 610, it displays the setting screen 1010 (FIG. 10).

[0061] When the information processing device 104 receives a selection of "custom command" (1021 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays the setting screen 1020 and edits the operating process information in the RAM 203.

[0062] "Custom Command" is a function that executes commands to execute batches or applications when an abnormality is detected. By using this function, it is possible to execute batch processing, instruct emails to be sent to an email client, instruct calls using an automated voice call, and link with other applications or external systems.

[0063] When the information processing device 104 receives a selection of "Analysis result notification (video monitoring server)" (1031 in FIG. 10) from the user in the function list 1011, the information processing device 104 displays a setting screen 1030.

[0064] "Analysis Result Notification (Video Surveillance Server)" is a function that notifies the video surveillance server of the analysis results. This function is used when the video surveillance server is configured to notify users of abnormalities, or when analyzing and managing events such as abnormalities on the video surveillance server.

[0065] Based on the values ​​of the address 1032 and port number 1033 input by the user on the setting screen 1030, the operating process information of the RAM 203 is edited.

[0066] When the information processing device 104 receives a press of the connection confirmation button 1034 from the user, it performs a connection test via the communication I / F controller 208 using the information in the RAM 203 . Returning to the explanation of Figure 4.

[0067] In S404, when the information processing apparatus 104 receives a press of the OK button 631 on the setting screen 630 (FIG. 6), it saves the operating process information registered in the RAM 203 in the external memory 212 as an operating process setting file.

[0068] In S405, the information processing apparatus 104 determines whether or not a setting file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406, and if it is registered, the process proceeds to S407.

[0069] In S406, the information processing apparatus 104 displays the setting screen 630, and upon receiving a press of the add button 632 for the monitoring process task from the user, registers the monitoring task in the RAM 203 and displays the setting screen 1110 (FIG. 11).

[0070] In S407 , when the information processing apparatus 104 receives a press on the monitoring task 1111 to be edited on the setting screen 1110 , it displays the setting screen 1120 . The monitoring task setting acceptance process performed in S407 will be described below with reference to FIGS.

[0071] When the information processing device 104 receives input of a name 1121 and a description 1122 from the user on a setting screen 1120 and a change to a toggle button for enabling this task 1123, it edits the operating process information in the RAM 203 and displays a setting screen 1130.

[0072] When the information processing device 104 receives a user's press of the input tab 1124 on the setting screen 1120, it displays the setting screen 1200 (FIG. 12) or the setting screen 1300 (FIG. 13) depending on the setting of the operation process.

[0073] If the input setting for the operating process is "Read video file (FFmpeg)" or "Read video file (OpenCV)", the setting screen 1200 (FIG. 12) is displayed.

[0074] When the information processing device 104 receives a user's click on the video folder 1201, selection of the folder, input of the frame acquisition interval 1202, or change of the toggle button for deleting processed video 1203, it edits the operating process information in the RAM 203 and displays the setting screen 1210.

[0075] When the input setting for the operation process is "load live video," the setting screen 1300 (FIG. 13) is displayed. When the information processing device 104 receives a selection of a list item for the camera 1301 from the user, it edits the operation process information in the RAM 203 and displays the setting screen 1310.

[0076] When the information processing device 104 receives a press of the notification tab 1125 from the user on the setting screen 1120, it displays the setting screen 1400 or the setting screen 1500 depending on the setting of the operation process.

[0077] If the notification setting for the operating process is "custom command," a setting screen 1400 (FIG. 14) is displayed. When the information processing device 104 receives input of a command path 1401 from the user and a change to the toggle button for asynchronous execution 1402, it edits the operating process information in the RAM 203 and displays a setting screen 1410.

[0078] When the notification setting for the operating process is "Analysis result notification (video monitoring server)," a setting screen 1500 (FIG. 15) is displayed. The information processing device 104 accepts input of a name 1501 and a camera ID 1502 from the user, and displays a screen 1510. At this time, if the user presses a test event button 1511, a test notification process is executed to check whether the notification works correctly with the input content. At this time, if the setting for the detection frame display toggle button 1512 is enabled, pseudo detection result information for testing purposes is also added to the notification content. Returning to the explanation of Figure 4.

[0079] In S408, when the information processing device 104 receives a user pressing the OK button 1602 on the setting screen 1600 (FIG. 16), it saves the monitoring task information registered in the RAM 203 as a monitoring task setting file in the external memory 212 and displays the setting screen 1610.

[0080] In S409, when the user presses the analysis tab 1611 on the setting screen 1610, the setting screen 1700 (FIG. 17) is displayed. At this time, when the user presses the adjustment button 1701, the analysis setting adjustment screen 1710 is displayed.

[0081] Thereafter, the information processing device 104 executes the process shown in FIG. The abnormality detection parameter setting process for the monitoring task in FIG. 4(b) will be described.

[0082] In S421, when the information processing apparatus 104 receives a user's press of the basic settings button 1711 on the analysis settings adjustment screen 1710 (FIG. 17), it displays the basic analysis settings screen 1720 and proceeds to S422. If the basic setting button has not been pressed, the process proceeds to S428.

[0083] In S422, when the information processing device 104 receives a change to the slide bars for the number of rows 1801 (FIG. 18) and the number of columns 1802 from the user, it updates the display of the grid (dashed lines) on the screen divided by the number of rows 1801 and the number of columns 1802, as shown in screen 1800 (FIG. 18), and saves the setting contents in RAM 203.

[0084] Specifically, on the screen 1800, the number of rows 1801 is set to 10 and the number of columns 1802 is set to 10, so the image to be analyzed is displayed divided into a grid of 10 rows and 10 columns.

[0085] The more grids there are, the more pinpoint detection becomes possible, but this increases the number of processes, such as the image difference (abnormality) detection process (Figure 23) described below, which may increase the time required for processing.

[0086] In S423, when the information processing device 104 receives a user's press 1811 on any area separated by a grid on the screen, a press of the Set All Masks button 1812, or a press of the Cancel All Masks button 1813, the information processing device 104 displays the presence or absence of a mask (notification suppression area) as shown on a screen 1810, and saves the setting contents in the RAM 203. That is, this step is a step showing an example of a process for receiving a designation of an area included in an image.

[0087] If there are many mask areas, the mask target area can be easily set by first masking all areas with the mask all setting button 1812 and then specifying the area to be unmasked.

[0088] If the masked area is small, the mask target area can be easily set by releasing the mask for all areas with the release all mask button 1813 and then specifying the area to be masked.

[0089] Furthermore, at this time, if a change in the mask color inversion 1821 toggle button by the user is accepted, the mask drawing color is changed as shown on screen 1820. Specifically, the mask area is represented in transparent black on screen 1810, but is represented in transparent white on screen 1820. That is, this step is a step showing an example of a process for accepting a designation of a display method for an area included in an image. Also, this step is a step showing an example of a process for displaying the area for which designation has been accepted in the display method for which designation has been accepted.

[0090] This allows the user to select the color of the mask area depending on the color tone and brightness of the image to be analyzed, by representing the mask area in transparent black when the image to be analyzed is bright, and in transparent white when the image to be analyzed is dark. This makes it easy to identify the mask area and also allows the user to understand the appearance of the mask area.

[0091] In this embodiment, the color of the mask area is "transparent black" or "transparent white," but this is not limited to this method. Other colors may be used, the user may specify the color, patterns such as diagonal lines may be used to vary the display, the area may be surrounded by a frame, or other methods may be used to distinguish between the monitored area and the mask area.

[0092] In S424, when the information processing device 104 receives a change from the user to the grid correction (shift grid) toggle button 1901 (Figure 19) or 1911, it updates the display of the frame 1902 (with frame) or 1912 (without frame) indicating the detection range and saves the setting contents in RAM 203.

[0093] In this embodiment, grid correction is set for the monitoring process, but this is not limited to this method, and grid correction may be set directly for the network camera or video monitoring server that is the image input source.

[0094] In S425 , when the information processing apparatus 104 receives a numerical input 1921 of the number of reference image frames from the user, the information processing apparatus 104 saves the setting content in the RAM 203 .

[0095] In S426, when the information processing device 104 receives a press of the back button 1803 from the user while the basic setting screen 1800 (FIG. 18) for analysis settings is displayed, it determines whether or not the basic settings have been changed. If there have been changes, the process proceeds to S427. If there have been no changes, the process proceeds to S428.

[0096] In S427, the information processing device 104 reflects the contents set in the RAM 203 in the monitoring task setting file in the external memory 212. The information processing device 104 also detects any changes to the monitoring task setting file and discards the reference image in the analysis process (in S2306, it is determined that the base information needs to be updated). In S428, when the information processing apparatus 104 receives a change to the slide bar for each item of the threshold 2001 on the analysis setting screen 2000 (FIG. 20), it saves the setting content in the RAM 203.

[0097] In this embodiment, the user sets each threshold value, but this method is not limited to this. The user may select a data set for each threshold value, or the user may set the object to be monitored or the abnormality to be detected so that the threshold parameters are automatically set, or the information processing device 104 may determine the object to be monitored or the abnormality to be detected and automatically set the threshold parameters, etc.

[0098] This makes it possible to easily set a threshold value for detecting an abnormality.

[0099] In S429, when the information processing device 104 receives a press of the Apply button 2002 from the user, it reflects the contents set in the RAM 203 in the monitoring task setting file in the external memory 212. The information processing device 104 also detects changes to the monitoring task setting file and changes the parameter values ​​used in the detection process (changing the threshold values ​​used in S2310). The abnormality detection process in FIG. 21 will be described.

[0100] In S2101, the information processing apparatus 104 executes an abnormality detection process if the external memory 212 contains a setting file for the monitoring task (task registration information 2201 (FIG. 22)) and the toggle 2202 for the task validation setting is set to valid.

[0101] In S2102, the information processing apparatus 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and repeats the process.

[0102] In S2103, the information processing apparatus 104 acquires input information according to the set input method (set on the setting screen 1200 or the setting screen 1300).

[0103] Specifically, if the input setting for the operation process (setting screen 1200) is "Load video file (FFmpeg)" or "Load video file (OpenCV)", the video file placed in the specified video folder is moved to the working area on the external memory 212 as input information.

[0104] Also, if the input setting for the operation process is "Read live video (video monitoring server)" (setting screen 1300), video acquisition processing is executed for the specified video acquisition destination, and if a still image is acquired, the process proceeds to S2106.

[0105] In S2104, if the information processing apparatus 104 has moved the video file on the external memory 212 in S2103, the process proceeds to S2105. If the information processing apparatus 104 has executed the video acquisition process and acquired a still image, the process proceeds to S2106.

[0106] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into consecutive still images for each frame, and files it on the external memory 212 or stores it as data on the RAM 203, and then proceeds to S2106.

[0107] In S2106, the information processing device 104 executes the process shown in FIG. Now, the first embodiment will be described.

[0108] The image difference (abnormality) detection process of FIG. 23 will be described.

[0109] In S2301, if the information processing apparatus 104 has divided the moving image file into successive still images for each frame (S2105), it repeats the subsequent processes until the processes for all the generated still images are completed.

[0110] In S2302, the information processing device 104 reads a still image to be processed and expands it on the RAM 203. That is, this step is a step showing an example of a process of acquiring images captured at the same angle of view. That is, this step is a step showing an example of a process of sequentially acquiring information related to the determination target image.

[0111] In S2303, the information processing apparatus 104 calculates information on hue, saturation, and lightness from the image data expanded on the RAM 203 by calculation.

[0112] In S2304, the information processing device 104 performs each histogram calculation process (FIG. 24) to calculate a histogram from the image data. Here, the histogram calculation process of FIG. 24 will be described.

[0113] In S2401, the information processing apparatus 104 proceeds to S2402 if the grid correction set in S424 is valid among the settings saved in the RAM 203. If it is invalid, the information processing apparatus 104 proceeds to S2403.

[0114] In S2402, the information processing device 104 sets, as target areas for histogram calculation processing, not only the areas divided by the grid 2500 (FIG. 25) set in S422, but also areas divided by grids 2501 shifted by half the size of one area in the horizontal and vertical directions. That is, this step is a step showing an example of processing for dividing the acquired image.

[0115] Specifically, as shown in 2510 and 2520, if an abnormality occurs near the 3-row x 4-column grid set in S422, since the abnormality spans multiple areas, the threshold value will not be exceeded in each area, and all areas will be determined to be normal, which may result in low accuracy in abnormality detection.

[0116] As in S2402, by determining whether an abnormality exists not only in the area separated by the normal grid 2500 set in S422 but also in the area separated by the corrected grid 2501 (shifted grid), it becomes possible to easily improve the accuracy of detecting an abnormality even when an abnormality occurs near the normal grid 2500 set in S422.

[0117] In this embodiment, the grids are shifted horizontally and vertically by half the size of one region. However, this is not limited to this method. It is also possible to create two new grids that are shifted horizontally and vertically by 1 / 3 and 2 / 3 of the size of one region, or to create a new hexagonal grid instead of a lattice (square) grid. This step is an example of a process for dividing an image using two or more division methods. This is an example in which an area divided by a first division method partially overlaps with at least one of the areas divided by a second division method.

[0118] This makes it possible to easily improve the accuracy of detecting an abnormality even when an abnormality occurs near the grid 2500.

[0119] In S2403, the information processing apparatus 104 calculates a histogram of luminance information from the luminance (Lightness) information acquired in S2303 for each processing target region set in S422 and S2402.

[0120] In S2404, the information processing apparatus 104 calculates a histogram of hue information from the hue and saturation information acquired in S2303 for each processing target region set in S422 and S2402.

[0121] In S2405, the information processing apparatus 104 calculates a histogram of saturation information from the saturation information acquired in S2303 for each processing target region set in S422 and S2402.

[0122] In S2406, the information processing device 104 calculates edge gradient information from the luminance (Lightness) information acquired in S2303 for each processing target area set in S422 and S2402, and calculates an edge gradient intensity histogram and an edge gradient angle histogram using the edge gradient information.

[0123] That is, S2304 and FIG. 24 are steps showing an example of processing for comparing the reference image and the image to be analyzed based on the feature amounts of the images.

[0124] S2304 and FIG. 24 are steps showing an example of a process of comparing the reference image and the image to be analyzed based on information relating to at least one of luminance, hue, saturation, edge gradient strength, and edge gradient angle.

[0125] Returning to the explanation of FIG.

[0126] In S2305, the information processing apparatus 104 calculates the degree of similarity for each processing target area set in S422 and S2402 using the histogram of the reference information and histogram intersection, which is one of the methods for comparing histograms.

[0127] In this case, the reference information is a histogram calculated in S2308, which will be described later. If there is no reference histogram data, the similarity is set to 100%.

[0128] In this embodiment, histogram intersection is used as the method for comparing histograms, but the method is not limited to this, and other methods such as Bhattacharyya coefficients may also be used.

[0129] Furthermore, in this embodiment, anomalies are detected by comparing histograms, but this is not limited to this method, and anomalies may be detected using other methods, such as template matching or feature point detection.

[0130] In S2306, if the number of data items already set as the reference image does not satisfy the "number of reference image frames" set in S425, the information processing device 104 determines that the reference image needs to be updated and proceeds to S2307. If data items that satisfy the "number of reference image frames" have already been set as the reference image, the process proceeds to S2309.

[0131] The information processing device 104 discards the reference image and initializes the number of set data items to 0 in the following cases:

[0132] When the basic settings are changed by the user of the S427 When the user clicks the Recreate Reference Image button 2003 This allows the user to easily instruct the re-creation of the reference image.

[0133] In S2307, the information processing device 104 determines whether the data is suitable as a reference image based on the similarity calculated in S2305. If there are no areas with low similarity, the process proceeds to S2308. If there are many areas with low similarity, the data is determined to be unsuitable as a reference image and the process ends.

[0134] In S2308, the information processing apparatus 104 adds the histogram calculated in S2304 to the histogram of the reference information currently held, recalculates the average value, updates the histogram of the reference information, and ends the process.

[0135] In S2309, the information processing apparatus 104 compares the similarity calculated in S2305 for each area to be processed and set in S422 with the threshold set in S428, and records areas below the threshold as abnormal areas in the RAM 203.

[0136] That is, this step is an example of a process of determining whether a change has occurred in a corresponding area of ​​a plurality of acquired images based on a change between the corresponding areas of the plurality of images.

[0137] That is, this step is a step showing an example of a process of determining whether there is a change that should be notified based on a comparison between the reference image and the image to be analyzed.

[0138] That is, this step is an example of a process of determining whether or not a predetermined change has occurred based on a comparison between information relating to the reference image and information relating to the image to be determined.

[0139] Specifically, in the threshold 2001 (FIG. 20), when the brightness threshold is set to 22%, if the brightness similarity of the area to be processed is less than 22%, the area is determined to be abnormal.

[0140] In this embodiment, similarity<threshold value, but this is not limiting and similarity≦threshold value may be satisfied.

[0141] In this embodiment, similarity is used, but it is not limited to similarity and may be amount of change. In this case, if amount of change is greater than or equal to the threshold, the region is determined to be abnormal.

[0142] In other words, this step is a step that indicates an example of a change to be notified, where the reference image and the image to be analyzed are compared and the similarity is less than a predetermined threshold, or the similarity is equal to or less than a predetermined threshold, or the amount of change exceeds a predetermined threshold, or the amount of change is equal to or greater than a predetermined threshold.

[0143] This step also compares corresponding regions of the reference image and the image to be analyzed, and shows an example of whether the similarity between the regions is less than a predetermined threshold, or whether the similarity between the regions is equal to or less than a predetermined threshold, or whether the amount of change between the regions exceeds a predetermined threshold, or whether the amount of change between the regions is equal to or greater than a predetermined threshold.

[0144] In S2310, the information processing device 104 compares the similarity for each area (area separated by a correction grid) set as the area to be processed in S2402, calculated in S2305, with the threshold set in S428, and detects areas that fall below the threshold.

[0145] At this time, if an abnormality is detected, the process proceeds to S2311. If no abnormality is detected, the process proceeds to S2312.

[0146] In S2311, the information processing device 104 determines (2620) that all areas 2610 (four areas in the case of FIG. 26) that overlap with the detection area 2600 have detected an abnormality, since the area 2600 (FIG. 26) where the abnormality was detected is different from the area to be notified to the user that was set in S422, and records the detected abnormality in RAM 203.

[0147] As a result, the area for which an abnormality is notified is the area divided by the normal grid 2500 set in S422, regardless of whether grid correction is performed or not. In other words, this step is an example of a process for outputting a determination result based on the area divided by the first division method when it is determined that a change has occurred in the area divided by the second division method.

[0148] In this way, it is possible to notify users and external applications of abnormalities based on areas that are always separated by a constant grid. In other words, because the notification method is constant regardless of whether grid correction is performed or not, it has the advantage that the notification is easy for users to understand, and for external applications, the method for receiving abnormality notifications does not need to be complicated, so the reception logic and method for outputting abnormalities can be constant.

[0149] In S2312, the information processing device 104 compares the abnormal area recorded on the RAM with the mask area set in S423, and if there is an abnormal area other than the mask area, proceeds to S2313. If there is no abnormal area or if all of the abnormal areas are mask areas, the process ends. In other words, this step is a step that shows an example of a process of controlling whether or not to output the determination result based on the area whose designation has been accepted.

[0150] In this embodiment, the histogram calculation process (S2304) is performed even in a masked area, but this is not limited to this method, and the histogram calculation process (S2304) may not be performed for a masked area. By doing so, the number of times each histogram calculation process is performed when there is a masked area is reduced, thereby shortening the overall processing time.

[0151] On the other hand, when performing histogram calculation processing even in a masked area, as in this embodiment, there are advantages such as being able to record abnormalities in the masked area and suggest changes to the masked area to the user, and being able to stably meet non-functional requirements by always keeping the processing speed constant.

[0152] In S2313, the information processing device 104 performs an abnormality notification process (FIG. 27) and notifies the user and an external application of information about the abnormal area. The abnormality notification process in FIG. 27 will be described.

[0153] In S2701, the information processing device 104 determines whether there is an abnormality to be notified based on criteria such as the continuity of abnormality detection. If it is determined that there is an abnormality to be notified, the process proceeds to S2702. If it is determined that there is no abnormality to be notified, the process proceeds to S2703.

[0154] Specifically, criteria are needed to determine whether an abnormality should be reported if an abnormality is detected in only one image frame, or if an abnormality is detected in several consecutive images over a period of several seconds.

[0155] For example, the information processing device 104 may accept the user's setting of criteria for notifying an abnormality in advance, such as how many consecutive frames an abnormality must be detected before notifying an abnormality, or may determine what the image being analyzed is (a factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and based on the object or environment, the information processing device 104 may determine the criteria for notifying an abnormality.

[0156] In step S2702, the information processing apparatus 104 records in the RAM 203 area information in which the abnormality to be notified has occurred.

[0157] In S2703, the information processing apparatus 104 deletes the abnormality information to be notified that is recorded on the RAM 203.

[0158] In S2710, the information processing apparatus 104 passes the still image information read in S2302 to the processing unit that draws the setting screen.

[0159] In S2711, the information processing apparatus 104 passes the abnormality information recorded in S2702 to the processing unit that draws the setting screen.

[0160] In S2712, the information processing apparatus 104 determines whether the setting screen is being drawn by a user operation. If the setting screen is being drawn, the process proceeds to S2713. If the setting screen is not being drawn, the process ends.

[0161] In S2713, the information processing apparatus 104 checks the detection area drawing color setting content 2801 in the RAM 203. If there is a designation regarding the drawing method, the process proceeds to S2714. If there is no designation, the process proceeds to S2715.

[0162] In S2714, the information processing device 104 changes the highlighting during the drawing process in S2715 (described later) based on the setting contents 2801 (FIG. 28). That is, this step is an example of a process of identifying and outputting which of multiple thresholds was used to determine that a change occurred.

[0163] The setting content 2801 is a setting for how to highlight the display depending on the detection factor of the abnormality detection (the item that fell below the threshold in S2309). That is, this step is an example of a process for accepting a designation of a method for identifying and outputting for each type of threshold.

[0164] Here, control is performed to prevent overlapping of the same highlighting settings so that the highlighting for each threshold can be distinguished (for example, an error message is displayed when the same highlighting is selected for different thresholds, or a highlighting that has already been selected cannot be newly selected, etc.). In other words, this step is an example of a process for controlling so that the same output method specification is not accepted for different threshold types.

[0165] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0166] In this embodiment, highlighting is performed using color, but the present invention is not limited to this method. The highlighting method can be set to use a pattern such as diagonal lines, to surround the information with a frame such as dotted or thick lines, or to set different notification methods such as voice or email, or to set the notification method using methods other than highlighting in addition to the type of highlighting.

[0167] In S2715, the information processing apparatus 104 develops an image for drawing in the drawing memory on the RAM 203 by highlighting the still image information acquired in S2710 and the area containing the abnormality information acquired in S2711.

[0168] In S2716, the information processing device 104 draws the drawing image created in S2715 on the setting screen 2810 (FIG. 28). That is, this step is a step showing an example of a process of outputting a determination result as to whether or not there has been a change in the area. That is, this step is a step showing an example of a process of outputting a result of a process based on an image comparison between the reference image and the target image.

[0169] At this time, if the rendering method is changed in S2714, the highlighting method for the abnormal area is changed and displayed as shown in abnormal area 2811(1), (2), and (4).

[0170] This allows the user to easily know which threshold was used to determine the abnormality.

[0171] The highlighting of the abnormal region and the display of the masked region may be performed on the same screen (not shown).

[0172] Furthermore, although it was stated in S2312 that the anomaly region is not displayed in the masked region, it is also possible to highlight the anomaly detection region (not shown) or highlight it differently (not shown) in the masked region in response to an instruction from the user. This is because, even in a region designated as a masked region, this can trigger the user to decide that the mask should be removed depending on the content of the anomaly detection. In other words, it becomes possible for the user to easily set the anomaly detection desired.

[0173] Furthermore, the information processing device 104 may have a function of informing the user on the screen 1810 or 1820 of the type of abnormality occurring in the masked region without receiving an instruction from the user.

[0174] Furthermore, there may be a function that suggests to the user to change the mask settings based on the number and content of anomaly detections, or that the information processing device 104 automatically changes the mask settings. This enables more accurate anomaly detection. That is, this step is a step that shows an example of a process of changing the area whose designation has been accepted based on the area whose designation has been accepted and the determination result of that area.

[0175] In S2720, the information processing device 104 notifies the abnormality information recorded in S2702 to the output destination set in S407 (for example, a video monitoring server, a mailer, software capable of instructing a call using an automated voice call, etc.).

[0176] That is, this step is a step showing an example of a process of notifying the determination result that there has been a change that should be notified.

[0177] That is, this step is a step showing an example of processing for outputting the results of processing based on image comparison between the reference image and the target image.

[0178] Figure 29 shows an example of the anomaly notification dashboard screen, which displays a table listing all the anomalies that have been detected so far. The screen displays the detection time, monitoring task name, and coordinates of the detected area. As a result, the user can easily know which threshold value was used to determine the abnormality.

[0179] Furthermore, it is possible to improve the accuracy of detecting abnormalities near the grid.

[0180] It also becomes possible to easily know what the basis for detecting an abnormality was.

[0181] This concludes the description of the first embodiment. Next, a second embodiment for updating the reference image will be described.

[0182] FIG. 30 is a diagram illustrating an example of an outline of updating the reference image in the second embodiment.

[0183] When three image frames are used as reference images, conventionally (top of Figure 30), feature values ​​(for each grid) are extracted from the three reference images, and the average value (for each grid) is calculated to create a histogram of the reference images (S2308 in Figure 23). Here, it is assumed that the average feature value of the three reference images is 4.0 (top right of Figure 30). In the improved second embodiment (bottom of Figure 30), in anomaly detection systems that previously could not automatically update the feature amounts of the reference image at the timing desired by the user, by setting a value (10 in this case) in the setting called "number of interval frames", it is now possible to automatically update the feature amounts of the reference image using the feature amounts of images at 10-frame intervals.

[0184] Specifically, if the feature value of the image to be updated is 7.0, and the average feature value of the three previous reference images was 4.0, then by replacing one of the reference images, the average feature value of the new reference image is updated to 5.0 (4.0 + 4.0 + 7.0) / 3 (Figure 30, bottom right). This update process will continue automatically every 10 frames until you change the "number of frames between frames."

[0185] In this embodiment, the reference image is held as a feature, but this method is not limited to this. The reference image may be stored as an image and an image comparison may be performed between the reference image and the image to be analyzed.

[0186] This concludes the explanation of FIG. Fig. 31 is an example of a flowchart obtained by rewriting the flowchart of Fig. 23 of the first embodiment for the second embodiment. Description of the same processes as in Fig. 23 will be omitted.

[0187] In S3101, the information processing device 104 performs a process for initializing the reference image automatic update (FIG. 32).

[0188] In S3102, the information processing device 104 performs the process of automatically updating the reference image (FIG. 33).

[0189] This concludes the explanation of FIG. FIG. 32 is an example of a flowchart of the initialization process for updating the reference image in the second embodiment.

[0190] In S3201, the information processing apparatus 104 initializes a variable of the number of interval frames to 0 for determining the timing of automatic updating of the reference image.

[0191] In S3202, the information processing apparatus 104 receives the setting of the value of the frame interval number setting value 3211.

[0192] That is, this step is a step showing an example of a process of accepting a first predetermined setting related to updating information related to the reference image.

[0193] That is, the interval frame number setting value 3211 is a setting for specifying information relating to a determination target image to be used to update information relating to the reference image, from information relating to determination target images that are sequentially acquired.

[0194] If the interval frame number setting value 3211 is 1 or more, the reference image is updated using the image to be analyzed when the interval frame number variable is equal to or greater than the interval frame number setting value 3211. Note that if the interval frame number setting value 3211=0, the reference image is not automatically updated.

[0195] In S3203, the information processing apparatus 104 receives the setting of an anomaly detection ratio threshold 3212 for determining whether or not to automatically update the reference image.

[0196] That is, the abnormality detection ratio threshold 3212 is a second predetermined setting related to information related to the reference image.

[0197] That is, this step is a step showing an example of a process of accepting a second predetermined setting related to information related to the reference image.

[0198] The anomaly detection rate is the number of areas where anomalies are detected / the number of areas divided by grids in the analysis target image used for updating the reference. If anomalies are detected in all areas, it is 100%, and if no areas have anomalies, it is 0%.

[0199] In the case where the abnormality detection rate threshold 3212=100, when the variable of the number of interval frames>the set value of the number of interval frames 3211, the reference image is always updated using the image to be analyzed.

[0200] When the anomaly detection rate threshold 3212 is 60, the reference image is updated using the analysis target image only when the variable for the number of interval frames is greater than the set value for the number of interval frames 3211 and the number of grids in which anomalies are detected in the analysis target image used for updating the reference is less than 60%.

[0201] When the abnormality detection rate threshold 3212=0, the reference image is not updated even when the variable of the number of interval frames is greater than the set value of the number of interval frames 3211.

[0202] By having this anomaly detection rate threshold 3212, if the image used for updating the reference image is an abnormal image, such as one containing large noise (i.e., if the number of grids equal to or greater than the anomaly detection rate threshold 3212 is judged to be "abnormal"), the reference image will not be updated.

[0203] This prevents abnormal images, such as images containing noise greater than a threshold or images that are far different from the reference image, from being included in the reference image, thereby preventing excessive abnormality notifications.

[0204] This completes the explanation of FIG. FIG. 33 is an example of a flowchart of the reference image update process in the second embodiment.

[0205] In S3301, the information processing device 104 determines whether to automatically update the reference image. If the interval frame number setting value 3211 set in S3202 is 0, the determination is NO, Fig. 33 is terminated, and the reference image is not automatically updated. If the interval frame number setting value 3211 is 1 or more, the determination is YES, and the process proceeds to S3302.

[0206] In S3302, the information processing apparatus 104 adds 1 to the variable of the number of interval frames used to determine the timing of automatic updating of the reference image.

[0207] In S3303, the information processing apparatus 104 determines whether the variable for the number of interval frames has reached the interval frame number setting value 3211 (whether it is time to automatically update the reference image). Specifically, if the variable for the number of interval frames is smaller than the set value for the number of interval frames 3211, the result is determined to be NO, the process of FIG. 33 is terminated, and the reference image is not automatically updated at this stage. If the variable of the number of interval frames is equal to or greater than the set value of the number of interval frames 3211, the answer is determined to be YES, and the process proceeds to S3304.

[0208] In this embodiment, it is determined whether the variable of the number of interval frames is equal to or greater than the set value of the number of interval frames 3211. However, this determination method is not limited to this, and the variable of the number of interval frames may be greater than the set value of the number of interval frames 3211.

[0209] In S3304, the information processing apparatus 104 calculates the ratio of abnormality detection areas (=number of areas where abnormality is detected / number of areas divided by grids) for the analysis target image used for updating the reference.

[0210] In S3305, the information processing apparatus 104 determines whether the anomaly detection ratio threshold 3212 set in S3203 is greater than the anomaly detection region ratio calculated in S3304.

[0211] If the anomaly detection rate threshold 3212>anomaly detection area rate, the answer is YES, so proceed to S3304; if the anomaly detection rate threshold 3212≦anomaly detection area rate, the answer is NO, so Figure 33 ends, and the reference image is not automatically updated using this image (because it is an abnormal image such as an image containing large noise above the threshold or an image that is far from the reference image).

[0212] In other words, this step is an example of a process that prevents the use of information related to the image to be judged in updating the information related to the reference image when it is determined, based on a comparison between information related to the reference image and information related to the image to be judged, that there is a change that is greater than or equal to the threshold value or less than or equal to or less than the threshold value.

[0213] In this embodiment, the reference image is updated when the abnormality detection rate threshold 3212>the abnormality detection rate, but this is not limited to this method, and other conditions may be used, such as the abnormality detection rate threshold 3212≧the abnormality detection rate, or the abnormality detection rate threshold 3212>the average of the abnormality detection rates.

[0214] In S3306, the information processing device 104 adds the histogram (of the image to be analyzed used for updating the reference) calculated in S2304 to the histogram of the currently stored reference information, recalculates the average value, and updates the histogram of the reference information.

[0215] That is, this step is a step showing an example of a process of updating information related to the reference image based on information related to the determination target image according to a first predetermined setting.

[0216] In other words, this step is an example of a process of updating information related to the reference image based on information related to the judgment target image that is acquired sequentially and information acquired based on the first predetermined setting.

[0217] In other words, this step is an example of a process of updating information related to the reference image using information related to the judgment target image related to the first predetermined setting from information related to the judgment target images acquired sequentially.

[0218] That is, this step is a step showing an example of a process of updating the information related to the determination target image according to a first predetermined setting by mixing it with the information related to the reference image at a predetermined ratio.

[0219] In other words, this step is an example of a process for updating the information related to the reference image based on the information related to the image to be judged when it is determined that there is a change related to the second specified setting based on a comparison between the information related to the reference image and the information related to the image to be judged.

[0220] In S3307, the information processing apparatus 104 clears the variable of the number of interval frames to 0, which is used to determine the timing of automatic updating of the reference image.

[0221] This completes the explanation of FIG.

[0222] As a result, the reference image can be easily updated automatically at the timing and intervals (=interval frame number setting value 3211) desired by the user.

[0223] This makes it possible to properly detect abnormalities even in images that change from moment to moment, such as in environments (e.g., outdoors) where proper abnormality detection would not be possible if the reference image were not updated.

[0224] Specifically, when using the same reference image outdoors during the day and at night, it is difficult to correctly detect abnormalities because the brightness of the entire image differs. By automatically updating the reference image periodically (for example, every minute), it becomes possible to properly detect abnormalities even in images whose brightness changes from moment to moment.

[0225] Furthermore, if the image being analyzed includes sky, the appearance of passing clouds will be judged as an abnormality, making it difficult to correctly detect abnormalities using the same reference image. However, by automatically updating the reference image periodically (for example, every 10 seconds), it becomes possible to properly detect abnormalities even in images in which clouds are passing by moment by moment.

[0226] In this embodiment, the reference image is updated based on the setting of the interval frame number setting value 3211, but this is not limited to the setting of the interval frame number setting value 3211. For example, the reference image may be automatically updated based on the setting of the type of object to be analyzed (e.g., water, smoke, fire, sky, etc.), the setting of the shooting environment (e.g., indoor, outdoor, underwater, telephoto shooting, zoom shooting, etc.), selecting a template suitable for the object to be analyzed and the shooting environment, and the setting value of that template.

[0227] As a result, overdetection will be reduced and abnormalities can be properly detected even in analysis objects and shooting environments that were previously unsuitable for detection.

[0228] As a result, it is possible to provide a mechanism for easily updating the reference image.

[0229] This concludes the description of the second embodiment. FIG. 34 is an example of a diagram illustrating an overview of anomaly detection using a trained model in the third embodiment.

[0230] The improved third embodiment (bottom of Figure 34) is an embodiment that enables object detection and anomaly detection using a trained model by specifying the program to be executed (Figure 37) in an anomaly detection system (top of Figure 34) that previously could only detect anomalies by comparing images.

[0231] This completes the explanation of FIG. Fig. 35 is an example of a flowchart obtained by rewriting the flowchart of Fig. 4(a) of the first embodiment for the third embodiment. Description of the same processes as in Fig. 4(a) will be omitted.

[0232] In S3501, the information processing apparatus 104 receives a selection of a value for the analysis method setting 3611 of the monitored process (FIG. 36) from the user.

[0233] If the value of the analysis method setting 3611 is "detection by image comparison", proceed to S409; if it is "object recognition (using a trained model by executing a program)", proceed to S3502.

[0234] Here, the analysis method setting screen 3630 (FIG. 36) will be explained.

[0235] When "Object recognition (using a trained model by executing a program)" is selected as the value of the analysis method setting 3611, the selection of a command 3631 for executing the program and a script path 3632 for the program is accepted.

[0236] You can select only one of these options, or you can select two or more. If you select two or more options, you can run the monitoring process simultaneously using two or more selected analysis methods.

[0237] In this embodiment, object detection processing is performed using a trained model, but this is not limited to object detection processing, and other processing using a trained model may be performed, such as anomaly detection processing using a trained model, correct / incorrect (normal product / defective product, etc.) determination processing using a trained model, prediction processing using a trained model, evaluation processing using a trained model, etc.

[0238] In this embodiment, the program is written in Python (registered trademark), but it is not limited to Python (registered trademark), and other programming languages ​​such as C, C++, and Java (registered trademark) may also be used.

[0239] In this embodiment, a program script is specified and executed, but this is not limited to this method. It is also possible to specify identification information for an application, API, web service, etc. and execute them.

[0240] In this embodiment, the analysis method is selected from the options of "detection by image comparison" and "object recognition (using a trained model by executing a program)", but it is not limited to these two options, and it may be a format in which an analysis execution terminal 3640 (for which an analysis method is specified) is selected, or a format in which a combination 3650 of an analysis execution terminal and an analysis method is selected.

[0241] The analysis execution terminal may include the information processing device 104, or may be only another terminal that does not include the information processing device 104.

[0242] That is, this step is a step showing an example of a process of accepting a selection of an information processing device that will analyze the target image and a combination of analysis processes.

[0243] In other words, this step is an example of a process for accepting at least one selection from options including options related to image comparison and options related to the trained model.

[0244] In other words, this step is an example of a process for accepting a selection from options including options related to image comparison and options related to the trained model.

[0245] In other words, this step is an example of a process for accepting the selection of two or more options from options including options related to image comparison and options related to the trained model.

[0246] As described above, if users are able to freely select the analysis execution terminal 3640 or the combination of analysis execution terminal and analysis method 3650, it will be possible to have other devices outside the anomaly monitoring system execute high-load monitoring processes, thereby enabling anomaly monitoring through load balancing.

[0247] Furthermore, since it becomes possible to perform analysis using different devices and different analysis methods for each monitoring process, it becomes possible to distribute the load and set up flexible monitoring settings.

[0248] It will also be possible to flexibly configure multiple devices to monitor a single monitoring target using different analysis methods.

[0249] In addition, it will be possible to flexibly use programs, libraries, trained models, etc. that are not available in the anomaly monitoring system, allowing for flexible system design. In some cases, it will even be possible to create a grid computing-like configuration.

[0250] In addition, in the case of a format in which a combination 3650 of an analysis execution terminal and an analysis method is selected, if the analysis execution terminal does not have a program / library / trained model, etc. of the analysis method, the anomaly monitoring system may send the program / library / trained model, etc. of the analysis method to the analysis execution terminal and have it executed, or the analysis execution terminal may download the program / library / trained model, etc. of the analysis method from another device, or the analysis execution terminal may use the program / library / trained model, etc. of the analysis method that exists in the other device.

[0251] Furthermore, when a script template save button 3633 is pressed, the script template of the program stored in the information processing device 104 (or an external device) can be saved (downloaded) to an arbitrary location.

[0252] By modifying the script template saved in this way, any trained model (3711 in Figure 37) can be used to detect the object the user desires (3712 in Figure 37, in this case, a "bear") and receive the detection results.

[0253] This means that anomaly detection systems, which previously could only detect anomalies by comparing images, can now detect a variety of objects using any trained model by running any program.

[0254] In this embodiment, people and bears are detected, but this is not limited to simple objects such as people and bears, and detection may also include the state or part of a living thing or object, such as a person walking, a person running, a person acting violently, a person who has fallen, a human child, a man in his 30s wearing red clothing, a woman in her 20s wearing a hat, a teenage boy with his hand raised, a right hand, a knee, a left foot, a standing bear, a baby bear, a swimming bear, etc.

[0255] In this embodiment, people and bears are detected by using a trained model obtained by executing a program. However, the present invention is not limited to people and bears, and may also include living things such as birds, cats, dogs, horses, and cows; vegetables such as carrots, cabbage, lettuce, potatoes, and broccoli; fruits such as apples, mandarin oranges, grapes, oranges, and melons; objects and structures such as cars, bicycles, buses, motorcycles, trucks, traffic lights, stop lines, crosswalks, buildings, schools, and hospitals; home appliances such as mobile phones, smartphones, televisions, radios, microwave ovens, toasters, ovens, and refrigerators; stationery and tools such as watches, pencils, ballpoint pens, notebooks, books, scissors, toothbrushes, combs, sprays, cans, bottles, balls, bats, gloves, and rackets; clothing such as ties, pants, skirts, shirts, and sweaters; works of art, copyrighted works, and products such as paintings, sculptures, photographs, and images; designs and trademarks such as goods and logos; and various other objects, living things, plants, crystals, atoms, molecules, elements, viruses, bacteria, and objects that can be identified by images.

[0256] In S3502, when the setting screen 1610 (FIG. 16) accepts a user's press of the analysis tab 1611, the setting screen 1700 (FIG. 17) is displayed. At this time, when the adjustment button 1701 is accepted from the user, the analysis setting adjustment screen 4210 (FIG. 42) is displayed. Thereafter, the information processing device 104 executes the detection parameter setting process for the trained model of the monitoring task in FIG. 38.

[0257] This concludes the explanation of FIG. FIG. 38 is an example of a flowchart of a parameter setting process for a trained model in the third embodiment.

[0258] In S3801, when the information processing apparatus 104 receives a user's press of the determination area button 4211 on the analysis setting adjustment screen 4210 (FIG. 42), it displays the analysis setting determination area screen 4220 (FIG. 42).

[0259] In S3802, the information processing device 104 accepts an operation from the user on the detection determination area setting screen 4220 ( FIG. 42 ) of the analysis settings, such as enclosing it with a rectangle 4221 or the like using a mouse, touch operation, or the like, and accepts the setting of the detection determination area 4221 of the trained model. This detection determination area 4221 can be set in multiple locations.

[0260] In this embodiment, a detection determination area setting screen 4220 accepts the setting of an area near the door as a detection determination area 4221 .

[0261] In this embodiment, the setting of the detection determination area 4221 is accepted, but the present invention is not limited to this method, and a method of accepting the setting of a detection non-determination area (mask area) may also be used.

[0262] That is, this step is an example of a process of accepting the designation of a target region for outputting the results, from among regions corresponding to the shape of the target image.

[0263] That is, this step is an example of a process for accepting designation of a target area, from among areas corresponding to the shape of the target image, for which output of the results is to be suppressed.

[0264] In S3803 , the information processing apparatus 104 stores the setting of the detection determination area 4221 in the RAM 203 .

[0265] In S3804, the information processing device 104 receives from the user the setting of a threshold value 4212 for the detection score on the analysis setting adjustment screen 4210. If there are multiple object types that can be detected by AI, the threshold value 4212 can be set for each object type.

[0266] Here, it is assumed that the setting of the threshold value 4212=55.0 for the object type person has been accepted.

[0267] In S3805, the information processing apparatus 104 stores the setting of the threshold 4212 in the RAM 203.

[0268] This completes the explanation of FIG. Fig. 39 is an example of a flowchart obtained by rewriting the flowchart of Fig. 21 of the first embodiment for the third embodiment. Description of the same processes as in Fig. 21 will be omitted.

[0269] In S3901, the information processing apparatus 104 acquires the value of the analysis method setting 3611 of the monitored process (FIG. 36) from the operating process information in the RAM 203.

[0270] If the value of the analysis method setting 3611 is "detection by image comparison", proceed to S2106; if it is "object recognition (use of trained model by program execution)", proceed to S3902.

[0271] In S3902, the information processing device 104 executes the detection process (FIG. 40) using the trained model.

[0272] This completes the explanation of FIG. FIG. 40 is an example of a flowchart of a detection process using a trained model in the third embodiment.

[0273] In S4001, if the information processing apparatus 104 has divided the moving image file into successive still images for each frame (S2105), it repeats the subsequent processes until the processes for all the generated still images are completed.

[0274] In S4002, the information processing device 104 reads a still image to be processed and develops it on the RAM 203. That is, this step is a step showing an example of processing for acquiring images captured at the same angle of view.

[0275] In S4003, the information processing device 104 performs processing to cause the detection process using the trained model to execute object detection processing (FIG. 41).

[0276] In S4004, the information processing device 104 acquires the detection score for each detected object received from the trained model.

[0277] In S4005, if the detection score for each detected object acquired in S4004 exceeds the threshold value 4212 set in S3805, the information processing device 104 records the detected object in the RAM 203 and proceeds to S4006. If the detection score does not exceed the threshold value 4212, the information processing device 104 proceeds to processing of the next still image.

[0278] In this embodiment, the judgment is made as to whether the detection score is greater than the threshold value 4212, but this judgment is not limited to this, and other judgments may be used, such as whether the detection score is greater than or equal to the threshold value 4212, or whether the average of the detection scores of all detected objects is greater than the threshold value 4212.

[0279] In S4006, if the area in which the detected object recorded on RAM is detected is within the determination area set in S3802, the information processing device 104 proceeds to S4007. If the detected object recorded on RAM is outside the determination area, the information processing device 104 proceeds to processing of the next still image. In other words, this step is a step showing an example of processing for controlling whether or not to output the determination result based on the area whose designation has been accepted.

[0280] In S4007, the information processing device 104 performs an abnormality notification process (FIG. 27) and notifies the user and / or an external application of information about the abnormal area.

[0281] In other words, this step is an example of a process in which the target image is input into a trained model and the results of processing using the trained model are output.

[0282] For example, if a "person" with a detection score of 72.1 is detected within the determination area set in S3802, the score exceeds the threshold value 4212=55.0, and therefore a notification is issued that a "person" has been detected.

[0283] This concludes the explanation of FIG. As a result, it will now be possible to set masks for object detection using trained models using a GUI similar to the mask area setting that was previously used for anomaly detection through image comparison.

[0284] In this embodiment, the detection judgment area 4221 and threshold value 4212 are set on the anomaly detection system side, and the anomaly detection system side determines whether or not to notify based on these settings. However, this is not limited to this method, and other methods may be used, such as sending the detection judgment area 4221 and threshold value 4212 set on the anomaly detection system side to the trained model, and having the trained model side determine whether or not to detect or notify.

[0285] Furthermore, in this embodiment, the detection judgment area 4221 and threshold value 4212 are set on the anomaly detection system side, and the anomaly detection system side determines whether or not to notify based on those settings, but this is not limited to this method, and other methods may be used, such as specifying the detection judgment area 4221 and threshold value 4212 in a program script, passing those settings to the trained model as arguments, and having the trained model side determine whether or not to detect or notify. FIG. 41 is an example of a flowchart of a process for causing a trained model to execute a detection process in the third embodiment.

[0286] In S4101, the information processing device 104 transmits the still image to be analyzed, which was developed on the RAM 203 in S4002, to the information processing device 106 in which the trained model is installed.

[0287] In S4102, the information processing device 106 receives the still image to be analyzed that has been transmitted by the information processing device 104.

[0288] In S4103, the information processing device 106 executes object detection inference on the received still image using the trained model.

[0289] In S4104, the information processing apparatus 106 acquires the detection result executed in S4103.

[0290] In S4105, the information processing device 106 transmits the detection result acquired in S4104 to the information processing device 104.

[0291] In S4106, the information processing device 104 receives the AI ​​detection result transmitted from the information processing device 106 in S4105 and expands it on the RAM 203.

[0292] This concludes the explanation of FIG.

[0293] As a result, anomaly detection systems that previously could only detect anomalies by comparing images can now detect objects and anomalies using trained models by specifying the program to be executed.

[0294] Furthermore, in anomaly detection systems that were previously only able to detect anomalies by comparing images, it will now be possible to detect a variety of objects and anomalies by running any program on any device. FIG. 43 is an example of a block diagram showing a software configuration in the second and third embodiments.

[0295] The information processing device 104 includes the following functional units.

[0296] The acquisition unit 4301 is a functional unit that sequentially acquires information related to the determination target image.

[0297] The first receiving unit 4302 is a functional unit that receives a first predetermined setting related to updating information related to the reference image.

[0298] The update unit 4303 is a functional unit that updates the information related to the reference image based on the information related to the determination target image according to the first predetermined setting.

[0299] The determination unit 4304 is a functional unit that determines whether or not a predetermined change has occurred based on a comparison between information relating to the reference image and information relating to the image to be determined.

[0300] The update unit 4303 is a functional unit that updates the information related to the reference image based on the information related to the determination target image sequentially acquired by the acquisition unit 4301 and the information acquired based on the first predetermined setting.

[0301] The update unit 4303 is a functional unit that updates the information related to the reference image using information related to the image to be determined that is related to the first predetermined setting, among the information related to the image to be determined that is sequentially acquired by the acquisition unit 4301.

[0302] The updating unit 4303 updates the information relating to the determination target image according to the first predetermined setting by mixing it with the information relating to the reference image at a predetermined ratio. The second receiving unit 4305 is a functional unit that receives a second predetermined setting related to information related to the reference image.

[0303] The update unit 4303 is a functional unit that updates the information related to the reference image based on the information related to the image to be judged when it is determined that there is a change related to the second specified setting based on a comparison between the information related to the reference image and the information related to the image to be judged.

[0304] The update unit 4303 is a functional unit that, based on a comparison between information related to the reference image and information related to the image to be judged, prevents information related to the image to be judged from being used to update the information related to the reference image when it determines that there is a change that is greater than or equal to the threshold value or less than or equal to or less than the threshold value.

[0305] The receiving unit 4306 is a functional unit that receives a selection of an information processing device that will analyze the target image and a combination of analysis processes.

[0306] The control unit 4307 is a functional unit that controls the information processing device of the combination selected and accepted by the accepting unit 4306 to execute analysis processing for the combination.

[0307] The receiving unit 4308 is a functional unit that receives the results of the analysis process from the information processing device that has been executed by the control unit 4307.

[0308] The output unit 4309 is a functional unit that outputs the result received by the receiving unit 4308 .

[0309] The reception unit 4306 is a functional unit that selects one of the options for the combination.

[0310] The receiving unit 4306 is a functional unit that receives the selection of two or more options for the combination.

[0311] The control unit 4307 is a functional unit that controls the information processing device for each of the two or more combinations selected by the receiving unit 4306 to execute analysis processing for the combination.

[0312] The output area receiving unit 4311 is a functional unit that receives the designation of a target area for outputting the results, from among areas corresponding to the shape of the target image.

[0313] The output unit 4309 is a functional unit that outputs the results related to the target region for outputting the results.

[0314] The mask area receiving unit 4312 is a functional unit that receives the designation of a target area, from among areas corresponding to the shape of the target image, for which the output of the result is to be suppressed.

[0315] The output unit 4309 is a functional unit that suppresses the output of the results related to the target region for which the output of the results is to be suppressed.

[0316] The threshold value receiving unit 4313 is a functional unit that receives the setting of a threshold value.

[0317] The first output unit 4314 is a functional unit that outputs the results of processing based on the image comparison between the reference image and the target image.

[0318] The second output unit 4315 is a functional unit that inputs the target image into a trained model and outputs the results of processing using the trained model.

[0319] The first output unit 4314 and the second output unit 4315 are functional units that output the results related to the target region for outputting the results.

[0320] The first output unit 4314 and the second output unit 4315 are functional units that suppress the output of the results related to the target region for which the output of the results is suppressed.

[0321] The first output unit 4314 and the second output unit 4315 are functional units that inhibit the output of the result whose numerical value is less than the threshold value or equal to or less than the threshold value.

[0322] The first output unit 4314 and the second output unit 4315 are functional units that inhibit the output of the result whose numerical value exceeds the threshold or is equal to or greater than the threshold.

[0323] The selection receiving unit 4316 is a functional unit that receives at least one selection from options including options related to the image comparison and options related to the trained model.

[0324] The selection receiving unit 4316 is a functional unit that accepts a selection alternatively from options including options related to the image comparison and options related to the trained model.

[0325] This concludes the explanation of FIG. As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0326] In this case, the program itself read from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0327] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon disks.

[0328] Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing a program it has read, but also cases are included in which an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0329] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0330] Furthermore, the present invention may be applied to a system consisting of multiple devices, or to a device consisting of a single device. It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or device. In this case, the system or device can enjoy the effects of the present invention by reading a recording medium containing a program for achieving the present invention into the system or device.

[0331] The program may be in the form of object code, program code executed by an interpreter, script data supplied to an OS (operating system), or the like.

[0332] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modifications.

Claims

1. a receiving means for receiving a selection of an information processing device that performs analysis of a target image and a combination of analysis processes; a control unit that controls the information processing device of the combination selected by the accepting unit to execute an analysis process of the combination; a receiving means for receiving a result of the analysis process from the information processing device executed by the control means; an output means for outputting the result received by the receiving means; An information processing system characterized by:

2. The receiving means selects one of the options of the combination.

2. The information processing system according to claim 1, wherein:

3. The accepting means selects two or more of the combination options, The control means controls the information processing device to execute an analysis process for each of the two or more combinations selected by the acceptance means.

2. The information processing system according to claim 1, wherein:

4. The information processing device includes an information processing device that accepts the selection.

2. The information processing system according to claim 1, wherein:

5. The analysis process is The program includes at least one of an analysis process using image comparison, a detection process using a trained model, an object detection process using a trained model, a success / failure determination process using a trained model, a prediction process using a trained model, an evaluation process using a trained model, an analysis process using an application, an analysis process using an API, an analysis process using a web service, a program that executes an analysis process using image comparison, a program that executes an analysis process using an application, a program that executes an analysis process using an API, a program that executes an analysis process using a web service, a program that executes a detection process using a trained model, a program that executes an object detection process using a trained model, a program that executes a success / failure determination process using a trained model, a program that executes a prediction process using a trained model, and a program that executes an evaluation process using a trained model.

2. The information processing system according to claim 1, wherein:

6. further comprising an output area receiving means for receiving a designation of a target area for outputting the results from among areas corresponding to the shape of the target image; The output means outputs the results relating to the target region for outputting the results.

2. The information processing system according to claim 1, wherein:

7. The method further includes a mask area receiving means for receiving a designation of a target area, among areas corresponding to the shape of the target image, for which output of the result is to be suppressed; The output means suppresses output of the result related to a target area for suppressing output of the result.

2. The information processing system according to claim 1, wherein:

8. further comprising a threshold value receiving means for receiving a threshold value setting, The output means suppresses output of the result whose numerical value is less than the threshold value or equal to or less than the threshold value.

2. The information processing system according to claim 1, wherein:

9. further comprising a threshold value receiving means for receiving a threshold value setting, The output means suppresses output of the result when the numerical value relating to the result exceeds the threshold value or is equal to or greater than the threshold value.

2. The information processing system according to claim 1, wherein:

10. a receiving step of receiving a selection of an information processing device that will analyze the target image and a combination of analysis processes; a control step of controlling the information processing device of the combination selected and accepted in the accepting step to execute an analysis process of the combination; a receiving step of receiving a result of the analysis process from the information processing device executed by the control step; an output step of outputting the result received in the receiving step; 1. A method for controlling an information processing system, comprising:

11. 10. A program for causing at least one computer to function as each of the means of the information processing system according to any one of claims 1 to 9.

Images