Information processing systems, information processing methods, and programs

Abstract

This system provides a mechanism for detection based on the relative positions of multiple objects. [Solution] A first object and a second object are detected within the image, and it is determined whether the positional relationship between the first object and the second object satisfies predetermined conditions. Based on the determination result, the detection result is output.

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program, and more particularly to a technique suitable for use in detecting abnormalities.

Background Art

[0002] Conventionally, there has been a mechanism for detecting an abnormality by performing image processing using AI.

[0003] In Patent Document 1, the camera monitoring unit identifies a substantially rectangular object grasped by the operator from the moving image by image recognition processing, and when the substantially rectangular object can be identified, further determines whether the substantially rectangular object is a smartphone by image recognition processing, and when it is a smartphone, a technique for detecting the event is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] In the object detection process using AI, when a first object (for example, a person) and a second object (for example, a skateboard) are detected, they are detected as separate objects. However, it has been difficult to recognize the first object and the second object as one object (for example, a person riding a skateboard). Patent Document 1 does not disclose or suggest the above problems.

[0006] Therefore, an object of the present invention is to provide a mechanism for detecting based on the positional relationship of a plurality of objects.

Means for Solving the Problems

[0007] To achieve the above objectives, the present invention A detection means for detecting a first object and a second object within an image, A determination means for determining whether the positional relationship between the first object and the second object satisfies predetermined conditions, An output means that outputs a detection result based on the determination result by the determination means, It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, a mechanism can be provided that detects based on the positional relationship of multiple objects. [Brief explanation of the drawing]

[0009] [Figure 1] This is a system configuration diagram of information processing system 100. [Figure 2] This is a hardware block diagram of the information processing device 104. [Figure 3] This is an example of a block diagram showing the software configuration. [Figure 4] This is an example of a flowchart for the setup process. [Figure 5] This is an example of the initial display of the settings screen. [Figure 6] This is an example of an overall view of the settings screen. [Figure 7] This is an example of the settings screen (input tab). [Figure 8] This is an example of the settings screen (input tab). [Figure 9] This is an example of the settings screen (analysis tab). [Figure 10] This is an example of the settings screen (notification tab). [Figure 11] This is an example of the settings screen (monitoring processes and monitoring tasks). [Figure 12] This is an example of the settings screen (input tab). [Figure 13] This is an example of the settings screen (input tab). [Figure 14] This is an example of the display of the settings screen (notification tab). [Figure 15] This is an example of the display of the settings screen (notification tab). [Figure 16] This is an example of the display of the settings screen (monitoring task). [Figure 17] This is an example of the display of the settings screen (analysis tab). [Figure 18] This is an example of a script for using a learned model. [Figure 19] This is an example of a flowchart of the parameter setting process for a learned model. [Figure 20] This is an example of the display of the analysis settings screen. [Figure 21] This is an example of a flowchart of the anomaly detection process. [Figure 22] This is an example of the display of task registration information. [Figure 23] This is an example of a flowchart of the detection process using a learned model. [Figure 24] This is an example of a flowchart of the process for causing a learned model to execute a detection process. [Figure 25] This is an example of a script for using a learned model. [Figure 26] This is a diagram showing an example of the superimposition determination process. [Figure 27] This is a diagram showing an example of the determination process of the positional relationship that satisfies a predetermined condition. [Figure 28] This is an example of a screen when a predetermined condition is satisfied. [Figure 29] This is an example of a screen when a predetermined condition is not satisfied. [Figure 30] This is an example of a flowchart of the anomaly notification process. [Figure 31] This is an example of the display of the settings screen (anomaly notification). [Figure 32] This is an example of the display of the anomaly notification dashboard screen.

Mode for Carrying Out the Invention

[0011] Figure 1 is a system configuration diagram showing an example of the configuration of the information processing system 100 in this embodiment.

[0012] Figure 1 shows a configuration in which the camera 102 and the information processing device 104 are connected via an image transfer cable (USB, Ethernet, CameraLink, etc.). However, a configuration of an information processing device equipped with a camera is also acceptable, even without this specific setup.

[0013] The information processing device 104 acquires an image obtained by the camera 102 capturing the target object via an image transfer cable, and performs processing related to the detection of abnormalities in the image.

[0014] Furthermore, the information processing system 100 of this embodiment may also have a configuration in which the camera 102 with the above-described configuration does not exist.

[0015] In that case, processing related to the detection of abnormalities in images is performed on images acquired from other terminals on the network or on images stored in the memory (RAM) of the information processing device 104.

[0016] The following describes the hardware configuration of an information processing device applicable to the information processing device 104 shown in Figure 1, using Figure 2 as an example.

[0017] Figure 2 is a block diagram showing a hardware configuration applicable to the information processing device 104 shown in Figure 1.

[0018] In Figure 2, 201 is the CPU, which comprehensively controls each device and controller connected to the system bus 204. The ROM 202 or external memory 212 stores the BIOS (Basic Input / Output System), which is the control program for the CPU 201, the operating system program (hereinafter referred to as the OS), and various programs necessary to implement the functions executed by each PC, as described later.

[0019] 203 is RAM, which functions as the main memory, work area, etc., of the CPU 201. The CPU 201 loads the necessary programs, etc., from ROM 202 or external memory 212 into RAM 203, and then executes the loaded programs to perform various operations.

[0020] 205 is an input controller that controls input from pointing devices such as the keyboard (KB) 210 and a mouse (not shown).

[0021] 206 is a video controller that controls the display on indicators such as display 211.

[0022] 207 is a memory controller that controls access to external memory 212, such as external storage devices (hard disks (HDs)) that store various types of data, flexible disks (FDs), or CompactFlash® memory connected to a PCMCIA card slot via an adapter.

[0023] 208 is a communication interface controller that controls the reception of image data from the external PC 213 via the network (TCP / IP). 209 is an image I / F controller that controls the reception of image data from camera 102 via an image transfer cable (USB, Ethernet, Camera Link, etc.).

[0024] The various programs described later for realizing this embodiment are stored in RAM 203 and executed by CPU 201.

[0025] Furthermore, the image data used when executing the above program is stored in ROM202, external memory212, external PC213, and camera102 depending on the application, and is stored in RAM203 via various controllers when the program is executed.

[0026] Figure 3 is an example of a block diagram showing the software configuration of an embodiment of this model.

[0027] The information processing device 104 includes the following functional units.

[0028] The acquisition unit 4301 is a functional unit that sequentially acquires information related to the image to be judged.

[0029] The first reception unit 4302 is a functional unit that receives the first predetermined setting related to the updating of information concerning the reference image.

[0030] The update unit 4303 is a functional unit that updates information relating to the reference image based on information relating to the judgment target image related to the first predetermined setting.

[0031] The determination unit 4304 is a functional unit that determines whether there is a predetermined change based on a comparison between the information relating to the reference image and the information relating to the image to be determined.

[0032] The update unit 4303 is a functional unit that updates the information relating to the reference image based on the information relating to the image to be judged, which has been sequentially acquired by the acquisition unit 4301, and the information acquired based on the first predetermined setting.

[0033] The update unit 4303 is a functional unit that updates the information relating to the reference image using the information relating to the first predetermined setting of the information relating to the image to be judged, which has been sequentially acquired by the acquisition unit 4301.

[0034] The update unit 4303 updates the information relating to the target image for determination related to the first predetermined setting by mixing it with the information relating to the reference image at a predetermined ratio. The second reception unit 4305 is a functional unit that receives a second predetermined setting related to information concerning the reference image.

[0035] The update unit 4303 is a functional unit that updates the information related to the reference image based on the information related to the image to be judged, when it determines that there has been a change in the second predetermined setting based on a comparison between the information related to the reference image and the information related to the image to be judged.

[0036] The update unit 4303 is a functional unit that, based on a comparison between the information relating to the reference image and the information relating to the image to be judged, determines that there is a change greater than or equal to, or less than or equal to, the threshold, and suppresses the use of the information relating to the image to be judged in updating the information relating to the reference image.

[0037] The reception unit 4306 is a functional unit that accepts the selection of an information processing device and analysis process combination for analyzing the target image.

[0038] The control unit 4307 is a functional unit that controls the information processing device to perform analysis processing on the combination selected by the reception unit 4306.

[0039] The receiving unit 4308 is a functional unit that receives the results of the analysis process from the information processing device executed by the control unit 4307.

[0040] The output unit 4309 is a functional unit that outputs the results received by the receiving unit 4308.

[0041] The reception unit 4306 is a functional unit that selectively selects one of the aforementioned combination options.

[0042] The reception unit 4306 is a functional unit that accepts the selection of two or more of the aforementioned combination options.

[0043] The control unit 4307 is a functional unit that controls the information processing device for each of the two or more combinations selected by the reception unit 4306 to perform analysis processing for that combination.

[0044] The output area receiving unit 4311 is a functional unit that receives the designation of a target area from among the areas corresponding to the shape of the target image, for which the result will be output.

[0045] The output unit 4309 is a functional unit that outputs the results related to the target area for which the above results are output.

[0046] The mask region receiving unit 4312 is a functional unit that receives the designation of a target region from among the regions corresponding to the shape of the target image in which the output of the result will be suppressed.

[0047] The output unit 4309 is a functional unit that suppresses the output of results related to the target region for which the output of the aforementioned results is to be suppressed.

[0048] The threshold reception unit 4313 is a functional unit that accepts the setting of a threshold.

[0049] The first output unit 4314 is a functional unit that outputs the results of processing based on image comparison between a reference image and a target image.

[0050] The second output unit 4315 is a functional unit that inputs the target image to the trained model and outputs the result of processing using the trained model.

[0051] The first output unit 4314 and the second output unit 4315 are functional units that output results related to the target area for which the above results are output.

[0052] The first output unit 4314 and the second output unit 4315 are functional units that suppress the output of results related to the target region for which the output of the results is to be suppressed.

[0053] The first output unit 4314 and the second output unit 4315 are functional units that suppress the output of the result if the numerical value related to the result is less than or equal to the threshold.

[0054] The first output unit 4314 and the second output unit 4315 are functional units that suppress the output of the result if the numerical value related to the result exceeds the threshold or is equal to or greater than the threshold.

[0055] The selection receiving unit 4316 is a functional unit that accepts at least one selection from a set of options, including the options related to the image comparison and the options related to the trained model.

[0056] The selection reception unit 4316 is a functional unit that accepts an alternative selection from a set of options, including the options related to the image comparison and the options related to the trained model.

[0057] This concludes the explanation of Figure 3.

[0058] The setup process shown in Figure 4(a) will be explained below.

[0059] In S401, the information processing device 104 determines whether a configuration file for the operating process exists in the external memory 212. If it is not registered, the process proceeds to S402; if it is registered, the process proceeds to S403.

[0060] In S402, the information processing device 104 displays the settings screen 510 (Figure 5), and when it receives a press of the add monitoring process button 511 from the user, it registers the operation process in RAM 203 and displays the settings screen 520.

[0061] In S403, when the information processing device 104 receives a press of the button 521 for selecting the operation process to be edited on the setting screen 502, it displays the setting screen 610 (Figure 6).

[0062] The setting acceptance process performed on setting screen 610 in S403 will be explained below using Figures 6 to 10.

[0063] When the information processing device 104 receives input from the user for the name 611 and description 612 of the operating process, it edits the operating process information in the RAM 203 and displays the settings screen 620.

[0064] When the information processing device 104 receives a press from the user on the settings screen 610, it displays the settings screen 710 (Figure 7).

[0065] When the information processing device 104 receives a selection from the user for "Video file loading (FFmpeg)" (721 in Figure 7) from the list of functions 711, it displays the settings screen 720. At this time, the settings screen 720 edits the operation process information of RAM 203 based on the location 722 and option 723 values ​​of the command entered by the user.

[0066] When the information processing device 104 receives a selection from the user for "Video file loading (OpenCV)" (731 in Figure 7) from the list of functions 711, it displays the settings screen 730 and edits the operation process information of RAM 203.

[0067] When the information processing device 104 receives a selection from the user for "Live video loading (video surveillance server)" (811 in Figure 8) from the list of functions 711, it displays the settings screen 810 (Figure 8). "Live video loading (video surveillance server)" is a function that loads live video captured by a network camera from a video surveillance server (not shown) that can be connected via the network.

[0068] Based on the computer name 812, username 813, and password 814 values ​​entered by the user on the settings screen 810, the operating process information of RAM 203 is edited. Also, if the connection confirmation 815 is pressed, a connection test is performed from the information processing device 104 to the video surveillance server via the communication I / F controller 208 using the information in RAM 203.

[0069] When the information processing device 104 receives a click from the user on the settings screen 610 to select the analysis tab 614 (Figure 6), it displays the settings screen 910 (Figure 9).

[0070] When the information processing device 104 receives a selection of "Detection by video comparison" from the function list 911 from the user, it displays the settings screen 920 and edits the operation process information of RAM 203. At this time, if the license is not authenticated, "No license" is displayed in the license information 922.

[0071] When the information processing device 104 receives a press of the license read button 923 from the user, if a valid license has been authenticated, it displays the settings screen 930, displays the license information 931, and edits the operation process information of the RAM 203.

[0072] When the information processing device 104 receives a notification from the user on the settings screen 610 by clicking the notification tab 615 (Figure 6), it displays the settings screen 1010 (Figure 10).

[0073] When the information processing device 104 receives a selection of a "custom command" (1021 in Figure 10) from the function list 1011 from the user, it displays the settings screen 1020 and edits the operation process information of the RAM 203.

[0074] A "custom command" is a function that executes commands to instruct batches or applications to run when an anomaly is detected. Using this function, it becomes possible to execute batch processing, instruct email sending to mail clients, instruct automated voice calls to be made, and integrate with other applications or external systems.

[0075] When the information processing device 104 receives a selection from the user for "Analysis Result Notification (Video Surveillance Server)" (1031 in Figure 10) from the function list 1011, it displays the settings screen 1030.

[0076] "Analysis Result Notification (Video Surveillance Server)" is a function that notifies the video surveillance server of the analysis results. This function is used when the video surveillance server is configured to notify users of anomalies, or when the video surveillance server analyzes and manages events such as anomalies.

[0077] Based on the values ​​of address 1032 and port number 1033 entered by the user on the settings screen 1030, the operating process information of RAM203 is edited.

[0078] When the information processing device 104 receives a connection confirmation 1034 from the user, it performs a connection test via the communication I / F controller 208 using the information in the RAM 203.

[0079] Returning to the explanation of Figure 4.

[0080] In S404, when the information processing device 104 receives a press of the OK button 631 on the setting screen 630 (Figure 6), it saves the operation process information registered in RAM 203 to external memory 212 as an operation process setting file.

[0081] In S405, the information processing device 104 determines whether a configuration file for the monitoring task exists in the external memory 212. If it is not registered, the process proceeds to S406; if it is registered, the process proceeds to S407.

[0082] In S406, the information processing device 104 displays the settings screen 630, and when it receives a press from the user of the Add task button 632 for the monitoring process, it registers the monitoring task in RAM 203 and displays the settings screen 1110 (Figure 11).

[0083] In S407, when the information processing device 104 receives a click on the monitoring task 1111 to be edited on the settings screen 1110, it displays the settings screen 1120.

[0084] The following describes the process for receiving settings for monitoring tasks performed in S407, using Figures 11 to 15.

[0085] When the information processing device 104 receives input from the user on the settings screen 1120, including a name 1121 and a description 1122, and a change in the toggle button 1123 for enabling this task, it edits the operation process information of the RAM 203 and displays the settings screen 1130.

[0086] When the information processing device 104 receives a press of the input tab 1124 from the user on the settings screen 1120, it displays either the settings screen 1200 (Figure 12) or the settings screen 1300 (Figure 13) depending on the settings of the operation process.

[0087] If the input setting for the operation process is "Load video file (FFmpeg)" or "Load video file (OpenCV)", the settings screen 1200 (Figure 12) will be displayed.

[0088] When the information processing device 104 receives a click on a video folder 1201 and a folder selection from the user, input of a frame acquisition interval 1202, and a change in the toggle button for deleting processed videos 1203, it edits the operation process information of RAM 203 and displays the settings screen 1210.

[0089] If the input setting for the operation process is "live video loading," the settings screen 1300 (Figure 13) is displayed. When the information processing device 104 receives a selection of a list item for camera 1301 from the user, it edits the operation process information in RAM 203 and displays the settings screen 1310.

[0090] When the information processing device 104 receives a notification from the user on the settings screen 1120 by clicking the notification tab 1125, it displays either the settings screen 1400 or the settings screen 1500, depending on the settings of the operation process.

[0091] If the notification setting for the operation process is "custom command," the settings screen 1400 (Figure 14) is displayed. When the information processing device 104 receives input of the command path 1401 from the user or a change in the toggle button for asynchronous execution 1402, it edits the operation process information in RAM 203 and displays the settings screen 1410.

[0092] If the notification setting for the operation process is "Analysis Result Notification (Video Surveillance Server)", the settings screen 1500 (Figure 15) is displayed. The information processing device 104 accepts the input of name 1501 and camera ID 1502 from the user and displays screen 1510. At this time, if the test event button 1511 is pressed by the user, a test notification process is executed to check whether the notification with the input content works correctly. At this time, if the setting of the detection frame display toggle button 1512 is enabled, test pseudo detection result information is also added to the notification content.

[0093] Returning to the explanation of Figure 4.

[0094] In S408, when the information processing device 104 receives a press of the OK button 1602 from the user on the setting screen 1600 (Figure 16), it saves the monitoring task information registered in RAM 203 as a monitoring task setting file in external memory 212 and displays the setting screen 1610.

[0095] In S409, the information processing device 104 accepts a selection from the user for the value of the monitoring process analysis method setting 1711 (Figure 17). If the value of the analysis method setting 1711 is "detection by image comparison", the process proceeds to S410; if it is "object recognition (using a trained model by program execution)", the process proceeds to S411.

[0096] Here, let me explain the analysis method setting screen 1730 (Figure 17).

[0097] If "Object Recognition (Use of a pre-trained model via program execution)" is selected as the value for the analysis method setting 1711, the system will accept the selection of the command 1731 for executing the program and the program's script path 1732.

[0098] You can select one of these options, or two or more. If you select two or more options, you can run the monitoring process simultaneously using the two or more selected analysis methods.

[0099] In this embodiment, object detection processing is performed using a trained model, but this is not limited to object detection processing. Other processing using a trained model may also be performed, such as anomaly detection processing using a trained model, correctness (normal product / defective product, etc.) processing using a trained model, prediction processing using a trained model, and evaluation processing using a trained model.

[0100] In this embodiment, the program is written in Python®, but it is not limited to Python®; other programming languages ​​such as C, C++, and Java® may also be used.

[0101] In this embodiment, we specify and execute a program script, but this method is not limited to that. You may also specify identification information for an application, API, web service, etc., and execute them.

[0102] In this embodiment, the analysis method is selected from the options of "detection by image comparison" and "object recognition (using a pre-trained model by program execution)." However, the system is not limited to these two options. It may also be possible to select an analysis execution terminal 1740 (for which the analysis method is specified), or to select a combination of an analysis execution terminal and an analysis method 1750.

[0103] The analysis execution terminal may include the information processing device 104, or it may consist only of other terminals that do not include the information processing device 104.

[0104] As described above, if users can freely select the analysis execution terminal 1740 or the combination of the analysis execution terminal and analysis method 1750, high-load monitoring processes can be executed on other devices outside the anomaly monitoring system, thereby enabling anomaly monitoring through load balancing.

[0105] Furthermore, since each monitoring process can be analyzed using different devices and different analysis methods, it becomes possible to flexibly configure monitoring settings along with load balancing.

[0106] Furthermore, it will become possible to flexibly configure monitoring of a single target using different analysis methods with multiple devices.

[0107] Furthermore, it becomes possible to flexibly utilize programs, libraries, and pre-trained models that are not available in the anomaly monitoring system's equipment, enabling more flexible system design. In some cases, it may even be possible to implement configurations similar to grid computing.

[0108] Furthermore, in the case where a combination of analysis execution terminal and analysis method 1750 is selected, if the analysis execution terminal does not have the program / library / trained model, etc. of the analysis method, the anomaly monitoring system may send the program / library / trained model, etc. of the analysis method to the analysis execution terminal and have it executed, or the analysis execution terminal may download the program / library / trained model, etc. of the analysis method from another device, or the analysis execution terminal may use the program / library / trained model, etc. of the analysis method that exists on another device.

[0109] Furthermore, when the script template save button 1733 is pressed, the script template of the program stored in the information processing device 104 (or external device) can be saved (downloaded) to any location.

[0110] By modifying the script template saved in this way, it becomes possible to use any pre-trained model (1811 in Figure 18) to detect an object desired by the user (1812 in Figure 18, in this case a "bear") and receive the detection results.

[0111] This allows anomaly detection systems, which previously could only detect anomalies through image comparison, to detect various objects using any pre-trained model by executing any program.

[0112] In this embodiment, while the detection is described as detecting people and bears, it is not limited to mere objects such as people and bears. It may also include detection of the state or parts of living beings or objects, such as a person walking, a person running, a person acting violently, a person who has fallen, a human child, a man in his 30s wearing a red shirt, a woman in her 20s wearing a hat, a teenage boy with his hand raised, a right hand, a knee, a left foot, a standing bear, a bear cub, a swimming bear, etc.

[0113] In this embodiment, we have stated that people and bears are detected by using a trained model obtained by program execution, but this is not limited to people and bears. It may also include living things such as birds, cats, dogs, horses, and cows; vegetables such as carrots, cabbage, lettuce, potatoes, and broccoli; fruits such as apples, oranges, grapes, and melons; objects and buildings such as automobiles, bicycles, buses, motorcycles, trucks, traffic lights, stop lines, pedestrian crossings, buildings, schools, and hospitals; home appliances such as mobile phones, smartphones, televisions, radios, microwave ovens, toasters, ovens, and refrigerators; stationery and tools such as clocks, pencils, ballpoint pens, notebooks, books, scissors, toothbrushes, combs, sprays, cans, bottles, balls, bats, gloves, and rackets; clothing such as ties, trousers, skirts, shirts, and sweaters; works of art, copyrighted works, and products such as paintings, sculptures, photographs, and images; designs and trademarks such as goods and logos; and various other objects, living things, plants, crystals, atoms, molecules, elements, viruses, bacteria, and other objects that can be identified by images.

[0114] In S410, the information processing device 104 accepts the setting of anomaly detection parameters for the monitoring task. Specifically, it accepts the setting of parameters for performing anomaly detection based on image differences.

[0115] In this embodiment, the explanation of anomaly detection based on image differences is omitted, but the parameter settings may be implemented by known methods. In S411, when the information processing device 104 receives a click of the user's analysis tab 1611 on the setting screen 1610 (Figure 16), it displays the setting screen (not shown). At this time, when the user clicks the adjustment button (not shown), it displays the analysis setting adjustment screen 2010 (Figure 20). Subsequently, the information processing device 104 executes the detection parameter setting process for the trained model of the monitoring task shown in Figure 19.

[0116] Figure 19 is an example of a flowchart for the parameter setting process of the trained model in this embodiment.

[0117] In S1901, when the information processing device 104 receives a press of the judgment area button 2011 from the user on the analysis settings adjustment screen 2010 (Figure 20), it displays the analysis settings judgment area screen 2020 (Figure 20).

[0118] In S1902, the information processing device 104 accepts user input such as drawing a rectangle 2021 around an area on the detection and judgment area setting screen 2020 (Figure 20) of the analysis settings using a mouse or touch, and accepts the setting of the detection and judgment area 2021 of the trained model. Multiple detection and judgment areas 2021 can be set.

[0119] In this embodiment, the detection and judgment area setting screen 2020 accepts the setting of the area near the door as the detection and judgment area 2021.

[0120] In this embodiment, the setting of the detection / determination area 2021 is accepted, but this method is not limited to this, and a method of accepting the setting of the detection / non-determination area (mask area) may also be used.

[0121] In other words, this step is an example of a process that accepts the specification of a target region from among the regions corresponding to the shape of the target image, for which the above result will be output.

[0122] In other words, this step is an example of a process that accepts the specification of a target region among the regions corresponding to the shape of the target image in which the output of the above result is to be suppressed.

[0123] In S1903, the information processing device 104 stores the settings for the detection and determination area 2021 in the RAM 203.

[0124] In S1904, the information processing device 104 accepts the setting of the detection score threshold 2012 from the user on the analysis settings adjustment screen 2010. If there are multiple types of objects that can be detected by the AI, the threshold 2012 can be set for each object type.

[0125] Here, we assume that a threshold of 2012 = 55.0 has been set for the object type person.

[0126] In S1905, the information processing device 104 stores the threshold value 2012 setting in the RAM 203.

[0127] This concludes the explanation of Figure 19.

[0128] The anomaly detection process shown in Figure 21 will be explained below.

[0129] In S2101, the information processing device 104 executes anomaly detection processing if the external memory 212 contains a configuration file for the monitoring task (task registration information 2201 (Figure 22)) and the task activation setting toggle 2202 is set to enabled.

[0130] In S2102, the information processing device 104 monitors whether there is new input information. If there is new input information, the process proceeds to S2103. If there is no new input information, the process returns to S2101 and repeats.

[0131] In S2103, the information processing device 104 acquires input information according to the configured input method (configured on the configuration screen 1200 or configuration screen 1300).

[0132] Specifically, if the input setting for the operation process (settings screen 1200) is "Read video file (FFmpeg)" or "Read video file (OpenCV)", the video files located in the specified video folder are used as input information and moved to the working area on external memory 212.

[0133] Furthermore, if the input setting for the operation process is "Live video loading (video monitoring server)" (settings screen 1300), the process of acquiring video from the specified video acquisition destination is executed, and if a still image is acquired, the process proceeds to S2106.

[0134] In S2104, if the information processing device 104 performed a video file transfer on the external memory 212 in S2103, it proceeds to S2105. If it has performed a video acquisition process and acquired a still image, it proceeds to S2106.

[0135] In S2105, the information processing device 104 reads the video file moved to the external memory 212 in S2103 using the means set in the input settings of the operation process, divides it into a series of still images for each frame, saves them as files on the external memory 212 or stores them as data on the RAM 203, and proceeds to S2106.

[0136] In S2106, the information processing device 104 obtains the value of the monitoring process analysis method setting 1711 (Figure 17) from the operation process information of RAM 203. If the value of the analysis method setting 1711 is "detection by image comparison", the process proceeds to S2107; if it is "object recognition (using a trained model by program execution)", the process proceeds to S2108.

[0137] In S2107, the information processing device 104 performs image difference detection processing.

[0138] In this embodiment, the explanation of anomaly detection based on image differences is omitted, but the image difference detection process may be implemented by known methods. In S2108, the information processing device 104 executes detection processing using the trained model. This will be explained in detail using Figure 23.

[0139] This concludes the explanation of Figure 21.

[0140] Figure 23 is an example of a flowchart of the detection process using the trained model in this embodiment.

[0141] In S2301, if the information processing device 104 has been splitting the video file into a series of still images frame by frame (S2105), it repeats the subsequent processing until processing is completed for all the generated still images.

[0142] In S2302, the information processing device 104 reads the still image to be processed and loads it onto the RAM 203. In other words, this step is an example of a process for acquiring images captured at the same angle of view.

[0143] In S2303, the information processing device 104 performs the process of causing the detection process using the trained model to execute the object detection process (Figure 24).

[0144] In S2304, the information processing device 104 obtains a detection score for each detected object received from the trained model.

[0145] In S2305, if the detection score for each detected object acquired in S2304 exceeds the threshold 2012 set in S1905, the information processing device 104 records the detected object on the RAM 203 and proceeds to S2306. If it does not exceed the threshold, it proceeds to the processing of the next still image.

[0146] In this embodiment, the determination is made as follows: detection score > threshold 2012. However, the determination is not limited to this, and other determinations such as detection score ≥ threshold 2012, or the average of the detection scores of all detected objects > threshold 2012, may also be used.

[0147] In S2306, the information processing device 104 proceeds to S2307 if the area in which the detected object recorded in RAM is located is within the determination area set in S1902. If the detected object recorded in RAM is outside the determination area, the process proceeds to the next still image processing. In other words, this step is an example of a process that controls whether or not to output a determination result based on the area that has been specified.

[0148] In S2307, the information processing device 104 performs an anomaly notification process (Figure 27) and notifies the user and / or external applications of information about the anomaly area.

[0149] In other words, this step is an example of a process in which the target image is input to a trained model and the result of processing using the trained model is output.

[0150] For example, if a "person" with a detection score of 72.1 is detected within the judgment area set in S1902, the score exceeds the threshold 2012 = 55.0, so a notification is sent indicating that a "person" has been detected.

[0151] This concludes the explanation of Figure 23.

[0152] As a result, it will now be possible to set masks for object detection using pre-trained models, using the same GUI as the mask region setting previously used for anomaly detection by image comparison.

[0153] In this embodiment, the anomaly detection system sets the detection judgment area 2021 and threshold 2012, and determines whether or not to notify based on these settings. However, this method is not the only way to do this, and other methods may be used, such as sending the detection judgment area 2021 and threshold 2012 set by the anomaly detection system to a trained model, where the trained model determines whether or not to detect or notify.

[0154] Furthermore, in this embodiment, the anomaly detection system sets the detection judgment area 2021 and threshold 2012, and determines whether or not to notify based on these settings. However, this method is not limited to this, and other methods may be used, such as specifying the detection judgment area 2021 and threshold 2012 in a program script, passing these settings as arguments to the trained model, and having the trained model determine whether or not to detect or notify.

[0155] Figure 24 is an example of a flowchart showing the process of having the trained model in this embodiment perform detection processing. In S2401, the information processing device 104 transmits the still image of the object to be analyzed, which was loaded onto the RAM 203 in S2302, to the information processing device 106, which is equipped with a trained model.

[0156] In S2402, the information processing device 106 receives the still image of the subject to analysis transmitted by the information processing device 104.

[0157] In S2403, the information processing device 106 performs object detection inference on the received still image using the trained model.

[0158] In S2404, the information processing device 106 acquires the detection results of two or more target objects within the still image.

[0159] In S2405, the information processing device 106 repeats the process until it has processed all possible combinations of objects from the detection results obtained in S2404. For example, if "person", "skateboard", and "ball" are detected, the processing in S2406 to S2409 is executed for each combination of "person" and "skateboard", "person" and "ball", "skateboard" and "ball", and "person", "skateboard", and "ball".

[0160] In S2406, the information processing device 106 determines whether or not parts of the detection areas of all the target objects overlap. If they overlap, the process proceeds to S2407; otherwise, it proceeds to S2409.

[0161] The specific processing method will be explained using Figure 26. Figure 26(a) is an example where the objects do not overlap, and Figure 26(b) is an example where the objects overlap. Assuming the top left of the image is the origin, we will assume that the detection area of ​​object 1 in Figure 26(a) has a top (x:y) of (60:40) and a bottom (x:y) of (60:80). Similarly, we will assume that the detection area of ​​object 2 has a top (x:y) of (60:90) and a bottom (x:y) of (60:100).

[0162] If the condition "Object 1's [bottom] <= Object 2's [top]" is true, it is determined that the objects are not superimposed. Similarly, if the condition "Object 2's [bottom] <= Object 1's [top]" is true, it is determined that the objects are not superimposed. If neither of these conditions is met, it is determined that the objects are superimposed.

[0163] In the example in Figure 26(a), the [bottom] of object 1 is (x:y)=(60:80) and the [top] of object 2 is (x:y)=(60:90). Comparing the y coordinate values, we get "80<=90". Therefore, we determine that object 1 and object 2 are not superimposed.

[0164] In the example in Figure 26(b), we assume that the detection region of object 2 is (x:y)=(60:75) at the top and (x:y)=(60:85) at the bottom. When calculated using the same method as above, neither condition is met. Therefore, it is determined that object 1 and object 2 are superimposed.

[0165] In this way, the system determines whether objects are overlapping based on a predetermined formula and the coordinate information of object 1 and object 2. In the example script in Figure 25, the description related to the overlap determination is in description section 2501.

[0166] Note that this calculation method is just one example; other methods are acceptable as long as they can determine whether or not objects are overlapping.

[0167] In this embodiment, we have described an example where overlap is determined if there is even a small overlap between the detected areas, but this method is not the only one we can use. For example, other methods are acceptable as long as it is possible to determine whether objects are overlapping or not, such as accepting a threshold setting and determining that overlap exists if the overlapping area is greater than or equal to the threshold.

[0168] By determining whether or not objects are superimposed, it is possible to narrow down the cases to be detected. Although it is possible to detect "a person riding a skateboard" using only the result of S2407, which will be described later, the objects that can be detected from an image are not limited to "people" and "skateboards". For example, if a "ball" is detected, processing related to the positional relationship between the "person" and the "ball" is unnecessary in the case where we want to detect "a person riding a skateboard". Therefore, by determining whether or not objects are superimposed, it is possible to skip unnecessary cases and perform processing efficiently.

[0169] In S2407, if the information processing device 106 determines that there is an overlap in S2406, it determines whether all the objects in question are in a positional relationship that satisfies predetermined conditions. If they are, the process proceeds to S2408; otherwise, it proceeds to S2409.

[0170] The specific processing method will be explained using Figure 27. The example in Figure 27 attempts to detect a person riding a skateboard, and aims to detect a state where "person (object 1)" and "skateboard (object 2)" are positioned in an upper-lower position. In the example in Figure 27, the predetermined condition is "object 1 < object 2". If this condition is met, it is determined that the positional relationship satisfies the predetermined condition (i.e., object 1 and object 2 are in an upper-lower position).

[0171] Here, the positional relationship can be that the objects are positioned vertically or horizontally on the image. Other positional relationships, such as diagonal, are also possible. In this embodiment, we will explain using a vertical positioning relationship as an example.

[0172] Assuming the top left of the image is the origin, the detection area of ​​object 1 in Figure 27(a) is assumed to be (x:y)=(60:40) at the top and (x:y)=(60:80) at the bottom. Similarly, the detection area of ​​object 2 is assumed to be (x:y)=(60:75) at the top and (x:y)=(60:85) at the bottom.

[0173] Substitute the y-coordinate value of each object into the following formula 1 to calculate the value. Below, detect1 refers to object 1, and detect2 refers to object 2.

[0174] [Formula 1] y1 = detect1[top] + int((detect1[bottom] - detect1[top]) * 0.9) y2 = detect2[top] + int((detect2[bottom] - detect2[top]) * 0.2)

[0175] Substituting each value and performing the calculation yields the following result.

[0176] [Formula 1 (after substitution)] y1 = 40 + (80-40)*0.9 =76 y² = 75 + (85 - 75) * 0.2 =77

[0177] Comparing the calculated values, we find that 76 < 77, which satisfies the predetermined condition "Object 1 < Object 2". Therefore, it is determined that Object 1 and Object 2 are in a positional relationship that satisfies the predetermined condition.

[0178] In the example in Figure 27(b), we assume that the detection area of ​​object 2 has a top (x:y) = (30:50) and a bottom (x:y) = (30:80). When calculated using the same method as above, the result for object 1 is 76 and the result for object 2 is 56. Since the predetermined condition "object 1 < object 2" is not met, it is determined that object 1 and object 2 do not have a positional relationship that satisfies the predetermined condition.

[0179] In this way, based on a predetermined mathematical formula and the coordinate information of object 1 and object 2, it is determined whether or not the objects are in a predetermined positional relationship. In the example script in Figure 25, the description related to the determination of the positional relationship is part 2502.

[0180] Note that the y-coordinate value was used here to calculate the vertical positional relationship; to find the horizontal positional relationship, use the x-coordinate value.

[0181] Note that the calculation method is just one example; other methods are acceptable as long as they can determine whether or not the objects are in a predetermined positional relationship.

[0182] In S2408, if it was determined in S2407 that the object was a match, the information processing device 106 integrates it as a new object detection result.

[0183] Specifically, Object 1 and Object 2 are merged and given a new class name. Additionally, the coordinate information for their respective detection areas is integrated and updated to represent the detection area for the new object. For example, "Person" and "Skateboard" are merged to obtain a new object detection result as "Skater" (although "Skater" is synonymous with "Person riding a skateboard," the merged object is referred to as "Skater"). By merging them as a new object, "Skater" is assigned a different ID and managed separately from "Person" and "Skateboard."

[0184] By integrating and managing these images as new objects, it becomes easier to view the detected images by linking them to an ID. For example, if the images reveal that the same person has been seen skateboarding in a prohibited area multiple times, measures such as issuing a warning to the skater can be considered.

[0185] Furthermore, the detection sensitivity can be adjusted for each object. For example, based on the score received when detecting a "skater" and the threshold related to the detection sensitivity, "skaters" can be excluded from detection. In other words, even if a "skater" is detected, if it does not meet the threshold, it can be decided not to notify it as an anomaly. By managing these as new objects, the values ​​that can be set for each individual object can be adjusted for the integrated new object as well. In other words, the flexibility of the anomaly detection system can be improved.

[0186] In S2409, if it is determined in S2407 or S2406 that the object in question is not applicable, the information processing device 106 controls the system to treat all such objects as undetected.

[0187] For example, if a person riding a skateboard is detected on a street where skateboarding is prohibited, an alert will be issued as an anomaly. However, if the person or the skateboard is detected individually, no anomaly will be issued. Therefore, if the conditions of S2407 or S2406 are not met, the respective objects will be reported as not detected.

[0188] In this embodiment, an example of outputting "not detected" has been described, but this method is not the only one that can be used. Any other method is acceptable as long as it distinguishes between objects in a predetermined positional relationship and individual objects. For example, the object could be detected as an individual object such as "person" or "skateboard," and the detection result could be sent to the information processing device 104. In this case, the information processing device 104 may control whether or not to notify an abnormality based on the received detection result.

[0189] If the processes in S2406-S2409 have been executed for all detected object combinations, proceed to S2410. If there are still object combinations remaining, return to S2406 and execute the processes in S2406-S2409 for the untested object combinations.

[0190] In S2410, the information processing device 106 transmits the detection results integrated in S2408 to the information processing device 104. It also transmits the results related to objects that were not detected in S2409.

[0191] In S2411, the information processing device 104 receives the new object detection results transmitted from the information processing device 106 in S2409 and loads them onto the RAM 203. It also receives the results related to objects that were not detected in S2409.

[0192] This concludes the explanation of Figure 24.

[0193] Figure 28 shows an example of an anomaly notification screen. The information processing device 104 notifies of an anomaly based on the newly received object detection result. Since a skater has been received as the new object detection result, the anomaly area 2801 is notified on the anomaly notification screen.

[0194] On the other hand, as shown in Figure 29, if the "person" 2901 and the "skateboard" 2902 are not in the predetermined positional relationship (i.e., the person is not riding the skateboard), it is not notified as an anomaly. In other words, an anomaly is notified when a new integrated object detection result (e.g., "person riding a skateboard") is received, and not otherwise. The anomaly notification process will be explained in Figure 30 and beyond.

[0195] While the example described shows how an anomaly can be notified via screen, the notification method is not limited to this. For example, notifications could be made via voice, or via both image and voice. Furthermore, the information processing system 100 may be connected to equipment for outputting sound, such as a speaker (a speaker will be used as an example below). If the analysis of images captured by camera 102 detects a "person riding a skateboard," an audio warning could be issued from a speaker near where camera 102 is installed. In short, any method that allows for notification in the event of an anomaly is acceptable.

[0196] In this embodiment, an example of notification when a new detection result is received has been described, but the timing of notification is not limited to this if it is possible to notify an anomaly. For example, instead of notifying an anomaly simply by detecting a new object, notification may be given only when the new object has moved. That is, if the position of the new object has changed before and after a frame, it is determined that it has moved and an anomaly is notified. Another example is to accept a frame interval setting and detect the position of new objects at each set frame interval. Based on the detected position of the new object and a threshold, if the amount of change in the position of the new object is greater than or equal to the threshold (this is just one example, and it may also be less than the threshold), it is determined that it has moved and an anomaly is notified.

[0197] As described above, this embodiment makes it possible to notify of an anomaly when objects are in a predetermined positional relationship. In conventional methods, each object was detected individually and notified as an anomaly, but this could not handle events that occurred due to combinations of objects. That is, when the goal was to detect "a person riding a skateboard," both "person" and "skateboard" would be detected, so even in cases such as "a person holding a skateboard," an anomaly would be detected. Therefore, by considering the positional relationship between objects, it becomes possible to accurately detect events that meet the conditions desired by the user, such as "a person riding a skateboard."

[0198] In this embodiment, an example of detecting a "person riding a skateboard" using "person" and "skateboard" has been described, but this is not the only example. For example, other detectable events that occur due to the positional relationship between objects may also be used, such as whether or not a helmet or hat is being worn, whether or not a person is riding a kick scooter or electric kick scooter, whether or not roller skates are being worn, or whether or not protective equipment is being worn in a factory, etc.

[0199] Alternatively, instead of detecting the relative positions of the objects themselves, the system may detect that Object 2 is superimposed on or in contact with a specific part of Object 1, or that a specific part of Object 1 and Object 2 are in a predetermined positional relationship, such as when a helmet or hat is attached to a person's head, when gloves or an umbrella are held in a person's hands, or when shoes or roller skates are attached to a person's feet.

[0200] The abnormality notification process shown in Figure 30 will be explained below.

[0201] In S3001, the information processing device 104 determines whether there is an anomaly to be notified based on judgment criteria such as the continuity of anomaly detection. If it determines that there is an anomaly to be notified, it proceeds to S3002. If it determines that there is no anomaly to be notified, it proceeds to S3003.

[0202] Specifically, a criterion is needed to determine whether to notify a system of an anomaly if it is detected in a single image, or if to notify a system of an anomaly if it is detected consecutively in images spanning several seconds.

[0203] For example, the information processing device 104 may accept the user's pre-set criteria for how many consecutive frames of anomaly detection will trigger a notification, or it may determine what the image being analyzed is of (factory, home appliance, plant, etc.) or whether it is indoors or outdoors, and then determine the criteria for notifying an anomaly based on the object and environment.

[0204] In S3002, the information processing device 104 records information about the region where the notification target abnormality has occurred on the RAM 203.

[0205] In step S3003, the information processing device 104 deletes the abnormal information to be notified that was recorded on the RAM 203.

[0206] In S3010, the information processing device 104 passes the still image information read in S2303 to the processing unit that draws the settings screen.

[0207] In S3011, the information processing device 104 passes the abnormal information recorded in S3002 to the processing unit that draws the settings screen.

[0208] In S3012, the information processing device 104 determines whether the settings screen is being displayed based on user input. If the settings screen is being displayed, the process proceeds to S3013. If the settings screen is not being displayed, the process terminates.

[0209] In S3013, the information processing device 104 checks the settings 3101 for the detection area drawing color on RAM 203. If there is a specification for the drawing method, proceed to S3014. If there is no specification, proceed to S3015.

[0210] In S3014, the information processing device 104 changes the highlighting during the drawing process in S3015, described later, based on the setting 3101 (Figure 31). In other words, this step is an example of a process that identifies and outputs which of several types of thresholds was used to determine that a change had occurred.

[0211] Setting 3101 specifies how to highlight items according to the detection factors for anomaly detection (items that fall below the threshold in S2310). In other words, this step is an example of a process that accepts the specification of how to identify and output each type of threshold.

[0212] Here, to ensure that the highlighting for each threshold is distinguishable, control is implemented to prevent the same highlighting settings from overlapping (e.g., displaying an error message if the same highlighting is selected for different thresholds, preventing the selection of a previously selected highlighting). In other words, this step demonstrates an example of a process that prevents the specification of the same output method for different types of thresholds.

[0213] Specifically, the highlighting method can be set for each type of threshold, such as (1) red when the brightness threshold is exceeded, (2) yellow when the hue threshold is exceeded, (3) green when the saturation threshold is exceeded, (4) blue when the edge gradient strength threshold is exceeded, and (5) purple when the edge gradient angle threshold is exceeded.

[0214] In this embodiment, highlighting is done using color, but this method is not limited to this. Other methods may be used, such as using patterns like diagonal lines, setting the highlighting method to surround items with dotted or thick lines, or setting different notification methods such as voice or email. It is also possible to set notification methods using methods other than highlighting.

[0215] In S3015, the information processing device 104 overlays highlighting on the areas containing the still image information acquired in S3010 and the abnormal information acquired in S3011, and then displays the image for drawing in the drawing memory on the RAM 203.

[0216] In S3016, the information processing device 104 draws the drawing image created in S3015 onto the setting screen 3110 (Figure 31). In other words, this step is an example of a process that outputs a result of determining whether or not there has been a change in the area.

[0217] If the drawing method is changed in S3014, the method of highlighting the abnormal region is changed and displayed as shown in abnormal region 3111(1)(2)(4).

[0218] This makes it easy for users to understand what threshold was used to determine that something was abnormal.

[0219] Alternatively, the highlighting of the abnormal area and the display of the masked area may be shown on the same screen (not shown).

[0220] Furthermore, although S2314 states that abnormal areas will not be displayed in the masked area, it is also possible to highlight the abnormal detection area (not shown) or highlight it in a different way (not shown) even in the masked area, based on user instructions. This is because it will give the user a reason to decide that the mask should be removed even in an area designated as a masked area, depending on the nature of the abnormal detection. In other words, it will be possible to easily configure the abnormal detection desired by the user.

[0221] Furthermore, the information processing device 104 may have a function to notify the user on screens 1810 and 1820 of what kind of abnormality has occurred in the mask area, even without receiving instructions from the user.

[0222] Furthermore, the system may have functions to suggest to the user that they change the mask settings based on the number and nature of anomaly detections, or for the information processing device 104 to automatically change the mask settings. Doing so would enable more accurate anomaly detection. In other words, this step is an example of a process that changes the specified area based on the specified area and the judgment result of that area.

[0223] In S3020, the information processing device 104 notifies the output destination set in S407 (e.g., video monitoring server, mailer, software that can instruct to make calls using automated voice communication) of the abnormal information recorded in S3002.

[0224] In other words, this step is an example of a process that notifies the result of a determination that a change requiring notification has occurred.

[0225] Figure 32 shows an example of the anomaly notification dashboard screen, which displays a list of anomalies detected so far in a table format. Here, the detection time, monitoring task name, and coordinates of the detected area are displayed.

[0226] As a result, users will be able to easily find out what threshold was used to determine that something was abnormal.

[0227] Furthermore, it will be possible to improve the accuracy of detecting anomalies near the grid.

[0228] Furthermore, it becomes easier to determine what criteria were used to detect the anomaly.

[0229] As described above, this embodiment provides a mechanism for detection based on the positional relationship of multiple objects.

[0230] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium containing a program that realizes the functions of the embodiments described above to a system or device, and by having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0231] In this case, the program read from the recording medium itself realizes the novel function of the present invention, and the recording medium on which that program is recorded constitutes the present invention.

[0232] For recording media used to supply programs, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, and the like.

[0233] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0234] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0235] Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device. It goes without saying that the present invention can also be applied when the results are achieved by supplying a program to a system or device. In this case, by reading a recording medium containing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0236] The above program may consist of object code, program code executed by an interpreter, script data supplied to the OS (operating system), and the like.

[0237] Furthermore, by downloading and reading the program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. It should be noted that all configurations combining the above-described embodiments and their modified forms are also included in the present invention. [Explanation of Symbols]

[0238] 100 Information Processing Systems 102 Cameras 104 Information Processing Device

Claims

1. A detection means for detecting a first object and a second object within an image, A determination means for determining whether the positional relationship between the first object and the second object satisfies predetermined conditions, If the determination means determines that the predetermined conditions are not met, the control means controls the first object and the second object to be processed as undetected. An output means that outputs a detection result based on the determination result by the determination means, An information processing system characterized by having the following features.

2. The system includes a determination means for determining a program for determining whether the positional relationship between the first object and the second object satisfies predetermined conditions. The information processing system according to claim 1, characterized by the following:

3. If the determination means determines that a predetermined condition is met, the control means controls the detection of the first object and the second object as a third object formed by integrating them. The information processing system according to claim 1, characterized by the following:

4. The output means outputs the detection result of the third object. The information processing system according to claim 3, characterized by the following:

5. The determination means determines whether or not the third object is moving. The output means outputs the detection result when it is determined by the determination means that movement is occurring. The information processing system according to claim 3, characterized by the following:

6. If the determination means determines that the predetermined conditions are not met, the control means controls the detection of the first object and the second object as separate objects. The information processing system according to claim 1, characterized by comprising the following:

7. The determination means determines whether the positional relationship satisfies predetermined conditions based on a predetermined mathematical formula and coordinate information relating to the first object and the second object. The information processing system according to claim 1, characterized by the following:

8. The determination means determines whether the positional relationship between a specific part of the first object and the second object satisfies predetermined conditions. The information processing system according to claim 1, characterized by the following:

9. The aforementioned predetermined condition is a condition in which the first object and the second object have a vertical or horizontal positional relationship within the image. The information processing system according to any one of claims 1 to 8.

10. The first object is a person, the second object is a skateboard, and the third object represents a skater. An information processing system according to any one of claims 3 to 5, characterized by the above.

11. If an abnormality is detected in the detection result output by the output means, the system includes a notification means that outputs a notification indicating that an abnormality has been detected. An information processing system according to any one of claims 1 to 8, characterized by the above.

12. A method for controlling an information processing system, The detection means of the information processing system includes a detection step in which it detects a first object and a second object in the image, The determination means of the information processing system includes a determination step of determining whether the positional relationship between the first object and the second object satisfies predetermined conditions, If the control means of the information processing system determines in the determination step that the predetermined conditions are not met, a control step is performed to control the first object and the second object to be processed as undetected. The output means of the information processing system outputs a detection result based on the determination result of the determination step, A control method for an information processing system characterized by comprising the following:

13. A program for causing at least one computer to function as one of the means of the information processing system described in any one of claims 1 to 8.

Images