Image monitoring system and image monitoring method

Abstract

To provide an image monitoring system and an image monitoring method that, while sufficiently suppressing processing needed for determination, make it possible to recognize an interaction between objects with high accuracy.SOLUTION: An image monitoring system 1 for detecting a phenomenon in a monitored region by using an image in which the monitored region is imaged, comprises at least one arithmetic-logic device and at least one storage device for storing a preset threshold for determining whether to perform interaction recognition. The arithmetic-logic device includes a position detection unit 32 for detecting an object position from the image, a calculation unit 33 for calculating a feature amount pertaining to a distance and / or an angle between each object in an object group to be recognized, a determination unit 34 for determining whether the feature amount reaches the threshold, and an interaction recognition unit 35 for recognizing an interaction with regard to an object group having reached the threshold.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image monitoring system and an image monitoring method for detecting a person or object from an image captured in a monitoring area and detecting a monitoring target based on the detection result. [Background technology]

[0002] In public spaces such as stations, airports, concert halls, and event facilities, terrorist acts and dangerous acts must be detected and dealt with promptly to ensure the safety of users. In addition, at work sites and construction sites, measures must be taken to prevent dangerous work from being carried out by workers to prevent workplace accidents. Since the number of monitors and security guards deployed at sites is limited, the demand for video surveillance using surveillance cameras is increasing. With the rise in awareness of security and prevention of workplace accidents, as well as the background of surveillance cameras becoming cheaper, more compact, and more energy-efficient, and with higher image resolution, the trend is for more surveillance cameras to be installed, and there is a need to monitor a huge amount of video with limited personnel.

[0003] For effective video surveillance and labor saving of surveillance personnel, automation of video surveillance is required. In particular, technology to recognize human behavior from video is one of the important technologies for automatic surveillance. For example, in public spaces, early detection of a subject's falling or crouching motion allows facility managers to quickly protect people in need of assistance within the facility. Early detection of running or violent behavior also contributes to maintaining security within the facility. Meanwhile, at work sites, recognition of dangerous work can be used to prevent workplace accidents and to understand work procedures to understand and streamline work mistakes.

[0004] To realize advanced behavior recognition, it is important to focus not only on people, but also on the objects and people around them and recognize their relationships with the target person. For example, in a workplace, there are many scenes that are difficult to recognize when using only information on a single person or a single person or object in the vicinity, such as assembly work using tools, improper use of a stepladder, or crawling under an object suspended by a crane, or in a public space, handing over, leaving behind, or taking away luggage.

[0005] On the other hand, as the number of surveillance cameras increases, the analysis server for performing image analysis also needs to be increased. Therefore, when it is difficult to increase the number of servers, it is required to deal with the problem by reducing the amount of analysis processing. In particular, in an environment where it is difficult to specify the installation method of cameras or to increase the number of cameras, if the number of recognition patterns is increased to accommodate various ways of seeing people, there is a problem that it leads to an increase in the amount of analysis processing and erroneous recognition. In response to this, for example, in the image processing device described in Patent Document 1, in order to provide an image processing device that enables highly accurate behavior recognition without increasing the processing load, a configuration is disclosed that recognizes behavior by using information on objects around a person in addition to information on the posture of the person. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-206321 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the technology disclosed in Patent Document 1, behavior recognition is attempted for various person-object pairs on the screen, so if the number of detected person-object pairs is large, the analysis process increases according to the number of pairs. For example, if the number of people and objects performing the interaction to be recognized in an image is twice as many as that in another comparison image, the number of combinations is four times that of the comparison image, so there is a problem that the number of inferences required to recognize the relationship increases exponentially. Therefore, the challenge is to enable high-precision interaction recognition while sufficiently suppressing the processing required for judgment.

[0008] Therefore, an object of the present invention is to provide an image monitoring system and an image monitoring method that enable highly accurate recognition of interactions between objects while sufficiently reducing the amount of processing required for judgment. [Means for solving the problem]

[0009] In order to solve the above problems, the image surveillance system of the present invention is an image surveillance system that detects events in a monitored area using images captured of the monitored area, and is equipped with at least one arithmetic device and at least one memory device that stores a preset threshold for determining whether or not to perform interaction recognition, and is characterized in that the arithmetic device has a position detection unit that detects the position of an object from the image, a calculation unit that calculates features related to the distance and / or angle between each object in an object group to be recognized, a determination unit that determines whether the feature reaches a threshold, and an interaction recognition unit that recognizes interactions for object groups that have reached the threshold.

[0010] In addition, the image monitoring method of the present invention is an image monitoring method that detects events in a monitored area using an image captured of the monitored area, wherein a position detection unit detects the position of an object from the image, a calculation unit calculates features related to the distance and / or angle between each object in an object group to be recognized, a judgment unit refers to a threshold condition that is preset to determine whether or not to perform interaction recognition and judges whether the feature reaches a threshold, and an interaction recognition unit recognizes an interaction for an object group that has reached the threshold. Effect of the Invention

[0011] According to the present invention, it is possible to provide an image monitoring system and an image monitoring method that enable highly accurate recognition of interactions between objects while sufficiently reducing the amount of processing required for determination. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram of an image monitoring system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an overall configuration of the image monitoring system shown in FIG. [Diagram 3] 1 is a diagram illustrating a hardware configuration of an image monitoring system according to a first embodiment of the present invention. [Figure 4] 1 is a functional block diagram of an image monitoring system according to a first embodiment of the present invention. [Diagram 5] FIG. 5 is an explanatory diagram of a determination unit shown in FIG. [Figure 6] 1 is a flowchart of an image analysis system according to a first embodiment of the present invention. [Figure 7] FIG. 13 is a diagram showing an example of a setting screen for threshold conditions. [Figure 8] FIG. 11 is a diagram showing an example of a screen for setting a monitoring importance level in the image monitoring system according to the second embodiment of the present invention. [Figure 9]FIG. 11 is a diagram showing a display screen in an image monitoring system according to a second embodiment of the present invention. [Figure 10] FIG. 11 is a diagram showing a search screen in the image monitoring system according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An embodiment of an image monitoring system according to the present invention will be described below. In one embodiment of the present invention, in order to reduce the processing load of the image analysis system and to increase the accuracy of recognition, the behavior recognition process is multi-staged, and whether or not to infer an interaction between an object and its surrounding objects is determined according to the result of the previous stage processing.

[0014] According to an embodiment of the present invention, in order to determine whether or not to recognize an interaction between an object detected on a screen and a peripheral object, first, feature amounts related to distance and angle are calculated from spatial positional relationships such as distance and angle between the target object and the peripheral object. Next, it is determined whether the feature amount reaches a preset threshold. If the threshold is reached, an interaction recognition process is performed on a group of objects. This makes it possible to reduce the number of inferences for interaction recognition that were conventionally performed on all candidates even when many objects are present on a screen, thereby reducing the processing load of the image monitoring system. In addition, by performing the process in multiple stages in this way, not only is the overall amount of calculations reduced, but it is also possible to exclude groups in which interactions may not occur in advance and perform detailed inference processing only on groups in which interactions are likely to occur, leading to improved recognition accuracy.

[0015] In this specification, the term "object" refers to an object or person detected by an object detection model in a certain monitoring area. The term "interaction" refers to a relationship between an object and another object. For example, examples of interactions between people include shaking hands and hugging. Examples of interactions between people and objects include grasping, leaving, or taking away an object. Examples of interactions between objects include approaching and separating. The term "candidates requiring monitoring" refers to an object or group thereof that has reached a threshold for performing detailed interaction recognition. The terms "candidates not requiring monitoring" refer to objects or groups thereof that do not reach a threshold for performing detailed interaction recognition. The term "event" refers to a scene in which a specific interaction is determined to have occurred as a result of interaction recognition. The term "objects requiring monitoring" refers to an object in which a specific interaction is determined to have occurred as a result of interaction recognition. The term "group" refers to a collection of two or more types of objects, and the number of associated objects does not matter. "User" refers to a person, such as a manager, monitor, or on-site security guard of the space in which the surveillance camera is installed, who can access the image surveillance system, operate the system settings, or check the footage and analysis results.

[0016] In one embodiment of the present invention, by performing interaction recognition only on candidates requiring monitoring, it is possible to reduce the processing load of the system and improve the recognition accuracy compared to a conventional image monitoring system that performs interaction recognition on all object groups. Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0017] Fig. 1 is an explanatory diagram of an image monitoring system according to a first embodiment of the present invention. As shown in Fig. 1, the image monitoring system 1 includes an image capturing system 2, an image analysis system 3, and a monitoring center system 4. The image capturing system 2 has at least one camera unit 21 installed in a monitored area 5.

[0018] In addition, the image analysis system 3 detects objects photographed within the monitoring area by analyzing the input image from the photographing system 2, and when a group of a predetermined object type, such as a pair of people or a pair of objects, is present, it calculates feature amounts from spatial positional relationships such as distance and angle relationships, and judges whether the feature amounts reach a predetermined threshold value. If the threshold value is reached, detailed interaction recognition processing is performed for the group of objects.

[0019] The monitoring center system 4 receives the analysis results from the image analysis system 3, effectively displays them to users such as monitors 6 and security guards, records all or part of the video, and transmits setting information regarding the above thresholds to the image analysis system 3. The monitoring center system 4 may also receive video transmitted from the photography system 2.

[0020] The photographing system 2, the image analysis system 3, and the monitoring center system 4 will be described below. Fig. 2 is a diagram showing the overall configuration of the image monitoring system shown in Fig. 1. As shown in Fig. 2, the imaging system 2 includes at least one camera unit 21 installed in the monitored area 5 (Fig. 1), and the captured images are sequentially input to the input unit 31 of the image analysis system 3. The camera unit 21 is a surveillance camera arranged so as to capture the entire area to be monitored (monitored area 5). There is no restriction on the type of surveillance camera, such as a fixed camera, a PTZ camera capable of pan-tilt-zoom (PTZ) operation, or a mobile camera such as a drone-mounted camera or a wearable camera.

[0021] When using a camera with an unfixed angle of view, such as a PTZ camera or a mobile camera, it is necessary to prevent the camera movement from affecting the model, especially when using a model that uses three-dimensional information in world coordinates for interaction recognition. For example, this can be addressed by pre-calibrating the camera and performing pre-processing such as estimating the object's position in the world coordinate system. In addition, it is not always necessary to perform video analysis on the entire area of ​​the image captured by the surveillance camera, and limiting the area to a portion of the surveillance area by masking is also effective in reducing the amount of analysis processing.

[0022] Moreover, the camera unit 21 and the input unit 31, as well as the camera unit 21 and the monitoring center system 4, are connected by wired communication and / or wireless communication, and the camera unit 21 continuously transmits frame images. When using a time-series data analysis model in which threshold calculation and / or interaction recognition are premised on the input of multiple frame images, it is desirable that the frame rate of the continuous transmission of frame images is equal to or higher than the required value of each model. However, if the deterioration in recognition accuracy that occurs when the frame rate falls below the required value is acceptable, the frame rate may be lower than the required value. In this case, it is preferable to perform processing to suppress the deterioration in accuracy, such as interpolation by interpolation or extrapolation of time-series data, in the input to each model. Moreover, the camera unit 21 and the image analysis system 3, as well as the camera unit 21 and the monitoring center system 4, do not have to correspond one-to-one. The image analysis system 3 and the monitoring center system 4 having one input unit 31 may receive video data from multiple camera units 21. Furthermore, when there are multiple image analysis systems 3, distributed processing of video analysis may be performed on the video from one camera unit 21. Even when the image analysis system 3 executes multiple processes for processing video data from a plurality of camera units 21, the frame rate from each camera unit 21 required by each process conforms to the above-mentioned constraints.

[0023] The camera unit 21 may be equipped with some or all of the functions of the image analysis system 3 described below. For example, the camera unit 21 includes at least one arithmetic unit and at least one storage unit, and performs edge processing for interaction recognition and other processes, and transmits only information related to objects to be monitored to the image analysis system 3, thereby making it possible to reduce the processing load of the image analysis system 3.

[0024] 2, the image analysis system 3 includes an input unit 31, a position detection unit 32, a calculation unit 33, a determination unit 34, an interaction recognition unit 35, an output control unit 36, and a storage unit 37. Note that in this embodiment, the image analysis system 3 is not limited to an on-premise system constructed on a server within an operation facility, and may be constructed on a server outside the facility by utilizing a cloud service, for example.

[0025] The input unit 31 receives an input of a video from the camera unit 21 and transmits the video data to the position detection unit 32. The video to be analyzed may not be directly input from the camera unit 21, but may be a video stored separately in a recorder, and the location where the video is stored does not matter. The position detection unit 32 performs object detection on the input image (video data) using a trained object detection model, and obtains a bounding box of a person or object. The calculation unit 33 calculates a feature amount related to a distance and / or an angle using information on the obtained bounding box. The determination unit 34 compares the feature amount calculated by the calculation unit 33 with a threshold value stored in the storage unit 37, and determines whether or not to perform interaction recognition between objects in the interaction recognition unit 35. For object groups that reach the threshold value in the determination unit 34, information is transmitted to the interaction recognition unit 35 as candidates that require monitoring. On the other hand, for object groups that do not reach the threshold value, information is not transmitted to the interaction recognition unit 35 as candidates that do not require monitoring. The interaction recognition unit 35 performs detailed interaction recognition processing using the position information based on the bounding box obtained by the position detection unit 32, or by adding other information obtained by additional image analysis to the position information. The output control unit 36 ​​controls information to be transmitted to the monitoring center system 4 according to the result of the interaction recognition. The memory unit 37 holds feature amounts related to distance and / or angle that serve as thresholds for comparison in the determination unit 34.

[0026] The monitoring center system 4 includes a recording unit 41, an image display unit 42, and a management control unit 43. The recording unit 41 has a function of storing image information and movement trajectory of the object to be monitored obtained by the video analysis by the image analysis system 3, information such as events caused by the person, attributes, occurrence area, and occurrence time as a database. The image display unit 42 displays information on the behavior of the object to be monitored at the current time and part or all of the frames at the time of the event occurrence. It also displays a frame image at the time of detection of the event or a video including the detection time, or an image or video showing the current position of the person involved in the event. It can also display images transmitted from the camera unit 21. The management control unit 43 has a function of allowing the user to input setting information in order to store the threshold value used by the determination unit 34 constituting the image analysis system 3 in the storage unit 37.

[0027] Fig. 3 is a diagram showing a hardware configuration of an image monitoring system according to this embodiment. As shown in Fig. 3, a camera unit 71 is connected to a computer 72 via a network. The computer 72 can communicate with a computer 73 via the network. Similarly, the computer 73 can communicate with a computer 74 via the network. The computers 72, 73, and 74 each have at least one arithmetic unit and at least one storage device. However, this configuration is not limited to this. For example, the camera unit 71 may be configured to communicate directly with the computer 74 without going through the computers 72 and 73.

[0028] The camera unit 71 functions, for example, as the imaging system 2 or the camera section 21, and the computer 72 functions as the image analysis system 3. The computer 73 functions, for example, as the monitoring center system 4. The computer 74 functions, for example, as a terminal used by an on-site monitor or the like.

[0029] At least one camera unit 71 is installed in the monitoring area, and transmits video data to a computer 72 as appropriate. The computer 72 includes a central processing unit (CPU) 721 as a calculation device, a random access memory (RAM) 722 as a main storage device, a hard disk drive (HDD) 723 or a solid state drive (SSD) as an auxiliary storage device, and a communication interface (IF) 724. If necessary, a graphical processing unit (GPU) is included as a calculation device. The computer 72 reads out various programs from the HDD 723, develops them in the RAM 722, and executes them with the CPU 721 or the GPU, thereby realizing an input unit 31, a position detection unit 32, a calculation unit 33, a determination unit 34, an interaction recognition unit 35, an output control unit 36, and a storage unit 37 that constitute an image analysis system 3 described later. The computer 72 also communicates with the camera unit 71 and the computer 73 via a predetermined communication interface 724. Although not shown, input / output devices such as a keyboard and a display may also be connected to the computer 72 via a predetermined IF. The computer 73 includes a CPU 731 as a calculation device, a RAM 732 as a main storage device, a HDD 733 as an auxiliary storage device, and a communication interface 734. The computer 73 implements the recording unit 41, the image display unit 42, and the management control unit 43 that constitute the monitoring center system 4 by reading various programs from the HDD 733, developing them in the RAM 732, and executing them by the CPU 731. The computer 73 is also connected to the computer 72 via a predetermined communication interface (IF) 734. Although not shown, input / output devices such as a keyboard and a display may also be connected to the computer 73 via a predetermined IF.

[0030] The computer 74 includes a CPU 741 as a calculation device, a RAM 742 as a main storage device, a HDD 743 as an auxiliary storage device, and a communication interface 744. The computer 74 reads various programs from the HDD 743, develops them in the RAM 742, and executes them by the CPU 741, thereby realizing a function as a terminal used by a user such as a site monitor. Although not shown, an input / output device such as a keyboard or a display may also be connected to the computer 74 via a predetermined IF. The input / output device receives input from a user such as a site monitor, and further presents the monitoring result to the user. An input / output device directly connected to the computer 73 may be used instead of the computer 74. A part or all of the processing of the image analysis system 3 may be processed on the side of the photographing system 2 (also called a surveillance camera). In that case, the photographing system 2 has a configuration including a part or all of the hardware of the camera unit 71 and the computer 72.

[0031] Fig. 4 is a functional block diagram of the image monitoring system according to this embodiment. The image analysis system 3 will be described in detail with reference to Fig. 4. The input unit 31, position detection unit 32, calculation unit 33, determination unit 34, interaction recognition unit 35, output control unit 36, and storage unit 37 that constitute the image analysis system 3 will be described below.

[0032] The input unit 31 sequentially receives images from at least one camera unit 21, and outputs the images to a downstream position detection unit 32. If the frame rate of the received images is high compared to downstream processing such as object position detection and interaction recognition, the frame rate may be thinned out to a frame rate sufficient for downstream processing in order to reduce the amount of analysis by the image analysis system 3.

[0033] The position detection unit 32 performs object detection on the input image using a trained object detection model, and obtains a bounding box of a person or object. The bounding box may be two-dimensional information on the vertical and horizontal directions of the image, but it is preferable that the bounding box is three-dimensional information in the world coordinate system in order to accurately obtain positional relationships such as distance and / or angle in subsequent processing. For object detection, there are cases where a method such as HOG (Histogram of Oriented Gradients) or YOLO (You Only Look Once) is used, and a method in which a circumscribing rectangle is used as one object region from key points for each object estimated by a skeleton estimation technique, and any method may be used in this embodiment. In addition, in order to obtain three-dimensional information of key points acquired in two dimensions, there are a method of estimating three-dimensional coordinates using a group of key points acquired in two dimensions as input, and a method of using depth estimation for each key point.

[0034] The calculation unit 33 calculates feature amounts related to distance and / or angle using the bounding box information acquired by the position detection unit 32. For example, in the case of a task to recognize an interaction between two objects, when calculating distance, the distance between the coordinates of the centers of both bounding boxes or the distance between any key points of both objects is calculated. When calculating angle, the angle between the centers of both bounding boxes or the angle between any key points of both objects is calculated as described above. A method such as object skeleton estimation is used to calculate key points.

[0035] The determination unit 34 compares the feature amount calculated by the calculation unit 33 with a threshold value stored in the storage unit 37, and determines whether or not the interaction recognition unit 35 recognizes an interaction between objects. The storage unit 37 stores settings in the storage unit 37, for example, such that the center-to-center distance of the bounding boxes between a person and a specific object such as a suitcase is 100 cm or less, or the person's height ratio is 0.5 or less. In the case of an angular relationship, for example, when comparing the relationship between the central coordinates of a person and a suitcase on an image, a setting is stored such that the detection range is 5 / 4 pi radians to 7 / 4 pi radians when the angular relationship with the suitcase is centered on the person. If the object group corresponds to these settings, it is transmitted to the interaction recognition unit 35 as a candidate for monitoring, together with the image, position information, and information regarding the distance and / or angle calculated by the calculation unit 33. On the other hand, an object group that does not correspond to the settings is treated as a candidate for not requiring monitoring, and is not transmitted to the interaction recognition unit 35. In addition, performing the judgment using multiple images, rather than performing the judgment for each image, is also suitable for stabilizing the recognition accuracy and reducing the amount of calculation. When the judgment process is performed for each image, it may be affected by the performance of the object detection model and occlusion, and position detection may not be performed normally, and there is a possibility that redundant interaction recognition processing may occur. In response to this, for example, a method may be used in which an arbitrary image is selected from several consecutive frames and judgment is performed, or a judgment process is performed for all frame images, and if a certain percentage of them are judged to be candidates for monitoring, information on that image or the frames before and after it is transmitted.

[0036] The interaction recognition unit 35 judges the presence or absence of an interaction between the object group and the type of the interaction. Detailed interaction recognition processing is performed using the position information based on the bounding box obtained by the position detection unit 32, or by adding other information obtained by additional image analysis to the position information. For example, if the calculation unit 33 has not calculated the key points of the object, the interaction recognition unit 35 can newly calculate them and use them in the recognition processing. When performing recognition using image information, bounding boxes, and position information of key points, a method of calculating feature amounts based on the three-dimensional positional relationship and moving speed of the object, the relative distance between the objects, the relative distance between any key points inside and outside the object, etc., and training SVM (Support Vector Machine), decision tree, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), GCN (Graph Convolutional Network), etc. can be mentioned. When constructing a recognition model, if the interaction to be recognized is an action in which it is important to capture the change over time, it is preferable to select a method capable of analyzing the change in time. For example, a method using convolution processing in the time direction, a method such as an RNN that can process current features while following past features, and the like can be used. These recognition models may be used not only alone but also in combination. A monitoring candidate in which a specific interaction has been determined to have occurred by the interaction recognition unit 35 is set as a monitoring object. At least one of a feature representing the amount of movement of an object per unit time calculated from the difference between previous and next image frames, a feature representing the type of object, a feature relating to the position of a rectangular area, and a feature obtained from an image may be calculated as the feature. The output control unit 36 ​​collects information such as images of the surroundings of the object and types of interactions, and transmits the information to the monitoring center system 4.

[0037] Fig. 5 is an explanatory diagram of the determination unit shown in Fig. 4. As shown in Fig. 5, in the monitored area 5 photographed by the camera unit 21, a person 51, a person 52, and a person 53 are photographed, and a stepladder 54 is photographed as an object. Here, it is considered to recognize an interaction between a person and the stepladder 54, that is, whether the person is working on the stepladder 54. In addition, in calculating the candidates to be monitored, it is assumed that an area 56 is designated in the storage unit 37 as a threshold value related to the distance centered on the designated stepladder 54. It is also assumed that an angle relationship is designated in advance in the storage unit 37.

[0038] The relationship 511 between the centers of the bounding boxes of the person 51 and the stepladder 54 is that the person and the specific object are within a certain distance and the angular relationship is within a specified range. At this time, the determination unit 34 determines this object group as a candidate for monitoring, and transmits related information to the interaction recognition unit 35 at the subsequent stage. On the other hand, the relationship 512 between the centers of the bounding boxes of the person 52 and the stepladder 54 is such that the person and the specific object are within a certain distance and the angular relationship is outside a predetermined range. In this case, the determination unit 34 determines this object group as a candidate not requiring monitoring, and the information is not transmitted to the interaction recognition unit 35 at the subsequent stage.

[0039] In addition, the relationship 513 between the centers of the bounding boxes of the person 53 and the stepladder 54 is such that the person and the specific object are outside a certain distance and the angular relationship is outside a predetermined range. In this case, similar to the relationship 512, the determination unit 34 determines this object group as a candidate not requiring monitoring, and the information is not transmitted to the interaction recognition unit 35 at the subsequent stage.

[0040] Next, the flow of processing of the image analysis system 3 constituting the image monitoring system 1 according to this embodiment will be described with reference to the flowchart shown in Fig. 6. It is assumed that the administrator of the monitoring center system 4 has set in advance in the storage unit 41 a threshold value related to the distance and / or angle used in the determination unit 34.

[0041] As shown in Fig. 6, in step S1, an image is input from the imaging system 2 to the image analysis system 3. Next, in step S2, object detection is performed by the position detection unit 32 constituting the image analysis system 3. If at least one object group to be recognized is detected in the monitored area 5 (step S3: YES), processing is performed for each detected object group in steps S4 to S9. On the other hand, if no object group to be recognized is detected in step S3, the process returns to step S1 to read the next frame image.

[0042] In step S5, the calculation unit 33 constituting the image analysis system 3 calculates a feature amount related to distance and / or angle. Next, in step S6, the determination unit 34 constituting the image analysis system 3 compares the calculated feature amount with a threshold value stored in the storage unit 37 to determine whether the threshold value is reached (step S7). If the determination result reaches the threshold value, an interaction recognition process is performed in step S8. On the other hand, if the threshold value is not reached (S7: NO), the process is performed again from step S5 for another object group. Finally, in step S10, as an output control to the monitoring center system 4, the output control unit 36 ​​performs a process of transmitting the object ID, image coordinates of the object to be monitored, etc. to the monitoring center system 4. After step S10, the process returns to step S1 to read the next frame image. It should be noted that the processing of the flow shown in FIG. 6 does not necessarily have to be performed by a single process, and may be performed using a plurality of processes in order to improve calculation efficiency.

[0043] Fig. 7 is a diagram showing an example of a setting screen for threshold conditions of feature quantities in this embodiment. Fig. 7 shows a GUI (Graphical User Interface) displayed by the management control unit 43 of the monitoring center system 4 in this embodiment, and allows the user to set thresholds. This setting information is stored in the storage unit 37 and used by the judgment unit 34.

[0044] In FIG. 7, a list of interactions to be recognized, distance conditions and angle conditions for each type of interaction, and a setting screen for the connection relationship between the two conditions are displayed. For example, in an area 81 of this display screen, the distance condition and the angle condition are each specified as "Yes" as a pre-determination condition for recognizing an interaction of inappropriate use of a stepladder. In the example of this figure, the distance condition column indicates that the threshold condition is met if a person is located within a distance of 300% of the reference length in the vertical, horizontal, or diagonal direction calculated from the size of the bounding box of the object. The angle condition column indicates that the threshold condition is met if a person is located within a range of 0.25 pi radians to 0.75 pi radians with respect to the positional relationship with the person relative to the object. In addition, for the connection relationship in the right column, it is specified whether the distance condition and the angle condition are to be connected as And / Or. In the example of the area 81, they are connected as And, and it is possible to set the threshold condition to be met when both the distance condition and the angle condition are satisfied.

[0045] Also, as shown in Fig. 7, area 82 shows a state in which only the distance condition is specified as a pre-determination condition for recognizing the interaction of holding the suitcase. The distance condition indicates that if the suitcase is located within 50% of the distance of the person, the threshold condition is met. On the other hand, the angle condition is not specified, and accordingly, the connection field in the right column shows a specification in which no input is required.

[0046] Also, as shown in Fig. 7, area 83 shows a state in which only an angle condition is specified as a pre-determination condition for recognizing an interaction of crawling under an object suspended by a crane. The angle condition indicates that the threshold condition is met if the person is located within a range of 1.25 pi radians to 1.75 pi radians in relation to the position of the person relative to the object as the base point. Meanwhile, no distance condition is specified, and accordingly, no input is required in the connection field in the right column.

[0047] 7 shows an example in which the distance condition is set in units of a ratio based on the size of the object, and the angle condition is set in units of pyramids with the object as the base point, but the setting method is not limited to this. For example, the distance condition may be set to a perimeter of "3m" based on an arbitrary object. In addition, FIG. 7 shows an example in which only the bounding box of an object is obtained in the position detection unit 32 constituting the image analysis system 3. However, if key points of an object are obtained in the position detection unit 32, it is also possible to set distance and angle conditions for at least one key point.

[0048] In this way, by introducing a preprocessing step to determine whether or not to recognize interactions based on various conditions, the number of inferences for interaction recognition is reduced, and the processing load on the video analysis server is reduced. In addition, by excluding object groups with clearly different spatial positional relationships from the recognition targets, the accuracy of interaction recognition is improved.

[0049] As described above, the image monitoring system 1 performs multi-stage behavior recognition processing with the aim of reducing the processing load on the image analysis system 3 and improving the accuracy of recognition, and determines whether or not to infer an interaction between an object and its surrounding objects depending on the results of the previous processing stage.

[0050] As described above, according to this embodiment, it is possible to provide an image monitoring system and an image monitoring method that enable highly accurate recognition of interactions between objects while sufficiently reducing the amount of processing required for determination. According to this embodiment, in order to determine whether or not to recognize an interaction between an object detected in the display screen and a peripheral object, first, feature amounts related to distance and angle are calculated from spatial positional relationships such as distance and angle between the target object and the peripheral object. Next, it is determined whether or not the feature amount reaches a predetermined threshold. If the threshold is reached, detailed interaction recognition processing is performed for the group of objects. This makes it possible to reduce the number of detailed interaction inferences for all candidate groups in which interactions may occur, even when many objects exist in the display screen, thereby reducing the processing load of the image monitoring system 1. In addition, by performing the processing in multiple stages in this way, not only is the overall amount of calculations suppressed, but also groups in which interactions may not occur can be excluded in advance and detailed inference processing can be performed only for groups in which interactions are likely to occur, leading to improved recognition accuracy. EXAMPLES

[0051] 8 is a diagram showing an example of a setting screen (GUI) for the monitoring importance in an image monitoring system according to a second embodiment of the present invention. This embodiment differs from the first embodiment in that the monitoring importance can be set from the viewpoint of an interaction type. Note that the functional blocks and configuration of the image monitoring system 1 are the same as those in the first embodiment, and therefore, in the following, explanations of parts that overlap with those in the first embodiment will be omitted.

[0052] In the above-mentioned first embodiment, the objective was to reduce the processing load of the image analysis system 3 constituting the image monitoring system 1 and to increase the accuracy of recognition, and since the monitoring importance for each interaction type was uniform, when multiple objects to be monitored appeared, it was left to the on-the-spot judgment of the monitor 6 to decide which object group was important, which could lead to differences in the order of response depending on the monitor 6. Therefore, in this embodiment, the monitoring importance can be set from the perspective of the interaction type, thereby improving the readability of the display screen and promoting effective video monitoring.

[0053] As shown in Fig. 8, there are three stages of monitoring importance: 1 point (low), 2 points (medium), and 3 points (high). However, the number of stages is not limited to this, and it is preferable that the setter be able to set it freely. Also, for interactions for which the monitoring importance is not set, the importance may be clearly indicated as 0 points.

[0054] The person setting the setting can press the pull-down field shown in area 84 and select a registered interaction from the list for each importance level column. The selected interaction type is added to the "Registered Interactions" list at the bottom by pressing the "Add" button in area 85. To delete a registered interaction after adding it, the user presses the check box corresponding to the interaction to be deleted and presses the "Delete registered item" button at the bottom of the display screen to delete it. In addition, the information set in this area is reflected in the memory unit 37 by pressing the "Save settings" button at the bottom of the display screen.

[0055] The monitoring importance is not limited to the type of interaction, but may be set in terms of the monitored area 5 where the interaction occurred, the attributes of the person who performed the interaction, etc. For example, regarding the location where the interaction occurred, the importance of the concourse may be set to 3 points (high) and the importance of the parking lot to 1 point (low), and the monitoring importance may be calculated by the sum or product of the location and the type of action.

[0056] FIG. 9 is a diagram showing a display screen in the image monitoring system according to the present embodiment, and is a diagram showing an example of a display on the image display unit 42 constituting the monitoring center system 4. As shown in FIG.

[0057] As shown in Fig. 8, a display method characterized by changing the size of the display screen in a situation in which monitoring importance scores are set in descending order of "taking away luggage," "leaving luggage behind," and "carrying luggage" is shown. Of the display screens 9, the people displayed on screen A (area 91), screen B (area 92), screen C (area 93), and screen D (area 94) are people who have performed interactions for which monitoring importance has been set and have been transmitted to the monitoring center system 4. Fig. 9 shows an example in which video images of all these people at the current time are displayed.

[0058] Although four display areas are illustrated in FIG. 9, the number and arrangement positions are not limited to this. In this embodiment, the area sizes of screen A (area 91) to screen D (area 94) are dynamically changed according to the determined monitoring importance. For example, when the monitoring importance of all persons is reset and none of the persons are displayed on the screen, person a who is "taking away luggage" at time "08:00:00" is first displayed on the largest screen. Next, person b who is "leaving luggage" at time "09:00:00" is determined to have a monitoring importance (2 points) lower than the monitoring importance (3 points) of person a, so the display area 92 of person b is smaller than the display area 91 of person a. In addition, the monitoring importance of person c who is "carrying luggage" at time "09:30:00" and person d who is "carrying luggage" at time "09:20:30" are both 1 point. Therefore, the display area 93 and the display area 94 are smaller than the display area 91 and the display area 92 .

[0059] In order to make it easier to distinguish the person displayed on the display screen 9 from other people photographed on the same screen, it is preferable to display the person after applying image processing such as superimposing a detection frame for the person or cropping the image. In addition, if multiple events are occurring due to limitations on the screen size, the screen may be made scrollable.

[0060] It is desirable to be able to switch the state of an event occurrence by selecting a row in the display screen 9 or area 95, in addition to the real-time tracking video as described above. Information about the displayed screen is output in the area 95, and in FIG. 9, the selection button in the area 96 corresponding to the display area 92 is pressed. By pressing this button, for example, the frame at the time of detection of an interaction caused by a person being tracked can be displayed, or the frame with the highest accuracy of the judgment between the start and end of the interaction can be displayed, or a short clip from the start to the end of the interaction can be played, which is suitable for the user to quickly grasp the event. Furthermore, when the staff has completed the response to the event confirmed as described above, when it is determined that no response is necessary, or when a false detection is obvious, the detected event can be deleted by selecting a row in the display screen 9 or area 95 and pressing the "delete" button.

[0061] Since events are expected to occur one after another over time, events that have not been processed by the user within a predetermined time from the time of detection may be automatically deleted from the display in order to improve the readability of the display screen 9. However, for particularly important events, processing may be performed to reduce user oversight, such as by issuing a warning after a certain time has elapsed since the event was detected or before the event is deleted.

[0062] In addition, the display screen 9 can be viewed not only by the person who is the subject of an alert, such as the monitor 6 at the monitoring center, who is expected to use a large display, but also by staff or security guards responding at the scene, who can view part or all of FIG. 9 at the scene by using a smartphone, tablet, AR goggles, etc.

[0063] The information on the object to be monitored may be stored in any location, such as the cloud or a storage server prepared in the monitoring center system 4. When an occlusion occurs in the object to be monitored or when the object to be monitored is framed out of the monitored area 5, it is also preferable to present information on the location and time when the object was last captured.

[0064] Fig. 10 is a diagram showing a search screen in the image monitoring system according to this embodiment. A search method using an interaction type, date and time, etc. will be described with reference to Fig. 10. Events for which the monitor 6 or on-site staff has completed their response in the image display unit 42 constituting the monitoring center system 4, or events for which it has been determined that no response is required, are deleted from the display output. Therefore, to check the events thereafter, a search function for the events from the recording unit 41, which is a database of information on the occurring events, is required.

[0065] As shown in FIG. 10, an area 97 indicates a search input screen, and an area 98 indicates an output screen. In the area 97, a search is narrowed down by interaction type and occurrence date and time. In this example, the check boxes for the interaction types have items for selecting the interaction to be searched for and inputting the occurrence date and time. Note that a narrowed down search may be performed based on the detection results of other video analysis tasks. For example, it is also effective to narrow down the search from the perspective of the attributes of the person who performed the interaction or the occurrence area. Also, each column does not necessarily require input, and may be left blank. For example, when a search is performed with all items left blank, it is possible to set it so that all information stored in the recording unit 41 is output as a search result.

[0066] In area 98, information on the search results such as the time of occurrence and the location of occurrence is displayed. If the person is confirmed to be staying within the monitored area 5 at the current time and can be tracked, information on the current location may be displayed. In the area displaying images, as in the GUI of Fig. 9, it is preferable to quickly grasp an event by displaying the frame at the time of detection of an interaction caused by a person being tracked, displaying the frame with the highest accuracy of determining the start and end of the interaction, or making it possible to play a short clip from the start to the end of the interaction.

[0067] By providing the above-mentioned search function, even if the event has been deleted from the display output by the image display unit 42 constituting the monitoring center system 4, it is possible to efficiently search for the event from the recording unit 41. In addition, since it is possible to search for similar cases and check the number of occurrences, it is possible to utilize this to take measures against and prevent events that are expected to occur in the future.

[0068] As described above, according to this embodiment, the image monitoring system 1 detects interactions from the monitoring video and sets the monitoring importance for each type of interaction, thereby enabling effective understanding of occurring events. In addition, by providing a new search unit, it is possible to search for similar cases, understand the number of occurrences, and take measures to deal with and prevent future incidents.

[0069] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to the embodiment having all of the described configurations.

[0070] In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another example to the configuration of one embodiment. In addition, the above-mentioned configurations, functions, processing units, processing methods, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits.

[0071] In addition, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. [Explanation of symbols]

[0072] 1...image surveillance system, 2...photography system, 21...camera unit, 3...image analysis system, 31...input unit, 32...position detection unit, 33...calculation unit, 34...judgment unit, 35...interaction recognition unit, 36...output control unit, 37...storage unit, 4...surveillance center system, 41...recording unit, 42...image display unit, 43...management control unit, 5...surveillance target area, 51, 52, 53...person, 54...ladder, 6...surveillant, 72, 73, 74...computer, 721, 731, 741...CPU, 722, 732, 742RAM, 723, 733, 743...HDD, 724, 734, 744...communication interface (IF)

Claims

1. 1. An image monitoring system that detects an event in a monitored area using an image captured in the monitored area, comprising: The present invention provides a method for detecting an interaction by using a computer, comprising: providing at least one computing device; and providing at least one storage device that stores a preset threshold value for determining whether or not to perform interaction recognition; The computing device includes: a position detection unit that detects a position of an object from the image; a calculation unit that calculates a feature amount related to a distance and / or an angle between each object in an object group to be recognized; a determination unit that determines whether the feature amount reaches a threshold value; and an interaction recognition unit that recognizes an interaction for an object group that has reached the threshold.

2. 2. The image monitoring system according to claim 1, The determination unit determines a group of objects in which the feature amount satisfies a preset condition as a monitoring candidate, The interaction recognition unit executes interaction recognition for the monitoring candidate, and when a predetermined interaction is detected, sets the object group or each object as a monitoring target object; An image monitoring system comprising a storage unit for storing information relating to the object to be monitored.

3. 2. The image monitoring system according to claim 1, An image monitoring system characterized by having a management control unit that can set distance conditions and / or angle conditions with respect to other objects for each type of interaction to be recognized, based on an arbitrary object that generates the interaction.

4. 2. The image monitoring system according to claim 1, An image monitoring system comprising an image display unit for displaying a frame image at the time of detection of the event or a video including the detection time, or an image or video showing the current position of a person involved in the event.

5. 2. The image monitoring system according to claim 1, the calculation unit detects two-dimensional or three-dimensional key points of an object in order to recognize the interaction, and calculates at least one of a feature representing an object's posture, a feature relating to a distance between any key points within the object and / or between objects, a feature representing an amount of movement of the object per unit time calculated from a difference between previous and next image frames, a feature representing an object type, a feature relating to a position of a rectangular area, and a feature obtained from an image; The image monitoring system is characterized in that the interaction recognition unit detects a type of interaction based on the calculated feature amount.

6. 2. The image monitoring system according to claim 1, the storage device stores, as importance information, information on at least one of an interaction type, a place where the interaction occurs, and an object attribute; The image monitoring system according to the present invention, wherein the computing device calculates a monitoring importance level for a person who has caused an interaction based on the importance level information.

7. 7. The image monitoring system according to claim 6, The image monitoring system according to the present invention, wherein the computing device has an image display unit that displays, in an emphasized manner, a person who has caused an interaction based on the monitoring importance.

8. 2. The image monitoring system according to claim 1, The image surveillance system is characterized in that the calculation device has a search unit that is capable of searching for information on the object that caused the interaction by searching the interaction detection record using information on at least one of the interaction type, the location where the interaction occurred, and the object attributes as a search query.

9. 2. The image monitoring system according to claim 1, The image surveillance system is characterized in that the arithmetic device is a plurality of arithmetic devices, some of which are implemented in a surveillance camera that captures images.

10. 1. An image monitoring method for detecting an event in a monitored area using an image captured in the monitored area, comprising: a position detection unit that detects a position of an object from the image; a calculation unit that calculates a feature amount related to a distance and / or an angle between each object in the object group to be recognized; a determination unit that determines whether the feature amount reaches a threshold value by referring to a threshold condition that is preset in order to determine whether or not to perform interaction recognition; An image monitoring method, comprising: an interaction recognition unit that recognizes an interaction for an object group that has reached the threshold value.

11. 11. The image monitoring method according to claim 10, The determination unit determines a group of objects in which the feature amount satisfies a preset condition as a monitoring candidate, The interaction recognition unit executes interaction recognition for the monitoring candidate, and when a predetermined interaction is detected, sets the object group or each object as a monitoring target object; A method for monitoring images, comprising the steps of: storing information relating to the object to be monitored;

12. 11. The image monitoring method according to claim 10, An image monitoring method characterized in that the management control unit is capable of setting distance conditions and / or angle conditions with respect to other objects for each type of interaction to be recognized, based on an arbitrary object that generates the interaction.

13. 11. The image monitoring method according to claim 10, An image monitoring method, characterized in that an image display unit displays a frame image at the time point when the event is detected or a video including the time point when the event is detected, or an image or video showing the current position of a person involved in the event.

14. 11. The image monitoring method according to claim 10, the calculation unit detects two-dimensional or three-dimensional key points of an object in order to recognize the interaction, and calculates at least one of a feature amount representing an object's posture, a feature amount relating to a distance between any key points within the object and / or between objects, a feature amount representing an object's movement amount per unit time calculated from a difference between previous and next image frames, a feature amount representing an object type, a feature amount relating to a position of a rectangular area, and a feature amount obtained from an image; The image monitoring method, wherein the interaction recognition unit detects a type of interaction based on the calculated feature amount.

Images