Image analysis system, method and program
The image analysis system addresses the issue of high false positives in work site monitoring by tracking and analyzing the relationship between people and objects, excluding stationary objects, and accurately identifying dangerous situations.
Patent Information
- Application Number
- JP2023073060
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2024-11-08
AI Technical Summary
Existing methods for detecting dangerous situations at work sites, such as interactions between workers and objects, suffer from high false positives and low recall, making them unsuitable for effective monitoring.
An image analysis system that tracks and analyzes the relationship between people and objects in time-series images, excluding stationary objects beyond a predetermined threshold to reduce false detections and determine dangerous situations accurately.
The system effectively reduces false positives by focusing on moving objects that pose a risk, thereby enhancing the accuracy of detecting dangerous work conditions and minimizing unnecessary tracking, thus improving the reliability of monitoring systems.
Smart Images

Figure 2024158121000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to techniques for recognizing people, objects, and relationships in images. [Background technology]
[0002] At work sites, it is important to detect dangers to people and deal with them before accidents occur. Dangerous situations at work sites include, for example, a worker crawling under a suspended load, using a stepladder in an inappropriate manner, climbing up and down a utility pole, and working at a height. In dangerous work recognition using images from the work site, whether a worker is in a dangerous situation is often determined based on the interrelationship between the worker and other workers or surrounding objects. For this reason, a method for recognizing interactions between people and objects (Human-Object Interaction: HOI recognition) is used for dangerous work recognition.
[0003] In HOI recognition, a predictive model is used to estimate the interaction between a person and an object. For example, after estimating the positions of the person and the object, the interaction between the person and the object is recognized by inputting RGB images of the bounding rectangles of the person and the object in one or more frames into the recognition model.
[0004] Patent Document 1 discloses a method for tracking objects in a video. The objects in Patent Document 1 include people. In the method of Patent Document 1, an object detected in a video in which a plurality of frames are arranged in a time series is classified as a moving object if the elapsed time from the detection of the object continues for a predetermined time or more, and an object that has elapsed for less than the predetermined time is classified as a moving object candidate, and a correspondence between an unrecognized object detected in a current frame and the moving object is checked, and if it is determined that the unrecognized object and the moving object correspond to each other, the unrecognized object and the moving object are associated, and if it is determined that the unrecognized object and the moving object do not correspond to each other, the correspondence between the unrecognized object and the moving object candidate is checked, and if it is determined that the unrecognized object and the moving object candidate correspond to each other, the unrecognized object and the moving object candidate are associated. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2019-003334 A Summary of the Invention [Problem to be solved by the invention]
[0006] When detecting the situation of a person in a workplace, etc., it is necessary to detect dangerous situations without overlooking them. In other words, due to the nature of the task, there is a demand to maximize the recall rate.
[0007] On the other hand, if there are too many false positives, where a system judges a situation to be dangerous when it is not, the system will not be able to function adequately as a surveillance system. For example, if there are many objects at a work site, there is a possibility that a large number of false positives will occur, indicating that people present there are in a dangerous situation. Therefore, there is a demand to reduce false positives as much as possible.
[0008] The method disclosed in Patent Document 1 is not a method for detecting the situation in which a person is placed, and is therefore not suitable for detecting the situation of a person from the interrelationship between a person and an object.
[0009] One objective of the present disclosure is to provide a technology for appropriately determining the situation of a person from the interrelationship between a person and an object in an image. [Means for solving the problem]
[0010] An image analysis system according to one aspect included in the present disclosure is an image analysis system that determines the situation of a person from time-series images, and has a memory that stores a software program and a processor that executes the software program, wherein the processor detects and tracks people and objects from images, and determines the situation of the person based on the relationship between the person and the object being determined, excluding objects whose stationary duration, which is an index value related to the time that the object remains stationary, has reached a predetermined threshold. Effect of the Invention
[0011] According to one aspect of the present disclosure, it is possible to provide a technology for appropriately determining the situation of a person from the interrelationship between the person and an object in an image. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram showing an overall overview of one embodiment of the present invention. [Diagram 2] 1 is a block diagram showing the functional configuration of an image analysis system, a camera unit, and a monitoring system in the present embodiment. [Diagram 3] 1 is a block diagram showing a hardware configuration of an image analysis system according to an embodiment of the present invention. [Figure 4] 4 is a flowchart showing a process for implementing an image analysis method by the image analysis system according to the present embodiment. [Diagram 5] 11 is a diagram showing how a tracking processing unit detects and tracks people and objects in an image. FIG. [Figure 6] FIG. 13 is a diagram showing an example of a tracking target table that records tracking information and the like relating to each object. [Figure 7] FIG. 13 is a diagram showing an example of a state in which rectangles overlap each other. [Figure 8] FIG. 2 is a diagram showing an example of a screen displayed on a video display unit of the surveillance system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0014] FIG. 1 is a diagram showing an overall overview of an embodiment of the present invention. As shown in FIG. 1, an image analysis system 10 in an embodiment of the present invention analyzes an image sent from a camera unit 20 that captures a work site where heavy machinery, materials, etc. are placed, detects people and objects contained in the image, tracks them for at least a predetermined time, and judges the situation of the people based on the relationship between the people and the objects. The image sent from the camera unit 20 and the situation of the people judged by the image analysis system 10 are sent to a monitoring system 30 in a monitoring center via the image analysis system 10. A monitor at the monitoring center monitors the situation of the people at the work site displayed on the image display unit of the monitoring system 30, and when an alert is issued as a result of the judgment by the image analysis system 10, for example, indicating that a worker is in a dangerous working state, the monitor notifies the worker or supervisor at the work site of the fact.
[0015] FIG. 2 is a block diagram showing the functional configuration of the image analysis system 10, the camera unit 20, and the monitoring system 30 in this embodiment.
[0016] As shown in FIG. 2, the image analysis system 10 in this embodiment has, as its functional configuration, an input unit 101, a position detection unit 102, a tracking processing unit 103, a judgment unit 104, a tracking target update unit 105, an interaction recognition unit 106, an output control unit 107, and a memory unit 108.
[0017] The input unit 101 acquires images captured by the camera unit 20 from the camera unit 20 by wireless communication or wired communication via a communication interface. In this specification and the claims, the term "image" is used to include not only one or more still images, but also a moving image (video) made up of multiple images that are consecutive in time series.
[0018] The position detection unit 102 performs a process of detecting one or more people and one or more objects contained in the image acquired by the input unit 101, and their positions within the image, using known object detection technology and AI with a trained model as described below.
[0019] The tracking processing unit 103 executes a process of tracking people and objects in the image detected by the position detection unit 102. The tracking processing unit 103 further calculates a first feature amount corresponding to the moving distance of the object between two image frames adjacent to each other in time series and a second feature amount corresponding to the number of consecutive frames during which the object in the image is tracked, and executes a process of calculating a "stationary duration" which is an index value regarding the time during which each object being tracked in the image remains stationary based on the first and second feature amounts. The stationary duration can be calculated based on the frame rate of the image and the number of consecutive frames. The tracking processing unit 103 further executes a process of assigning identification information to each person and object to be detected, and acquiring the number of each image frame in which each person or object appears and the coordinate position of each person or object in each image frame.
[0020] The determination unit 104 executes a process of determining whether or not to exclude each object in the image tracked by the tracking processing unit 103 from the target of tracking processing by the tracking processing unit 103, based on the "stillness duration" of each object calculated by the tracking processing unit 103. If the stillness duration is equal to or longer than a predetermined time, it is highly likely that the object is not moving and is placed in that position for storage or the like. Since such an object is unlikely to pose a danger to people nearby when moving, it is preferable to exclude it from the target of tracking processing. This is because if tracking is continued for an object whose stillness duration is equal to or longer than a predetermined time, a false detection may occur in which an interaction between a person and an object that is low in danger is recognized in the interaction recognition unit 106 described later.
[0021] The tracking target update unit 105 executes a process of excluding an object excluded by the determination unit 104 from the tracking targets of the tracking processing unit 103 among the objects in the image tracked by the tracking processing unit 103. The tracking target update unit 105 specifies the position of the object in the image at the time of executing the process of excluding the object from the tracking targets of the tracking processing unit 103. The tracking target update unit 105 also sets the object as a tracking target of the tracking processing unit 103 again after a predetermined time (an exclusion process period to be described later) has elapsed after executing the process of excluding the object from the tracking targets of the tracking processing unit 103.
[0022] The interaction recognition unit 106 executes a process of recognizing a mutual relationship between a person and an object in an image being tracked by the tracking processing unit 103. A specific example of an interaction between a person and an object in an image is, for example, a positional relationship between the person and the object in an image, and the interaction recognition unit 106 recognizes that a mutual relationship between the person and the object in an image has occurred when the distance between the person and the object in an image is equal to or less than a reference distance.
[0023] The output control unit 107 executes a process of outputting message information related to the interaction between the person and the object in the image when the interaction recognition unit 106 recognizes that an interaction between a person and an object in the image has occurred. The message information related to the interaction between the person and the object in the image is, for example, text data and / or image data such as "Dangerous Work" indicating that a person has approached a dangerous object.
[0024] The memory unit 108 stores multiple thresholds, such as a movement distance threshold range determined by predetermined upper and lower limits for the movement distance (first feature) of an object per frame between two image frames that are adjacent to each other in a chronological order, a threshold for the number of consecutive frames (second feature) which is the number of frames that continue to be tracked, a threshold for a stationary duration calculated based on the first and second features and used by the determination unit 104 to determine whether or not to exclude each object in the image from the target of tracking processing by the tracking processing unit 103, and a reference distance used by the interaction recognition unit 106 to recognize the interrelationship between people and objects in the image.
[0025] The camera unit 20 in this embodiment is an imaging device that captures images including still images and videos. One or more camera units 20 are installed so as to capture an image of the entire area, such as a work site, where the relationship between people and objects should be monitored. Image data captured by the camera unit 20 is transmitted to the input unit 101 of the image analysis system 10 by wireless communication or wired communication via a communication interface, as described above.
[0026] The monitoring system 30 in this embodiment has a recording unit 301, a video display unit 302, and a management control unit 303, as shown in FIG. 2. The monitoring system 30 is provided in a monitoring center. The image data captured by the camera unit 20 and the message information output from the output control unit 107 of the image analysis system 10 are transmitted to the monitoring system 30 via the image analysis system 10, and are recorded in the recording unit 301 of the monitoring system 30 and displayed on the video display unit 302. A supervisor at the monitoring center watches the image displayed on the video display unit 302 of the monitoring system 30 to monitor people and objects in the image, and notifies the site supervisor or worker at the work site when the message information is displayed on the video display unit 302. The management control unit 303 can determine some or all of various thresholds such as the threshold for the number of consecutive frames and the threshold range for the moving distance based on information (history information) previously acquired, and set them in the storage unit 108 of the image analysis system 10.
[0027] FIG. 3 is a block diagram showing the hardware configuration of the image analysis system 10 in this embodiment.
[0028] As shown in FIG. 3, the image analysis system 10 in this embodiment has a hardware configuration that is a general computer, and includes at least a processor 11, a main memory 12, a storage memory 13, a communication device 14, an input device 15, and a display device 16, which are interconnected via a bus 17.
[0029] The processor 11 is a computing device that controls the operation of the entire image analysis system 10, controls the transmission and reception of data between each element, and performs information processing required for application execution and authentication processing. For example, the processor 11 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs stored in the storage memory 13 and deployed in the main memory 12 to perform various information processing. The processor 11 executes the programs stored in the storage memory 13 to function as the input unit 101, position detection unit 102, tracking processing unit 103, determination unit 104, tracking target update unit 105, interaction recognition unit 106, and output control unit 107 of the image analysis system 10 described with reference to FIG. 2.
[0030] The main memory 12 includes volatile storage devices such as RAM (Random Access Memory) and DRAM (Dynamic Random Access Memory) and ROM (Read Only Memory). The main memory 12 is used as a work area for the processor 11, and also stores a BIOS (Basic Input / Output System) that is executed when the image analysis system 10 is started up, various setting information, and the like.
[0031] The storage memory 13 includes a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs such as application programs and programs for implementing functions according to the embodiments of the present invention. The storage memory 13 also at least temporarily stores information acquired by the communication device 14 and the input device 15, and various data (including intermediate data) generated by the processing operations of the processor 11. A database for storing such data may be constructed in the storage memory 13.
[0032] One or both of the main memory 12 and the storage memory 13 constitute the memory unit 108 of the image analysis system 10 shown in FIG.
[0033] The communication device 14 is a communication interface that enables the image analysis system 10 to communicate with the camera unit 20, the monitoring system 30, etc. via a communication network such as the Internet, an intranet, a LAN (Local Area Network), or a WAN (Wide Area Network), or a wired connection such as a USB (Universal Serial Bus).
[0034] The input device 15 is an input interface that accepts input by a user of the image analysis system 10 to operate the image analysis system 10, and can be composed of, for example, a keyboard, a pointing device such as a mouse or joystick, a touch panel, an audio microphone, or any combination of these.
[0035] The display device 16 is a display device that displays images input from the camera unit 20 and images generated by the processor 11 to provide visual information to a user of the image analysis system 10.
[0036] A bus 17 is commonly connected to the above elements, and transmits, for example, address signals, data signals and various control signals.
[0037] Next, an image analysis method by the image analysis system 10 according to this embodiment will be described with reference to Fig. 4 etc. Fig. 4 is a flowchart showing a process for carrying out the image analysis method by the image analysis system according to this embodiment.
[0038] In the image analysis system 10 according to this embodiment, first, the processor 11 functioning as the input unit 101 of the image analysis system 10 performs a process of accepting input of an image captured by the camera unit 20 and transmitted from the camera unit 20 (step S11).
[0039] As described above, as an example, the camera unit 20 is installed at a work site and captures images including people and objects present at the work site. Data of the images captured by the camera unit 20 is input from the camera unit 20 to the input unit 101 of the image analysis system 10.
[0040] Next, the processor 11 functioning as the position detection unit 102 of the image analysis system 10 detects people and objects included in the image input from the camera unit 20 to the input unit 101 (step S12).
[0041] The process of detecting people and objects can be performed using any known object detection technology. Object detection is a technology that identifies specific classes (categories such as "people" and "electric poles") that exist in an image by image processing, and also identifies the positions and areas of the objects to be detected in the image. Known object detection technologies include, for example, a method using Haar-Like features, a method using scale-invariant feature transform (SIFT), and a method using HOG features. In addition, a method may be used in which a learner detects people and objects using a trained model obtained by performing machine learning or deep learning using a convolutional neural network (CNN) using a large number of images including people and objects.
[0042] Furthermore, in step S12, the processor 11 functioning as the tracking processing unit 103 performs processing to continuously track the position and area of each person and object detected by the position detection unit 102 in the image. For example, multi-object tracking (MOT), which is a technology for tracking multiple specific objects in an image, can be used to track the person and object. According to the MOT technology, identification information (ID) is given to each of multiple specific objects detected in an image, and bounding boxes indicating the positions and areas of the objects are continuously generated or predicted, thereby tracking each object. A bounding box is a rectangular area circumscribing a detected object (person or object). The position of an object can be predicted based on the detected position of the object and its transition, for example, by a method using a Kalman filter. In this way, the tracking processing unit 103 tracks the position of an object based on the position of the detected object and the past transition of the object's position over time.
[0043] Fig. 5 shows a state in which detection and tracking processing is performed on a person and an object in an image by the tracking processing unit 103. In Fig. 5, with reference to reference numerals 401 and 402, a person and a utility pole as an object are detected and tracked, respectively, and bounding boxes are generated for them.
[0044] When the processor 11 starts the process of tracking the position and area of each person and object in the image, the processor 11 also performs a process of adding and recording tracking information and the like related to each object to be tracked in a tracked object table. Fig. 6 shows an example of a tracked object table that records tracking information and the like related to each object. The tracked object table is stored in the storage unit 108 of the image analysis system 10.
[0045] When the processor 11 starts the process of tracking the position and area of each object in the image, it assigns identification information (object ID in the tracking target list in FIG. 3) to each object to be tracked, and records in the "appearance frame number (coordinate position)" of the tracking target table a pair of the frame number of the image frame in which the object appears and information on the object's position in each image frame in association with the identification information of each object. The pair of the frame number and the information on the object's position is sequentially stored in the tracking target list while the object appears in the image.
[0046] The tracking target table further records an "exclusion flag" indicating whether the object is excluded from the tracking target in a process described later, and "final rectangular coordinates" which are coordinate information indicating the position and size of the bounding box of the object in the image at the time when the object is excluded from the tracking target. The final rectangular coordinates are expressed as "(x1, y1)-(x2, y2)" as a set of the coordinates (x1, y1) of the upper left corner and the coordinates (x2, y2) of the lower right corner of the bounding box of the object in the image at the time when the object is excluded from the tracking target. The tracking target table may further record the feature amount (first feature amount, second feature amount, and stationary duration) calculated for each object. Alternatively, the feature amount calculated for each object may be recorded and managed independently in a table separate from the tracking target list.
[0047] Returning to FIG. 4, the processor 11 functioning as the determination unit 104 of the image analysis system 10 next performs a process of determining whether or not one or more people and objects are included in the image in which the detection and tracking processes have been performed in the above step S12 (step S13). If it is determined that one or more people and objects are included in the image, the process proceeds to step S14, and if it is determined that one or more people and objects are not included in the image, the process returns to step S11. This is because if one or more people and objects are not included in the image, for example, if only a person is detected or if no person is detected and only one or more objects are detected, there is no interrelationship between the person and the object that the image analysis system 10 should determine. The processor 11 repeats the processes of steps S11 to S13 until it is determined that one or more people and objects are included in the image. In the example shown in FIG. 5, two people and many objects are included in the image.
[0048] Next, the processor 11 functioning as the tracking processing unit 103 calculates, for each object to be determined whose position is being tracked, a first feature value corresponding to the distance the object has moved between two image frames that are chronologically adjacent to each other, and a second feature value corresponding to the number of consecutive frames during which the object is being tracked, and performs a process of calculating a "stationary duration" relating to the time that each object being tracked in the image remains stationary based on the first and second feature values (steps S14, S15).
[0049] More specifically, the "still duration" relating to the time during which each object remains still is calculated by the number of consecutive frames (second feature), which is the number of frames that have been tracked in which the object's movement distance per frame between two chronologically adjacent image frames (first feature) remains within a movement distance threshold range determined by a predetermined upper and lower limit. The frame rate of the image is taken into consideration when calculating the still duration. The reason why the standard for the object's movement distance between two adjacent image frames calculated as the first feature is set to be within the movement distance threshold range determined by a predetermined upper and lower limit is that, for example, an object that moves beyond the upper limit in the image is likely to be caused by an image detection error, and also, in the case where the installed camera unit 20 vibrates at a distance below the lower limit due to the movement of heavy machinery at the work site or wind, and the object in the image appears to have moved slightly due to such vibration, the object is prevented from being counted as having moved.
[0050] The moving distance (first feature amount) of an object per frame between two image frames adjacent to each other in time series can be calculated by, for example, the processor 11 determining the center of gravity of a rectangular bounding box surrounding the object, and determining the moving distance of the center of gravity by assuming the center of gravity as the position of the object. Alternatively, the moving distance can be calculated by the processor 11 associating the outline of an object detected in an image or pixels contained in a rectangular bounding box surrounding the outline between frames, and determining the average moving distance for each pixel between frames. This makes it possible to recognize the rotation of an object as a movement.
[0051] Next, the processor 11 functioning as the determination unit 104 determines whether the "stillness duration" of each object calculated by the tracking processing unit 103 in steps S14 and S15 has reached a threshold value for stillness duration stored in the memory unit 108 of the image analysis system 10 (step S16), and removes objects whose "stillness duration" has reached the threshold value from the objects to be determined and advances the processing to step S17; and returns the processing to step S14 for objects whose "stillness duration" has not reached the threshold value, without removing them from the objects to be determined, as the objects whose movement has been confirmed (step S17).
[0052] When the stillness duration reaches the predetermined threshold time, it is highly likely that the object is not moving and is placed at that location for storage or the like. Since such an object is unlikely to be dangerous to a person nearby when moving, it is preferable to exclude it from the object for recognizing the mutual relationship with the person. This is because a false detection may occur in the interaction recognition unit 106 described later, in which an object with a low risk is recognized as having a mutual relationship with the person. On the other hand, an object whose stillness duration does not reach the predetermined threshold time may be a moving object and may be a dangerous object that may suddenly collide with a person, so it is not excluded from the judgment target but remains as a target for recognizing the mutual relationship with the person. The threshold value for the stillness duration is stored in the storage unit 108 of the image analysis system 10 as described above. For the object excluded from the judgment target, the tracking process by the tracking processing unit 103 is terminated.
[0053] Next, the processor 11 functioning as the tracked target update unit 105 performs a process of excluding the object whose stationary duration has been determined to have reached the threshold in the process of step S17 from the object for which determination regarding stationary duration is to be made (step S18).
[0054] In this embodiment, the process of excluding an object from the determination target is performed by changing the exclusion flag associated with the identification information of the object in the tracking target table shown in Fig. 6. If the exclusion flag is "0", the object is subjected to tracking processing as a determination target for "stationary duration", and if the exclusion flag is "1", the object is excluded from the determination target and is not subjected to tracking processing.
[0055] When it is determined in the process of step S17 that the stillness duration has reached the threshold, the tracking target update unit 105 records the final rectangular coordinates indicating the excluded object position information, which is the position of the excluded object in the image at that time, in the tracking target table, and updates the exclusion flag for the excluded object to "1". Then, the object is excluded from the determination target for "stillness duration", and the tracking processing unit 103 does not perform tracking processing for the object thereafter. Therefore, after the object is excluded from the determination target, the above-mentioned first and second feature amounts and stillness duration for the object are not updated. However, an object that was determined to be a determination target because its stillness duration did not reach a predetermined threshold in the past is not excluded from the determination target even if its stillness duration reaches the threshold thereafter. In other words, an object that has been detected to have moved even once in the past is not excluded from the determination target.
[0056] When an object in an image moves, the shape and size of the bounding box surrounding the object may change as the posture and orientation of the object changes. For example, in the case where a utility pole is hung from a crane at a work site and moved, when the utility pole hung from the crane rotates, the bounding box becomes a horizontally long rectangle when viewed perpendicular to the longitudinal direction, but becomes a square surrounding a circular cross section when viewed along the longitudinal direction. The final rectangular coordinates indicating the excluded object position information of the object when the stationary duration reaches the threshold value are the rectangular coordinates at the time when the object whose position and orientation have changed is excluded from the determination target.
[0057] 5 again, in the example shown in Fig. 5, the utility pole of the object indicated by reference numeral 401 is suspended by a crane and moved, and is maintained as an object whose stationary duration has not reached the threshold value. On the other hand, the other utility poles indicated by reference numerals 402 and 403 have been placed for storage, and are excluded from the objects whose stationary duration has reached the threshold value.
[0058] As described above, an object whose stillness duration reaches the threshold (an object that has not moved for a predetermined time) is subsequently excluded from the judgment target, and tracking processing is not required. Therefore, in order to recognize that an object detected in an image is an object that has been excluded from the judgment target, the processor 11 functioning as the tracked target update unit 105 performs processing to exclude an object from the judgment target. If an Intersection over Union (IoU) indicating the degree of overlap between a rectangle (bounding box indicating a true area) indicated by the excluded object position information of an object whose stillness duration reaches the threshold and a rectangle (bounding box indicating an area to be judged as an overlapping area) indicated by the excluded object position information of another object detected in the image is equal to or greater than a predetermined threshold, the other object is also judged to have been detected in the excluded object position information of the object whose stillness duration reaches the threshold, and is excluded from the judgment target. IoU is an index indicating the ratio of overlap of image areas, and the larger the IoU, the more the image areas overlap. More specifically, IoU takes a maximum value of 1 when the judgment target area completely overlaps the true area, and takes a minimum value of 1 when there is no overlapping area at all.
[0059] Fig. 7 shows an example of a state in which rectangles overlap each other. In the example shown in Fig. 7, a rectangle OBJ1 indicated by excluded object position information of an object whose stationary duration time has reached a threshold, and a rectangle OBJ2 of an object detected in an image are shown. The rectangles OBJ2 and OBJ1 overlap each other in an overlapping area OL. The processor 11 functioning as the tracked object update unit 105 calculates the IoU of the rectangle OBJ2 with respect to the rectangle OBJ1, and if the IoU is equal to or greater than a predetermined threshold, determines that the object detected from the image is an object excluded from the determination target, and excludes it from the determination target.
[0060] The tracked object update unit 105 may further be configured to not exclude an object that is a judgment target because its stillness duration does not reach a predetermined threshold value, even if its stillness duration reaches the threshold value thereafter. For example, in a case where a utility pole placed at a work site is temporarily moved to change its storage location, the utility pole continues to be still before and after the move, but moves temporarily during the move. Since an object that has been confirmed to have moved even once (an object that has been determined to have a stillness duration that did not reach a predetermined threshold value) is an object that may move thereafter, it is preferable to continue to be a judgment target even if it remains still for the stillness duration thereafter. In this example, an object with an exclusion flag of "0" in the table of FIG. 3 is an object that has been confirmed to have moved even once (an object that has been determined to have a stillness duration that did not reach a predetermined threshold value), and therefore is not excluded from the judgment target even if its stillness duration reaches the threshold value thereafter.
[0061] The tracking target update unit 105 may be further configured to cancel the exclusion of an object that has been excluded from the determination target and start calculating the stationary duration at each predetermined exclusion process cycle. The predetermined exclusion process cycle is not particularly limited, but is set to a time longer than the threshold value of the stationary duration.
[0062] An object that has been excluded from the judgment target because a stationary duration has elapsed during a certain time interval may move thereafter. For example, it is assumed that a utility pole that has been stored at a work site for a long time is hung from a crane and loaded onto a truck bed in order to be removed from the work site. The utility pole that has been stored at the work site for a long time is excluded from the judgment target after a predetermined stationary duration has elapsed, but if an object that has once been excluded from the judgment target is never considered as a judgment target thereafter, it may be impossible to recognize the relationship between the object and a person when the object is moved thereafter as described above, and as a result, it may be impossible to detect dangerous work.
[0063] In response to this, by canceling the exclusion of an object that has been excluded from the determination target at each predetermined exclusion processing cycle and starting calculation of the stationary duration time as described above, the object that has been excluded from the determination target can be included again as a determination target. If an object that remains stationary for a long period of time is continued to be included as a determination target, the load of the tracking processing and determination processing by the processor 11 increases, and the possibility of false detection of the correlation with a person increases. However, by canceling the exclusion of an object that has been excluded from the determination target at each predetermined exclusion processing cycle, these can be reduced.
[0064] Next, the processor 11 functioning as the interaction recognition unit 106 performs processing to recognize the interrelationship (interaction) between the person and object being tracked by the tracked target update unit 105 (step S19).
[0065] The mutual relationship between a person and an object is recognized as occurring when the distance between the person and the object in the image is closer than a reference distance, for example. The threshold value for this reference distance is stored in the storage unit 108 of the image analysis system 10 as described above. The object to be recognized by the interaction recognition unit 106 is an object that is recognized to have moved within the stationary duration time, and is therefore not excluded from the judgment target in step S17 and is continuously tracked, and may potentially pose a danger to the person. Therefore, if a person approaches such an object, there is a possibility that the object may cause danger, and therefore the interaction recognition process of step S19 is performed by the interaction recognition unit 106 to recognize that the person has approached such an object.
[0066] The distance between the person and the object in the image is calculated based on the distance between the bounding boxes of the person and the object generated by the tracking target update unit 105, for example. More specifically, when the center positions of the bounding boxes of the person and the object are closer than a reference distance, the interaction recognition unit 106 recognizes that a relationship has occurred between the person and the object. In the example shown in FIG. 5, as shown by reference numeral 401, the center of gravity of the bounding box indicating the utility pole, which is the object to be judged, and the center of gravity of the bounding box indicating the person nearby are closer than a reference distance, and it is recognized that a relationship has occurred between the person and the object. In this way, the image analysis system 10 judges the situation of the person (e.g., dangerous work, etc.) with respect to the object detected and tracked as the object to be judged in the image.
[0067] Next, the processor 11 functioning as the output control unit 107 executes a process of outputting message information related to the interaction between the person and the object in the image when the interaction recognition unit 106 recognizes that an interaction between the person and the object in the image has occurred (step S20).
[0068] The message information related to the interrelationship between the person and the object in the image is, for example, text data and / or image data such as "Dangerous Work" indicating that the person has approached a dangerous object. The timing at which the interrelationship between the person and the object in the image is recognized is the timing at which a dangerous work state has occurred, and it is preferable to distinguish this timing from other normal states. The processor 11 functioning as the output control unit 107 is configured to output the message information when it recognizes that the interrelationship between the person and the object in the image has occurred, thereby displaying the timing at which the interrelationship between the person and the object in the image has been recognized in a manner that is distinguishable from other timings. The processor 11 as the output control unit 107 also outputs the image input from the camera unit 20 to the image analysis system 10.
[0069] The image and the message information output from the output control unit 107 of the image analysis system 10 are transmitted to the monitoring system 30 of the monitoring center, stored in the recording unit 301 of the monitoring system 30, and displayed on the video display unit 302 consisting of an image display device or the like. A monitor at the monitoring center monitors the situation of people and objects at the work site in the image displayed on the video display unit 302 of the monitoring system 30, and when message information indicating that, for example, a worker at the work site is in a dangerous state with respect to an object is issued and displayed on the video display unit 302 as a result of the determination by the interaction recognition unit 106 of the image analysis system 10, the monitor can notify the worker or supervisor at the work site of the situation. The monitor can notify the worker or supervisor at the work site by, for example, instructing the target worker to evacuate from the place through an in-house speaker installed at the work site, or by calling the mobile phone of the worker or supervisor to warn them.
[0070] Fig. 8 shows an example of a screen displayed on the video display unit 302 of the monitoring system 30. The video display unit 302 shows a utility pole, which is an object to be determined and is being moved while suspended from a crane, and a bounding box 501 showing the utility pole, and a person near the utility pole and a bounding box 502 showing the person. In the example shown in Fig. 8, the bounding boxes 501 and 502 are closer than the reference distance, so the interrelationship between the person and object they show is recognized, and the text "Dangerous work" and an image accompanying it are displayed as message information. When such message information is displayed on the video display unit 302, a monitor at the monitoring center notifies the worker and supervisor at the work site as described above.
[0071] The video display unit 302 can display images transmitted in real time from the camera unit 20, and can also display an image selected for playback from among images stored in the recording unit 301 of the surveillance system 30. In the display example shown in Fig. 8, an image selected for playback from among images stored in the recording unit 301 of the surveillance system 30 is displayed, and playback of the selected image begins by clicking a playback button 503 displayed on the screen. The video display unit 302 may further display a timeline display bar 504 that displays a playback timeline of the image being played.
[0072] The video display unit 302 may also display a button 505 for switching on / off the display of bounding boxes showing people and objects. The display of bounding boxes can be switched on / off by clicking this button 505. However, even if the display of bounding boxes is turned off, the generation of bounding boxes continues for people and objects being tracked. The video display unit 302 further displays, as a set threshold value, a reference distance between a person and an object that is used by the interaction recognition unit 106 to recognize a mutual relationship between the two.
[0073] The storage unit 108 of the image analysis system 10 stores the image captured by the camera unit 20, message information output at the timing when the interrelationship between a person and an object in the image is recognized, and each threshold value. Each threshold value includes a moving distance threshold range in which the moving distance (first feature amount) of an object per frame between two chronologically adjacent image frames is determined by a predetermined upper and lower limit, and a threshold value of the number of consecutive frames (second feature amount) which is the number of frames that are tracked and continue to be within the moving distance threshold range. Meanwhile, the recording unit 301 of the surveillance system 30 can store the occurrence of the interrelationship between a person and an object in the image as well as the timing when the interrelationship between a person and an object in the image is recognized. The information stored in the storage unit 108 is accumulated as statistical information or history information.
[0074] In the monitoring system 30, for example, the management control unit 303 determines the consecutive frame number threshold and / or the movement distance threshold range based on past statistical information. As an example, when the image analysis process is executed by the image analysis system 10 from the next time onwards or at another work site, the management control unit 303 selects the value of each threshold used when an appropriate judgment was made based on the past statistical information data stored in the storage unit 108.
[0075] In the above-described embodiment, the image analysis processing is performed on people and objects (particularly utility poles) at a work site as an example. However, the application of the image analysis system 10 of this embodiment is not limited to this, and it can be applied to people and any object present in any other area.
[0076] As described above, according to one aspect of the present disclosure, there is provided an image analysis system for determining the situation of a person from time-series images, the image analysis system having a memory for storing a software program and a processor for executing the software program, in which the processor detects and tracks people and objects from images, and determines the situation of the person based on the relationship between the person and the object being determined, excluding objects whose stationary duration, which is an index value related to the time that the object remains stationary, has reached a predetermined threshold.
[0077] According to the image analysis system of the present disclosure configured in this manner, by excluding objects that have reached the stationary duration from the objects to be judged, it is possible to limit the objects to be judged for their interrelationship with a person, and to eliminate so-called false positives in which an interrelationship with an object that does not need to be judged is judged. As a result, it becomes possible to appropriately judge the situation of a person with respect to an object detected and tracked as a judgment target in an image.
[0078] The image analysis system of the present disclosure may further be configured such that the processor tracks the position of the object to be judged, records excluded object position information representing the position of the object when it is excluded from the judgment target, and thereafter does not track the object excluded from the judgment target, and does not treat the object detected at the position represented by the excluded object position information as a judgment target. This makes it possible to reduce the calculation processing burden on the processor since an object once excluded from the judgment target can be excluded from the judgment target without the processor performing tracking processing thereafter.
[0079] The image analysis system of the present disclosure may further be configured such that the excluded object position information is information indicating a rectangular area circumscribing the object, and the processor forms a rectangle circumscribing the object detected from the image, and if the intersection over union of the rectangle with the rectangle indicated by the excluded object position information is equal to or greater than a predetermined threshold, it is determined that the object detected from the image has been detected at a position indicated by the excluded object position information. This makes it possible to determine that an object for which a rectangle overlapping with the rectangle indicated by the excluded object position information has been generated has been detected at a position indicated by the excluded object position information, and to exclude the object from the determination target.
[0080] The image analysis system of the present disclosure may further be configured to not exclude an object that has been determined as a determination target because its stillness duration does not reach a predetermined threshold value even if its stillness duration subsequently reaches the threshold value. An object that has been determined as a determination target because it has moved even once during its stillness duration may move again even after some time has passed. With this configuration, an object that has moved even once can continue to be determined as a determination target thereafter.
[0081] The image analysis system of the present disclosure may further be configured such that the processor cancels the exclusion of an object that has been excluded from the determination target at each predetermined exclusion processing cycle and starts calculating the stationary duration. Even if an object is excluded from the determination target because it has been stationary for a certain period of time, it may move thereafter. By canceling the exclusion of an object at each predetermined exclusion processing cycle, it becomes possible to re-determine whether or not an object that has once been excluded from the determination target needs to be excluded from the determination target.
[0082] The image analysis system of the present disclosure may further be configured such that the processor tracks an object based on the detected object's position and past transitions in the object's position over time, and an object whose number of consecutive frames, which is the number of frames that it continues to track, is equal to or greater than a predetermined consecutive frame number threshold, and whose movement distance, which is the distance moved per frame, is within a movement distance threshold range determined by predetermined upper and lower limits, is not excluded from the list of objects to be determined as an object whose movement has been confirmed.
[0083] The image analysis system of the present disclosure may further be configured such that the processor generates a rectangle surrounding the object when detecting the object from the image, and calculates the moving distance by using the center of gravity of the rectangle as the position of the object. Alternatively, the processor may be configured to associate pixels included in the object detected from the image between frames, and to use the average of the movement amount for each pixel between frames as the moving distance. In this way, the image analysis system of the present disclosure can calculate the moving distance of an object in an image using various techniques.
[0084] The image analysis system of the present disclosure may further be configured such that the processor displays the timing at which an object whose movement has been confirmed is identified in a manner that makes it possible to distinguish the timing from other timings, thereby making it possible to, for example, regard the timing at which an object has been confirmed to have started moving as the timing at which a dangerous operation has started and reflect this in the display.
[0085] The image analysis system of the present disclosure may further be configured such that the processor determines the consecutive frame number threshold and / or the movement distance threshold range based on past statistical information, so that, for example, when the image analysis system is used to monitor the work site again or another work site, it is possible to perform object detection and tracking processing with appropriate detection accuracy using each threshold determined based on the past statistical information.
[0086] The above-described embodiments of the present invention are merely illustrative examples of the present invention, and are not intended to limit the scope of the present invention to these embodiments. Those skilled in the art can implement the present invention in various other forms without departing from the scope of the present invention. [Explanation of symbols]
[0087] 10...image analysis system, 11...processor, 12...main memory, 13...storage memory, 14...communication device, 15...input device, 16...display device, 20...camera unit, 30...surveillance system, 101...input unit, 102...position detection unit, 103...tracking processing unit, 104...determination unit, 105...tracking target update unit, 106...interaction recognition unit, 107...output control unit, 108...memory unit, 301...recording unit, 302...video display unit, 303...management control unit
Claims
1. An image analysis system for determining a situation in which a person is placed from a time series of images, comprising: a memory for storing a software program; a processor for executing said software program; having The processor, Detecting and tracking people and objects from the images; The remaining objects are subjected to judgment after excluding the objects whose stationary duration, which is an index value related to the time the object continues to be stationary, has reached a predetermined threshold value; determining a state of the person based on a correlation between the person and the object to be determined; Image analysis system.
2. The processor, Tracking the position of the object to be determined; Recording excluded object position information indicating the position of the object when the object was excluded from the determination target; Thereafter, the object that has been excluded from the determination target is not tracked, and the object detected at the position represented by the excluded object position information is not included in the determination target. The image analysis system according to claim 1 .
3. the excluded object position information is information indicating a rectangular area circumscribing the object, The processor forms a rectangle circumscribing the object detected from the image, and if an intersection over union of the rectangle with the rectangle indicated by the excluded object position information is equal to or greater than a predetermined threshold, determines that the object detected from the image is detected at a position indicated by the excluded object position information. The image analysis system according to claim 2 .
4. The object that has been determined as the object to be determined because the stationary duration does not reach a predetermined threshold is not excluded from the object to be determined even if the stationary duration subsequently reaches the threshold. The image analysis system according to claim 1 .
5. The processor, canceling the exclusion of the object that has been excluded from the determination target at each predetermined exclusion process cycle, and starting the calculation of the stationary duration. The image analysis system according to claim 1 .
6. The processor, Tracking the object based on the detected object's position and past transitions of the object's position over time; If the number of consecutive frames, which is the number of frames that have been continuously tracked, is equal to or greater than a predetermined consecutive frame number threshold, and the moving distance, which is the distance moved per frame, is within a moving distance threshold range determined by a predetermined upper and lower limits, the object is not excluded from the determination target as an object whose movement has been confirmed. The image analysis system according to claim 1 .
7. The processor, generating a rectangle surrounding an object when the object is detected from the image; The moving distance is calculated by taking the center of gravity of the rectangle as the position of the object. The image analysis system according to claim 6.
8. The processor, Pixels included in the object detected from the image are associated with each other between frames, and an average value of the amount of movement between frames for each pixel is set as the movement distance. The image analysis system according to claim 6.
9. The processor displays a timing when the object whose movement has been confirmed is identified in a manner that can be distinguished from other timings. The image analysis system according to claim 5 .
10. The image analysis system according to claim 6 , wherein the processor determines the consecutive frame number threshold and / or the movement distance threshold range based on past statistical information.
11. An image analysis method for determining a situation in which a person is placed from time-series images, comprising: Detecting and tracking people and objects from the images; The remaining objects are subjected to judgment after excluding the objects whose stationary duration, which is an index value related to the time the object continues to be stationary, has reached a predetermined threshold value; determining a situation of the person based on a correlation between the person and the object to be determined, Image analysis methods.
12. An image analysis program for determining a situation in which a person is placed from time-series images, comprising: Detecting and tracking people and objects from the images; The remaining objects are subjected to judgment after excluding the objects whose stationary duration, which is an index value related to the time the object continues to be stationary, has reached a predetermined threshold value; determining a situation of the person based on a correlation between the person and the object to be determined; Image analysis program.
Citation Information
Patent Citations
Object tracking method and object tracking program and object tracking system
JP2019003334A
