System, method and program

Abstract

To determine the proximity of people and objects at low cost. [Solution] A person and an object are detected in an image, the distance between the person and the object in the image is calculated, the depth difference between the person and the object in the image is calculated, and based on the distance and depth difference, it is determined whether the person and the object are approaching each other.

Description

[Technical Field]

[0001] The present invention relates to a system, a method, and a program. [Background technology]

[0002] Construction sites are full of dangers, and safety management is an important theme for construction companies. To prevent danger, it is important to inform the person in charge or the worker of the extent to which unsafe behavior, which does not result in an accident but is not considered safe, has occurred, and to prevent it from happening in the first place. The most common technology for detecting unsafe behavior involves attaching cameras to heavy machinery and issuing warnings based on the images captured by the cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-170041 Summary of the Invention [Problem to be solved by the invention]

[0004] However, attaching a camera to each piece of heavy machinery requires various costs, such as money and labor, and is therefore not realistic, particularly at work sites where there are a large number of pieces of heavy machinery and they are frequently replaced. Furthermore, there are many blind spots when looking from heavy machinery, making it difficult to properly detect unsafe behavior. Even if the number of cameras attached to heavy machinery were increased, this problem would not be completely solved, and the cost issue mentioned above would become even greater. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a system for analyzing video captured by a camera, which detects a person and an object in the video, calculates a distance between the person and the object in the video, calculates a depth difference between the person and the object in the video, and determines whether the person and the object are close to each other based on the distance and the depth difference. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the server device. [Figure 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the client device. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of a camera. [Figure 5] FIG. 5 is a flowchart illustrating an example of information processing executed by the server device. [Figure 6] FIG. 6 is a diagram showing an example of a video. [Figure 7] FIG. 7 is a diagram showing an example of a process for excluding a heavy machine driver from a video. [Figure 8] FIG. 8 is a diagram illustrating an example of a depth map of an image. [Figure 9] FIG. 9 is a diagram illustrating an example of a generated record. [Figure 10] FIG. 10 is a diagram showing an example of a thumbnail screen displayed on the output unit of the client device. [Figure 11] FIG. 11 is a diagram showing an example of a viewer screen 1010 displayed on the output unit of the client device. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various features shown in the following embodiments (including modified examples, the same applies hereinafter) can be combined with each other.

[0008] <Embodiment 1> 1. System configuration diagram Fig. 1 is a diagram showing an example of the system configuration of an information processing system 1000. As shown in Fig. 1, the information processing system 1000 includes, as a system configuration, a server device 100, a client device 110, and a camera 120. The server device 100, the client device 110, and the camera 120 are communicatively connected via a network 150. The network 150 includes any one of a wide area network (WAN), a local area network (LAN), and the Internet, or any combination thereof. The network 150 includes both a wired network and a wireless network.

[0009] The information processing system 1000 is a system that analyzes video captured by a camera 120. The information processing system 1000 is an example of a cloud camera system.

[0010] The server device 100 is a device that provides cloud functions. For example, the server device 100 is a device that analyzes video captured by a camera 120 and displays the data on a client device 110 or the like. The server device 100 executes the main processing of the first embodiment, which will be described later.

[0011] Camera 120 is a camera of a cloud camera system installed for the purpose of monitoring and / or recording at a construction site. Camera 120 is fixed and installed in a location that captures the entire construction site, for example. Note that cameras 120 do not include those attached to heavy machinery (construction machinery). In other words, cameras 120 are installed in a fixed state relative to the ground surface. More specific installation locations include poles, protective walls, scaffolding, building pillars, walls, ceilings, etc. at construction sites. Mobile cameras are also included if they are installed in a similar manner using tripods, various attachments, etc. By specifying that cameras 120 are not cameras attached to heavy machinery, it is possible to limit the analysis of images from cameras that capture a wider area from a bird's-eye view, such as fixedly installed surveillance cameras. This makes it possible to capture approaching situations over a wider area that are difficult to grasp from the heavy machinery itself, thereby expanding the field of view for safety monitoring.

[0012] A construction site 160 is a location where civil engineering or construction work is actually carried out. A construction site can also be described as a place where workers (people) 180 and heavy equipment (objects) 170 work together to create structures or infrastructure. A construction site can also be described as a workplace where people, machines, and materials work together to carry out production activities under safety management. Heavy equipment is a machine that can move under its own power and performs tasks such as excavation, transportation, leveling, and lifting in civil engineering or construction work. Examples of heavy equipment include hydraulic excavators, bulldozers, wheel loaders, crane trucks (mobile cranes), dump trucks, and road rollers.

[0013] 1 shows only one camera 120 connected to the network 150 for simplicity, but multiple cameras 120 may be connected to the network 150. The information processing system 1000 can also be said to be a system that monitors or manages the safety of people at a construction site.

[0014] The client device 110 is a device used to check the video captured by the camera 120 and stored in the server device 100. The client device 110 is an example of an information terminal device. The client device 110 may be a PC, a tablet terminal, or a smartphone. The client device 110 may be any device that allows the video captured by the camera 120 to be checked. For simplicity, FIG. 1 shows only one client device 110 connected to the network 150, but multiple client devices 110 may be connected to the network 150.

[0015] Here, the claimed information processing system may be composed of multiple devices or may be composed of a single device. When the claimed information processing system is composed of a single device, an example of that device is server device 100. When the claimed information processing system is composed of multiple devices, an example of the multiple devices is server device 100 and client device 110, or a cloud server composed of multiple server devices that provide the functions of server device 100.

[0016] 2. Hardware Configuration (1) Hardware Configuration of Server Device 100 FIG. 2 is a diagram illustrating an example of a hardware configuration of the server device 100. As shown in FIG. 2, the server device 100 includes, as its hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0017] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire server device 100 and executes processing based on input information and the like.

[0018] The storage unit 220 is any one of a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), a solid state drive (SSD), etc., or any combination thereof, and stores programs and data used when the control unit 210 executes processing based on the programs. The storage unit 220 is an example of a storage medium. Examples of data used when the control unit 210 executes processing based on the programs include video data (video) sent from the camera 120, various thresholds described below, records, etc.

[0019] In the present specification, the data used by control unit 210 when executing processing based on a program is described as being stored in storage unit 220, but the data may also be stored in a storage unit of another device that can communicate with server device 100. The data may be stored in a storage unit of any device as long as it can be referenced and / or acquired by control unit 210. When control unit 210 executes processing based on a program stored in storage unit 220, the functions of server device 100 and the processing of the flowchart shown in Fig. 5, which will be described later, are realized.

[0020] The communication unit 230 connects the server device 100 to the network 150 and controls communication with other devices.

[0021] The hardware configuration of each of the control unit 210, the storage unit 220, and the communication unit 230 is not limited to one. For example, multiple control units may be included in the server device 100. The same applies to the client devices and cameras described below.

[0022] (2) Hardware Configuration of the Client Device 110 FIG. 3 is a diagram illustrating an example of the hardware configuration of the client device 110. As shown in FIG. 3, client device 110 includes, as its hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, a communication unit 350, and an internal bus 360. Control unit 310, storage unit 320, input unit 330, output unit 340, and communication unit 350 are electrically connected via internal bus 360.

[0023] The control unit 310 is a CPU or the like, and controls the entire client device 110, and executes processing based on input information and the like.

[0024] The storage unit 320 is any one of an HDD, a ROM, a RAM, an SSD, etc., or any combination thereof, and stores programs, data used when the control unit 310 executes processing based on the programs, etc.

[0025] In the present specification, the data used by control unit 310 when executing processing based on a program is described as being stored in storage unit 320, but the data may also be stored in a storage unit of another device that can communicate with client device 110. The data may be stored in a storage unit of any device as long as it can be referenced and / or acquired by control unit 310. The functions of client device 110 are realized when control unit 310 executes processing based on a program stored in storage unit 320.

[0026] The input unit 330 is a device that inputs information to the client device 110 in response to an operation by an operator. The input unit 330 accepts an operation input made by the operator. The operation input is transferred as a command signal to the control unit 310 via the internal bus 360. The control unit 310 may execute predetermined control and / or calculation based on the transferred command signal as necessary. The input unit 330 may be included in the housing of the client device 110 or may be externally attached. For example, the input unit 330 may be implemented as a touch panel display integrated with the output unit 340. When the input unit 330 is implemented as a touch panel display, the operator can input tap operations, swipe operations, etc. to the input unit 330. Instead of a touch panel display, a switch button, a mouse, a trackpad, a keyboard, etc. may be used as the input unit 330.

[0027] The output unit 340 is, for example, a display unit typified by a display, and is a device that outputs (displays) information as a screen of a graphical user interface (GUI) that can be operated by an operator. The output unit 340 may be included in the housing of the client device 110 or may be externally attached. More specifically, the output unit 340 may be implemented as a display device such as a liquid crystal display, an organic electro-luminescence (EL) display, or a plasma display. It is preferable that these display devices are implemented by selectively using them depending on the type of client device 110.

[0028] The communication unit 350 connects the client device 110 to the network 150 and controls communication with other devices.

[0029] In the present specification, a personal computer (computer) is used as an example of the client device 110. However, the client device may also be a notebook PC, a smartphone, a tablet computer, etc. The client device may be any device that can display a screen as described below and accept operations from an operator via the screen, etc.

[0030] (3) Hardware configuration of the camera 120 FIG. 4 is a diagram showing an example of the hardware configuration of the camera 120. As shown in FIG. 4, camera 120 includes, as its hardware configuration, a control unit 410, a storage unit 420, an image capturing unit 430, a communication unit 440, and an internal bus 450. Control unit 410, storage unit 420, image capturing unit 430, and communication unit 440 are electrically connected via internal bus 450.

[0031] The control unit 410 is a CPU or the like, and controls the entire camera 120, and executes processing based on input information and the like.

[0032] The storage unit 420 is any one of an HDD, a ROM, a RAM, an SSD, etc., or any combination thereof, and stores programs, data used when the control unit 410 executes processing based on the programs, etc. The storage unit 420 is an example of a storage medium.

[0033] In the specification, the data used by control unit 410 when executing processing based on a program is described as being stored in storage unit 420, but the data may also be stored in a storage unit of another device that can communicate with camera 120. The data may be stored in a storage unit of any device as long as it can be referenced and / or acquired by control unit 410. The functions of camera 120 are realized by control unit 410 executing processing based on the program stored in storage unit 420.

[0034] The image capturing unit 430 is a camera that captures an image of a subject included in the image capturing range. The image capturing unit 430 is equipped with a stereo camera, a ToF (Time of Flight) sensor, a LiDAR (Light Detection and Ranging), or the like, and thereby acquires depth information indicating the distance to the subject along with the image. Examples of subjects include people and heavy machinery at a construction site. For the sake of simplicity, the following description will be given assuming that the camera 120 includes a stereo camera unless otherwise specified.

[0035] The communication unit 440 connects the camera 120 to the network 150 and controls communication with other devices.

[0036] 3. Information Processing The information processing of the first embodiment will be described below.

[0037] (1) Overview of the process The information processing system 1000 is a system that analyzes video captured by the camera 120. The control unit 210 detects a person and an object in the video, calculates the distance between the person and the object, calculates the difference (depth difference) between the depth of the person and the depth of the object in the video, and executes processing to determine whether the person and the object are approaching each other based on the distance and difference.

[0038] By performing this type of processing, the approach of a person and an object detected from the video can be determined using not only the distance between them but also the difference in depth, making it possible to determine the approach with higher accuracy and fewer false positives. In particular, by using information in the depth direction (depth), it is possible to appropriately evaluate the risk of approach in three dimensions, rather than simply two-dimensional proximity. This can contribute to preventing contact and accidents between people and heavy machinery.

[0039] (2) Details of the processing FIG. 5 is a flowchart showing an example of information processing executed by the server device 100. In step S510, the control unit 210 executes a process to detect people and heavy machinery (objects) from the video transmitted from the camera 120. An example of an object is the main body or part of the heavy machinery. The main body of the heavy machinery refers to the basic structural parts of the machine, excluding attachments used to perform work. A part of the heavy machinery refers to an attachment of the heavy machinery or the individual parts that make up the attachment of the heavy machinery. An attachment of the heavy machinery refers to an interchangeable tool that is attached to the main body of the heavy machinery to perform a specific task. Examples of attachments for hydraulic excavators include buckets, breakers, grapples, crushers, magnets, etc. Examples of attachments for bulldozers include rippers, winches, rakes, etc. Examples of attachments for wheel loaders include forks, grapples, sweepers / brushes, snow plows / snow blowers, etc. Examples of attachments for crane trucks include jibs, clamshell / orange peel buckets, etc. Examples of attachments for dump trucks include sprayers, sprinkler tanks, etc. Examples of attachments for road rollers include shell kits, asphalt cutters, etc. For the sake of simplicity, the following description will be given using heavy machinery as an example of an object unless otherwise specified. Identifying an object as the main body or part of heavy machinery increases the effectiveness of the information processing system 1000 in the extremely important field of safety management, namely, preventing personal injury accidents in specific work environments involving heavy machinery, such as construction sites and factories.

[0040] The control unit 210 uses an object detection model based on deep learning to detect people and heavy machinery from video. The object detection model is a trained model that uses video (still images or video) as input data and the type of detected object (people or heavy machinery) and a rectangular area (bounding box) indicating where the object is located in the video as output data. Note that a bounding box refers to the smallest rectangular area that surrounds a specific object in an image.

[0041] As another example, the control unit 210 may detect people and heavy machinery through image processing. The control unit 210 may detect people from video using, for example, a Histogram of Oriented Gradients (HOG) feature. Furthermore, the control unit 210 may find a candidate area for heavy machinery from the video based on a predetermined color, slide an image of a template for heavy machinery prepared in advance in the storage unit 220 or the like over the candidate area in the video, and search for a location that most resembles heavy machinery (a rectangular area for heavy machinery). Alternatively, the control unit 210 may extract the outline of an object using edge detection (a process for finding color boundaries), identify heavy machinery based on predetermined rules such as a "square area of ​​a specific size" and / or a "long arm-like shape," and extract the rectangular area for the heavy machinery.

[0042] In step S520, the control unit 210 executes a process of tracking the detected person and heavy machinery (object). For example, the control unit 210 treats each frame of the video as an independent image, and in the processing of step S510, it surrounds the positions of people and heavy machinery present in the frame with bounding boxes and assigns class labels such as "person," "excavator," etc. In step S520, the control unit 210 assigns a unique ID (identification) to each of the people and heavy machinery detected in step S510, and performs tracking processing to follow them across frames.

[0043] More specifically, the control unit 210 predicts the likely position of the object (person or heavy machinery) that was tracked in the previous frame in the current frame. This prediction is made using a technique called a Kalman filter. The Kalman filter is a technique that mathematically predicts the next position of an object from its past velocity and acceleration. The control unit 210 compares the predicted object position with the position of the actually detected object to determine which is the same object. The degree of overlap of the bounding boxes is used as a criterion for this determination. The control unit 210 determines that the predicted object's bounding box and the detected object's bounding box are the same object if the overlap between them is equal to or greater than a threshold. The control unit 210 may obtain feature amounts of the shape of the object in addition to the position information, and determine whether or not the object is the same based on the position information and feature amounts.

[0044] If the association of an object between frames is successful, the control unit 210 updates the Kalman filter with the object's latest position information and continues to assign the same ID. For an object that does not match any prediction, the control unit 210 determines that the object has newly appeared on the screen, issues a new ID, and begins tracking. For an ID that has not matched any object over a predetermined number of frames, the control unit 210 determines that the object corresponding to the ID has disappeared from the screen, and ends tracking.

[0045] In step S530, the control unit 210 determines whether the heavy equipment (object) is moving by calculating the speed of the detected and tracked heavy equipment. More specifically, the control unit 210 calculates the time using the following formula: Elapsed time (seconds) = number of frames / frame rate (fps) The control unit 210 obtains the three-dimensional coordinates (x, y, z) of the heavy equipment in the real world for each frame. Then, based on the three-dimensional coordinates (x1, y1, z1) and (x2, y2, z2) at two points in time, the control unit 210 calculates the movement distance in three-dimensional space using the following formula: Distance (m)=√{(x2−x1) 2+(y2-y1) 2 +(z2-z1) 2} The control unit 210 calculates the speed using the following formula based on the calculated movement distance and the calculated elapsed time. Speed ​​(m / s) = Distance traveled (m) / Elapsed time (s) If necessary, the control unit 210 converts the speed per second (m / s) to speed per hour (km / h). Control unit 210 determines whether the calculated speed is equal to or greater than a threshold. If control unit 210 determines that the calculated speed is equal to or greater than the threshold, it determines that the object is moving and proceeds to step S540. If control unit 210 determines that the calculated speed is not equal to or greater than the threshold, it ends the processing of the flowchart shown in FIG.

[0046] The threshold here is an example of the third threshold. The process of step S530 is an example of a process that executes a process for determining whether a person and heavy equipment are approaching each other based on the difference in distance and depth when the speed of the heavy equipment is equal to or greater than a third threshold. By executing the process for determining whether a person and heavy equipment are approaching each other based on the difference in distance and depth only when the speed of the object is equal to or greater than a specific threshold, it is possible to narrow down the approach determination to particularly high-risk situations (e.g., heavy equipment moving at high speed). This allows resources to be concentrated on detecting situations that are likely to lead to serious accidents, excluding approaches when the object is stationary or moving at low speed, thereby improving the responsiveness of the information processing system 1000 and the importance of detection.

[0047] In step S540, the control unit 210 determines whether or not the heavy machinery (object) is operating based on the pixel difference. The control unit 210 extracts only the bounding box area of ​​the heavy machinery being tracked from the current frame (t) and the previous frame (t-1). This extracted small area is called a ROI (Region of Interest). To improve the accuracy of the comparison, the control unit 210 performs preprocessing such as grayscale conversion and smoothing (blur processing) on ​​the ROI image.

[0048] The control unit 210 subtracts the brightness values ​​of pixels at the same positions in the two pre-processed ROI images. Difference pixel value = |Pixel value of ROI(t) - Pixel value of ROI(t-1)| In areas where there is no change, the subtraction result will be close to 0, and the area will appear black in the difference image. In areas where there is a change, the subtraction result will be large, and the area will appear white in the difference image.

[0049] The control unit 210 sets a binarization threshold. The control unit 210 converts pixels in the difference image that have changed more than the threshold into "white (changed)" and pixels that have changed less than the threshold into "black (no change)," thereby making only the changed parts clear. This threshold can be said to be a threshold for "deeming slight changes to be noise." The control unit 210 counts the total number of white pixels in the binarized image. If the number of white pixels is equal to or greater than a preset motion determination threshold (for example, 1% of the entire ROI), the control unit 210 determines that there is motion, and if the number of white pixels is less than the preset motion determination threshold, the control unit 210 determines that there is no motion. If control unit 210 determines that there is motion, it advances the process to step S550, and if it determines that the calculated speed is not equal to or greater than the threshold, it ends the process of the flowchart shown in FIG.

[0050] The motion determination threshold is an example of a fourth threshold. The processing of step S540 is an example of a process that executes a process to determine whether a person and heavy equipment are approaching each other based on the difference in distance and depth when the difference in the time series of pixels corresponding to an object becomes greater than or equal to a fourth threshold. By executing the process of determining whether a person and heavy machinery are approaching each other only when the difference in the time series of pixels corresponding to the object (degree of movement) is equal to or exceeds a specific threshold, it is possible to limit approach determination to situations in which the object is moving (changing). This allows priority to be given to detecting situations in which there is a large visual change, i.e., the object is active or making sudden movements, further reducing false positives and making it more difficult to overlook potentially dangerous changes.

[0051] In step S550, control unit 210 obtains the distance between the detected (extracted) person and the heavy machinery (object) (the distance between two points in the two-dimensional image), and determines whether the distance between the person and the object is short. FIG. 6 is a diagram showing an example of a video 600. As shown in FIG. Rectangular region 610 is the bounding box of the person extracted from video 600. Rectangular region 620 is the bounding box of the heavy machinery extracted from video 600. Note that, when the control unit 210 extracts the bounding boxes of the person and the heavy equipment, it may execute an exclusion process to determine whether the bounding box of the person belongs to the driver inside the heavy equipment.

[0052] FIG. 7 is a diagram showing an example of a process for excluding a heavy machine driver from a video 700. The control unit 210 calculates the area of ​​the overlapping region between the bounding box of the person (rectangular region 720) and the bounding box of the heavy equipment (rectangular region 710), and divides this area by the area of ​​the bounding box of the person to calculate the overlap rate. If this overlap rate is equal to or greater than a predetermined threshold (e.g., 80%), the bounding box of the person is considered to represent the driver and is excluded from subsequent distance calculations.

[0053] This process is an example of a process in which, when a person is enclosed by an object, the process of determining whether a person and heavy machinery are approaching each other is not executed; that is, when a person is not enclosed by an object, the process of determining whether a person and an object are approaching each other is executed based on the difference in distance and depth. When a person is enclosed by heavy machinery (for example, the person is riding on heavy machinery), the process of determining whether a person and heavy machinery are approaching each other based on the difference in distance and depth is not executed, thereby eliminating erroneous warnings and recordings in situations that are not intended to be warning targets (operations within a safe range). This enables appropriate operation consistent with the purpose of approach judgment, improving convenience for users of the information processing system 1000 and the reliability of warnings.

[0054] The control unit 210 calculates the center coordinates (x, y) of the bounding boxes of the person and the heavy machinery, respectively, based on the following formulas. Center x = upper left x coordinate of bounding box + (bounding box width / 2) Center y = y coordinate of the top left corner of the bounding box + (height of the bounding box / 2)

[0055] The control unit 210 determines the center of the person (x1, y1) and the center of the heavy equipment (x2, y2), and then uses Pythagoras' theorem to determine the straight-line distance 630 between the two points. The distance obtained by this calculation is in pixels. This distance simply indicates how far apart they are on the screen. If the straight-line distance 630 between the two points is less than or equal to a threshold, the control unit 210 determines that there is a bounding box (Bbox) of a person near the heavy equipment, and if the straight-line distance 630 between the two points is not less than or equal to the threshold, it determines that there is no bounding box of a person near the heavy equipment. The threshold here is an example of a first threshold. If the control unit 210 determines that there is a bounding box of a person near the heavy equipment, it determines that the distance between the person and the object is close and proceeds to step S560, and if it determines that there is no bounding box of a person near the heavy equipment, it determines that the distance between the person and the object is not close and ends the processing of the flowchart shown in Figure 5.

[0056] As described above, the camera 120 is a stereo camera. Therefore, in step S560, the control unit 210 acquires from the camera 120 a depth map of an image included in or associated with the image. As another example, the control unit 210 may generate a depth map from an image using a trained model. This trained model is a trained model that has learned a large amount of images and depth information of the images (information about how far each pixel is apart). In other words, this trained model is a trained model that has learned using images as input data and depth information of the images as output data. The control unit 210 may generate a depth map of the image based on the depth information of the image. As another example, a depth map may be generated using millimeter wave radar, LIDAR, infrared light, or the like.

[0057] FIG. 8 is a diagram illustrating an example of a depth map 800 of an image. Rectangular region 810 is the bounding box of the person in the depth map that corresponds to rectangular region 610. Rectangular region 820 is the bounding box of the heavy equipment in the depth map that corresponds to rectangular region 620.

[0058] The control unit 210 acquires a representative depth value from each of the detected bounding boxes. For example, the control unit 210 acquires the depth value of the pixel at the center of the bounding box. As another example, the control unit 210 may calculate the average depth value of all pixels in the bounding box and acquire the average value as the depth value.

[0059] The control unit 210 calculates the difference (absolute value) between the depth of the person and the depth of the heavy equipment. The distance obtained by this calculation is in real-world units such as meters. This allows the control unit 210 to obtain practical information that is directly related to safety management, such as that the person and the heavy equipment are approximately 5 m apart in the depth direction.

[0060] In step S570, control unit 210 determines whether the difference (depth difference) between the depth of the person and the depth of the heavy equipment is equal to or less than a threshold. If control unit 210 determines that the difference between the depth of the person and the depth of the heavy equipment is equal to or less than the threshold, control unit 210 proceeds to step S580, and if it determines that the difference between the depth of the person and the depth of the heavy equipment is not equal to or less than the threshold, control unit 210 ends the processing of the flowchart shown in FIG. The threshold here is an example of the second threshold.

[0061] The processing of steps S550 and S570 is an example of processing for determining whether a person and heavy equipment are approaching each other based on the difference in distance and depth when the distance between the person and the heavy equipment is equal to or less than a first threshold and the difference in depth between the person and the heavy equipment is equal to or less than a second threshold. By performing the processing for determining whether a person and heavy equipment are approaching each other only when the condition that the distance and the difference in depth are each equal to or less than a specific threshold is satisfied, it is possible to suppress the execution of unnecessary processing and reduce the processing load of the information processing system 1000. Furthermore, by limiting warnings and recording to truly dangerous situations (sufficient proximity), it is possible to improve the accuracy and reliability of approach determination and prevent distraction due to false alarms.

[0062] In step S580, the control unit 210 determines that the person and the heavy equipment are in close proximity to each other. The processing of step S580 can also be said to be processing in which the control unit 210 issues an event indicating that the person and the heavy equipment are in close proximity to each other. The process of step S580 is an example of a process for determining an action.

[0063] In step S590, the control unit 210 executes output corresponding to the event. For example, the control unit 210 generates a record relating to an event that a person and a heavy machine are in close proximity to each other, and stores the record in a predetermined storage area such as the storage unit 220. The record format is, for example, CSV format.

[0064] 9 is a diagram showing an example of a record 900 generated as an example of the output of step S590. The record 900 includes, as items, a record ID 910, a start time 911, an end time 912, a real start time 913, a real end time 914, a start timestamp URL 915, an end timestamp URL 916, a class label 917, a class ID 918, a detection type 919, and an approach level 920.

[0065] The record ID 910 is identification information that identifies a record. The start time 911 is the time when detection starts in the video. The start time 911 can also be said to be the start time of an event in the video where a person and heavy machinery are in close proximity to each other. The end time 912 is the time when detection ends in the video. The end time 912 can also be said to be the time when the event in the video ends, in which a person and heavy machinery are in close proximity to each other.

[0066] The actual start time 913 is the time of detection start in standard time. The actual start time 913 can also be said to be the time of start of an event in which a person and heavy machinery are in close proximity to each other in standard time. The actual end time 914 is the time in standard time when detection ends. The actual end time 914 can also be said to be the time in standard time when the event of a person and heavy machinery being in close proximity to each other ends.

[0067] The start timestamp URL 915 is a timestamp corresponding to the time of the start of detection. The end timestamp URL 916 is a timestamp corresponding to the time when detection ends. The timestamp is information for displaying the video at the time of detection, and includes information for identifying the web app, camera, and time.

[0068] The class label 917 is a label for identifying the type of the detected object. The class ID 918 is an ID for identifying the detected object.

[0069] The detection type 919 is a type of detection, which includes a motion detection and a proximity detection. The proximity level 920 is an item included when the detection type 919 is proximity determination, and indicates the degree of proximity between a person and heavy machinery, for example, on a scale of 1 to 5. Level 1 indicates the lowest degree of proximity, and the degree of proximity increases stepwise as the number increases to levels 2, 3, and 4, with level 5 indicating the highest degree of proximity.

[0070] By recording detailed information (time, video information, type of object, degree of approach, etc.) when the processing in step S580 is executed and outputting it as a record, it becomes easier to understand the situation when an accident occurs, investigate the cause, and review safety measures. This provides objective data that is useful for improving work processes and formulating new safety standards.

[0071] 10 is a diagram showing an example of a thumbnail screen 1100 displayed on the output unit 340 of the client device 110 as an example of the output at step S590. In response to a request from the client device 110, the control unit 210 generates the thumbnail screen 1100 as shown in FIG. 10 and transmits it to the client device 110. The control unit 310 of the client device 110 displays the received thumbnail screen 1100 on the output unit 340.

[0072] A thumbnail screen 1100 displays a plurality of thumbnail images relating to video (in the example of FIG. 10, thumbnail image 1110, thumbnail image 1120, thumbnail image 1130, thumbnail image 1140, thumbnail image 1150, and thumbnail image 1160). Each thumbnail image is a thumbnail image of a video corresponding to the time when it was determined that a person and heavy machinery were approaching each other. In this way, by displaying thumbnails of footage corresponding to the time when it was determined that a person and heavy machinery were approaching each other, the content of the footage can be grasped in detail before viewing the main footage, which allows for quicker access to necessary information and reduces the time required to confirm the incident. For simplicity, in FIG. 10, the thumbnail screen 1100 displays only a list of thumbnail images, but information such as the title, camera name, date and time may also be displayed in association with the thumbnail images.

[0073] When one thumbnail image is selected from the plurality of thumbnail images, the control unit 310 transmits to the server device 100 a request to display a viewer screen for the video corresponding to the thumbnail image.

[0074] 11 is a diagram showing an example of a viewer screen 1010 displayed on the output unit 340 of the client device 110 as an example of the output of step S590. In response to a request from the client device 110 to display a viewer screen for a video corresponding to the thumbnail image, the control unit 210 generates the viewer screen 1010 as shown in FIG. 11 and transmits it to the client device 110. The control unit 310 of the client device 110 displays the received viewer screen 1010 on the output unit 340.

[0075] The viewer screen 1010 includes a display area 1020. The display area 1020 displays video captured by the corresponding camera 120 and corresponding to the thumbnail image. The video in the example of FIG. 11 includes people and heavy machinery. The viewer screen 1010 also includes a timeline 1030. The timeline 1030 indicates the time of the video displayed in the display area 1020. More specifically, the timeline displays a time axis in the shape of a strip from the start of the video to the end of the video. In the embodiment, the timeline displays the elapsed time from the start as a scale. However, the timeline may also display the time or date and time as a scale.

[0076] The playback position display GUI component 1050 is a GUI component that indicates the shooting date and time of the video displayed in the display area 1020. When the playback position display GUI component 1050 is at the leftmost position, it is the start position of the video playback, and when it is at the rightmost position, it is the end position of the video playback. The playback position display GUI component 1050 indicates the position on the timeline 1030 of the video being played in the display area 1020. The operator of the client device 110 can also operate the playback position display GUI component 1050 to specify the shooting date and time of the video displayed in the display area 1020. The flag 1040 is a flag that indicates the period from the start time of an event, displayed on the timeline 1030, in which a person included in the video is in close proximity to heavy machinery, to the end time of the event in which the person included in the video is in close proximity to heavy machinery. The time at the leftmost position of the flag 1040 indicates the start time of the event, and the time at the rightmost position of the flag 1040 indicates the end time of the event. In this way, by clearly displaying on the video timeline the time when the process that determined that a person and heavy machinery were approaching each other was executed, it is possible to quickly identify important locations where approach events occurred from long video recordings, which greatly improves the efficiency of video review and reduces the burden on administrators.

[0077] Another example of the output in step S590 may be a warning or notification. For example, at the construction site 160 where the camera 120 is installed, workers 180 and the like may be notified by sounding a warning from a speaker, turning on a patrol lamp, or displaying a message on signage. Also, for example, a remote site supervisor may be notified by sending a notification via email, chat, a pop-up, or a speaker sound from the client device 110.

[0078] As described above, according to the first embodiment, by determining the approach of a person and an object using not only the distance between them detected from the video but also the difference in depth, it is possible to make an approach determination with higher accuracy and fewer false positives. In particular, by using information in the depth direction (depth), it is possible to appropriately evaluate not only the risk of simple two-dimensional approach but also the risk of three-dimensional approach. This can contribute to preventing contact and accidents between people and heavy machinery.

[0079] (Variation) The camera 120 may perform some or all of the processes performed by the server device 100 described above.

[0080] <Additional Notes> It may be provided in the following manner. (Appendix 1) A system for analyzing video captured by a camera, Detecting people and objects in the video; calculating a distance between the person and the object in the video; determining a depth difference between the person and the object in the video; determining whether the person and the object are approaching each other based on the distance and the depth difference; system. (Appendix 2) 2. The system of claim 1, When the distance is equal to or less than a first threshold and the depth difference is equal to or less than a second threshold, it is determined that the person and the object are approaching each other. system. (Appendix 3) 10. The system of claim 1 or 2, The determination is performed when the velocity of the object is equal to or greater than a third threshold. system. (Appendix 4) 10. The system of claim 1, further comprising: performing the determination when a difference between the time series of pixels corresponding to the object is equal to or greater than a fourth threshold value; system. (Appendix 5) 5. The system of any one of claims 1 to 4, performing the determination when the person is not contained within the object; system. (Appendix 6) 6. The system of any one of claims 1 to 5, outputting a record including at least one of the following information: a time when a determination that the person and the object are approaching each other started, information for displaying an image corresponding to the time, a time when a determination that the person and the object are approaching each other ended, a type of the object, and a degree of approach between the person and the object; system. (Appendix 7) 7. The system of any one of claims 1 to 6, comprising: and identifiably displaying the time at which it is determined that the person and the object are approaching each other on the timeline of the video. system. (Appendix 8) 10. The system of any one of Supplementary Notes 1 to 7, comprising: displaying a thumbnail of an image corresponding to the time when it is determined that the person and the object are approaching each other; system. (Appendix 9) 9. The system of any one of Supplementary Notes 1 to 8, comprising: If the determination is made, a warning is issued. system. (Appendix 10) 10. The system of any one of Supplementary Notes 1 to 9, comprising: The object is a body or part of a heavy machine. system. (Appendix 11) 11. The system of any one of claims 1 to 10, further comprising: The camera is installed in a fixed state relative to the ground surface. system. (Appendix 12) A method executed by a system for analyzing video captured by a camera, comprising: Detecting people and objects in the video; determining a distance between the person and the object; determining a difference between the depth of the person and the depth of the object in the video; execute a process of determining whether the person and the object are approaching each other based on the distance and the difference; method. (Appendix 13) A program that analyzes video captured by a camera, On the computer, detecting people and objects in the video; determining a distance between the person and the object; determining a difference between the depth of the person and the depth of the object in the video; executing a process for determining whether the person and the object are approaching each other based on the distance and the difference; A program to execute.

[0081] While various embodiments of the present invention have been described, they are presented as examples and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. The embodiments and modifications thereof are intended to be included within the scope and spirit of the invention, and are also intended to be included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0082] 100: Server device 110: Client device 120: Camera 150: Network 210: Control unit 220: Storage section 230: Communications Department 1000: Information Processing Systems

Claims

1. A system for analyzing video captured by a camera, Detecting people and objects in the video; calculating a distance between the person and the object in the video; determining a depth difference between the person and the object in the video; determining whether the person and the object are approaching each other based on the distance and the depth difference; When the distance is equal to or less than a first threshold and the depth difference is equal to or less than a second threshold, it is determined that the person and the object are approaching each other. system.

2. A system for analyzing video captured by a camera, Detecting people and objects in the video; calculating a distance between the person and the object in the video; determining a depth difference between the person and the object in the video; determining whether the person and the object are approaching each other based on the distance and the depth difference; performing the determination when the person is not contained within the object; system.

3. A system for analyzing video captured by a camera, Detecting people and objects in the video; calculating a distance between the person and the object in the video; determining a depth difference between the person and the object in the video; determining whether the person and the object are approaching each other based on the distance and the depth difference; a time when it is determined that the person and the object are approaching each other is displayed in a distinguishable manner on a timeline of the video; system.

4. 4. The system according to claim 1, further comprising: The determination is made when the velocity of the object is equal to or greater than a third threshold. system.

5. 4. The system according to claim 1, further comprising: performing the determination when a difference in the time series of pixels corresponding to the object is equal to or greater than a fourth threshold value; system.

6. 4. The system according to claim 1, further comprising: outputting a record including at least one of the following information: a time when a determination that the person and the object are approaching each other started, information for displaying an image corresponding to the time, a time when a determination that the person and the object are approaching each other ended, a type of the object, and a degree of approach between the person and the object; system.

7. 4. The system according to claim 1, further comprising: displaying a thumbnail of an image corresponding to the time when it is determined that the person and the object are approaching each other; system.

8. 4. The system according to claim 1, further comprising: If the determination is made, a warning is issued. system.

9. 4. The system according to claim 1, further comprising: The object is a body or part of a heavy machine. system.

10. 4. The system according to claim 1, further comprising: The camera is installed in a fixed state relative to the ground surface. system.

11. A method executed by a system for analyzing video captured by a camera, comprising: Detecting people and objects in the video; calculating a distance between the person and the object in the video; determining a depth difference between the person and the object in the video; determining whether the person and the object are approaching each other based on the distance and the depth difference; When the distance is equal to or less than a first threshold and the depth difference is equal to or less than a second threshold, it is determined that the person and the object are approaching each other. method.

12. A program that analyzes video captured by a camera, On the computer, detecting people and objects in the video; determining a distance between the person and the object in the video; determining a depth difference between the person and the object in the video; determining that the person and the object are approaching each other when the distance is equal to or less than a first threshold and the depth difference is equal to or less than a second threshold; A program to execute.

Images