Dangerous work detection system, and dangerous work detection method

Abstract

To provide a dangerous work detection system for efficiently detecting a dangerous work.SOLUTION: A dangerous work detection system detects a dangerous work for a person working with use of an object. The dangerous work detection system includes an arithmetic unit, and a memory unit for storing a program to be executed by the arithmetic unit. The arithmetic unit determines an orientation of the object in a first frame image of image data. The arithmetic unit selects one dangerous work detection method to determine a dangerous work state of the person in the first frame image corresponding to the orientation of the object from a plurality of dangerous work detection methods.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a dangerous operation detection system. [Background technology]

[0002] As a measure to ensure work safety, work is filmed with cameras, the footage is taken home to determine whether dangerous work was performed, and the footage is used to improve future work. As the number of sites to be filmed increases, the amount of video analysis processing is also increasing.

[0003] A technology related to the present disclosure is disclosed in JP 2020-10154 A. JP 2020-10154 A detects work that is estimated to be dangerous from a video of a worker's work, thereby reducing the user's workload. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-10154 A Summary of the Invention [Problem to be solved by the invention]

[0005] There are various ways in which a worker performs a task using a specific object. In tasks using an object, there may be multiple work states that are considered dangerous. For example, in tasks using a stepladder, there are work states that are considered dangerous, such as standing on the top of the ladder and sitting on the top of the ladder. With conventional recognition technology, there is a concern that many erroneous judgments may occur. Advanced recognition using multiple pieces of information can handle complex cases, but if one tries to perform all tasks without erroneous judgments using a single advanced model, it will require high-load processing every time, which will take a lot of processing time. [Means for solving the problem]

[0006] One aspect of the present invention is a dangerous work detection system for a person performing work using an object, the system including a computing device and a storage device for storing a program executed by the computing device, wherein the computing device determines the orientation of the object in a first frame image of video data, and selects one dangerous work determination method from a plurality of dangerous work determination methods for determining the dangerous work state of the person for the first frame image in accordance with the orientation of the object. Effect of the Invention

[0007] According to one aspect of the present invention, dangerous work can be efficiently detected. Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. Furthermore, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0008] [Figure 1] An example of classification of work states using a stepladder is shown below. [Diagram 2] FIG. 2 is a logical configuration diagram of the dangerous operation detection system. [Diagram 3] 2 shows an example of the configuration of a video analysis device. [Figure 4] 1 shows an example of a skeleton definition model. [Figure 5A] Shown is a stepladder facing sideways. [Figure 5B] Shows a stepladder at an angle. [Figure 5C] Shows a stepladder facing forward. [Figure 6] 1 shows a flowchart of an example of processing by a video analysis device. [Figure 7A] Here is an example of a bounding box for a person and a stepladder with an IoU of 0. [Figure 7B] 13 shows an example of a bounding box of a person and a stepladder with an IoU greater than 0 in a frame image where the IoU is 0 in the immediately preceding frame image. [Figure 7C] Shows tabletop standing position. [Figure 8] 4 shows an example of the configuration of a danger determination unit of the recognition processing unit. [Figure 9] 13 shows a flowchart of an example of processing by a danger determination unit of the recognition processing unit. [Figure 10] 1 shows an example of a GUI screen displayed to a work manager on a user terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. For convenience, if necessary, the description will be divided into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is related to the other as a partial or complete modification, detail, supplementary explanation, etc. Furthermore, in the following, when the number of elements, etc. (including the number, numerical value, amount, range, etc.) is mentioned, it is not limited to the specific number, and may be more than or less than the specific number, unless otherwise specified or when it is clearly limited to a specific number in principle.

[0010] The system of an embodiment of this specification may be a physical computer system (one or more physical computers) or a system built on a group of computing resources (multiple computing resources) such as a cloud infrastructure. The computer system or the group of computing resources may include one or more interface devices (including, for example, a communication device and an input / output device), one or more storage devices (including, for example, a memory (main memory) and an auxiliary storage device), and one or more arithmetic devices.

[0011] When a function is realized by a computing device executing a program including instruction codes, the defined processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be at least a part of the computing device. Processing described with the function as the subject may be processing performed by the computing device or a system having the computing device. The program may be installed from a program source.

[0012] The program source may be, for example, a program distribution computer or a computer-readable storage medium (for example, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0013] A dangerous operation detection system according to an embodiment of the present specification will be described below. The dangerous operation detection system detects a work state that is estimated to be dangerous when a person is performing work using a specific object, and notifies the user. This can increase the safety of work using a specific object.

[0014] There are various types of work that a person performs using a specific object. In the following, an example of detecting a dangerous state in work performed using a stepladder will be described. The features of the embodiment described below can also be applied to danger detection in work using other objects, such as work in the cage of an aerial work vehicle or work on a lift. The embodiment of this specification can be used for safety monitoring at work sites, including electric power companies and construction companies.

[0015] The system according to an embodiment of the present specification analyzes a video of a person working using a stepladder and detects a dangerous work state in the video. The dangerous work state may be defined by a user depending on the object being used, and a state that deviates from the work procedure defined by the user may be determined to be a dangerous work state.

[0016] A system according to an embodiment of the present specification executes a process of detecting objects used, such as a stepladder and a person, in video data. The detected stepladder and person are represented by bounding boxes in a video frame. The bounding box is a rectangle that indicates the position and size of the object (stepladder or person), and is a rectangle that surrounds the object so as to be in contact with the outer shape of the object. A label indicating the type of object, in this case, a stepladder or a person, is assigned to the bounding box. The bounding box indicates the positional relationship between the person and the stepladder.

[0017] In addition to the bounding box of the person, the system detects the person's posture and uses it to detect unsafe work conditions. The posture of the person can be known, for example, from the person's skeleton. By using the person's skeleton information, it becomes possible to more appropriately detect various types of unsafe work conditions.

[0018] The system according to an embodiment of the present specification selects a method for determining a dangerous work state from a plurality of determination methods according to the orientation of a stepladder, which is an object to be used. This makes it possible to detect a dangerous work state efficiently and appropriately. In order to facilitate understanding of this embodiment, several work states using a stepladder will be described.

[0019] Fig. 1 shows an example of classification of work states using a stepladder. A person 191 performs work standing on a stepladder 192. Work using the stepladder 192 is classified into a safe work state 180 and a dangerous work state 170. Fig. 1 shows one example of a safe work state and three examples of a dangerous work state.

[0020] Specifically, an example of a safe working state 180 is one-sided standing 181. A person 191 has both feet on one side of a stepladder 192. Examples of dangerous working states 170 are a top-standing position 171, a top-sitting position 172, and a top-straddling position 173. The top-standing position 171 is a state in which a person 191 has his / her feet on the top of the stepladder 192. The top-sitting position 172 is a state in which a person 191 sits on the top. The top-straddling position 173 is a state in which a person 191 is standing with his / her feet on both sides of the stepladder 192.

[0021] An example of a dangerous operation detection system according to the present specification will be described below. The dangerous operation detection system in this example receives frame images from video data (moving images) captured by a single surveillance camera, and judges whether the images satisfy predetermined conditions for dangerous operation detection.

[0022] The dangerous work detection system determines the orientation of a stepladder, which is an object used in the work, and selects a dangerous work determination method from among a plurality of dangerous work determination methods based on the determination result for determining dangerous work from the image. The dangerous work detection system performs dangerous work determination processing for the image using the selected dangerous work determination method. Different dangerous work determination methods use different dangerous work determination models. The dangerous work determination model may be either a machine learning model or a rule-based model.

[0023] In the example described below, the selectable dangerous work determination models are composed of two types of dangerous work determination models corresponding to different orientations of the stepladder, specifically, a side dangerous work determination model and a non-side dangerous work determination model.

[0024] 2 is a logical configuration diagram of the dangerous operation detection system of this embodiment. The dangerous operation detection system includes a user terminal 20, a video analysis device 30, and a photographing device 40. The video analysis device 30 is connected to the user terminal 20 and the photographing device 40 via a network.

[0025] The image capturing device 40 captures images of the work site and transmits the image data to the image analyzing device 30. At the work site, one or more people can perform work, and each of the one or more people can perform work using a stepladder.

[0026] The user terminal 20 displays the video analyzed by the video analysis device 30 to the work supervisor. The user terminal 20 includes a video display unit 201 and a setting unit 203. The user terminal 20 may have a computer configuration similar to the hardware configuration of the video analysis device 30 described below. In other words, the user terminal 20 may include an input device, a display device, a calculation device, a storage device, and a communication interface. The input device accepts input to the dangerous work detection system by a user. The display device outputs the calculation results by the dangerous work detection system. In addition to the display device, other output devices may be implemented.

[0027] The setting unit 203 transmits the threshold value input by the user to the video analysis device 30. The threshold value is used in the dangerous work judgment process, which will be described in detail later. The video display unit 201 displays the risk judgment result on the display device of the user terminal 20. An example of the displayed screen will be described later. The video display unit 201 may, for example, display a frame image that has been judged to be a dangerous work state on the display device.

[0028] The video analysis device 30 includes a video input unit 301, a rectangle overlay determination unit 321, a storage unit 322, an orientation determination unit 323, and a recognition processing unit 324. The rectangle overlay determination unit 321 includes a person detection unit 302, a ladder detection unit 303, and a danger determination unit 304. The person detection unit 302, the ladder detection unit 303, and the danger determination unit 304 can each use a machine learning model or a rule-based model. The machine learning model can use, for example, a neural network.

[0029] The memory unit 322 stores ladder state information 332, video data 331, and threshold information 333. The video data 331 is, for example, a recorded video stored by the video input unit 301. The threshold information 333 is sent from the setting unit 203 of the user terminal 20 and stored in the memory unit 322. The orientation determination unit 323 includes an orientation estimation unit 305 and a model determination unit 306. The recognition processing unit 324 includes a skeleton detection unit 307 and a danger determination unit 308.

[0030] The rectangle overlap determination unit 321 executes a determination process based on the overlap between the bounding box of the person and the bounding box of the stepladder. The person detection unit 302 detects people in the frame images and assigns the bounding box and the person label.

[0031] The stepladder detection unit 303 detects a stepladder in a frame image and assigns its bounding box and a stepladder label. The danger determination unit 304 includes information on the stepladder's bounding box in a frame image in which the person and the stepladder are in a predetermined positional relationship, together with the frame image, in the stepladder state information 332. The danger determination unit 304 determines whether the work is in a dangerous state based on the positional relationship between the person's bounding box and the stepladder's bounding box. Here, the person's skeletal information is not referenced.

[0032] The orientation determination unit 323 determines the orientation of the stepladder and selects a model to be used for risk determination. The orientation estimation unit 305 estimates the orientation of the stepladder in the frame image. More specifically, the orientation estimation unit 305 obtains information on the frame image and the bounding box of the stepladder from the stepladder state information 332, and estimates the orientation of the stepladder. The orientation estimation unit 305 can estimate the orientation of the stepladder by using a machine learning model or a rule base. The machine learning model can be, for example, a neural network.

[0033] The model determination unit 306 determines a model for determining a dangerous work state according to the orientation of the stepladder. In one embodiment of the present specification, two models are prepared, one of which is a model when the orientation of the stepladder is sideways, and the other is a model when the orientation of the stepladder is not sideways. The orientation of the stepladder is based on the orientation in the frame image, that is, the direction in which the image capture device 40 is facing.

[0034] The recognition processing unit 324 judges a dangerous work state using the judgment model specified by the model determination unit 306. The skeleton detection unit 307 detects the skeleton of a person in a frame image. The skeleton detection unit 307 can estimate the orientation of the stepladder by using a machine learning model or a rule base. The machine learning model can be, for example, a neural network.

[0035] The risk determination unit 308 includes a plurality of models, and determines a risky work state using a designated determination model among them. The model can perform risk determination using a machine learning model or a rule base. The machine learning model can use, for example, a neural network.

[0036] The danger determination unit 308 determines whether the work is in a dangerous state based on the positional relationship between the bounding box of the person and the bounding box of the stepladder, and the posture information obtained from the skeleton information of the person. Note that the position information indicated by the skeleton information may be used instead of the bounding box.

[0037] 3 shows an example of the configuration of the video analysis device 30. The video analysis device 30 includes an input device 310, an output device 314, a calculation device 311, a memory 312, and an auxiliary storage device 313. The user terminal 20 can have a hardware configuration similar to the hardware configuration shown in FIG.

[0038] The input device 310 includes a communication interface with a surveillance camera and accepts image input. The input device 310 also includes a human interface such as a mouse and a keyboard. It accepts input to the dangerous operation detection system by a user. The output device 314 includes a display or printer that outputs the results of calculations by the video analysis device 30, or a communication interface with another device such as a server.

[0039] The auxiliary storage device 313 stores various programs for implementing the analysis processing by the video analysis device 30, and the results of the processing, etc. The auxiliary storage device 313 stores video data 331 and threshold information 333.

[0040] Various programs and data stored in the auxiliary storage device 313 are deployed in the memory 312. In the example of Fig. 3, the memory 312 stores processing programs for the video input unit 301, the rectangle superimposition determination unit 321, the orientation determination unit 323, and the recognition processing unit 324. The memory 312 further stores ladder state information 332. The calculation device 311 executes the programs loaded in the memory 312 and operates as a corresponding functional unit, and is composed of, for example, one or more CPUs, GPUs, etc. The processing and calculations described below are executed by the calculation device 311.

[0041] As described above, the recognition processing unit 324 detects the skeleton of a person and uses that information to determine whether a task is in a dangerous state. Figure 4 shows an example of a skeleton definition model. The rectangles in the figure represent keypoints, and the lines connecting the keypoints represent skeletons. The skeleton detection unit 307 receives an image as input, and outputs the detection results of each keypoint as coordinates within the image.

[0042] The skeleton definition model 100 defines keypoints for a head 101 , a neck 102 , a right shoulder 103 , a left shoulder 104 , a buttocks 105 , a right knee 106 , a left knee 107 , a right ankle 108 , a left ankle 109 , a right elbow 110 , a right wrist 111 , a left elbow 112 , and a left wrist 113 .

[0043] The skeletal definition model 100 further defines a skeleton 151 between the head and neck, a skeleton 152 between the neck and right shoulder, a skeleton 153 between the neck and left shoulder, a skeleton 154 between the neck and buttocks, a skeleton 155 between the buttocks and the right knee, a skeleton 156 between the buttocks and the left knee, a skeleton 157 between the right knee and right ankle, a skeleton 158 between the left knee and left ankle, a skeleton 159 between the right shoulder and right elbow, a skeleton 160 between the right elbow and right wrist, a skeleton 161 between the left shoulder and left elbow, and a skeleton 162 between the left elbow and left wrist.

[0044] The skeleton definition model 100 shown in Fig. 4 is an example of a skeleton definition model 100 that can be used. The skeleton definition model 100 may further include other keypoints, such as eyes, nose, mouth, ears, etc., or may omit some of the keypoints shown in Fig. 4.

[0045] Next, the processing of the video analysis device 30 will be described. As described with reference to Fig. 1, a plurality of dangerous work states can be defined in the work states on a plurality of stepladders. Different postures of the person 191 may look similar depending on the orientation of the stepladder with respect to the image capture device.

[0046] 5A to 5C show examples of the orientation of a stepladder as viewed from the camera. FIG. 5A shows a stepladder facing sideways, FIG. 5B shows a stepladder facing diagonally, and FIG. 5C shows a stepladder facing frontally. A stepladder has a structure in which, for example, two ladders are joined together in a V-shape and connected by a top plate. The ladder includes two vertical bars and a number of steps (horizontal bars) connecting them. The side of the stepladder is the surface seen in the direction in which the steps connect the two vertical bars.

[0047] When the ladder is facing sideways as shown in Fig. 5A, the worker's legs are less likely to be blocked during dangerous work. Therefore, the video analysis device 30 determines whether the work situation in one frame image is dangerous or not from the positional relationship between the person and the ladder in the frame image and information on the person's posture. This can reduce processing time.

[0048] When the stepladder is not oriented to the side as illustrated in FIG. 5B or 5C, the worker's legs are likely to be blocked during dangerous work. Therefore, the side posture during dangerous work may be similar to the non-side posture during safe work. The video analysis device 30 judges whether the work state in one frame image is dangerous or not based on information obtained from a frame sequence consisting of multiple frame images. By judging the dangerous work state based on the change in the posture of the worker (person), the dangerous work state can be more appropriately detected even when the stepladder is not oriented to the side.

[0049] In this way, by switching the process for determining whether a work is dangerous depending on the orientation of the stepladder during work, it is possible to more efficiently and appropriately detect dangerous work.

[0050] 6 shows a flowchart of an example of processing by video analysis device 30. Video analysis device 30 stores video data 331 received from image capture device 40 in auxiliary storage device 313. Video analysis device 30 determines whether each frame image of video data 331 shows a dangerous working state by a worker.

[0051] First, the danger determination unit 304 of the rectangle overlap determination unit 321 determines whether a person and a stepladder are detected in the current image frame (S11). More specifically, the person detection unit 302 executes a person detection process in the frame image, and further, the stepladder detection unit 303 executes a stepladder detection process in the frame image. The danger determination unit 304 acquires these detection results and determines whether a person and a stepladder are detected in the frame image. Note that multiple people and multiple stepladders may be detected in the frame image. The following description may be performed for each combination of people and stepladders.

[0052] If either one has not been detected (S11: NO), the danger determination unit 304 resets the ladder state information 332 (S12), and further determines that no dangerous work is occurring in the frame image (S13).

[0053] When both a person and a stepladder are detected (S11: YES), the danger determination unit 304 determines the overlap (IoU: Intersection over Union) between the bounding box of the person and the bounding box of the stepladder (S14).

[0054] In this example, if IoU is 0, that is, if the two bounding boxes are separated and do not overlap (S14: NO), the danger determination unit 304 resets the ladder state information 332 (S12) and further determines that there is no dangerous work in the frame image (S13). This enables efficient processing. Figure 7A shows an example of the bounding boxes 251, 252 of the person and the ladder, where IoU is 0. In this way, if the person and the ladder are separated, dangerous work cannot occur.

[0055] If IoU is greater than 0, that is, if there is overlap between the two bounding boxes (S14: YES), the danger determination unit 304 determines whether the ladder state information 332 is in a reset state (S15). If the ladder state information 332 is in a reset state (S15: YES), information indicating the bounding box of the detected ladder is stored in the ladder state information 332 together with the frame image (S16).

[0056] FIG. 7B shows an example of bounding boxes 251, 252 of a person and a stepladder with an IoU greater than 0 in a frame image in which the IoU is 0 in the immediately preceding frame image. The stepladder state information 332 can be referred to in a later danger determination process. In this way, by acquiring the stepladder bounding box 252 when the overlap between the person and the stepladder is small and retaining that information, it becomes possible to use a more accurate stepladder bounding box for danger determination. Note that the threshold value of the stepladder state information 332 may be a value greater than 0. The stepladder state information 332 may be omitted, and the stepladder may be detected in the target frame image.

[0057] In step S15, if the ladder state information 332 is not in the reset state (S15: NO), step S16 is skipped.

[0058] Next, the risk determination unit 304 compares the IoU with a predetermined threshold (S17). The threshold is included in the threshold information 333 and is specified in advance by the user. This threshold is greater than the threshold in step S14. If the IoU is equal to or less than the threshold (S17: NO), the risk determination unit 304 determines that there is no dangerous work in the frame image (S13). For example, before the person stands on the stepladder, dangerous work cannot occur.

[0059] If IoU is greater than the threshold (S17: YES), the risk determination unit 304 determines whether the frame image indicates a dangerous work state based on the positional relationship between the bounding boxes of the person and the stepladder (S18). The risk determination unit 304 can use a machine learning model using a neural network or the like or a rule-based model.

[0060] In this way, the risk determination unit 304 determines whether the person is within a predetermined range of the stepladder, and if the determination result indicates that the person is outside the predetermined range, it determines that there is no dangerous work. If the person is within the predetermined range of the stepladder, the risk determination unit 304 executes a dangerous work determination based on the positional relationship between the person and the stepladder. This enables efficient processing.

[0061] The model determines whether the positional relationship between the two input bounding boxes corresponds to a predefined positional relationship. If the positional relationship corresponds to the predefined positional relationship (S18: YES), the danger determination unit 304 determines that the frame image shows a dangerous work state (S19). An example of the predefined positional relationship is the top board standing position 171 shown in FIG. 7C. The top board standing position 171 can be determined by referring only to the positional relationship between the bounding boxes 251, 252 of the person and the stepladder, without referring to the person's skeletal information. This enables efficient dangerous work determination.

[0062] In step S18, if it is determined that the positional relationship between the two input bounding boxes does not correspond to the default positional relationship (S18: NO), the orientation estimation unit 305 estimates the orientation of the stepladder (S20). The orientation estimation unit 305 obtains the frame image and the information of the stepladder's bounding box stored in step S16 from the stepladder state information 332, and estimates the orientation of the stepladder. In this example, the orientation estimation unit 305 determines whether the orientation of the stepladder is sideways.

[0063] The model determination unit 306 selects a model for performing risk judgment based on the estimation result of the orientation estimation unit 305. If the orientation of the stepladder is estimated to be sideways (S20: YES), the model determination unit 306 specifies a model for performing dangerous work judgment on a frame-by-frame basis. The recognition processing unit 324 uses the specified model to determine whether the frame image shows a dangerous work state (S21).

[0064] If the ladder is estimated to be facing other than sideways (S20: NO), the model determination unit 306 designates a model for performing dangerous work judgment using the frame sequence. The recognition processing unit 324 uses the designated model to refer to a frame sequence consisting of multiple frames preceding the frame in question, and judges whether the frame image in question indicates a dangerous work state (S22).

[0065] Next, the processes S21 and S22 performed by the recognition processing unit 324 will be described in detail. Fig. 8 shows an example configuration of the danger determination unit 308 of the recognition processing unit 324. The danger determination unit 308 includes a side dangerous work determination model 401 and a non-side dangerous work determination model 402. The side dangerous work determination model 401 performs dangerous work determination using, for example, the bounding boxes of a person and a stepladder in one frame image and the person's skeletal information as input. Some of the input information, for example the person's bounding box, may be omitted.

[0066] The non-side dangerous work determination model 402 can determine a dangerous work state based on a change in a person's posture. For example, the non-side dangerous work determination model 402 executes dangerous work determination using the bounding boxes of a person and a stepladder in one frame image and the skeletal information of the person in a frame sequence as input. Some of the input information, for example the bounding box of the person, may be omitted.

[0067] 9 shows a flowchart of an example of processing by the risk determination unit 308. First, the risk determination unit 308 receives designation of a dangerous operation determination model to be used from the model determination unit 306 (S31).

[0068] If the specified model is the side dangerous work judgment model 401 (S32: YES), the danger judgment unit 308 acquires the bounding box of the stepladder from the stepladder state information 332 (S33). The stepladder bounding box indicated by the stepladder state information 332 is deemed to match the stepladder bounding box in the current frame image.

[0069] The danger determination unit 308 further acquires skeletal information of the person in the current frame image from the skeleton detection unit 307, and acquires a bounding box of the person in the current frame image from the person detection unit 302 (S34). The skeleton detection unit 307 receives, for example, the bounding box of the person in one frame image and the frame image, and outputs the coordinates of the person's keypoints as skeletal information.

[0070] The risk determination unit 308 uses the side dangerous work determination model 401 to determine whether the current frame image indicates a dangerous work state (S35). Specifically, the risk determination unit 308 inputs the stepladder bounding box from the stepladder state information 332, the bounding box of the person in the current frame image, and the person's skeletal information to the side dangerous work determination model 401. The side dangerous work determination model 401 outputs a determination result as to whether the person working on the stepladder is in a dangerous work state.

[0071] If the specified model is the non-side dangerous work judgment model 402 (S32: NO), the danger judgment unit 308 acquires the bounding box of the stepladder from the stepladder state information 332 (S36). The bounding box of the stepladder indicated by the stepladder state information 332 is deemed to match the bounding box of the stepladder in the current frame image.

[0072] The danger determination unit 308 acquires a bounding box of the person in the current frame image from the person detection unit 302. The danger determination unit 308 further acquires skeleton information of the person in the frame sequence from the skeleton detection unit 307 (S37).

[0073] The frame sequence may be, for example, a predetermined number of frames from the current frame or the frame when the ladder state information 332 was set. The skeletal information referenced may be different from the skeletal information used in the side dangerous work determination model 401. For example, only a part of the skeleton definition model 100 shown in Fig. 4, for example, only the leg keypoints, may be referenced. The leg keypoints may be all or a part of the right knee 106, left knee 107, right ankle 108, and left ankle 109.

[0074] The risk determination unit 308 uses the non-side dangerous work determination model 402 to determine whether the current frame image indicates a dangerous work state (S38). Specifically, the risk determination unit 308 inputs the ladder bounding box from the ladder state information 332, the bounding box of the person in the current frame image, and skeletal information of the frame sequence to the non-side dangerous work determination model 402.

[0075] The non-side dangerous work determination model 402 outputs a determination result as to whether a person working on a stepladder is in a dangerous work state. The skeletal information of the frame sequence input to the model may be, for example, the average movement amount per unit time of the legs in the frame sequence and the movement amount per unit time from the current frame to the immediately previous frame. The position of the legs in each frame of the frame sequence may be input to the model. The input to the non-side dangerous work determination model 402 may include a bounding box of the person in each sequence of the frame sequence.

[0076] As described above, by changing the method of using the human skeleton for each stepladder orientation to perform dangerous work judgment, it is possible to detect a wider range of dangerous stepladder work compared to the method of only referring to the bounding box. In addition, by using a high-load judgment process (skeleton judgment, etc.) only when necessary depending on the stepladder orientation, it is possible to reduce processing time while preventing erroneous judgment.

[0077] Fig. 10 shows an example of a GUI (Graphical User Interface) screen displayed to the work manager on the user terminal 20. The GUI screen displays a frame image determined to be a dangerous work state. The example shown in Fig. 10 shows a person 501 and a stepladder 502 in a dangerous work state, as well as their bounding boxes and the person's skeletal information.

[0078] The GUI screen further shows the IoU threshold value set by the work manager. The work manager can set the IoU threshold value on another GUI screen. The playback time bar 503 shows the time of the displayed frame image.

[0079] When the work manager selects fast forward button 504, the video display unit 201 displays the next frame image of the dangerous work state. Since the frame image of the dangerous work state is automatically extracted and displayed, the work manager can confirm the dangerous work in a short time. In response to the work manager's selection of button 505, the video display unit 201 switches between showing and hiding the bounding boxes of the person 501 and stepladder 502. This makes it possible to display an image that is easy for the work manager to see.

[0080] The video analysis device 30 may analyze the video from the image capture device 40 in real time and notify the on-site work supervisor of danger in real time on a GUI screen. In other words, the video analysis device 30 analyzes the video data while the worker is working and notifies the work supervisor of dangerous work.

[0081] The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0082] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a computing device interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0083] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0084] 20 User terminal 30 Video Analysis Equipment 40 Imaging Equipment 301 Video input section 302 Person detection unit 303 Ladder detector 304 Risk Assessment Department 305 Orientation Estimation Unit 306 Model Determination Unit 307 Skeleton detection unit 308 Risk Assessment Department 310 Input Device 311 Arithmetic equipment 312 Memory 313 Auxiliary storage 314 Output Device 321 Rectangle overlap determination unit 322 Storage section 323 Orientation determination unit 324 Recognition Processing Unit 331 Video Data 332 Ladder status information 333 Threshold Information

Claims

1. A dangerous operation detection system for a person performing an operation using an object, comprising: A computing device; A storage device for storing a program executed by the arithmetic unit, The computing device includes: determining an orientation of the object in a first frame image of video data; a dangerous work detection system that selects one dangerous work determination method from a plurality of dangerous work determination methods for determining a dangerous work state of the person for the first frame image depending on an orientation of the object.

2. The dangerous operation detection system according to claim 1, A dangerous work detection system, wherein the computing device determines whether the person is in a dangerous work state based on the position of the person, the position of the object, and the posture of the person in the first frame image when the orientation of the object is a first orientation.

3. The dangerous operation detection system according to claim 1, The dangerous work detection system, wherein the computing device determines whether the person is in a dangerous work state based on a change in the person's posture in a frame sequence consisting of a plurality of frame images selected from the first frame image and a frame image prior to the first frame image when the orientation of the object is a second orientation.

4. 3. The dangerous operation detection system according to claim 2, The computing device includes: storing, in the storage device, position information of the object in a second frame image which is a frame image preceding the first frame image and in which the distance between the person and the object is shorter than the distance between the person and the object in the first frame image; A dangerous work detection system that uses the position indicated by the stored object position information as the position of the object when determining whether the person is in a dangerous work state.

5. The dangerous operation detection system according to claim 1, The computing device includes: storing, in the storage device, information on the orientation of the object in a second frame image which is a frame image preceding the first frame image and in which the distance between the person and the object is closer than the distance in the first frame image; A dangerous operation detection system that determines the orientation indicated by the stored object orientation information as the orientation of the object in the first frame image.

6. The dangerous operation detection system according to claim 4 or 5, In the second frame image, an overlap between a bounding box of the object and a bounding box of the person is greater than 0; A dangerous work detection system, wherein in the frame image immediately before the second frame image, the overlap between the bounding box of the object and the bounding box of the person is zero.

7. The dangerous operation detection system according to claim 1, The computing device includes: Before selecting one of the hazardous work assessment methods, determining the positions of the object and the person in the first frame image; A dangerous work detection system that, when the person is present within a specified range of the object in the first frame image, determines whether the person is in a dangerous work state based on the positional relationship between the object and the person, without referring to the person's posture.

8. The dangerous operation detection system according to claim 7, A dangerous operation detection system, wherein the predetermined range is user-settable.

9. The dangerous operation detection system according to claim 1, the object is a stepladder; The computing device includes: When the direction of the stepladder is sideways, it is determined whether the person is in a dangerous working state based on the position of the person, the position of the object, and the posture of the person in the first frame image; A dangerous work detection system that determines whether the person is in a dangerous work state based on changes in the person's posture in a frame sequence consisting of a plurality of frame images selected from the first frame image and a frame image prior to the first frame image when the orientation of the object is not to the side.

10. The dangerous operation detection system according to claim 1, The dangerous work detection system, wherein the computing device, when a dangerous work is detected, displays an image indicating the dangerous work on a display device.

11. A method for detecting a dangerous operation of a person performing an operation using an object, comprising: the system determining an orientation of the object in a first frame image of video data; A method in which the system selects one dangerous work determination method from a plurality of dangerous work determination methods for determining a dangerous work state of the person for the first frame image depending on an orientation of the object.

Images