Worker detection device, worker detection method, and program

Abstract

To provide a worker detector, worker detection method, and program making it possible to alleviate a load on a monitoring person in inspection work of an electric power facility, and allowing a worker to safely carry out inspection work.SOLUTION: A detector 100 includes a determination unit 153 that determines whether a worker exists in either a detection area, which is defined in a fixed point image acquired by an acquisition unit 152, or a danger area located inside the detection area, and a rendering unit 154 that, when it is determined by the determination unit 153 that a worker exists in either the detection area or danger area, renders frames representing the detection area and danger area in the fixed point image acquired by the acquisition unit 152 and a mark indicating the position of the worker.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a worker detection device, a worker detection method, and a program. [Background technology]

[0002] On power transmission towers supporting multiple lines, inspection work is sometimes carried out by shutting off only some of the lines to maintain power supply to consumers. In such cases, a hazard zone is set up to prevent workers from accidentally approaching lines that are in operation, and a supervisor monitors workers' approach to the hazard zone. To reduce the burden of such monitoring work by supervisors, it is conceivable to apply a system that detects the presence of workers to power transmission towers. For example, Patent Document 1 discloses a system that, when an area and equipment name within a facility are specified, calculates an alarm area based on the alarm distance corresponding to the equipment name, and issues an alarm if a moving object enters the calculated alarm area. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2017-122948 [Overview of the project] [Problems that the invention aims to solve]

[0004] The system described in Patent Document 1 only issues an alarm to the monitor when a moving object enters the alarm area. Therefore, even if applied to the detection of workers on power transmission towers, there is a problem in that the monitor cannot determine whether a worker is likely to enter the alarm area before the worker actually enters it. For this reason, there is room for improvement in reducing the burden on monitors during power transmission tower inspection work and enabling workers to perform inspection work more safely. Such problems exist not only in the monitoring of power transmission towers but also in the monitoring of other power facilities.

[0005] The present invention has been made based on such a background, and an object thereof is to provide a worker detection device, a worker detection method, and a program that can reduce the burden on a monitor in an inspection operation of power facilities and enable an operator to perform an inspection operation more safely.

Means for Solving the Problems

[0006] In order to achieve the above object, the worker detection device according to the present invention includes an acquisition unit that periodically acquires fixed-point images of power facilities, a determination unit that determines whether a worker exists in either a detection area set in the fixed-point image acquired by the acquisition unit or a danger area inside the detection area, a drawing unit that draws a frame indicating the detection area and the danger area and a mark indicating the position of the worker in the fixed-point image acquired by the acquisition unit when it is determined by the determination unit that a worker exists in either the detection area or the danger area, and is provided with.

Effects of the Invention

[0007] According to the present invention, it is possible to provide a worker detection device, a worker detection method, and a program that can reduce the burden on a monitor in an inspection operation of power facilities and enable an operator to perform an inspection operation more safely.

Brief Description of the Drawings

[0008] [Figure 1] It is a schematic diagram showing the configuration of a detection system according to an embodiment of the present invention. [Figure 2] It is a block diagram showing the hardware configuration of a detection device according to an embodiment of the present invention. [Figure 3] (a) to (c) are diagrams showing an example of an output image used as learning data. [Figure 4] It is a conceptual diagram of a neural network used by a detection device according to an embodiment of the present invention. [Figure 5](a) is a diagram showing an example of a data table of an image data storage unit according to an embodiment of the present invention, and (b) is a diagram showing an example of a data table of a detection / risk area storage unit according to an embodiment of the present invention. [Figure 6] It is a diagram showing an example of a bounding box, a detection area, and a risk area drawn on a fixed-point image. [Figure 7] (a) is a diagram showing an example of a normal risk area, and (b) is a diagram showing an example of a risk area during a single-sided power outage. [Figure 8] It is a diagram showing a procedure for optimizing the weights between layers of a neural network by a detection device according to an embodiment of the present invention. [Figure 9] (a) is a diagram showing a state where an operator is detected outside the risk area, and (b) is a diagram showing a state where an operator is detected inside the risk area. [Figure 10] It is a block diagram showing the hardware configuration of a monitoring device according to an embodiment of the present invention. [Figure 11] (a) to (c) are diagrams showing a procedure for setting a detection area and a risk area. [Figure 12] It is a flowchart showing the flow of learning processing according to an embodiment of the present invention. [Figure 13] It is a flowchart showing the flow of optimization processing according to an embodiment of the present invention. [Figure 14] It is a flowchart showing the flow of setting processing according to an embodiment of the present invention. [Figure 15] It is a flowchart showing the flow of detection processing according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, an operator detection device, an operator detection method, and a program according to an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same or equivalent parts are denoted by the same reference numerals. Hereinafter, a case of monitoring a transmission tower as a monitoring target will be described as an example.

[0010] The worker detection system according to this embodiment acquires fixed-point images by repeatedly photographing the same location on the power transmission tower to be monitored, and detects whether a worker is present in a detection area set on the power transmission tower based on these fixed-point images. The power transmission tower is a truss structure tower made of steel, and each power transmission line is supported by multiple crossarms extending to the left and right. The following explanation will use a double-circuit tower as an example, which has three crossarms extending to the left and three power transmission lines that constitute one circuit, installed on both the left and right sides.

[0011] As shown in Figure 1, detection system 1 is an example of a worker detection system comprising a camera 2, a detection device 100, and a monitoring device 200. The camera 2 and the detection device 100 are connected to each other via a wired or wireless communication line, and the detection device 100 and the monitoring device 200 are connected to each other via a wired or wireless communication line.

[0012] Camera 2 repeatedly photographs the same location on the power transmission tower being monitored from the same spot at a fixed sampling period, generating a series of fixed-point images (videos) arranged in chronological order. The generated fixed-point images are then transmitted to the detection device 100 in real time each time. Camera 2 is, for example, mounted on a tripod in the same location.

[0013] Camera 2 is a device that captures an image through a lens onto an image sensor, converts it into an electrical signal, and outputs it. Camera 2 is, for example, a video camera. The video camera is connected to the detection device 100 via a capture device (not shown). The capture device converts the HDMI (High-Definition Multimedia Interface: registered trademark) signal output from the video camera into an electrical signal that the detection device 100 can process. By using a video camera that can automatically adjust the aperture without manual intervention, an image with a constant brightness can be obtained even outdoors where the brightness changes. In addition, power transmission towers located far away can be monitored using the zoom function of the video camera.

[0014] The detection device 100 is an example of a worker detection device that acquires fixed-point images by repeatedly photographing the same location on the power transmission tower to be monitored, and detects whether a worker is present in a detection area set on the power transmission tower by performing image analysis processing on the fixed-point images. The image analysis processing uses a detection model (trained model) generated by machine learning.

[0015] The monitoring device 200 receives fixed-point images transmitted from the detection device 100 and displays them to the monitor, and sets detection areas and hazard areas on the fixed-point images based on instructions from the monitor. The detection area is the region in the fixed-point image where workers are detected, and the hazard area is set inside the detection area and is the region where a warning is issued to the monitor when a worker is detected. The monitoring device 200 may be located at the same site as the workers, or it may be located at a monitoring center away from the site.

[0016] Next, the hardware configuration of the detection device 100 will be described with reference to Figure 2. The detection device 100 is, for example, a general-purpose computer. The detection device 100 comprises an operation unit 110, a display unit 120, a communication unit 130, a storage unit 140, and a control unit 150. Each part of the detection device 100 is interconnected via an internal bus (not shown).

[0017] The operation unit 110 receives user instructions and supplies operation signals corresponding to the received operations to the control unit 150. The operation unit 110 includes, for example, a mouse and a keyboard.

[0018] The display unit 120 displays various images to the user based on image data supplied from the control unit 150. For example, the display unit 120 displays a fixed-point image of a power transmission tower taken by camera 2.

[0019] The communication unit 130 is a communication interface for the detection device 100 to communicate with external devices. The communication unit 130 communicates with external devices via a communication network such as the Internet. The communication unit 130 also has input / output terminals and communicates with external devices via a communication cable. The input / output terminals are, for example, USB (Universal Serial Bus).

[0020] The memory unit 140 includes, for example, RAM (Random Access Memory), ROM (Read Only Memory), flash memory, and a hard disk. The memory unit 140 stores programs executed by the control unit 150 and various types of data. The memory unit 140 also temporarily stores various types of information and functions as work memory for the control unit 150 to execute processing. Furthermore, the memory unit 140 includes a learning data storage unit 141, a detection model storage unit 142, an image data storage unit 143, and a detection / danger area storage unit 144.

[0021] The training data storage unit 141 stores training data used as training data for machine learning. The training data includes multiple datasets, each dataset containing one input image of a power transmission tower and a worker, and one output image (ground truth image) that includes a bounding box surrounding the worker present in the input image. The output image is generated manually, for example, based on the input image. The bounding box is a sub-region enclosed by a rectangular boundary line around the target object region (in this case, the worker) relative to the outer region.

[0022] Both the input and output images are RGB images. An RGB image consists of three maps: red, green, and blue, each containing numerous pixels with values ​​(RGB values) ranging from 0 to 255. In each dataset, the input and output images have the same number of pixels and the same aspect ratio.

[0023] The input image includes a cropped image of the entire crossarm of the power transmission tower, as shown in Figure 3(a), from a horizontal image (original image) taken by camera 2; an image cropped from the original image so that the worker is near the center, as shown in Figure 3(b); and a vertical image cropped from the original image so that the entire power transmission tower is near the center, as shown in Figure 3(c). The image cropped with the worker at the center is, for example, a square image. Note that in Figures 3(a) to (c), the output image including the bounding box is shown, but the image obtained by removing the bounding box from the output image corresponds to the input image.

[0024] When obtaining input image samples, photograph the power transmission tower from various locations and angles, ensuring that at least one worker is included in each image. Ensure that objects other than the worker and the power transmission tower, such as power lines and insulators, are also visible in the surrounding images. This improves the accuracy of worker detection in the detection model generated by machine learning. Furthermore, while the training data includes vertically cropped images of the entire power transmission tower, as will be discussed later, the detection area for workers may be set to cover the entire vertically cropped power transmission tower. Using such images allows machine learning to be performed in an environment closer to actual monitoring conditions.

[0025] The output image is an image containing information about the pixel positions of a rectangular bounding box relative to the input image. The bounding box is, for example, a rectangle (circumscribed rectangle) that is in contact with the outside of a worker. Since the bounding box is a rectangle without tilt on the screen, its position and size can be determined by the pixel positions of two diagonally opposite vertices, for example, the top-left and bottom-right vertices.

[0026] Furthermore, in the training data, data augmentation may be performed as a preprocessing step for machine learning to increase the number of data points in the dataset. Data augmentation is a process that increases the amount of data in the training data by transforming or combining sample images that were originally in the training data dataset. For example, data augmentation can be performed by randomly flipping the sample images horizontally, changing the brightness, or rotating them. In addition, both the training data and the models generated based on it should be treated as general-purpose, and there should be no need to separate training samples or prepare individual models based on the size, type, location, climate, weather, brightness, number of workers, etc., of the power transmission towers being photographed.

[0027] Returning to Figure 2, the detection model storage unit 142 stores the detection model generated by machine learning based on the training data stored in the training data storage unit 141. The detection model is a model that takes the pixel value of each pixel of the input image as input and outputs the pixel value of each pixel of the output image.

[0028] The detection model uses a multi-layered neural network consisting of multiple interconnected nodes. Therefore, the detection model memory unit 142 stores optimized weight coefficients that indicate the strength of the connections between the nodes constituting the neural network. While detection models such as FCOS (Fully Convolutional One-Stage Object Detectionaster), SSD (Single Shot MultiBox Detector), and YOLO (You Only Look Once) can be used, FCOS is preferred considering the detection performance and speed in detecting workers on power transmission towers.

[0029] As shown in Figure 4, a neural network comprises an input layer into which input data is received, an output layer to which output data is output, and at least one hidden layer positioned between the input and output layers. The arrows between neurons in each layer represent the parameter connections between the input and output layers.

[0030] The input layer includes a plurality of input neurons. The number of input neurons in the input layer corresponds to the number of input data. The input data is composed of pixel values of each pixel of the input image. If the number of pixels of each image is m pixels × n pixels, it can be expressed as I(1,1), I(1,2), …, I(m,n). Since the input image is an RGB image, each input data I(i,j) can be expressed as a two-dimensional array (R value, G value, B value). The input data input to the input neurons in the input layer is respectively input to each intermediate neuron in the intermediate layer.

[0031] Each intermediate layer includes a plurality of intermediate neurons. Each neuron in the intermediate layer, when receiving an input value I

[0033] , i , , from each neuron in the previous stage, calculates the product I i of the input value I i [[ID=???]]and the weight coefficient W i as I i ×W i , calculates the sum value of the products I i ×W i calculated for each neuron in the previous stage, and outputs an output value by substituting the sum value of the product I

[0032] i into an activation function. The activation function is a function that represents the non-linear relationship between input and output in a certain neuron. For example, it is a sigmoid function, a max function, or a Gaussian function.

[0032]

[0033] The output layer includes a plurality of output neurons that respectively output output data. The number of output neurons corresponds to the number of output data. Each neuron in the output layer, when receiving an input value I i from each neuron in the intermediate layer at the rearmost position, calculates the product I i in the same way as each neuron in the intermediate layer, calculates the product I i of the input value I i and the weight coefficient W i as I i ×W i , calculates the sum value of the products I <000001? ×W i calculated for each neuron in the previous stage, and outputs an output value by substituting the sum value of the product I i ×W i into an activation function.

[0033] It should be noted that there seems to be some unclear or incorrect notations in the original text (such as the "I i " etc. which might need further clarification in the original context for a more accurate translation). Also, in the translation, the "I i " etc. are left as they are as per the requirement to preserve the 7 - digit tags exactly.The output data consists of the pixel values ​​of each pixel in the output image, and the number of pixels corresponds to the number of pixels in the input image, which is m pixels × n pixels, so it can be expressed as O(1,1), O(1,2), ..., O(m,n). Also, since the output image is an RGB image, each output data O(i,j) can be expressed as a two-dimensional array (R value, G value, B value) in the same way as the input data I(i,j).

[0034] As shown in Figure 5(a), the image data storage unit 143 stores the image data acquired from the camera 2 in order, associated with the date and time of capture. In addition, when a bounding box is drawn on a fixed point image acquired from the camera 2 through a process described later, the image data storage unit 143 stores the data of the pixel positions of the bounding box, associated with the fixed point image.

[0035] As shown in Figure 5(b), the detection / danger area storage unit 144 stores the pixel positions indicating the detection area and danger area set in the fixed-point image. Both the detection area and the danger area are rectangular regions. The pixel positions indicating the detection area are represented by the pixel positions of two diagonally opposite vertices, for example, the pixel positions of the top-left vertex and the bottom-right vertex. The same applies to the pixel positions indicating the danger area.

[0036] As shown in Figure 6, the detection area is set to the minimum range, for example, including the crossarms of the power transmission tower. Although the image captured by camera 2 is a horizontal image, the detection target is a worker on the power transmission tower, so by setting the detection area to the area around the crossarms of the power transmission tower, the detection speed in the worker detection process can be improved. If the power transmission tower is tall or if camera 2 is installed near the power transmission tower, camera 2 will be tilted to look up at the power transmission tower, which may cause the worker on the screen to become small and undetectable. In such cases, it is advisable to reduce the detection area to a range in which the worker can be detected.

[0037] Regarding the hazardous area, under normal circumstances, the area is set to surround the entire crossarm on both the left and right sides, as shown in Figure 7(a). During inspection work, the area is set to surround only the crossarm on one side that is currently transmitting power, as shown in Figure 7(b). The reason for this is that under normal circumstances, it is easy to see that there is a risk of electric shock from both the left and right crossarms, whereas during inspection work, one circuit on one side is shut off, which could cause workers to mistakenly identify the left and right sides of the crossarm.

[0038] Returning to Figure 2, the control unit 150 includes a processor and controls each part of the detection device 100. The processor includes a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The control unit 150 executes the learning process in Figure 12, the optimization process in Figure 13, the setting process in Figure 14, and the detection process in Figure 15, respectively, by executing the program stored in the memory unit 140. Functionally, the control unit 150 includes a learning unit 151, an acquisition unit 152, a determination unit 153, a drawing unit 154, and an output unit 155.

[0039] The learning unit 151 acquires the training data stored in the training data storage unit 141, generates a detection model using machine learning with the acquired training data, and stores it in the detection model storage unit 142. The learning unit 151 generates the detection model by performing supervised learning using all or part of the multiple datasets included in the training data as training data.

[0040] More specifically, the learning unit 151 uses multiple datasets included in the training data as training data to adjust the weight coefficients that represent the connection state of each layer in the neural network. The procedure involves first inputting the pixel values ​​of each pixel in the input image from the training data into the input layer, and then comparing the pixel values ​​of each pixel in the output image output to the output layer with the pixel values ​​of each pixel in the output image from the training data. The weight coefficients are then optimized so that the difference between the pixel values ​​of each pixel in the output image output to the output layer and the pixel values ​​of each pixel in the output image from the training data is as small as possible. For example, backpropagation can be used to optimize the weight coefficients.

[0041] The procedure for optimizing the weight coefficients W using backpropagation will be explained below with reference to Figure 8. First, input data I(i,j), which consists of the pixel values ​​of each pixel in the input image, is input to the input neurons of the input layer. Based on the output data O(i,j), which consists of the pixel values ​​of each pixel in the output image output from the output neurons, and the output data O'(i,j), which consists of the pixel values ​​of each pixel in the manually created output image, the mean squared error (MSE) of the output data O(i,j) and O'(i,j) is calculated.

[0042] MSE is a type of evaluation function that scores and evaluates the performance of a learning model, and is expressed by the following equation (1). MSE = Σ(O(i,j) - O'(i,j)) 2 / (m×n) …(1) However, i = 1 to m and j = 1 to n. Also, in the initial MSE calculation, the weight coefficients between each neuron, which were initialized for the O(i,j) calculation of the output data, are used. The initial values ​​of the weight coefficients between each neuron can be given, for example, as random numbers between -0.1 and +0.1.

[0043] Next, the weight coefficients W of the neural network are optimized by minimizing the MSE. The optimized weight coefficients when the MSE is at its minimum value are given by W.opt This is how it is expressed. To minimize the MSE, for example, one can use gradient descent. Gradient descent calculates the slope of the MSE and updates the weight coefficient W in the opposite direction to the magnitude of that slope, and by repeating this procedure, the value of the MSE is reduced and the weight coefficient W is optimized. opt This is a method to obtain the current weight coefficient W i , the updated weight coefficient is W i+1 ΔW i Expressed in this way, these satisfy the following equation (2). W i+1 =W i +ΔW i …(2)

[0044] The slope of the MSE is expressed as the partial derivative (∂E / ∂W) obtained by differentiating the MSE with respect to the weight coefficient W, and therefore the change in the weight coefficient ΔW i This can be expressed by the following equation (3). ΔW i =-η×(∂E / ∂W i ) …(3) However, η is a coefficient, and 0 < η < 1. Substituting equation (3) into equation (2), the updated weight coefficient is W i+1 This allows us to obtain the optimized weight coefficient W. opt The detection model is stored in the detection model storage unit 142. The above is the procedure for optimizing the weight coefficient W using the backpropagation method.

[0045] Returning to Figure 2, the acquisition unit 152 periodically acquires fixed-point images of the power transmission tower transmitted from the camera 2 and stores them in the image data storage unit 143, associating them with the date and time of capture. The acquisition unit 152 also acquires data on the pixel positions indicating the detection area and danger area in the fixed-point images transmitted from the monitoring device 200 and stores it in the detection / danger area storage unit 144. Note that the acquisition of data by the acquisition unit 152 also includes reading the data stored in the storage unit 140 into the work memory.

[0046] The determination unit 153 uses the detection model stored in the detection model storage unit 142 to perform image analysis processing on the latest fixed-point image and determines whether a worker is present in either the detection area or the hazard area stored in the detection / hazard area storage unit 144. Specifically, each time a fixed-point image of a power transmission tower is acquired, the fixed-point image is cropped to cut out the frame of the detection area. Next, the detection model is used to determine whether a worker is present in the detection area of ​​the cropped image, and if it is determined that a worker is present in the detection area, it is determined whether a worker is present in the hazard area.

[0047] In this process, the pixel values ​​of each pixel in the cropped fixed-point image are input to the input neurons of the detection model's input layer, causing the output neurons to output the pixel values ​​of each pixel in the output image. As a result, if a bounding box surrounding a worker is drawn in the output image, it can be determined that the worker is present in at least the detection area. In addition, if at least a portion of the bounding box drawn in the fixed-point image overlaps with the hazard area, it is determined that the worker is present in the hazard area. If it is determined that a worker is present in either the detection area or the hazard area, the pixel positions of the bounding box generated at that time are associated with the fixed-point image being determined and stored in the image data storage unit 143.

[0048] The drawing unit 154 overlays the frames indicating the detection area and danger area stored in the detection / danger area storage unit 144 onto the fixed point image. If the determination unit 153 determines that a worker is present in either the detection area or the danger area of ​​the fixed point image, it additionally overlays the frame indicating the boundary box surrounding the worker, stored in the image data storage unit 143, onto the fixed point image at the time of detection. If a worker is present in the detection area of ​​the fixed point image, the color of the frames indicating the danger area and boundary box is set to the normal color (e.g., green), as shown in Figure 9(a). If a worker is present in the danger area of ​​the fixed point image, the color of the frames indicating the danger area and boundary box is set to the warning color (e.g., red), as shown in Figure 9(b).

[0049] Hereinafter, images generated when no workers are detected in the detection area or hazardous area will be referred to as "non-detection images," images generated when workers are detected in the detection area but not in the hazardous area will be referred to as "detection images," and images generated when workers are detected in the hazardous area will be referred to as "warning images." Note that the images shown in Figures 9(a) and (b) were created to verify whether it was possible to detect when a worker entered a hypothetical hazardous area set up in a safe area away from the power lines, and these differ from actual monitoring conditions.

[0050] Returning to Figure 2, the output unit 155 outputs the fixed-point image acquired by the acquisition unit 152 directly to the outside if the detection area and danger area settings have not been completed. If the detection area and danger area settings have been completed, it outputs one of the non-detection image, detection image, or warning image generated by the drawing unit 154 to the outside. In the output by the output unit 155, for example, the image to be output is displayed on the display unit 120 and the image is also transmitted to the monitoring device 200 in real time. The above describes the configuration of the detection device 100.

[0051] Next, the hardware configuration of the monitoring device 200 will be described with reference to Figure 10. The monitoring device 200 is, for example, a general-purpose computer. The monitoring device 200 comprises an operation unit 210, a display unit 220, a communication unit 230, a storage unit 240, and a control unit 250. Each part of the monitoring device 200 is interconnected via an internal bus (not shown).

[0052] The operation unit 210 receives user instructions and supplies operation signals corresponding to the received operations to the control unit 250. The display unit 220 displays various images to the user based on image data supplied from the control unit 250. The operation unit 210 and the display unit 220 are configured, for example, by a touch panel. The touch panel displays an operation screen that accepts predetermined operations and supplies operation signals to the control unit 250 corresponding to the position where the user makes a contact operation on the operation screen.

[0053] The display unit 220 overlays frames indicating the detection area and hazard area onto the fixed-point image of the power transmission tower captured by camera 2. The position and size of the frames indicating the detection area and hazard area can be changed by the monitor operating the control unit 210. In addition, when a worker is detected by the detection device 100, the display unit 220 overlays a frame indicating the boundary box onto the fixed-point image of the power transmission tower captured by camera 2.

[0054] The communication unit 230 is a communication interface for the monitoring device 200 to communicate with external equipment, such as the detection device 100. The communication unit 230 communicates with external equipment via a communication network, such as the Internet.

[0055] The storage unit 240 includes, for example, RAM, ROM, flash memory, and a hard disk. The storage unit 240 stores programs executed by the control unit 250 and various types of data. The storage unit 240 also temporarily stores various types of information and functions as a work memory for the control unit 250 to execute processing. The storage unit 240 includes an image data storage unit 241 and a detection / danger area storage unit 242. The image data storage unit 241 stores data of either non-detected images, detected images, or warning images, associated with the shooting date and time of the respective original fixed-point images. The detection / danger area storage unit 242 has the same or equivalent configuration as the detection / danger area storage unit 144.

[0056] The control unit 250 includes a processor and controls each part of the detection device 100. The processor is, for example, a CPU. The control unit 250 executes the setting process shown in Figure 14 by executing a program stored in the storage unit 240. Functionally, the control unit 250 includes an acquisition unit 251, a setting unit 252, and an output unit 253.

[0057] The acquisition unit 251 periodically acquires fixed-point images of the power transmission towers from the detection device 100 which is performing the setting process. The acquisition unit 251 also periodically acquires either non-detected images, detected images, or warning images from the detection device 100 which is performing the detection process, and stores them in the image data storage unit 241.

[0058] The setting unit 252 sets detection areas and danger areas in the fixed-point images periodically captured by the camera 2, based on operation signals from the operation unit 210. Specifically, the setting unit 252 overlays frames indicating the detection areas and danger areas onto the fixed-point images and displays them on the display unit 220. Furthermore, when the position and size of the frames indicating the detection areas and danger areas are adjusted by the operation unit 210, the setting unit 252 changes the position and size of the frames indicating the detection areas and danger areas displayed on the display unit 220 according to that operation. Once the position and size of the frames indicating the detection areas and danger areas are finalized, the setting unit 252 stores the position and size of the frames indicating the detection areas and danger areas in the detection / danger area storage unit 242.

[0059] The procedure for setting detection areas and hazardous areas by the monitor will be explained below with reference to Figure 11. Note that the fixed-point images shown in Figure 11 were created by setting a hypothetical hazardous area in a safe area away from the power lines and verifying whether it was possible to detect when a worker entered the hazardous area, and therefore do not reflect actual monitoring conditions.

[0060] First, when the monitor operates the control unit 210 to start a program that causes the detection device 100 to perform detection processing, a frame indicating the detection area, as shown in Figure 11(a), is displayed on the fixed-point image of the power transmission tower shown on the display unit 220. At this point, the frame indicating the detection area is set to its initial position.

[0061] Next, when the monitor operates the control unit 210 to move the symbols ○ in the upper left and lower right corners of the frame indicating the detection area on the fixed-point image, the color of the frame changes (for example, from green to blue). After adjusting the position and size of the frame indicating the detection area and pressing the "Settings button" on the screen, the detection area is set as shown in Figure 11(b), and the position and size of the frame indicating the detection area are stored in the detection / danger area storage unit 242.

[0062] Once the detection area is set, a frame indicating the hazardous area is displayed on the fixed-point image shown on the display unit 220. The monitor adjusts this frame indicating the hazardous area using the operation unit 210 in the same procedure as for the frame indicating the detection area, and when they press the "Settings button" on the screen, the hazardous area is set as shown in Figure 11(c), and the position and size of the frame indicating the hazardous area are stored in the detection / hazardous area storage unit 242. The above describes the procedure for setting the detection area and the hazard area.

[0063] Returning to Figure 10, once the position and size of the frame indicating the detection area and danger area are set by the setting unit 252, the output unit 253 transmits the pixel position data indicating the position and size of the detection area and danger area to the detection device 100. Also, once the acquisition unit 251 acquires any of the non-detected image, detected image, or warning image, the output unit 253 displays it on the display unit 220. The above describes the configuration of the monitoring device 200.

[0064] (Learning process) Next, with reference to the flowchart in Figure 12, the flow of the learning process performed by the detection device 100 according to the embodiment will be explained. The learning process is the process of generating a detection model based on the learning data. The learning process starts when the user has stored the learning data in the learning data storage unit 141 and has instructed the start of the learning process by operating the operation unit 110. The learning data is generated by preparing multiple horizontal images of power transmission towers, cropping the horizontal images into three types of images shown in Figures 3(a) to (c), and surrounding the workers in these images with frames indicating boundary boxes.

[0065] First, the learning unit 151 acquires the learning data stored in the learning data storage unit 141 (step S11), and then performs an optimization process to optimize the weight coefficients included in the learning data (step S12). The flow of the optimization process performed by the detection device 100 according to this embodiment will be explained below with reference to the flowchart in Figure 13.

[0066] (Optimization process) First, the learning unit 151 initializes the weight coefficients between all neurons in each layer of the neural network (step S21). The initial values ​​of the weight coefficients between each neuron can be given, for example, as random numbers between -0.1 and +0.1.

[0067] Next, the learning unit 151 inputs the pixel values ​​of each pixel in the input image from the training data to the input neurons of the input layer, outputting the pixel values ​​(predicted values) of each pixel in the output image output from the output neurons of the output layer, and calculates the MSE between the predicted pixel values ​​and the pixel values ​​(set values) of each pixel in the output image from the training data, which have been manually bounded (step S22).

[0068] Next, the learning unit 151 determines whether the MSE is below the threshold (step S23). If it is determined that the MSE is below the threshold (step S23; Yes), the process returns. On the other hand, if it is determined that the MSE is not below the threshold (step S23; No), the learning unit 151 determines the respective weight coefficients W i In contrast, the change in the weight coefficient ΔW calculated by equation (2) i By adding these together, the value of each weight coefficient is W i+1 The data is updated (step S24), and the process returns to step S22. The above is the flow of the optimization process.

[0069] Returning to Figure 12, the learning unit 151 then processes each of the finally obtained updated weight coefficients W. i+1 Each of these is an optimized weight coefficient W optThe detected model is stored in the detection model storage unit 142 (step S13), and the process is terminated. The above is the flow of the learning process.

[0070] (Setup process) Next, referring to the flowchart in Figure 14, the flow of the setting process performed collaboratively by the detection device 100 and the monitoring device 200 according to the embodiment will be explained. The setting process is the process of setting the detection area and the danger area for a fixed point image based on the user's operation instructions. First, as a preprocessing step, camera 2 is installed in front of the power transmission tower to be monitored, camera 2 is connected to the detection device 100 and the monitoring device 200 so that they can communicate with each other, and the fixed point image captured by camera 2 is set to be transmitted to the monitoring device 200 via the detection device 100.

[0071] First, the acquisition unit 152 of the detection device 100 acquires a fixed-point image of the power transmission tower to be monitored and stores it in the image data storage unit 143 (step S31). Next, the output unit 155 transmits the fixed-point image acquired in step S31 to the monitoring device 200 in real time (step S32).

[0072] When the acquisition unit 251 of the monitoring device 200 receives a fixed-point image from the detection device 100 (step S41), the setting unit 252 overlays frames indicating the detection area and danger area onto the fixed-point image received in step S41 and displays them on the display unit 220 (step S42). When the position and size of the frames indicating the detection area and danger area are adjusted by an operation signal from the operation unit 210, the setting unit 252 changes the position and size of the frames indicating the detection area and danger area displayed on the display unit 220 according to that operation. The monitor adjusts and confirms the position and size of the frames indicating the detection area and danger area on the fixed-point image.

[0073] Next, when the setting unit 252 receives an operation signal from the operation unit 210 indicating that the position and size of the frame showing the detection area and danger area have been determined (step S43), the output unit 253 stores the pixel position data indicating the position and size of the detection area and danger area in the detection / danger area storage unit 242 and transmits it to the detection device 100 (step S44), and then terminates the process.

[0074] When the acquisition unit 152 of the detection device 100 receives pixel position data indicating the location and size of the detection area and the danger area from the monitoring device 200 (step S33), it stores this data in the detection / danger area storage unit 144 (step S34) and terminates the process. The above is the flow of the setup process.

[0075] (Detection process) Next, with reference to the flowchart in Figure 15, the flow of the detection process performed by the control unit 150 of the detection device 100 according to the embodiment will be explained. The detection process involves acquiring fixed-point images by repeatedly photographing the same location on the power transmission tower to be monitored, and then performing image analysis processing on these fixed-point images to detect whether or not there are workers in the detection area set on the power transmission tower. The detection process starts when the user operates the operation unit 110 to instruct the start of the detection process after the setting process has been completed.

[0076] First, the acquisition unit 152 begins the periodic acquisition of fixed-point images of the power transmission towers to be monitored (step S51). Each time the acquisition unit 152 acquires a fixed-point image, it stores it in the image data storage unit 143, associating it with the date and time of acquisition.

[0077] Next, the determination unit 153 uses the detection model stored in the detection model storage unit 142 and constructed using a neural network to perform image analysis on the latest fixed-point image and determines whether a worker is present in the detection area set on the fixed-point image by the monitoring device 200 (step S52). Specifically, the latest fixed-point image is cropped with a frame indicating the detection area, and the cropped image is used as the input image, with each pixel value being input to the input layer of the neural network. Then, the optimized weight coefficients W between all neurons in each layer of the neural network are calculated. opt The calculations are performed sequentially on the intermediate layer and output layer, and the pixel value of each pixel in the output image is estimated.

[0078] If it is determined that a worker is present in the detection area (step S52; Yes), the process proceeds to step S53. On the other hand, if it is determined that no worker is present in the detection area (step S52; No), the drawing unit 154 generates a non-detection image in which frames indicating the detection area and the hazard area are drawn in normal colors in the fixed-point image at the time of detection (step S54).

[0079] If the result of step S52 is Yes, the determination unit 153 determines whether the worker detected to be present in the detection area is present in the hazardous area set in the fixed-point image (step S53). If it is determined that a worker is present in the hazardous area (step S53; Yes), the drawing unit 154 draws a frame indicating the detection area in the fixed-point image at the time of detection, and generates a warning image in which the boundary box surrounding the worker and the frame indicating the hazardous area are drawn in a warning color (step S55).

[0080] On the other hand, if it is determined that there are no workers in the hazardous area (step S53; No), the drawing unit 154 generates a detection image in which a boundary box surrounding the worker, the detection area, and a frame indicating the hazardous area are drawn in normal colors in the fixed-point image at the time of detection (step S56). In both steps S55 and S56, data indicating the position of the boundary box surrounding the worker is stored in the image data storage unit 143.

[0081] Next, the output unit 155 transmits one of the non-detected images, detected images, or warning images generated by the processing in steps S54 to S56 to the monitoring device 200 managed by the supervisor (step S57), and returns the process to step S52. When the acquisition unit 251 of the monitoring device 200 receives one of the non-detected images, detected images, or warning images from the detection device 100, the output unit 253 displays one of the images on the display unit 220. The supervisor refers to the image displayed on the display unit 220 and gives instructions to the worker as appropriate using a wireless device. The above is the flow of the detection process.

[0082] As described above, the detection device 100 according to the embodiment includes a determination unit 153 that determines whether a worker is present in either the detection area or the hazardous area inside the detection area, which are set in the fixed-point image acquired by the acquisition unit 152, and a drawing unit 154 that, when the determination unit 153 determines that a worker is present in either the detection area or the hazardous area, draws frames indicating the detection area and the hazardous area and a frame indicating a boundary box surrounding the worker in the fixed-point image acquired by the acquisition unit 152. Therefore, it can indicate the location of a worker in the detection area and issue a warning when a worker enters the hazardous area, thereby alerting the observer.

[0083] The present invention is not limited to the embodiments described above, and the following modifications are also possible.

[0084] (modified version) In the above embodiment, a video camera was used as camera 2 and connected to the PC via a capture device, but the present invention is not limited to this. For example, a USB camera or an industrial camera can also be used as camera 2. However, USB cameras generally have a fixed magnification that leans towards wide-angle, and industrial cameras require adjustment by program, etc., making them more difficult to set up compared to video cameras.

[0085] In the above embodiment, RGB images were used in the learning and detection processes, but the present invention is not limited thereto. For example, to reduce the computational load, grayscale images with pixel values ​​from 0 to 255 (256 gradations) may be used.

[0086] In the above embodiment, the position and size of the frame indicating the bounding box, detection area, and danger area were represented by the pixel positions of two opposing vertices, but the present invention is not limited to this. For example, the position and size of the frame in a fixed-point image may be represented by the position of the center point and the width and height of the bounding box.

[0087] In the above embodiment, gradient descent was used as the update algorithm to reduce the evaluation function, but the present invention is not limited to this. For example, stochastic gradient descent or momentum methods may be used.

[0088] In the above embodiment, the weight coefficients were optimized by determining whether the evaluation function, MSE, was below a threshold; however, the present invention is not limited to this. Functions other than MSE may be used as the evaluation function. Furthermore, the weight coefficients may be determined to be optimized when the number of training iterations reaches an upper limit.

[0089] In the above embodiment, a detection area and a hazard area were set in a fixed-point image, it was determined whether a bounding box surrounding a worker could be drawn in the detection area, and if a bounding box surrounding a worker was drawn, it was determined whether the frame indicating the bounding box overlapped with the hazard area. However, the present invention is not limited to this. For example, a hazard area may be set in a fixed-point image, and a detection model may be used to determine whether a worker is present in the hazard area.

[0090] In the above embodiment, when a worker is present in the detection area or hazard area, a frame indicating a boundary box surrounding the worker is drawn on the fixed-point image, but the present invention is not limited to this. The mark indicating the worker's position is not limited to a boundary box, and may be, for example, a circular or elliptical frame surrounding the worker. Furthermore, the mark indicating the worker's position is not limited to a frame surrounding the worker, and may be, for example, a symbol.

[0091] In the above embodiment, the color of the boundary block and the frame indicating the hazardous area were changed depending on whether a worker was detected within the detection area or within the hazardous area. However, the present invention is not limited to this. For example, the frame may be made to flash when a worker is detected within the hazardous area.

[0092] In the above embodiment, the monitoring device 200 was managed by a supervisor, and the supervisor was alerted by displaying either a non-detection image, a detection image, or a warning image on the display unit 220. However, the present invention is not limited to this. For example, the monitoring device 200 may be carried by a supervisor or worker as a wearable device, and the wearable device may alert the supervisor or worker by generating at least one of sound and vibration. Specifically, the wearable device may not generate sound or vibration when it acquires a non-detection image, but may generate sound or vibration when it acquires a detection image or a warning image. When a warning image is acquired, it may generate a louder sound or vibration than when a detection image is acquired, or the duration of the sound or vibration may be extended. The wearable device may be, for example, a wristwatch-type wearable device worn on the arm of a supervisor or worker.

[0093] In the above embodiment, both the training data and the model generated based thereon were treated as general-purpose, but the present invention is not limited to this. For example, training samples may be divided or models may be prepared individually based on the size, type, location, climate, weather, brightness, number of workers, etc., of the power transmission tower to be photographed.

[0094] In the above embodiment, the learning unit 151 generated a trained model by performing machine learning using a neural network, but the present invention is not limited to this. The trained model defines the relationship between input data and output data, and any model that can obtain output data from the output layer by supplying input data to the input layer may be a model constructed using methods other than machine learning. For example, the input and output images may be converted to grayscale, and HoG (Histgrams of Oriented Gradients) features may be used.

[0095] In the above embodiment, the detection device 100 was equipped with the functions of the learning unit 151, but the present invention is not limited thereto. For example, a device separate from the detection device 100 may be equipped with the functions of the learning unit 151. In this case, the external device generates a detection model by learning the relationship between input images and output images based on training data, and the detection device 100 can acquire the detection model generated by the external device via communication by the communication unit 130.

[0096] In the above embodiment, the optimized weight coefficients obtained by the learning process using the training data were stored directly in the detection model storage unit 142, but the present invention is not limited thereto. For example, after performing the learning process using a portion of the training data, the validity of the optimized weight coefficients may be evaluated using the remaining training data, and if the weight coefficients are evaluated as valid, they may be stored in the detection model storage unit 142.

[0097] In the above embodiment, various data were stored in the storage units 140 and 240 of the detection device 100 and the monitoring device 200, but the present invention is not limited thereto. For example, all or part of the various data may be stored in an external control device or computer via a communication network.

[0098] In the above embodiment, the detection device 100 and the monitoring device 200 operated based on programs stored in the storage units 140 and 240, respectively, but the present invention is not limited thereto. For example, a functional configuration realized by a program may be realized by hardware.

[0099] In the above embodiment, the detection device 100 and the monitoring device 200 were, for example, general-purpose computers, but the present invention is not limited thereto. For example, the detection device 100 may be implemented as a computer located on the cloud.

[0100] In the above embodiment, the processing performed by the detection device 100 and the monitoring device 200 was realized by the device having the above-described physical configuration executing a program stored in the storage units 140 and 240. However, the present invention may be realized as a program, or as a storage medium on which the program is recorded.

[0101] Alternatively, a device that performs the above-mentioned processing operations may be configured by distributing a program for executing the above-mentioned processing operations on a computer-readable non-temporary recording medium such as a flexible disk, CD-ROM (Compact Disk Read-Only Memory), DVD (Digital Versatile Disk), or MO (Magneto-Optical Disk), and then installing that program on a computer.

[0102] In the above embodiment, transmission towers were the target of monitoring, but the present invention is not limited to this. Other power facilities, such as various equipment installed in power plants and substations, and distribution poles, may also be the targets of monitoring.

[0103] The embodiments described above are illustrative, and the present invention is not limited thereto. Various embodiments are possible without departing from the spirit of the invention as described in the claims. The components described in the embodiments and modifications can be freely combined. Furthermore, inventions equivalent to the invention described in the claims are also included in the present invention. [Explanation of symbols]

[0104] 1. Detection System 2 cameras 100 detection device 150 Control Unit 151 Learning Department 152 Acquisition Department 153 Judgment section 154 Drawing section

Claims

1. An acquisition unit that periodically acquires fixed-point images of power equipment, A determination unit determines whether a worker is present in either the detection area set in the fixed-point image acquired by the acquisition unit or in the hazardous area located inside the detection area. If the determination unit determines that a worker is present in either the detection area or the hazard area, the drawing unit draws frames indicating the detection area and the hazard area and marks indicating the worker's position in the fixed-point image acquired by the acquisition unit. A worker detection device equipped with the following features.

2. The aforementioned power equipment is a transmission tower, The detection area is set to surround all the crossarms installed on the transmission tower. The aforementioned hazardous area is set up to surround the crossarms that support the power transmission lines being transmitted, which are installed on the power transmission towers. The worker detection device according to claim 1.

3. The aforementioned transmission tower supports multiple transmission lines that constitute different circuits at each of the crossarms extending to the left and right, The aforementioned hazardous area is set up to surround the crossarm of the other line while excluding the crossarm of the one line when one line is experiencing a power outage and the other line is receiving power. The worker detection device according to claim 2.

4. The aforementioned mark is a frame indicating a boundary box surrounding a worker in the fixed-point image acquired by the acquisition unit. The worker detection device according to claim 1.

5. The drawing unit changes the color of the frame indicating the hazardous area and the frame indicating the boundary box surrounding the worker depending on whether it is determined that a worker is present in the detection area or in the hazardous area. The worker detection device according to claim 4.

6. The determination unit generates a bounding box surrounding a worker in the fixed-point image acquired by the acquisition unit when a worker is present in either the detection area or the hazard area, based on a trained model that outputs the pixel value of each pixel in the output image in response to the input pixel value of each pixel in the input image. A worker detection device according to any one of claims 1 to 5.

7. The aforementioned trained model is generated by performing machine learning using the training data. The aforementioned training data dataset includes an input image obtained by cropping an image of power equipment to include workers, and an output image obtained by drawing a bounding box around the workers on the input image. The worker detection device according to claim 6.

8. A worker detection method performed by a worker detection device, The steps include periodically acquiring fixed-point images of power equipment, The steps include determining whether a worker is present in either the detection area set in the acquired fixed-point image or the hazardous area located inside the detection area, If it is determined that a worker is present in either the detection area or the hazard area, the acquired fixed-point image is to draw a frame indicating the detection area and the hazard area, and a mark indicating the position of the worker. A worker detection method including the following.

9. Computers, A means for periodically acquiring fixed-point images of power equipment. A determination means for determining whether a worker is present in either the detection area set in the fixed-point image acquired by the acquisition means or in the hazardous area located inside the detection area. If the determination means determines that a worker is present in either the detection area or the hazard area, the drawing means draws frames indicating the detection area and the hazard area and marks indicating the worker's position in the fixed-point image acquired by the acquisition means. A program designed to function as such.

Images