Production work scenario understanding method considering risk scenario dependency
By extracting visual and topological features from power operation images using deep residual networks and encoder-decoder convolutional networks, the problem of low recognition and understanding accuracy in existing technologies is solved, achieving higher precision in understanding power operation scenarios and reducing the false alarm rate of safety risks.
Patent Information
- Application Number
- CN202310073880.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-18
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2043-01-18
AI Technical Summary
Existing intelligent understanding methods for power production operation scenarios ignore the inherent correlation features between operation elements, resulting in low accuracy in recognition and understanding.
A deep residual network is used to extract feature maps of job elements, and a region proposal network is used to obtain candidate region boxes. Visual background features and spatial topological relationship features are extracted through a global visual feature perception module and an encoder-decoder convolutional network, and feature fusion is performed to achieve scene understanding.
It improved the accuracy of identifying power operation scenarios and reduced the false alarm rate of production safety risks.
Smart Images
Figure CN116229048B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image processing, and relates to a production operation scene understanding method considering risk scene dependency. BACKGROUND
[0002] With the continuous expansion of the scale of the power system, safety accidents frequently occur in the operation process of the system. As another risk factor of illegal behavior in the power production site, the operation scene usually jointly acts with other risk factors such as protective tools and dynamic illegal behavior of personnel to cause the occurrence of production safety illegal cases, and part of the production safety risks show strong scene dependency. However, the operation scene of power production is complex and diverse, and the operation scene is usually composed of multiple elements such as personnel, equipment, protective tools and the production interaction relationship among the elements. How to fully excavate and utilize the characteristics of multiple elements such as personnel, equipment, protective tools and environment in the power production image and the interaction relationship among the elements is of great importance to reduce the false alarm rate of production safety risk and improve the level of fine prevention and control of safety risk.
[0003] Scene understanding is originally derived from the concept in the field of computer vision. Its basic meaning is to realize the perception of the real world by simulating human visual function through computer vision technology, which is an important means for computer technology to perceive the real world. However, with the wide application of deep learning in the field of computer vision, great breakthroughs have been made in target detection, semantic segmentation and other technologies, which have gradually become an independent subdivision direction in the field of computer vision. At present, the meaning of scene understanding in the field of computer vision mainly focuses on image description, which is a hot issue of the integration of machine vision technology and natural language processing and other fields. Compared with the scene understanding (image description) in the field of computer vision which describes and explains the whole image, the understanding of the safety operation scene of power production identifies the specific operation scene or process that the operation personnel is performing from the features of the image, and is rich in certain power professional knowledge, and the understanding task is higher. The current intelligent understanding method of the operation scene of power production mostly only considers the extraction of the visual features of the operation elements themselves, and ignores the internal correlation features, resulting in low recognition and understanding accuracy. SUMMARY
[0004] The purpose of the application is to provide a production operation scene understanding method considering risk scene dependency, which solves the problem of low recognition and understanding accuracy in the prior art.
[0005] The technical scheme adopted by the application is a production operation scene understanding method considering risk scene dependency, comprising the following steps:
[0006] Step 1, a work element feature map of the power work image is extracted by using a deep residual network, and a work element candidate region frame in the work element feature map is obtained by using a region proposal network;
[0007] Step 2, visual features and visual background features of the work element feature map are extracted by a global visual feature perception module, and the global visual features are obtained by fusing the visual features and the visual background features;
[0008] Step 3, each work element candidate region frame in the power work image is taken as an entity node, and an association relationship between a plurality of work element candidate region frames is taken as an edge, so as to obtain a power work scene undirected graph;
[0009] Step 4, a spatial topological relationship feature between each work element in the power work scene undirected graph is extracted by using an encoding-decoding convolutional network;
[0010] Step 5, the power work scene is understood by fusing the global visual features of the power work image and the spatial topological relationship features between the work elements.
[0011] The application also has the characteristics that:
[0012] The global visual feature perception module comprises an interested candidate region pooling layer and a multi-scale feature fusion attention layer, the interested candidate region pooling layer is used for extracting visual features of the work element feature map, and the multi-scale feature fusion attention layer is used for extracting visual background features of the work element feature map.
[0013] The processing process of the interested candidate region pooling layer is that the work element feature map and the work element candidate region frame are input, the work element candidate region frame is first mapped to the work element feature map to obtain a plurality of candidate regions with different sizes, then each candidate region is divided into the same number MxN parts, then each part is subjected to maximum pooling, and a target candidate region feature map with the same size, i.e., the visual features of the work element, is output.
[0014] The processing process of the multi-scale feature fusion attention layer is that the visual background features of the work element feature map are extracted by three asymmetric convolution branches with different scales, on each asymmetric convolution branch, first, a 1x1xk asymmetric convolution kernel is used to perform convolution operation on the work element feature map to obtain a first feature map, then a 1xkx1 asymmetric convolution kernel is used to perform convolution operation on the first feature map to obtain a second feature map, then a kx1x1 asymmetric convolution kernel is used to perform convolution operation on the second feature map to obtain a third feature map, after each convolution operation, the feature map is subjected to normalization operation, and finally, the outputs of the three asymmetric convolution branches are activated to obtain the visual background features.
[0015] In step 3, the association relationship between multiple candidate regions of task elements is defined as the spatial distance between two candidate regions of task elements, i.e., the minimum distance between two candidate regions of task elements. The minimum distance between two candidate regions of task elements is determined as follows: a. If the rectangles of two candidate regions of task elements intersect, the distance between these two rectangles is considered to be 0; b. If the rectangles of two candidate regions of task elements do not intersect, but have partially overlapping coordinates in the X or Y axis direction of the matrix, the minimum distance is the minimum parallel distance or perpendicular distance between the two rectangles; c. If the rectangles of two candidate regions of task elements do not intersect and have no overlapping coordinates in the X and Y axis directions, the minimum distance is the distance between the two vertices of the two rectangles that are closest to each other.
[0016] The minimum distance between two candidate regions of job elements is calculated as follows:
[0017]
[0018] In the above formula, x i y i , where are the x and y coordinates of the minimum distance point of the rectangle within the candidate region of the task element, and n is the dimension of the distance coordinate.
[0019] The encoder-decoder convolutional network consists of an encoder and a decoder. The encoder is composed of multiple graph convolutional encoding modules, and the decoder is composed of the same number of graph convolutional decoding modules as the encoder. Each encoding module includes a graph pooling layer and a graph convolutional layer. The graph pooling layer reduces the size of the undirected graph of the task scene to represent high-dimensional features of the scene. Each decoding module includes a graph inverse pooling layer and a graph convolutional layer. The graph inverse pooling layer restores the undirected graph of the task scene to an undirected topological structure with higher resolution. The graph convolutional layer performs first-order neighbor aggregation information of the undirected graph nodes in the scene through graph convolution operations based on the memory adjacency matrix, thereby extracting the topological structure information around the undirected graph nodes.
[0020] The fusion method in step 5 is as follows: First, the global visual features and spatial topological relationship features are matrix-scaled using point convolution to obtain global visual feature vectors and spatial topological relationship feature vectors of the same dimension. Then, the weighted feature fusion method is used to fuse the global visual feature vectors and spatial topological relationship feature vectors to obtain a fused vector. Finally, the softmax classification function is used to classify the fused vector for the scene, thus completing the power operation scene recognition.
[0021] The beneficial effects of this invention are as follows: This invention considers the production operation scenario understanding method based on risk scenario dependence. It uses the deep residual network ResNet50 to extract the operation element feature map of power operation images, which can break the layer limit of the convolutional network model to a certain extent and effectively improve the gradient diffusion phenomenon caused by the increase of the number of convolutional layers. It makes full use of the external visual features and internal correlation features of the elements such as workers, safety protection tools, power equipment or environment in the power operation scenario, which can more accurately characterize and explore the complex operation interaction relationship features between workers and various elements, improve the accuracy of recognition, and thus reduce the false alarm rate of production safety risks. Attached Figure Description
[0022] Figure 1 This is a flowchart of the production operation scenario understanding method that considers the dependence of risk scenarios in this invention;
[0023] Figure 2 This is a parameter diagram of the deep residual network architecture in the production operation scenario understanding method that considers the dependence of risk scenarios in this invention;
[0024] Figure 3 This is a structural diagram of the global visual feature perception module in the production operation scenario understanding method that considers the dependence of risk scenarios in this invention;
[0025] Figure 4 This is a schematic diagram illustrating the minimum distance between multiple candidate regions of operational elements in the production operation scenario understanding method that considers risk scenario dependence in this invention; Detailed Implementation
[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0027] Methods for understanding production operation scenarios that consider the dependence of risk scenarios, such as Figure 1 As shown, it includes the following steps:
[0028] Step 1: Extract the feature map of operation elements from the power operation image using the deep residual network ResNet50, and obtain the candidate region bounding box of the operation element in the feature map using the region proposal network. The architecture parameter diagram of the deep residual network ResNet50 is shown in the table below. Using the deep residual network ResNet50 to extract the feature map of operation elements from the power operation image can break the layer limit of the convolutional network model to a certain extent and effectively improve the gradient vanishing phenomenon caused by the increase of the number of convolutional layers.
[0029] Parameters of the ResNet50 Deep Residual Network Architecture
[0030]
[0031] Step 2: Extract the visual features and visual background features of the task element feature map through the global visual feature perception module, and fuse the visual features and visual background features to obtain the global visual features; for example... Figure 2 As shown, the global visual feature perception module includes a region of interest pooling layer and a multi-scale feature fusion attention layer. The region of interest pooling layer is used to extract the visual features of the task element feature map, and the multi-scale feature fusion attention layer extracts the visual background features of the task element feature map.
[0032] Specifically, the processing of the candidate region pooling layer is as follows: input the job element feature map and the job element candidate region box, first map the job element candidate region box onto the job element feature map to obtain multiple candidate regions of different sizes, then divide each candidate region into the same number of M×N parts (M=N=7 in this embodiment), then perform max pooling on each part, and output the target candidate region feature map of the same size, that is, the visual features of the job element.
[0033] The multi-scale feature fusion attention layer processes the following: Visual background features are extracted from the feature map of the task element using three asymmetric convolutional branches of different scales. In each asymmetric convolutional branch, a 1×1×k asymmetric convolution kernel is first used to convolve the feature map of the task element to obtain the first feature map. Then, a 1×k×1 asymmetric convolution kernel is used to convolve the first feature map to obtain the second feature map. Next, a k×1×1 asymmetric convolution kernel is used to convolve the second feature map to obtain the third feature map. k is the feature scale parameter; in this embodiment, k is 3, 5, and 7 respectively. After each convolutional operation, a normalized batch normalization (BN) operation is performed on the feature map. Finally, the outputs of the three asymmetric convolutional branches are activated to obtain the visual background features.
[0034] Step 3: Treat each candidate region bounding box of the power operation image as an entity node, and treat the relationship between multiple candidate region bounding boxes of the power operation as edges to obtain an undirected graph of the power operation scene.
[0035] Specifically, such as Figure 3 As shown, the association between multiple candidate regions of a task element is the spatial distance between two candidate regions, i.e., the minimum distance between two candidate regions. The minimum distance between two candidate regions is determined as follows: a. If the rectangles of two candidate regions intersect, then the distance between these two rectangles is considered to be 0. See Figure 3 a. If the rectangles of two candidate regions for a given task do not intersect, but partially overlap in the X or Y axis of the matrix, then the minimum distance is the minimum parallel or perpendicular distance between the two rectangles. (See...) Figure 3In D12, c, if the bounding boxes of the two candidate regions for job elements do not intersect and have no overlapping coordinates in the X and Y axes, then the minimum distance is the distance between the two vertices of the two bounding boxes that are closest to each other. Figure 3 D13.
[0036] The minimum distance between two candidate regions of job elements is calculated as follows:
[0037]
[0038] In the above formula, x i y i , , are the x and y coordinates of the minimum distance point of the rectangular bounding box of the candidate region of the task element, respectively, and n is the dimension of the distance coordinate. For a two-dimensional image, n = 2.
[0039] Step 4: Use an encoder-decoder convolutional network to extract the spatial topological relationship features between various operation elements in the undirected graph of the power operation scenario;
[0040] like Figure 4 As shown, the encoder-decoder convolutional network includes an encoder and a decoder. The encoder consists of multiple graph convolutional encoding modules, and the decoder consists of the same number of graph convolutional decoding modules as the encoder. Each encoding module includes a graph pooling layer (gPooling layer) and a graph convolutional layer (GCO layer). The graph pooling layer reduces the size of the undirected graph of the task scene to represent high-dimensional features of the scene. Each decoding module includes a graph unpooling layer (gUnpooling layer) and a graph convolutional layer. The graph unpooling layer restores the undirected graph of the task scene to an undirected topological structure with higher resolution. The graph convolutional layer performs first-order neighbor aggregation information of the undirected graph nodes in the scene through graph convolution operations based on the memory adjacency matrix, thereby extracting the topological structure information around the undirected graph nodes.
[0041] Step 5: Understand the power operation scene by fusing the global visual features of power operation images and the spatial topological relationship features between operation elements.
[0042] The specific fusion method is as follows: First, the global visual features and spatial topological relationship features are matrix-scaled using point convolution to obtain global visual feature vectors and spatial topological relationship feature vectors with the same dimension. Then, a weighted feature fusion method is used to fuse the global visual feature vectors and spatial topological relationship feature vectors to obtain a fused vector. Finally, the softmax classification function is used to classify the fused vector for the scene, thereby realizing the understanding and recognition of power operation scenes.
[0043] The weighting calculation in the weighted feature fusion method is as follows:
[0044] X fusion=LeakyReLU(βX) visual +(1-β)X spatial )
[0045] In the above formula, β is the weighting coefficient for weighted feature fusion. Feature vector weighting is achieved by adding corresponding elements of the two feature vectors. X fusion The fused feature vector of the two features, X visual Global visual feature vector, X spatial Spatial topological relationship feature vector;
[0046] Leaky ReLU is a nonlinear activation function unit, and its specific calculation formula is as follows:
[0047]
[0048] In the above formula, x is (βX) visual +(1-β)X spatial ).
[0049] Through the above methods, this invention considers the production operation scenario understanding method based on risk scenario dependence. It selects the deep residual network ResNet50 to extract the operation element feature map of power operation images, which can break the layer limit of convolutional network models to a certain extent and effectively improve the gradient vanishing phenomenon caused by the increase of convolutional layers. It makes full use of the external visual features and internal correlation features of the elements such as workers, safety protection tools, power equipment or environment in the power operation scenario, which can more accurately characterize and explore the complex operation interaction relationship features between workers and various elements, improve the accuracy of recognition, and thus reduce the false alarm rate of production safety risks.
Claims
1. A method for understanding production operation scenarios that considers the dependence of risk scenarios, characterized in that, Includes the following steps: Step 1: Use a deep residual network to extract the feature map of power operation elements from the image, and use a region proposal network to obtain the candidate region boxes of operation elements in the feature map of operation elements; Step 2: Extract the visual features and visual background features of the task elements in the task element feature map through the global visual feature perception module, and fuse the visual features and visual background features of the task elements to obtain the global visual features. The global visual feature perception module includes a region of interest pooling layer and a multi-scale feature fusion attention layer. The region of interest pooling layer is used to extract the visual features of the task elements in the task element feature map, and the multi-scale feature fusion attention layer extracts the visual background features of the task element feature map. The processing of the pooling layer for the candidate region of interest is as follows: input the feature map of the job element and the candidate region bounding box of the job element, first map the candidate region bounding box of the job element onto the feature map of the job element to obtain multiple candidate regions of different sizes, then divide each candidate region into the same number of M×N parts, then perform max pooling on each part, and output the target candidate region feature map of the same size, that is, the visual features of the job element. The processing procedure of the multi-scale feature fusion attention layer is as follows: visual background features are extracted from the feature map of the task element through three asymmetric convolutional branches of different scales. In each asymmetric convolutional branch, the feature map of the task element is first convolved with an asymmetric convolutional kernel of 1×1×k to obtain a first feature map. Then, the first feature map is convolved with an asymmetric convolutional kernel of 1×k×1 to obtain a second feature map. Next, the second feature map is convolved with an asymmetric convolutional kernel of k×1×1 to obtain a third feature map. After each convolutional operation, the feature map is normalized. Finally, the outputs of the three asymmetric convolutional branches are activated to obtain the visual background features. Step 3: Take each candidate region box of the power operation image as an entity node, and take the spatial distance between multiple candidate region boxes of the power operation as an edge to obtain an undirected graph of the power operation scene. In step 3, the association relationship between multiple candidate regions of the task elements is defined as the spatial distance between two candidate regions of the task elements, that is, the minimum distance between two candidate regions of the task elements. The minimum distance between two candidate regions of the task elements is determined as follows: a. If the rectangles of two candidate regions of the task elements intersect, the distance between the two rectangles is considered to be 0; b. If the rectangles of two candidate regions of the task elements do not intersect, but have partially overlapping coordinates in the X or Y axis direction of the matrix, the minimum distance is the minimum parallel distance or perpendicular distance between the two rectangles; c. If the rectangles of two candidate regions of the task elements do not intersect and have no overlapping coordinates in the X and Y axis directions, the minimum distance is the distance between the two vertices of the two rectangles that are closest to each other. The minimum distance between two candidate regions of job elements is calculated as follows: In the above formula, x i y i These are the x and y coordinates of the minimum distance point between the bounding boxes of the candidate regions for the task elements, respectively, where n is the dimension of the distance coordinates; Step 4: Use an encoder-decoder convolutional network to extract the spatial topological relationship features between each operation element in the undirected graph of the power operation scenario; Step 5: Understand the power operation scene by fusing the global visual features of the power operation image and the spatial topological relationship features between the operation elements.
2. The method for understanding production operation scenarios considering risk scenario dependence according to claim 1, characterized in that, The encoder-decoder convolutional network includes an encoder and a decoder. The encoder consists of multiple graph convolutional encoding modules, and the decoder consists of the same number of graph convolutional decoding modules as the encoder. Each encoding module includes a graph pooling layer and a graph convolutional layer. The graph pooling layer reduces the size of the undirected graph of the task scene to represent high-dimensional features of the scene. Each decoding module includes a graph depooling layer and a graph convolutional layer. The graph depooling layer restores the undirected graph of the task scene to an undirected topological structure with higher resolution. The graph convolutional layer performs first-order neighbor aggregation information of the undirected graph nodes in the scene through graph convolution operations based on the memory adjacency matrix, thereby extracting the topological structure information around the undirected graph nodes.
3. The method for understanding production operation scenarios considering risk scenario dependence according to claim 1, characterized in that, The fusion method in step 5 is as follows: First, the global visual features and spatial topological relationship features are matrix-scaled using point convolution to obtain global visual feature vectors and spatial topological relationship feature vectors with the same dimension. Then, a weighted feature fusion method is used to fuse the global visual feature vectors and spatial topological relationship feature vectors to obtain a fused vector. Finally, the softmax classification function is used to classify the fused vector for the scene, thus completing the power operation scene recognition.