A binocular stereo matching detection method and device based on channel attention mechanism

Abstract

The present invention proposes a binocular stereo matching detection method and device based on a channel attention mechanism. This method relies on existing power text data and substation image videos of live working taken by a binocular camera carried by an operator, and preprocesses the substation image videos; constructs a substation power knowledge graph based on the power text data and substation image video data; constructs a real-time binocular stereo matching model based on the attention mechanism to perform binocular stereo matching on the live image videos in the substation power knowledge graph, and finally calculates the depth distance according to the binocular ranging process to achieve the detection of the safe distance from live electricity. The present invention performs feature extraction and binocular stereo matching by integrating the attention mechanism, and finally calculates the depth distance according to the binocular ranging process to achieve the detection of the safe distance from live electricity.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and particularly to a binocular stereo matching detection method and device based on a channel attention mechanism. Background Art

[0002] With the development of computer vision binocular stereo matching technology, this technology has been gradually applied to State Grid power services to achieve the detection of fine-grained near-electric safety distances. Since electric shock accidents occur from time to time, and "preventing electric shock" is one of the most typical and common safety control requirements for preventing personal safety incidents. To improve the ability to prevent power grid safety risks, when operating personnel approach equipment of each voltage level, since the surrounding equipment is in a live state, to ensure the personal safety of the operating personnel, it is necessary to calculate the red line range of the safety distance according to the environment where the personnel and the tools, materials, etc. they hold are located. Currently, there are mainly two ways to judge the distance to live equipment. The first is to utilize the electric field coupling principle between the high-voltage electric field and the sensor, and the second is to use an ultrasonic rangefinder to directly measure the distance to the live equipment. However, the above two methods have limitations. The current substation scene elements are complex, and there are high requirements for the real-time information of live equipment and operating personnel.

[0003] Aiming at the problems of complex data types, large data volume in the substation business scenario data, and the inapplicability of traditional database solutions, a multi-modal data knowledge graph construction technology based on multi-modal sharing and fusion is studied; based on the constructed power knowledge graph, aiming at the disadvantages of low efficiency and poor intelligent means in substation manual inspection, a multi-stage binocular stereo matching detection technology based on a channel attention mechanism is studied. Relying on the substation power image multi-modal database, and performing preprocessing to obtain high-quality binocular images and video monitoring of live operations, through the fusion attention mechanism for feature extraction and binocular stereo matching, multi-scale image and video features are obtained, the matching accuracy of multi-stage objects of different scales is improved, and finally, the depth distance is calculated according to the binocular ranging process to achieve near-electric safety distance detection. Summary of the Invention

[0004] In order to solve the above deficiencies in the prior art, the purpose of the present invention is to provide a binocular stereo matching method and device based on a channel attention mechanism, study the multi-modal power image database technology, obtain binocular video monitoring of live operations and perform preprocessing, construct a power image knowledge graph, and perform binocular stereo matching through the fusion attention mechanism. Finally, the depth distance is calculated according to the binocular ranging process to achieve near-electric safety distance detection.

[0005] The present invention discloses a binocular stereo matching detection method based on a channel attention mechanism, including the following steps:

[0006] Step S1: Based on the existing power text data and the substation image and video of live working captured by the binocular cameras carried by the operators, preprocess the substation image and video.

[0007] Step S2: Construct a substation power knowledge graph according to the power text data and substation image and video data.

[0008] Step S3: Construct a real-time binocular stereo matching model based on the attention mechanism. Based on the substation power knowledge graph, use the attention mechanism to perform binocular stereo matching on the live image and video in the substation power knowledge graph. The real-time binocular stereo matching model based on the attention mechanism includes an attention module and a disparity optimization module.

[0009] The attention module relies on the video image data set in the substation power knowledge graph, uses the channel attention and spatial attention modules to enhance the features of the image, and uses the residual module to extract the features of the image frame. The output is the feature vector at each pixel in the downsampled image, and a feature map is obtained.

[0010] The disparity optimization module uses the obtained attention feature map and continues to perform calculations using two-dimensional convolutional blocks to reduce the network parameters during training, making the real-time binocular stereo matching model based on the attention mechanism more lightweight. It selects the method of gradually magnifying the disparity Figure 3 step sampling. First, two feature maps are input for convolutional operations, and then the prediction results obtained through the residual block are combined with the coarse-grained disparity prediction. Finally, a high-resolution feature disparity map is obtained, increasing the receptive field, obtaining the context information of the multi-scale image, improving the inference speed of the real-time binocular stereo matching model based on the attention mechanism, meeting the real-time requirements, and at the same time obtaining a high-quality disparity map.

[0011] Step S4: Near-electricity safety distance detection: Based on the obtained high-quality disparity map, restore the three-dimensional geometric information of the substation operators, use triangulation to calculate the depth information between the operators and the substation equipment in the image, form a three-dimensional point cloud, and finally realize the distance calculation between the operators and the substation equipment.

[0012] Further, the image preprocessing includes image correction, image denoising, and intelligent image and video analysis.

[0013] Further, for the image correction: decompose the power video monitoring into multiple image frames. For the image frames with obvious edges, use the correction algorithm based on contour extraction; for the image frames with unclear edges but neat arrangement, use the correction algorithm based on Hough line detection.

[0014] Further, for the image denoising: for the problem of unclear image frames and noise in the substation image and video, use the Gaussian filtering method for processing.

[0015] Furthermore, the intelligent image and video analysis: Based on the decomposed multi-frame images, model the foreground and background at the pixel level, use the probability density distribution of RGB pixels. When the foreground object does not change, generate a background model through continuous N frames, and then extract the images of the operating personnel in the picture who are in a moving state; At the same time, based on the obtained foreground images of the operating personnel, use pattern recognition technologies based on shape features and color features to identify specific objects. If it is detected that the distance between the operating personnel and the power equipment exceeds the set threshold, a warning of violation behavior will be issued.

[0016] Furthermore, construct the substation power knowledge graph as follows:

[0017] S21. First, perform knowledge extraction operations on text information such as the information of operating personnel in the substation, safety requirements at the operation site, and operating parameters of substation equipment, and realize the linkage of power cross-modal data;

[0018] Perform natural language processing, word segmentation, part-of-speech tagging, and syntactic analysis on the input power text data and substation image and video data. Select a learning-based method, use the long short-term memory network algorithm to automatically extract entity relationships from the power text data after sequence tagging, and use the k-means clustering algorithm to merge entities with the same meaning to achieve entity disambiguation. Finally, save the data in the form of triples;

[0019] Tag the substation image and video data, link the entities corresponding to the substation image and video data in the relational database through the numbers and names in the relational database, and realize the knowledge fusion under different modalities to form a power knowledge base;

[0020] S22. Experts review the power knowledge base that has completed knowledge fusion and modify and improve it according to the expert review opinions;

[0021] S23. Based on the improved substation power knowledge base, visualize the power knowledge base based on the Neo4j graph database to form a complete substation power knowledge graph.

[0022] Further preferably, the attention module first compresses the spatial dimension of the input feature map. After max pooling and average pooling operations, it then downsamples the image frames, uses two 3×3 convolutional kernels while keeping the number of channels at 32 to obtain the local channel attention cross-channel correlation. Through this channel attention, a channel attention feature map F1 is obtained. The attention feature map F1 is multiplied by the feature map to obtain the first fused feature map. For spatial attention, max pooling and average pooling operations are applied along the channel axis based on the first fused feature map, and then the resulting features are input into a convolutional operation. Finally, a spatial attention feature map F2 is obtained using an activation function. Then, the spatial attention feature map F2 is multiplied by the first fused feature map to obtain the final attention feature map.

[0023] The present invention also provides a binocular stereo matching detection device based on a channel attention mechanism, including a binocular camera for image and video acquisition, an image preprocessing module, a substation power knowledge graph module, a binocular stereo matching module, a proximity electrical safety distance calculation module, and an alarm module;

[0024] The substation power knowledge graph module constructs a substation power knowledge graph based on power text data and substation image and video data;

[0025] The binocular stereo matching module incorporates a real-time binocular stereo matching model based on the attention mechanism. The binocular stereo matching module uses the attention mechanism to perform binocular stereo matching on the live image and video in the substation power knowledge graph;

[0026] The proximity electrical safety distance calculation module restores the three-dimensional geometric information of the substation operators based on the obtained high-quality disparity map, calculates the depth information of the operators and substation equipment in the image using triangulation to form a three-dimensional point cloud, and finally realizes the calculation of the distance between the operators and substation equipment;

[0027] The alarm module notifies the operators of the calculation results of the proximity electrical safety distance calculation module.

[0028] The image preprocessing module includes an image correction module, an image denoising module, and an intelligent image and video analysis module.

[0029] The present invention has the following beneficial effects: relying on the substation power image multi-modal database, preprocessing to obtain high-quality live operation binocular image and video monitoring, constructing a power image knowledge graph, extracting features and performing binocular stereo matching through the fusion of the attention mechanism, and finally calculating the depth distance according to the binocular ranging process to achieve proximity electrical safety distance detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other features, objects, and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments read in conjunction with the accompanying drawings:

[0031] Figure 1 is a schematic flow chart of the present invention;

[0032] Figure 2 is a schematic flow chart of power image preprocessing;

[0033] Figure 3 is the construction process of the substation power knowledge graph library;

[0034] Figure 4 are the relevant knowledge included in the substation power knowledge graph library;

[0035] Figure 5 is a binocular stereo matching network structure based on the attention mechanism;

[0036] Figure 6 is a schematic flow chart of the near-electrical safety distance detection based on the attention mechanism;

[0037] Figure 7 is a real-time stereo matching flow chart. Specific implementation method

[0038] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the relevant invention and not for limiting the invention. Additionally, it should be noted that for the convenience of description, only the parts related to the invention are shown in the drawings.

[0039] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0040] As Figure 1 shown, an embodiment of the present invention discloses a binocular stereo matching detection method based on a channel attention mechanism, including the following steps:

[0041] Step S1, relying on the existing power text data and the substation image and video of live working taken by the binocular camera carried by the operator, perform image preprocessing on the substation image and video, including image correction, image denoising, and intelligent image and video analysis. The operation process of image preprocessing is as Figure 2 shown.

[0042] (1) Image correction. Decompose the power video monitoring into multiple image frames. Secondly, for the image frames with obvious edges, use a correction algorithm based on contour extraction; for the image frames with unclear edges but neat arrangement, use a correction algorithm based on Hough line detection.

[0043] (2) Image denoising. Aiming at the problems of unclear image frames and noise in substation image videos, the Gaussian filtering method is used to improve the clarity of image frames, increase the compression rate of substation image videos, and ensure the quality of substation image videos.

[0044] (3) Intelligent image and video analysis. Based on substation image videos, the movement of objects in the picture is detected through foreground extraction technology, and different behaviors are distinguished by using pattern recognition technology, such as wires, item leftovers, perimeters, etc. Targeted modeling is carried out on the objects to be monitored, so as to detect specific objects in substation image videos and related applications. First, according to the decomposed multi-frame image frames, the foreground and background pixels are modeled at the pixel level. Using the probability density distribution of RGB pixels, when the foreground object does not change, a background model is generated by modeling through continuous N frames, and then the image of the operating personnel in the moving state in the picture is extracted. At the same time, based on the obtained foreground operating personnel image, pattern recognition technologies based on shape features and color features are used to identify specific objects such as wires, safety helmets and other items. If it is detected that the distance between the operating personnel and the power equipment exceeds the set threshold, a warning of illegal behavior is given.

[0045] Step S2, construct a substation power knowledge graph according to power text data and substation image video data. The specific process is as Figure 3 shown:

[0046] S21. First, perform knowledge extraction operations on text information such as the information of operating personnel in the substation, safety requirements at the operation site, and operating parameters of substation equipment, and realize the link of power cross-modal data;

[0047] Perform natural language processing, word segmentation, part-of-speech tagging, and syntactic analysis on the input power text data and substation image video data. Select a learning-based method, and use the long short-term memory network algorithm to automatically extract entity relationships from the power text data after sequence tagging. And use the k-means clustering algorithm to merge entities with the same meaning to achieve entity disambiguation, and finally save the data in the form of triples, such as (power equipment, contains, overhead line), (overhead line, contains, pole and accessories), (pole and accessories, contains, insulator).

[0048] Based on the entity relationship data in the power field, perform cross-modal data linking. First, label the substation image video data, and link the entities corresponding to the substation image video data in the relational database through the numbers and names in the relational database to achieve knowledge fusion in different modalities and form a power knowledge base.

[0049] S22. Experts review the power knowledge base that has completed knowledge fusion and modify and improve it according to the expert review opinions; S23. Based on the improved substation power knowledge base, visualize the power knowledge base based on the Neo4j graph database to form a complete substation power knowledge graph. The substation power knowledge graph is as follows Figure 4 shown. The circles represent the substation knowledge point entities. In the figure, "contains" represents a relationship between entities. The substation power knowledge graph contains basic information of operators, safety requirements at the operation site, operation parameter information of substation equipment, 3D maps of substations, operation qualifications of operators, knowledge of substation equipment components, and picture and video data of other modalities corresponding to each part. The relationships between the constructed knowledge points are not limited to a single "contains" relationship.

[0050] Step S3: Construct a real-time binocular stereo matching model based on the attention mechanism. Based on the substation power knowledge graph, use the attention mechanism to perform binocular stereo matching on the live image and video in the substation power knowledge graph to increase the matching accuracy in weak texture areas;

[0051] The real-time binocular stereo matching model based on the attention mechanism in this embodiment consists of two main modules: an attention module and a disparity optimization module.

[0052] The network structure of the attention module is as follows Figure 5 shown. Relying on the video image data set in the substation power knowledge graph, use the channel attention and spatial attention modules to enhance the features of the image, and use 5 residual modules to extract the features of the image frame. The output is a 32-dimensional feature vector at each pixel in the downsampled image, obtaining a feature map. Different from the traditional end-to-end feature extraction method, the network introducing channel attention and spatial attention first compresses the spatial dimension of the input feature map, performs max pooling and average pooling operations, then downsamples the image frame, uses 2 3×3 convolutional kernels, keeps the number of channels at 32, obtains the local channel attention cross-channel correlation, and obtains the channel attention feature map F1 through this channel attention. The attention feature map F1 is multiplied by the feature map to obtain the first fusion feature map. The spatial attention then applies max pooling and average pooling operations along the channel axis on the basis of the first fusion feature map, and then inputs the obtained features for convolutional operations. Finally, the spatial attention feature map F2 is obtained using the activation function. Then, the spatial attention feature map F2 is multiplied by the first fusion feature map to obtain the final attention feature map. Through channel attention and spatial attention, features such as the edges, contours, and contrast of the image become clearer. The attention module establishes the correlation between different channels and between different positions in the same channel, enabling the real-time binocular stereo matching model based on the attention mechanism to give more attention weights in challenging areas and retain more details.

[0053] The network structure of the disparity optimization module is as followsFigure 6 As shown in the figure. Using the obtained attention feature map, continue to perform calculations using a two-dimensional convolutional block to reduce the network parameters during training, making the real-time binocular stereo matching model based on the attention mechanism more lightweight. Select to use the method of gradually magnifying the disparity Figure 3 step sampling. First, input two feature maps, perform convolutional operations, and then combine the prediction results obtained through the residual block with the coarse-grained disparity prediction. Finally, obtain a high-resolution feature disparity map, increase the receptive field, obtain the context information of the multi-scale image, improve the inference speed of the real-time binocular stereo matching model based on the attention mechanism, meet the real-time requirements, and at the same time obtain a high-quality disparity map.

[0054] Step S4, near-electricity safety distance detection. Based on the obtained high-quality disparity map, restore the three-dimensional geometric information of the substation operators, use triangulation to calculate the depth information between the operators and the substation equipment in the image, form a three-dimensional point cloud, and finally realize the distance calculation between the operators and the substation equipment. The real-time stereo matching process is as Figure 7 shown.

[0055] This embodiment also provides a binocular stereo matching detection device based on the channel attention mechanism, including a binocular camera for image and video acquisition, an image preprocessing module, a substation power knowledge graph module, a binocular stereo matching module, a near-electricity safety distance calculation module, and an alarm module;

[0056] The substation power knowledge graph module constructs a substation power knowledge graph according to power text data and substation image and video data;

[0057] The binocular stereo matching module is built with a real-time binocular stereo matching model based on the attention mechanism. The binocular stereo matching module uses the attention mechanism to perform binocular stereo matching on the live image and video in the substation power knowledge graph;

[0058] The near-electricity safety distance calculation module restores the three-dimensional geometric information of the substation operators based on the obtained high-quality disparity map, uses triangulation to calculate the depth information between the operators and the substation equipment in the image, forms a three-dimensional point cloud, and finally realizes the distance calculation between the operators and the substation equipment;

[0059] The alarm module notifies the operators of the calculation results of the near-electricity safety distance calculation module.

[0060] The image preprocessing module includes an image correction module, an image denoising module, and an intelligent image and video analysis module.

[0061] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

Claims

1. A binocular stereo matching detection method based on channel attention mechanism, characterized in that, it includes the following steps: S1: Relying on the existing power text data and the substation image video of live working taken by the binocular camera carried by the operator, preprocess the substation image video; S2: Construct a substation power knowledge graph according to the power text data and substation image video data; S3: Construct a real-time binocular stereo matching model based on the attention mechanism. Based on the substation power knowledge graph, use the attention mechanism to perform binocular stereo matching on the live image video in the substation power knowledge graph; the real-time binocular stereo matching model includes an attention module and a disparity optimization module; The attention module relies on the video image data set in the substation power knowledge graph, uses the channel attention and spatial attention modules to enhance the image features, and uses the residual module to extract the features of the image frame. The output is the feature vector at each pixel in the downsampled image, and an attention feature map is obtained; The disparity optimization module uses the obtained attention feature map and continues to calculate using a two-dimensional convolutional block to reduce the network parameters during training, making the real-time binocular stereo matching model more lightweight. It uses a three-step sampling method of gradually magnifying the disparity map. First, two feature maps are input for convolutional operations, and then the prediction result obtained through the residual block is combined with the coarse-grained disparity prediction. Finally, a high-resolution feature disparity map is obtained, increasing the receptive field, obtaining the context information of the multi-scale image, improving the inference speed of the real-time binocular stereo matching model to meet the real-time requirement, and at the same time obtaining a high-quality disparity map; S4: Near-electricity safety distance detection: Based on the obtained high-quality disparity map, restore the three-dimensional geometric information of the substation operator, use triangulation to calculate the depth information between the operator and the substation equipment in the image to form a three-dimensional point cloud, and finally realize the distance calculation between the operator and the substation equipment; Construct the substation power knowledge graph as follows: S21. First, perform knowledge extraction operations on the text information of the operator information, job site safety requirements, and substation equipment operation parameters in the substation, and realize the link of power cross-modal data; Perform natural language processing on the input power text data and substation image video data, including word segmentation, part-of-speech tagging, and syntactic analysis. Select a learning-based method, use the power text data after sequence annotation, and adopt the long short-term memory network algorithm to automatically extract entity relationships, and use the k-means clustering algorithm to merge entities with the same meaning to achieve entity disambiguation. Finally, save the data in the form of triples; Tag the substation image video data, and link the entities corresponding to the substation image video data in the relational database through the numbers and names in the relational database to realize knowledge fusion in different modalities and form a substation power knowledge base; S22. Experts review the substation power knowledge base and modify and improve it according to the expert review opinions; S23. Based on the improved substation power knowledge base, visualize the power knowledge base based on the Neo4j graph database to form a complete substation power knowledge graph.

2. A binocular stereo matching detection method based on channel attention mechanism according to claim 1, characterized in that, the image preprocessing includes image rectification, image denoising, and intelligent image and video analysis.

3. A binocular stereo matching detection method based on channel attention mechanism according to claim 2, characterized in that, the image rectification: decomposes the power video monitoring into multiple image frames, and for the image frames with obvious edges, uses a rectification algorithm based on contour extraction; for the image frames with unclear edges but neat arrangement, uses a rectification algorithm based on Hough line detection.

4. A binocular stereo matching detection method based on channel attention mechanism according to claim 2, characterized in that, the image denoising: for the problem of unclear and noisy image frames in the substation image video, uses the method of Gaussian filtering to process.

5. A binocular stereo matching detection method based on channel attention mechanism according to claim 2, characterized in that, the intelligent image and video analysis: according to the decomposed multiple image frames, models the foreground and background at the pixel level, uses the probability density distribution of RGB pixels, when the foreground object does not change, generates a background model through continuous N frames, and then extracts the image of the operating personnel in the picture in a moving state; at the same time, based on the obtained foreground operating personnel image, uses pattern recognition technologies based on shape features and color features to identify specific objects, and if it is detected that the distance between the operating personnel and the power equipment exceeds the set threshold, issues a warning for violation behaviors.

6. A binocular stereo matching detection method based on channel attention mechanism according to claim 1, characterized in that, the attention module first compresses the spatial dimension of the input feature map, through max pooling and average pooling operations, then downsamples the image frame, uses 2 3×3 convolutional kernels, keeps the number of channels as 32, obtains the local channel attention cross-channel correlation, obtains the channel attention feature map F1 through this channel attention, multiplies the channel attention feature map F1 with the feature map to get the first fusion feature map, the spatial attention then applies max pooling and average pooling operations along the channel axis on the basis of the first fusion feature map, then inputs the obtained features for convolutional operations, and finally uses the activation function to get the spatial attention feature map F2, and then multiplies the spatial attention feature map F2 with the first fusion feature map to get the final attention feature map.

7. A binocular stereo matching detection device based on channel attention mechanism, using the detection method described in any one of claims 1-6, characterized in that, the detection device includes a binocular camera for image and video acquisition, an image preprocessing module, a substation power knowledge graph module, a binocular stereo matching module, a proximity electrical safety distance calculation module, and an alarm module; the substation power knowledge graph module constructs a substation power knowledge graph according to power text data and substation image and video data; The binocular stereo matching module incorporates a real-time binocular stereo matching model based on the attention mechanism. The binocular stereo matching module uses the attention mechanism to perform binocular stereo matching on the live image videos in the substation power knowledge graph; The near-electrical safety distance calculation module restores the three-dimensional geometric information of the substation operators based on the obtained high-quality disparity map, calculates the depth information between the operators and the substation equipment in the image using triangulation, forms a three-dimensional point cloud, and finally realizes the distance calculation between the operators and the substation equipment; The alarm module notifies the operators of the calculation results of the near-electrical safety distance calculation module.

8. The binocular stereo matching detection device based on the channel attention mechanism according to claim 7, characterized in that, The image preprocessing module includes an image correction module, an image denoising module, and an intelligent image video analysis module.

Images