Power grid equipment detection method and device based on multi-modal feature fusion and electronic equipment
Through the multi-modal feature fusion power grid equipment detection method, multiple network models are used to extract and fusion features, combined with AIFI module and YOLOv8 model for detection, solving the problems of time consumption and misjudgment of traditional power inspections, and improving the detection efficiency and accuracy.
Patent Information
- Application Number
- CN202411025027.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional manual power inspection methods are time-consuming and labor-intensive, and are prone to subjective misjudgment and omissions.
The multimodal feature fusion method is used to obtain the original data of the power grid, perform target annotation and normalization processing, and combine multiple network models to extract and fusion the multimodal data, using AIFI module and YOLOv8 model for Classification and detection.
It improves the richness and diversity of data, enhances the robustness of detection results, reduces the complexity of model parameters and calculation volume, and solves the time-consuming and misjudgment problems of traditional methods.
Smart Images

Figure CN118918437A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of power grid equipment detection, and in particular, to a multi-modal feature fusion power grid equipment detection method, device, computer-readable storage medium and electronic device. Background Art
[0002] As an important part of the power system, power equipment undertakes the key tasks of transmitting, distributing and converting electric energy. These devices include transformers, switchgear, generators, transmission lines, etc. Their normal operation is crucial to the reliability and stability of the power system. However, open-air power equipment faces many challenges in harsh environments for a long time, such as equipment aging, failure and damage. In order to ensure the stable operation of the power system, it is necessary to regularly detect and identify the points of power grid equipment according to the inspection route, and to diagnose and maintain the inspection equipment.
[0003] The traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgments and omissions. Summary of the invention
[0004] The main purpose of this application is to provide a multi-modal feature fusion power grid equipment detection method, device, computer-readable storage medium and electronic device, so as to at least solve the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions.
[0005] In order to achieve the above object, according to one aspect of the present application, a multi-modal feature fusion power grid equipment detection method is provided, the method comprising:
[0006] Acquire raw data of the power grid, and perform target labeling and normalization processing on the raw data of the power grid to obtain multimodal data, wherein the raw data of the power grid includes image data of the target object collected by a camera, temperature data of the target object collected by an infrared device, and point cloud data of the target object collected by a radar sensor;
[0007] Using a variety of network models to extract features from the multimodal data, and respectively obtain feature information of the target object, the feature information including image features, temperature features and morphological features;
[0008] Based on the features of different modalities, feature fusion is performed on the feature information of the target object to obtain fused features;
[0009] The AIFI module and YOLOv8 model are used to classify and detect the fusion features to obtain the final detection results.
[0010] Optionally, target labeling and normalization processing are performed on the raw data of the power grid to obtain multimodal data, including:
[0011] The boundary position of the target area in the original data of the power grid is marked, and the image data of different modes in the original data of the power grid are size normalized to obtain multi-modal data.
[0012] Optionally, in the process of performing a size normalization operation on image data of different modes in the original data of the power grid to obtain multimodal data, the method further includes:
[0013] Mosaic image enhancement technology is used to perform multimodal image enhancement processing on image data of different modes in the original data of the power grid.
[0014] Optionally, multiple network models are used to extract features from the multimodal data to obtain feature information of the target object, including:
[0015] Using AKConv convolution technology to perform feature extraction processing on the feature information of the target object to obtain initial features, and using YOLOv8 target detection algorithm to identify the initial features to obtain the image features;
[0016] Using the FCOS model to perform feature extraction processing on the feature information of the target object to obtain the temperature feature;
[0017] The GK means algorithm is used to perform feature extraction processing on the feature information of the target object to obtain the morphological features.
[0018] Optionally, performing feature fusion on the feature information of the target object based on features of different modalities to obtain fused features includes:
[0019] The SENet technology is used to perform feature fusion on the feature information of the target object to obtain the fused feature.
[0020] Optionally, in the process of using SENet technology to perform feature fusion on the feature information of the target object to obtain the fused feature, the method further includes:
[0021] According to A=σ(ω 2 δ(ω 1 ,R)), determine the weight of the feature map of each channel;
[0022] Wherein, A is the weight of the feature map, R is a one-dimensional real number obtained after performing a global average pooling operation on the feature information of the target object, ω 1 and ω 2 is the fully connected layer, σ is the sigmoid activation function, and δ is the ReLU activation function.
[0023] Optionally, the AIFI module and the YOLOv8 model are used to classify and detect the fusion features to obtain a final detection result, including:
[0024] The AIFI module is used to classify the fusion features based on the self-attention mechanism to obtain a multi-scale feature fusion result;
[0025] The YOLOv8 model is used to perform detection processing on the multi-scale feature fusion result to obtain the final detection result.
[0026] According to another aspect of the present application, a multi-modal feature fusion power grid equipment detection device is provided, the device comprising:
[0027] an acquisition unit, used to acquire raw data of a power grid, and to perform target labeling and normalization processing on the raw data of the power grid to obtain multimodal data, wherein the raw data of the power grid includes image data of a target object acquired by a camera, temperature data of the target object acquired by an infrared device, and point cloud data of the target object acquired by a radar sensor;
[0028] A first processing unit, configured to extract features from the multimodal data using a plurality of network models, and obtain feature information of the target object, wherein the feature information includes image features, temperature features, and morphological features;
[0029] A second processing unit, configured to perform feature fusion on the feature information of the target object based on features of different modalities to obtain fused features;
[0030] The third processing unit is used to classify and detect the fusion features using the AIFI module and the YOLOv8 model to obtain a final detection result.
[0031] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the multi-modal feature fusion power grid equipment detection methods.
[0032] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the multimodal feature fusion power grid equipment detection methods.
[0033] By applying the technical solution of the present application, a plurality of network models are used to extract features from the multimodal data, and feature information of the target object is obtained respectively, wherein the feature information includes image features, temperature features and morphological features; feature fusion is performed on the feature information of the target object based on features of different modalities to obtain fused features; an AIFI module and a YOLOv8 model are used to classify and detect the fused features to obtain final detection results, thereby improving the richness and diversity of the collected data and the robustness of the detection results. By using the AIFI module to form a lightweight YOLOv8 model, the complexity of model parameters and calculation amount is reduced, thereby solving the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0035] Figure 1 A schematic flow chart of a multi-modal feature fusion power grid equipment detection method provided according to an embodiment of the present application is shown;
[0036] Figure 2 A schematic diagram showing how to extract features from multimodal data using multiple network models to obtain feature information of target objects according to an embodiment of the present application;
[0037] Figure 3 A schematic flow chart of another multi-modal feature fusion power grid equipment detection method provided according to an embodiment of the present application is shown;
[0038] Figure 4 A structural block diagram of a multi-modal feature fusion power grid equipment detection device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0042] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:
[0043] Mosaic image enhancement is a technique used to improve image quality and visual effects. It usually involves segmenting the image and then performing different processing on each segmented block, such as enhancing contrast, sharpening edges, adjusting brightness, etc. In this way, the image can be made clearer and deeper while retaining the details and features of the original image. Mosaic image enhancement is often used in digital image processing, computer vision, and image analysis.
[0044] The YOLOv8 target detection algorithm is a target detection algorithm based on deep learning. It is the latest version of the YOLO series of target detection algorithms with faster speed and higher accuracy.
[0045] AKConv convolution is a special convolution operation that takes into account the spatial correlation of the input image during the convolution calculation process and can better capture the feature information in the image.
[0046] The FCOS model is a target detection model. The FCOS model based on sparse convolution uses sparse convolution to reduce the amount of calculation and the number of parameters while maintaining high detection accuracy.
[0047] The GK means method is a clustering algorithm, which is an improved version of the traditional K-means algorithm. It optimizes the clustering results by introducing graph structure information and is suitable for clustering tasks of graph data.
[0048] SENet (Squeeze-and-Excitation Network) is a deep learning network structure for image classification tasks, which aims to improve the performance of the network on specific tasks. The main idea of SENet is to introduce a "Squeeze-and-Excitation" block in each network module to adaptively learn the correlation between feature channels and weight the features of different channels according to these correlations, thereby improving the representation ability of the network. By introducing the SENet module, the network can more effectively utilize the information between different channels, thereby improving the performance and generalization ability of the network. SENet has achieved good results in multiple image classification tasks and has become an important technological breakthrough in the field of deep learning.
[0049] Anchor Box is a bounding box used to predict the position and size of a target in target detection. In the task of target detection, Anchor Box is usually used to generate multiple preset bounding boxes in order to detect targets of different sizes and scales. Anchor Box can be defined at different positions, sizes, and scales, and used to match targets in the image to determine the position and size of the target. Anchor Box is widely used in some deep learning-based target detection methods, such as Faster R-CNN, SSD (Single Shot MultiBox Detector), etc.
[0050] As introduced in the background technology, as an important part of the power system, power equipment undertakes the key tasks of transmitting, distributing and converting electric energy. These devices include transformers, switchgear, generators, transmission lines, etc. Their normal operation is crucial to the reliability and stability of the power system. However, open-air power equipment faces many challenges in harsh environments for a long time, such as equipment aging, failure and damage. In order to ensure the stable operation of the power system, it is necessary to accurately detect and identify the points of the power grid equipment according to the inspection route on a regular basis, and to diagnose and maintain the inspection equipment. In order to solve the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions, the embodiments of the present application provide a multi-modal feature fusion power grid equipment detection method, device, computer-readable storage medium and electronic device.
[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0052] In this embodiment, a multimodal feature fusion power grid equipment detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0053] Figure 1 FIG. 1 is a flow chart of a multi-modal feature fusion power grid equipment detection method provided according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0054] Step S101, acquiring raw data of a power grid, and performing target labeling and normalization processing on the raw data of the power grid to obtain multimodal data, wherein the raw data of the power grid includes image data of a target object collected by a camera, temperature data of the target object collected by an infrared device, and point cloud data of the target object collected by a radar sensor;
[0055] Step S101 performs target labeling and normalization processing on the above-mentioned original data of the power grid to obtain multimodal data, including:
[0056] The boundary position of the target area in the above-mentioned original data of the power grid is marked, and the image data of different modes in the above-mentioned original data of the power grid are resized to obtain multi-modal data.
[0057] Specifically, multimodal data of RGB images, temperature distribution images and point cloud information images of the target object are collected, the boundary positions of the target areas in the data are annotated, and the image data of different modes are resized and normalized. In order to enhance the diversity of the data, Mosaic image enhancement technology is used for multimodal image enhancement processing.
[0058] In the process of performing a size normalization operation on the image data of different modes in the above-mentioned original data of the power grid to obtain multi-modal data, the above-mentioned method further includes:
[0059] Mosaic image enhancement technology is used to perform multimodal image enhancement processing on image data of different modes in the above-mentioned power grid original data.
[0060] Specifically, in order to enhance the diversity of data, Mosaic image enhancement technology is used to perform multimodal image enhancement processing.
[0061] Step S102, using a variety of network models to perform feature extraction on the multimodal data to obtain feature information of the target object, wherein the feature information includes image features, temperature features and morphological features;
[0062] Step S102, i.e., using a variety of network models to extract features from the multimodal data to obtain feature information of the target object, including:
[0063] The feature information of the target object is extracted by using AKConv convolution technology to obtain initial features, and the initial features are identified by using YOLOv8 target detection algorithm to obtain the image features.
[0064] Using the FCOS model to perform feature extraction processing on the feature information of the target object to obtain the temperature feature;
[0065] The GK means algorithm is used to perform feature extraction processing on the feature information of the target object to obtain the morphological features.
[0066] Specifically, Figure 2 As shown in the figure, the improved YOLOv8 target detection algorithm is used to identify the power grid equipment image. On the basis of the original model, considering the problem that the target objects in the power grid equipment image have large size differences, AKConv convolution is used to extract image features; usually, there are a lot of background data in infrared images, and the gray value is close to or equal to zero. Therefore, the infrared temperature distribution image of the target object is extracted based on the sparse convolution FCOS model, which reflects the temperature distribution and reduces the calculation amount of the model; the front-end information collected by the millimeter wave sensor is processed based on Doppler to obtain the distance and speed of the target object, and the peak data corresponding to the target object is processed into angle information to form the point cloud data of the target object. When extracting features from radar data, noise interference is considered, so the GK means algorithm is proposed to cluster the feature vectors and remove abnormal noise according to the Gaussian distribution.
[0067] In the feature extraction stage, image features, temperature features and point cloud features are represented as P_img, P_temp and P_shape respectively, and the fused feature representation is obtained by splicing: U = Contact (P_img, P_temp, P_shape), where U is the fused feature.
[0068] Input U into SENet and get
[0069] Among them, U c is the feature processed by the cth convolution kernel, V c is the cth convolution kernel, X i is the i-th input of SENet, is a two-dimensional convolution kernel, and X is the input of SENet.
[0070] Step S103, performing feature fusion on the feature information of the target object based on features of different modalities to obtain fused features;
[0071] Step S103, that is, fusing the feature information of the target object based on features of different modalities to obtain fused features, includes:
[0072] The SENet technology is used to perform feature fusion on the feature information of the target object to obtain the fused feature.
[0073] Specifically, when processed by SENet, first, the fusion features are subjected to global average pooling to obtain a one-dimensional real number that matches the number of input feature channels, and the real number has the receptive field of the entire channel, where the element R in the real number c It can be expressed as:
[0074]
[0075] Among them, R c is a one-dimensional real number obtained after average pooling transformation, F sq is the global average pooling, W is the width of the feature map, H is the height of the feature map, and U c (w,h) is processed by the cth convolution kernel.
[0076] The compressed real numbers autonomously obtain the importance of different elements through the activation function and learn the relationship between channels. In the algorithm iteration process, the weights of effective channels are continuously enhanced and the weights of irrelevant channels are reduced. The mathematical relationship is as follows:
[0077] A=F ex (R,ω)=σ(g(R,ω))=σ(ω 2 δ(ω 1 ,R));
[0078] Where A is the weight of the feature map, Fex() is an activation function that stimulates the importance of each real number, g() is the convolution operation of the input feature map z and the ω weight, R is the one-dimensional feature real number obtained after the global average pooling operation on the feature information of the target object, ω is the weight matrix, ω 1 and ω 2 are two fully connected layers, σ is the sigmoid activation function, and δ is the ReLU activation function.
[0079] At this time, the output assigns different importance weights to different channels, and acts on the original feature channel based on the product relationship. The adjusted feature map is:
[0080]
[0081] in, and F scale is the feature map, A c ·U c is the product of the corresponding channels.
[0082] In one embodiment of the present application, in the process of using SENet technology to fuse the feature information of the target object to obtain the fused feature, the method further includes:
[0083] According to A=σ(ω 2 δ(ω 1 ,R)), determine the weight of the feature map of each channel;
[0084] Where A is the weight of the feature map, R is a one-dimensional real number obtained by performing a global average pooling operation on the feature information of the target object, and ω 1 and ω 2 is a fully connected layer, σ is a sigmoid activation function, and δ is a ReLU activation function.
[0085] Step S104, using the AIFI module and the YOLOv8 model to classify and detect the above fusion features to obtain the final detection result.
[0086] In the above steps, by adopting a variety of network models to extract features from the above multimodal data, characteristic information of the above target objects is obtained respectively, and the above characteristic information includes image features, temperature features and morphological features; the above characteristic information of the above target objects is fused based on the features of different modes to obtain fused features; the AIFI module and the YOLOv8 model are used to classify and detect the above fused features to obtain the final detection results, thereby improving the richness and diversity of the collected data, and also improving the robustness of the detection results. By adopting the AIFI module to form a lightweight YOLOv8 model, the complexity of model parameters and calculation amount is reduced, thereby solving the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions.
[0087] Step S104, i.e., using the AIFI module and the YOLOv8 model to classify and detect the above fusion features, to obtain the final detection results, including:
[0088] The AIFI module is used to classify the above fusion features based on the self-attention mechanism to obtain the multi-scale feature fusion result;
[0089] The YOLOv8 model is used to detect the above multi-scale feature fusion results to obtain the above final detection results.
[0090] Specifically, the AIFI module processes high-level features and internal scale interactions in the image based on the self-attention mechanism, so that the model can take into account other relevant parts of the data when processing a specific part of the data, and focus on high-level image features, thereby more effectively capturing effective information; after the fused features are input into the YOLOv8 detection model, the target box is matched and adjusted by applying the Anchor Box, the feature map is processed and enhanced, and multi-scale feature fusion is performed based on the AIFI module, and finally the target detection result is generated through the prediction layer.
[0091] With the rapid development of information technology and visual technology, the monitoring and maintenance of power equipment are gradually moving towards automation and intelligence. Through drone inspections or the deployment of monitoring devices on power equipment, real-time collection of operating data and status information is carried out. These data are modeled and processed through image processing and machine learning algorithms to achieve monitoring and evaluation of equipment status and performance. At the same time, power equipment is widely distributed and has different characteristics, and the detection and identification of power grid equipment requires timely response so that maintenance and management measures can be taken in a timely manner. This places high demands on the lightweight of the network model and the accuracy of target recognition, so as to meet the real-time and robustness requirements that traditional methods cannot meet.
[0092] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the multimodal feature fusion power grid equipment detection method of the present application will be described in detail below in combination with specific embodiments.
[0093] This embodiment relates to a specific multi-modal feature fusion power grid equipment detection method, such as Figure 3 As shown, the following steps are included:
[0094] Step S1: acquiring the original data of the power grid, marking the boundary position of the target area in the original data of the power grid, and performing a size normalization operation on the image data of different modes in the original data of the power grid to obtain multimodal data. The original data of the power grid includes the image data of the target object collected by the camera, the temperature data of the target object collected by the infrared device, and the point cloud data of the target object collected by the radar sensor;
[0095] Step S2: Use AKConv convolution technology to extract the feature information of the target object to obtain initial features, and use YOLOv8 target detection algorithm to identify the initial features to obtain image features; use FCOS model to extract the feature information of the target object to obtain temperature features; use GK means algorithm to extract the feature information of the target object to obtain morphological features, and the feature information includes image features, temperature features and morphological features;
[0096] Step S3: Using SENet technology to fuse the feature information of the target object to obtain fused features;
[0097] Step S4: Using the AIFI module to classify the fused features based on the self-attention mechanism to obtain the multi-scale feature fusion result;
[0098] Step S5: Use the YOLOv8 model to detect the multi-scale feature fusion result to obtain the final detection result.
[0099] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0100] The embodiment of the present application also provides a power grid equipment detection device with multimodal feature fusion. It should be noted that the power grid equipment detection device with multimodal feature fusion in the embodiment of the present application can be used to execute the power grid equipment detection method for multimodal feature fusion provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0101] The following is an introduction to the multi-modal feature fusion power grid equipment detection device provided in an embodiment of the present application.
[0102] Figure 4 is a structural block diagram of a multi-modal feature fusion power grid equipment detection device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:
[0103] The acquisition unit 41 is used to acquire the original data of the power grid, and perform target labeling and normalization processing on the original data of the power grid to obtain multimodal data, wherein the original data of the power grid includes image data of the target object acquired by the camera, temperature data of the target object acquired by the infrared device, and point cloud data of the target object acquired by the radar sensor;
[0104] A first processing unit 42 is used to extract features from the multimodal data using a variety of network models to obtain feature information of the target object, where the feature information includes image features, temperature features, and morphological features;
[0105] A second processing unit 43 is used to perform feature fusion on the feature information of the target object based on features of different modalities to obtain fused features;
[0106] The third processing unit 44 is used to classify and detect the above fusion features using the AIFI module and the YOLOv8 model to obtain a final detection result.
[0107] In the above-mentioned device, by adopting a variety of network models to extract features from the above-mentioned multimodal data, characteristic information of the above-mentioned target object is obtained respectively, and the above-mentioned characteristic information includes image features, temperature features and morphological features; the above-mentioned characteristic information of the above-mentioned target object is feature fused based on the features of different modes to obtain fused features; the AIFI module and the YOLOv8 model are used to classify and detect the above-mentioned fused features to obtain the final detection results, thereby improving the richness and diversity of the collected data, and also improving the robustness of the detection results. By adopting the AIFI module to form a lightweight YOLOv8 model, the complexity of model parameters and calculation amount is reduced, thereby solving the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions.
[0108] In one embodiment of the present application, the acquisition unit includes a first processing module,
[0109] The first processing module is used to mark the boundary position of the target area in the above-mentioned original power grid data, and perform size normalization operation on the image data of different modes in the above-mentioned original power grid data to obtain multi-modal data.
[0110] In one embodiment of the present application, the first processing module includes a processing submodule, and in the process of performing a size normalization operation on the image data of different modes in the original power grid data to obtain multimodal data, the method further includes:
[0111] The processing submodule is used to perform multi-modal image enhancement processing on image data of different modes in the above-mentioned original data of the power grid by using Mosaic image enhancement technology.
[0112] In one embodiment of the present application, the first processing unit includes a second processing module, a third processing module and a fourth processing module.
[0113] The second processing module is used to use AKConv convolution technology to perform feature extraction processing on the feature information of the target object to obtain initial features, and use YOLOv8 target detection algorithm to identify the initial features to obtain the image features;
[0114] The third processing module is used to perform feature extraction processing on the feature information of the target object using the FCOS model to obtain the temperature feature;
[0115] The fourth processing module is used to perform feature extraction processing on the feature information of the target object by using the GK means algorithm to obtain the morphological features.
[0116] In one embodiment of the present application, the second processing unit includes a fifth processing module,
[0117] The fifth processing module is used to use SENet technology to perform feature fusion on the feature information of the target object to obtain the fused feature.
[0118] In one embodiment of the present application, the fifth processing module includes a determination submodule, in the process of using SENet technology to perform feature fusion on the feature information of the target object to obtain the fusion feature,
[0119] Determine the submodule for A=σ(ω 2 δ(ω 1 ,R)), determine the weight of the feature map of each channel;
[0120] Where A is the weight of the feature map, R is a one-dimensional real number obtained by performing a global average pooling operation on the feature information of the target object, and ω 1 and ω 2 is the fully connected layer, σ is the sigmoid activation function, and δ is the ReLU activation function.
[0121] In one embodiment of the present application, the third processing unit includes a sixth processing module and a seventh processing module.
[0122] The sixth processing module is used to classify and process the above fusion features based on the self-attention mechanism using the AIFI module to obtain a multi-scale feature fusion result;
[0123] The seventh processing module is used to use the YOLOv8 model to perform detection processing on the above-mentioned multi-scale feature fusion result to obtain the above-mentioned final detection result.
[0124] The above-mentioned multimodal feature fusion power grid equipment detection device includes a processor and a memory, and the above-mentioned acquisition unit, the first processing unit, the second processing unit and the third processing unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in the form of any combination.
[0125] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omission.
[0126] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0127] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the power grid equipment detection method of multimodal feature fusion.
[0128] An embodiment of the present invention provides a processor, which is used to run a program, wherein the power grid equipment detection method based on multimodal feature fusion is executed when the program is running.
[0129] The embodiment of the present invention provides a device, the device includes a processor, a memory and a program stored in the memory and can be run on the processor, and the processor implements at least the following steps when executing the program: obtaining the original data of the power grid, and performing target labeling and normalization processing on the original data of the power grid to obtain multimodal data, the original data of the power grid includes image data of the target object collected by the camera, temperature data of the target object collected by the infrared device, and point cloud data of the target object collected by the radar sensor; using multiple network models to extract features from the multimodal data to obtain feature information of the target object, respectively, the feature information includes image features, temperature features and morphological features; based on the features of different modalities, the feature information of the target object is feature fused to obtain fused features; using AIFI module and YOLOv8 model to classify and detect the fused features to obtain the final detection results. The device in this article can be a server, PC, PAD, mobile phone, etc.
[0130] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes at least the following method steps: acquiring raw data of a power grid, and performing target annotation and normalization processing on the raw data of the power grid to obtain multimodal data, wherein the raw data of the power grid includes image data of a target object collected by a camera, temperature data of the target object collected by an infrared device, and point cloud data of the target object collected by a radar sensor; using a variety of network models to perform feature extraction on the multimodal data to obtain feature information of the target object, respectively, and the feature information includes image features, temperature features, and morphological features; performing feature fusion on the feature information of the target object based on features of different modalities to obtain fused features; and using an AIFI module and a YOLOv8 model to classify and detect the fused features to obtain final detection results.
[0131] The present application also provides an electronic device, the electronic device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any of the above-mentioned multimodal feature fusion power grid equipment detection methods. By using a variety of network models to extract features from the above-mentioned multimodal data, the feature information of the above-mentioned target object is obtained respectively, and the above-mentioned feature information includes image features, temperature features and morphological features; based on the features of different modes, the above-mentioned feature information of the above-mentioned target object is feature fused to obtain fusion features; the AIFI module and the YOLOv8 model are used to classify and detect the above-mentioned fusion features to obtain the final detection results, thereby improving the richness and diversity of the collected data, and also improving the robustness of the detection results. By using the AIFI module to form a lightweight YOLOv8 model, the complexity of model parameters and calculations is reduced, thereby solving the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions.
[0132] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0138] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0139] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0140] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0141] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:
[0142] 1) The multimodal feature fusion power grid equipment detection method of the present application uses a variety of network models to extract features from the multimodal data to obtain feature information of the target object, including image features, temperature features and morphological features; based on the features of different modes, the feature information of the target object is fused to obtain fused features; the AIFI module and the YOLOv8 model are used to classify and detect the fused features to obtain the final detection results, thereby improving the richness and diversity of the collected data and the robustness of the detection results. The AIFI module is used to form a lightweight YOLOv8 model to reduce the complexity of model parameters and calculations, thereby solving the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions.
[0143] 2) The multimodal feature fusion power grid equipment detection device of the present application uses a variety of network models to extract features from the above-mentioned multimodal data to obtain feature information of the above-mentioned target objects, including image features, temperature features and morphological features; based on the features of different modes, the feature information of the above-mentioned target objects is feature fused to obtain fused features; the AIFI module and the YOLOv8 model are used to classify and detect the above-mentioned fused features to obtain the final detection results, thereby improving the richness and diversity of the collected data, and also improving the robustness of the detection results. By using the AIFI module to form a lightweight YOLOv8 model, the complexity of model parameters and calculation amount is reduced, thereby solving the problem that the traditional manual power inspection method is not only time-consuming and labor-intensive, but also prone to subjective misjudgment and omissions.
[0144] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A multi-modal feature fusion power grid equipment detection method, characterized in that: include: Acquire raw data of the power grid, and perform target labeling and normalization processing on the raw data of the power grid to obtain multimodal data, wherein the raw data of the power grid includes image data of the target object collected by a camera, temperature data of the target object collected by an infrared device, and point cloud data of the target object collected by a radar sensor; Using a variety of network models to extract features from the multimodal data, and respectively obtain feature information of the target object, the feature information including image features, temperature features and morphological features; Based on the features of different modalities, feature fusion is performed on the feature information of the target object to obtain fused features; The AIFI module and YOLOv8 model are used to classify and detect the fusion features to obtain the final detection results.
2. The method according to claim 1, characterized in that The raw data of the power grid is subjected to target labeling and normalization processing to obtain multimodal data, including: The boundary position of the target area in the original data of the power grid is marked, and the image data of different modes in the original data of the power grid are size normalized to obtain multi-modal data.
3. The method according to claim 2, characterized in that In the process of performing a size normalization operation on image data of different modes in the original data of the power grid to obtain multi-modal data, the method further includes: Mosaic image enhancement technology is used to perform multimodal image enhancement processing on image data of different modes in the original data of the power grid.
4. The method according to claim 1, characterized in that A variety of network models are used to extract features from the multimodal data to obtain feature information of the target object, including: Using AKConv convolution technology to perform feature extraction processing on the feature information of the target object to obtain initial features, and using YOLOv8 target detection algorithm to identify the initial features to obtain the image features; Using the FCOS model to perform feature extraction processing on the feature information of the target object to obtain the temperature feature; The GK means algorithm is used to perform feature extraction processing on the feature information of the target object to obtain the morphological features.
5. The method according to claim 1, characterized in that The feature information of the target object is subjected to feature fusion based on features of different modalities to obtain fused features, including: The SENet technology is used to perform feature fusion on the feature information of the target object to obtain the fused feature.
6. The method according to claim 5, characterized in that In the process of using SENet technology to fuse the feature information of the target object to obtain the fused feature, the method further includes: According to A = σ(ω2δ(ω1,R)), determine the weight of the feature map of each channel; Among them, A is the weight of the feature map, R is a one-dimensional real number obtained after performing a global average pooling operation on the feature information of the target object, ω1 and ω2 are fully connected layers, σ is the sigmoid activation function, and δ is the ReLU activation function.
7. The method according to any one of claims 1 to 6, characterized in that The AIFI module and YOLOv8 model are used to classify and detect the fusion features to obtain the final detection results, including: The AIFI module is used to classify the fusion features based on the self-attention mechanism to obtain a multi-scale feature fusion result; The YOLOv8 model is used to perform detection processing on the multi-scale feature fusion result to obtain the final detection result.
8. A multi-modal feature fusion power grid equipment detection device, characterized in that: include: an acquisition unit, used to acquire raw data of a power grid, and to perform target labeling and normalization processing on the raw data of the power grid to obtain multimodal data, wherein the raw data of the power grid includes image data of a target object acquired by a camera, temperature data of the target object acquired by an infrared device, and point cloud data of the target object acquired by a radar sensor; A first processing unit, configured to extract features from the multimodal data using a plurality of network models, and obtain feature information of the target object, wherein the feature information includes image features, temperature features, and morphological features; A second processing unit, configured to perform feature fusion on the feature information of the target object based on features of different modalities to obtain fused features; The third processing unit is used to classify and detect the fusion features using the AIFI module and the YOLOv8 model to obtain a final detection result.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-modal feature fusion power grid equipment detection method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a power grid equipment detection method for executing the multimodal feature fusion described in any one of claims 1 to 7.
Citation Information
Patent Citations
Multi-dimensional feature integrated building point cloud hierarchical clustering segmentation method
CN106780509A
Target multi-dimensional detection method based on four-class multi-modal data fusion
CN113971801A
Visible light, infrared and radar fusion target detection method based on deep learning
CN114254696A
Multi-mode power equipment state detection method and model
CN116627691A
Power equipment detection method based on cross-modal multilevel feature fusion
CN117350964A
Cited By
Deep fusion system and method for multi-modal data of power distribution network
CN119622640A
