Substation equipment detection method and machine readable storage medium

Abstract

The embodiment of the invention provides a substation equipment detection method and a machine readable storage medium, and belongs to the technical field of target recognition. The equipment detection method comprises the following steps: acquiring a depth camera image and a point cloud data image about substation equipment; performing feature extraction on the depth camera image and the point cloud data image to obtain depth image features and point cloud features corresponding to the depth image features; fusing the depth image features and the corresponding point cloud features; and detecting the position of the equipment needing to be detected according to the fused features. The equipment detection method can accurately identify the position of the substation equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of target identification, and in particular to a substation equipment detection method and a machine-readable storage medium. Background Art

[0002] Substation is an important part of the power system, which mainly includes the following functions: 1. Voltage conversion: One of the main functions of the substation is to change the voltage level. Electricity needs a higher voltage to reduce losses when it is transmitted over long distances, and it needs to be reduced when it reaches the user end to meet safety and usage requirements. Substations use transformers to increase (boost) and reduce (step-down) voltage. 2. Power distribution: Substations distribute electric energy to different areas and users. Through substations, electricity can be transmitted to different power consumption areas such as factories, residential areas, and commercial areas. 3. Power control: Substations can control and regulate the flow of electricity to ensure the stable operation of the power system. This includes power switch control, load distribution, and fault isolation. 4. Power protection: Substations are equipped with protection devices, such as relay protection devices, which can quickly cut off the faulty part in the event of a fault to protect the safety of the power system. 5. Power metering: Substations are also responsible for power metering, that is, measuring and recording the consumption of electricity, which is crucial for the calculation of electricity bills and the management of power resources. Due to the special role of substations, identification and modeling of their equipment is a major issue, but in the identification and modeling process of the prior art, it is difficult to accurately detect and locate the equipment targets of substations. Therefore, a method that can accurately locate substation equipment is needed. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a substation equipment detection method and a machine-readable storage medium, wherein the equipment detection method can accurately identify the location of the substation equipment.

[0004] In order to achieve the above object, an embodiment of the present invention provides a substation equipment detection method, the equipment detection method comprising: Acquire depth camera images and point cloud data images of substation equipment; Performing feature extraction on the depth camera image and the point cloud data image to obtain depth image features and point cloud features corresponding thereto; Fusing the depth image features with corresponding point cloud features; The location of the device to be detected is detected based on the fused features.

[0005] Optionally, feature extraction is performed on the depth camera image and the point cloud data image to obtain depth image features and point cloud features corresponding thereto, including: Reprojecting the point cloud scan information into two images in the depth camera image by calibrating the parameters to form two sparse point cloud data images corresponding to the depth camera image; Using a sequence of morphological operations and a Gaussian blur operation, holes in the sparse point cloud data image are filled with depth values ​​from nearby valid point clouds to obtain a dense point cloud data image; The dense point cloud data image is normalized using the maximum depth value in the dataset so that the depth value is between 0 and 1; The normalized dense point cloud data image and the corresponding depth camera image are input into the feature encoder to extract the corresponding point cloud features and depth image features.

[0006] Optionally, the depth camera image and the point cloud data image each have a feature encoder, and the architectures of the two feature encoders are the same.

[0007] Optionally, fusing the depth image feature and the corresponding point cloud feature includes: Inputting the depth image features and point cloud features into a feature encoder; The feature encoder obtains high-level features based on the features of the depth camera image and the point cloud data image; The depth image features and point cloud features are obtained, and the high-level features, depth image features and point cloud features are upsampled according to formula (1): Formula (1), in, Represents high-level features, represents the upsampled high-level feature map, represents the upsampling operation by nearest neighbor interpolation, Represents the feature extractor For deep image features, represents a 1*1 convolutional layer, represents the upsampled deep image feature map, Indicates For point cloud features, Represents the upsampled point cloud feature map; After extracting the high-level features, deep image features and point cloud features, the fusion features are calculated according to formula (2): Formula (2), in, represents a 3*3 convolutional layer, Represents the fused feature map.

[0008] Optionally, fusing the depth image features with corresponding point cloud features includes: Get the extracted The deep image features and point cloud features of the pair; According to formula (3), we can obtain The weight of the point cloud features at the level: Formula (3), in, Indicates The weight of the point cloud features at the level; Reintegrate the features according to formula (4): Formula (4).

[0009] Optionally, the depth camera image can be divided into a left image and a right image, and the two images can be respectively fused with the corresponding point cloud data image to obtain a fused feature map.

[0010] Optionally, the location of the device to be detected is detected based on the fused features, including: The two images of the depth camera image are respectively fused with the corresponding point cloud data image, and the obtained fusion feature map is input into the stereo RPN; The stereo RPN classifies the fused feature map and regresses the bounding box residual of each input position; Connect the regressed left and right feature maps at each level; The range of interest is extracted from the connected left and right feature maps to obtain the position of the border.

[0011] Optionally, the location of the device to be detected is detected based on the fused features, including: Get the position and parallax of the frame; Concatenate the left and right feature maps at each disparity level to form a matching cost volume; Using a 3D convolutional layer trained from the matching cost volume and downsampling the feature representation from the matching cost volume; Through network regularization, the downsampled depth is mapped to each depth of the border; By taking a weighted sum of the normalized probability for each depth of the border, the depth of the center of the border is obtained according to formula (5): Formula (5), in, Indicates the number of levels of depth, represents the normalized probability, Represents each depth in the bounding box, Indicates the depth of the bounding box center.

[0012] On the other hand, the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute a substation equipment detection method as described above.

[0013] Through the above technical solution, the substation equipment detection method and machine-readable storage medium provided by the present invention can obtain the depth camera image and point cloud data image of the substation equipment, and then extract the features of the depth camera image and the point cloud data image, so as to obtain the depth image features and the remaining corresponding point cloud features. After obtaining the depth image features and the point cloud features, the depth image features and the point cloud features can be fused, and then the position of the equipment to be detected can be detected based on the fused features. This method can accurately detect the position of the target.

[0014] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 is a flow chart of a substation equipment detection method according to an embodiment of the present invention; Figure 2 is a flow chart of acquiring point cloud features and depth image features of a substation equipment detection method according to an embodiment of the present invention; Figure 3 is a first flow chart of feature fusion of a substation equipment detection method according to an embodiment of the present invention; Figure 4 is a second flow chart of feature fusion of a substation equipment detection method according to an embodiment of the present invention; Figure 5 is a first flow chart of the position of the detection frame of the substation equipment detection method according to an embodiment of the present invention; Figure 6 It is a second flow chart of the position of the detection frame of the substation equipment detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0017] Figure 11 is a flow chart of a substation equipment detection method according to an embodiment of the present invention. In the present invention, the process of the equipment detection method may include: In step S1 , a depth camera image and a point cloud data image of a substation device are acquired.

[0018] In step S2, feature extraction is performed on the depth camera image and the point cloud data image to obtain depth image features and corresponding point cloud features.

[0019] In step S3, the depth image features and the corresponding point cloud features are fused.

[0020] In step S4, the position of the device to be detected is detected based on the fused features.

[0021] In the present invention, when performing target detection on substation equipment, a depth camera image and a point cloud data image of the substation equipment can be obtained, and then feature extraction can be performed on the depth camera image and the point cloud data image, so that the depth image features and the remaining corresponding point cloud features can be obtained. After obtaining the depth image features and the point cloud features, the depth image features and the point cloud features can be fused, and then the position of the device to be detected can be detected based on the fused features. This method can accurately detect the position of the target.

[0022] In one embodiment of the present invention, Figure 2 As shown, the process of obtaining the depth image features and the corresponding point cloud features may include: In step S5, the point cloud scanning information is reprojected into two images in the depth camera image by calibration parameters to form two sparse point cloud data images corresponding to the depth camera image.

[0023] In step S6, a sequence of morphological operations and a Gaussian blur operation are used to fill holes in the sparse point cloud data image with depth values ​​from nearby valid point clouds to obtain a dense point cloud data image.

[0024] In step S7 , the dense point cloud data image is normalized using the maximum depth value in the data set so that the depth value is between 0 and 1.

[0025] In step S8, the normalized dense point cloud data image and the corresponding depth camera image are input into a feature encoder to extract corresponding point cloud features and depth image features.

[0026] In the present invention, when acquiring the depth image features and the corresponding point cloud features, the point cloud scanning information can be reprojected into two images in the depth camera image by calibration parameters, so that two sparse point cloud data images corresponding to the depth camera image can be formed. The point cloud data image is sparse, so it needs to be completed to make it dense. After obtaining the sparse point cloud data image, a morphological operation sequence and a Gaussian blur operation can be used to fill the holes in the sparse point cloud data image with the depth values ​​of the point clouds of nearby valid points to make it dense, so that a dense point cloud data image can be obtained. After obtaining the dense point cloud data image, the maximum depth value in the data set can be used for normalization so that the depth value is between 0 and 1, and then the normalized dense point cloud data image and the corresponding depth camera image can be input into the feature encoder to extract the corresponding point cloud features and depth image features.

[0027] In the present invention, the depth camera image and the point cloud data image each have a feature encoder, and the architectures of the two feature encoders are the same, and the left input view and the right input view of the encoding weights are shared, so that the same quantization operation can be performed on the left input view and the right input view.

[0028] In one embodiment of the present invention, Figure 3 As shown, the first process of feature fusion may include: In step S9, the depth image features and the point cloud features are input into the feature encoder.

[0029] In step S10, the feature encoder obtains high-level features based on the features of the depth camera image and the point cloud data image.

[0030] In step S11, the deep image features and point cloud features are obtained, and the high-level features, deep image features and point cloud features are upsampled according to formula (1): Formula (1), in, Represents high-level features, represents the upsampled high-level feature map, represents the upsampling operation by nearest neighbor interpolation, Represents the feature extractor For deep image features, represents a 1*1 convolutional layer, represents the upsampled deep image feature map, Indicates For point cloud features, Represents the upsampled point cloud feature map.

[0031] In step S12, after extracting high-level features, deep image features and point cloud features, the fusion features are calculated according to formula (2): Formula (2), in, represents a 3*3 convolutional layer, Represents the fused feature map.

[0032] In the present invention, after obtaining the depth image features and the corresponding point cloud features, the two features need to be fused. In the fusion process, the depth image features and the point cloud features can be first input into the feature encoder, and then the feature encoder can obtain a higher-level spatially coarser but semantically more powerful high-level features based on the features of the depth camera image and the point cloud data image. Based on the acquired depth image features, point cloud features and high-level features, the high-level features, depth image features and point cloud features can then be upsampled according to formula (1), and finally the upsampled results can be brought into formula (2) to calculate the fused features.

[0033] In one embodiment of the present invention, Figure 4 As shown, the second process of feature fusion may include: In step S13, the extracted The deep image features and point cloud features of the pair.

[0034] In step S14, the first The weight of the point cloud features at the level: Formula (3), in, Indicates The weight of the point cloud features at the level.

[0035] In step S15, the features are re-integrated according to formula (4): Formula (4).

[0036] In the present invention, it can be seen from the above formulas (1) and (2) that the features between different modalities are in an equal position when fused, rather than weighted, which may result in the importance of different modalities not being reflected. Therefore, the point cloud features can be added to the deep image features according to the attention mechanism, and weights can be set for each feature level. The weights of the point cloud features at each feature level can be obtained by formula (3), and then the weights can be brought into formula (4) to obtain a more accurate fusion feature map.

[0037] In one embodiment of the present invention, the depth camera image can be divided into a left image and a right image, and the two images can be respectively fused with the corresponding point cloud data image to obtain a fused feature map.

[0038] In one embodiment of the present invention, Figure 5 As shown, the first process of detecting the position of the border may include: In step S16, the two images of the depth camera image are respectively fused with the corresponding point cloud data image, and the obtained fusion feature map is input into the stereo RPN.

[0039] In step S17, the stereo RPN classifies the fused feature map and regresses the bounding box residual of each input position.

[0040] In step S18, the regressed left and right feature maps are connected at each level.

[0041] In step S19, the range of interest is extracted from the connected left and right feature maps to obtain the position of the border.

[0042] In the present invention, after the fusion feature map obtained by fusing the two images of the depth camera image with the corresponding point cloud data image is input into the stereo RPN, the stereo RPN can classify the fusion feature map and regress the frame residual of each input position. Then the regressed left and right feature maps can be connected at each level, and the range of interest can be extracted from the connected left and right feature maps to obtain the position of the frame.

[0043] In the present invention, the second process of detecting the position of the frame may include: In step S20, the position and disparity of the frame are obtained.

[0044] In step S21, the left and right feature maps at each disparity level are connected to form a matching cost volume.

[0045] In step S22, a 3D convolutional layer is used to train from the matching cost volume and downsample the feature representation from the matching cost volume.

[0046] In step S23, the depth obtained by downsampling is corresponded to each depth of the border through network regularization.

[0047] In step S24, each depth of the frame is weighted and summed according to the normalized probability to obtain the depth of the center of the frame according to formula (5): Formula (5), in, Indicates the number of levels of depth, represents the normalized probability, Represents each depth in the bounding box, Indicates the depth of the bounding box center.

[0048] In the present invention, after obtaining the position and disparity of the border of the target, the left and right feature maps at each disparity level can be connected to form a matching cost volume. Then, a 3D convolutional layer can be used to train from the matching cost volume and downsample the feature representation from the matching cost volume. After downsampling, the depth obtained by downsampling can be corresponded to each depth of the border through network regularization, and then each depth of the border can be weighted summed according to the normalized probability, and finally the center depth of the border can be obtained by formula (5).

[0049] On the other hand, the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute a substation equipment detection method as described above.

[0050] Through the above technical solution, the substation equipment detection method and machine-readable storage medium provided by the present invention can obtain the depth camera image and point cloud data image of the substation equipment, and then extract the features of the depth camera image and the point cloud data image, so as to obtain the depth image features and the remaining corresponding point cloud features. After obtaining the depth image features and the point cloud features, the depth image features and the point cloud features can be fused, and then the position of the equipment to be detected can be detected based on the fused features. This method can accurately detect the position of the target.

[0051] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0052] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0053] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0054] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0055] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0056] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0057] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0058] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0059] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A substation equipment detection method, characterized in that: The device detection method comprises: Acquire depth camera images and point cloud data images of substation equipment; Performing feature extraction on the depth camera image and the point cloud data image to obtain depth image features and point cloud features corresponding thereto; Fusing the depth image features with corresponding point cloud features; The location of the device to be detected is detected based on the fused features.

2. The device detection method according to claim 1, characterized in that: Performing feature extraction on the depth camera image and the point cloud data image to obtain depth image features and corresponding point cloud features, including: Reprojecting the point cloud scan information into two images in the depth camera image by calibrating the parameters to form two sparse point cloud data images corresponding to the depth camera image; Using a sequence of morphological operations and a Gaussian blur operation, holes in the sparse point cloud data image are filled with depth values ​​from nearby valid point clouds to obtain a dense point cloud data image; The dense point cloud data image is normalized using the maximum depth value in the dataset so that the depth value is between 0 and 1; The normalized dense point cloud data image and the corresponding depth camera image are input into the feature encoder to extract the corresponding point cloud features and depth image features.

3. The device detection method according to claim 2, characterized in that: The depth camera image and the point cloud data image each have a feature encoder, and the architectures of the two feature encoders are the same.

4. The 3D object detection algorithm according to claim 3, characterized in that: Fusion of the depth image features and corresponding point cloud features includes: Inputting the depth image features and point cloud features into a feature encoder; The feature encoder obtains high-level features based on the features of the depth camera image and the point cloud data image; The depth image features and point cloud features are obtained, and the high-level features, depth image features and point cloud features are upsampled according to formula (1): Formula (1), in, Represents high-level features, represents the upsampled high-level feature map, represents the upsampling operation by nearest neighbor interpolation, Represents the feature extractor For deep image features, represents a 1*1 convolutional layer, represents the upsampled deep image feature map, Indicates For point cloud features, Represents the upsampled point cloud feature map; After extracting the high-level features, deep image features and point cloud features, the fusion features are calculated according to formula (2): Formula (2), in, represents a 3*3 convolutional layer, Represents the fused feature map.

5. The device detection method according to claim 4, characterized in that: The depth image features and the corresponding point cloud features are fused, including: Get the extracted The deep image features and point cloud features of the pair; According to formula (3), we can obtain The weight of the point cloud features at the level: Formula (3), in, Indicates The weight of the point cloud features at the level; Re-integrate the features according to formula (4): Formula (4).

6. The device detection method according to claim 4, characterized in that: The depth camera image can be divided into a left image and a right image, and the two images can be respectively fused with the corresponding point cloud data image to obtain a fused feature map.

7. The device detection method according to claim 6, characterized in that: The location of the device to be detected is detected based on the fused features, including: The two images of the depth camera image are respectively fused with the corresponding point cloud data image, and the obtained fusion feature map is input into the stereo RPN; The stereo RPN classifies the fused feature map and regresses the bounding box residual of each input position; Connect the regressed left and right feature maps at each level; The range of interest is extracted from the connected left and right feature maps to obtain the position of the border.

8. The device detection method according to claim 7, characterized in that: The location of the device to be detected is detected based on the fused features, including: Get the position and parallax of the frame; Concatenate the left and right feature maps at each disparity level to form a matching cost volume; Using a 3D convolutional layer trained from the matching cost volume and downsampling the feature representation from the matching cost volume; Through network regularization, the downsampled depth is mapped to each depth of the border; By taking a weighted sum of the normalized probability for each depth of the border, the depth of the center of the border is obtained according to formula (5): Formula (5), in, Indicates the number of levels of depth, represents the normalized probability, Represents each depth in the bounding box, Indicates the depth of the bounding box center.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which are used to enable a machine to execute a substation equipment detection method as described in any one of claims 1-8.

Images