Multi-mode power transmission corridor hidden danger target identification method and system, medium and equipment

Abstract

The embodiment of the invention discloses a multi-mode power transmission corridor hidden danger target identification method and system, a medium and equipment. The method comprises the following steps: collecting LiDAR point cloud data and visible light image data of a power transmission corridor; extracting elevation features, curvature features and reflection intensity features of the LiDAR point cloud data; extracting material features of the visible light image data; determining an elevation mutation region according to the elevation feature and the curvature feature, extracting a region texture of the elevation mutation region from the material feature, and performing attention weight calculation on the elevation mutation region and the region texture to obtain a material sensing result; determining a high-reflection area in the material features, extracting the reflection intensity of the high-reflection area from the reflection intensity features, and performing attention weight calculation on the high-reflection area and the reflection intensity to obtain an elevation correction result; performing weighted fusion on the material sensing result and the elevation correction result to obtain fusion features; the hidden danger target of the power transmission corridor is identified through the fused features, and the identification precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of transmission corridor hidden danger monitoring technology, and in particular to a multi-modal transmission corridor hidden danger target identification method, system, medium and equipment. Background Art

[0002] At present, the monitoring of hidden dangers in transmission corridors mainly relies on satellite remote sensing images combined with deep learning technology. By inputting satellite image data (such as multispectral or panchromatic images with a resolution of 0.5-2 meters), the convolutional neural network is used to directly output the classification results and binary images of hidden danger targets.

[0003] While this approach has achieved promising results, it still has some significant drawbacks. First, the spatial resolution of satellite imagery is typically low, making it difficult to capture small targets around transmission corridors, such as broken insulators or small hanging objects. Second, these images primarily rely on two-dimensional spectral characteristics and lack three-dimensional geometric information, leading to problems in identifying tree obstacles and distinguishing between ground film and hardened ground. Finally, the long revisit period of satellite imagery makes it difficult to promptly respond to temporary hidden dangers, such as the intrusion of construction vehicles. Summary of the Invention

[0004] Based on this, it is necessary to propose a multimodal transmission corridor hidden danger target identification method, system, medium and equipment to address the above problems.

[0005] A multimodal transmission corridor hidden danger target identification method, the method comprising:

[0006] Collect LiDAR point cloud data and visible light image data of the transmission corridor.

[0007] Extracting three-dimensional geometric features of the LiDAR point cloud data, wherein the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features.

[0008] Extracting two-dimensional texture features of the visible light image data, wherein the two-dimensional texture features include material features.

[0009] An elevation mutation area is determined according to the elevation features and the curvature features, a regional texture of the elevation mutation area is extracted from the material features, and attention weights are calculated for the elevation mutation area and the regional texture to obtain a material perception result.

[0010] Determine a high-reflection area in the material feature, extract the reflection intensity of the high-reflection area from the reflection intensity feature, perform attention weight calculation on the high-reflection area and the reflection intensity, and obtain an elevation correction result.

[0011] The material perception results and elevation correction results are weightedly fused to obtain fusion features.

[0012] The fusion features are used to identify hidden danger targets in the transmission corridor.

[0013] The extraction of three-dimensional geometric features of the LiDAR point cloud data includes elevation features, curvature features, and reflection intensity features, specifically including:

[0014] The LiDAR point cloud data is voxelized to obtain a grid whose number of LiDAR point clouds is greater than or equal to a preset threshold.

[0015] A convolution operation is performed on the LiDAR point cloud data within the grid to obtain intermediate features, wherein the LiDAR point cloud data includes height value, curvature value and reflection intensity.

[0016] A downsampling operation is performed on the intermediate features, and the downsampled intermediate features are upsampled to the original resolution to obtain three-dimensional geometric features of the LiDAR point cloud data, where the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features.

[0017] The step of extracting the two-dimensional texture features of the visible light image data includes material features, specifically including:

[0018] A depthwise separable convolution operation is performed on the visible light image data to obtain deep features.

[0019] Position encoding is performed on the deep features in the X and Y directions, and position information is added to the deep features to obtain a position-encoded feature map.

[0020] The deep features are processed by 1×1 convolution operation and Sigmoid activation function to generate an attention weight map.

[0021] The attention weight map is weighted and multiplied with the position-encoded feature map to obtain a two-dimensional texture feature, where the two-dimensional texture feature includes a material feature.

[0022] The step of determining the elevation change zone according to the elevation characteristics and curvature characteristics specifically includes:

[0023] The elevation differences between adjacent grids are determined according to the elevation features.

[0024] Adjacent grids whose elevation difference is greater than a second preset threshold are determined as candidate mutation areas.

[0025] The curvature value of the candidate mutation region is obtained according to the curvature feature.

[0026] If it is determined that the curvature value of the candidate mutation region is greater than or equal to a third preset threshold, a sharp edge exists and the candidate mutation region is an elevation mutation region.

[0027] The step of calculating the attention weights of the elevation mutation region and the regional texture to obtain the material perception result specifically includes:

[0028] according to The attention weights of the elevation mutation area and the regional texture are calculated to obtain the material perception results, where F L→R is the material perception result, Q L It is the area of ​​sudden elevation changes. is the transposed amount of the regional texture, V R For regional textures.

[0029] The step of calculating the attention weight of the high-reflective area and the reflection intensity to obtain the elevation correction result specifically includes:

[0030] according to The attention weight is calculated for the high reflective area and the reflection intensity to obtain the elevation correction result, where F R→L is the elevation correction result, Q R Highly reflective area. is the transposed value of the reflection intensity, V R is the reflection intensity.

[0031] The weighted fusion of the material perception result and the elevation correction result to obtain the fusion feature specifically includes:

[0032] The fusion weight is determined according to the point cloud density of the LiDAR point cloud data.

[0033] The geometry-guided material perception result and the texture-guided elevation correction result are weightedly fused according to the fusion weight to obtain a fusion feature.

[0034] A multi-modal transmission corridor hidden danger target identification system, comprising:

[0035] The data acquisition module is used to collect LiDAR point cloud data and visible light image data of the transmission corridor.

[0036] The three-dimensional geometric feature extraction module extracts the three-dimensional geometric features of the LiDAR point cloud data, wherein the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features.

[0037] The two-dimensional texture feature extraction module is used to extract the two-dimensional texture features of the visible light image data, where the two-dimensional texture features include material features.

[0038] The material perception result acquisition module is used to determine the elevation mutation area based on the elevation features and curvature features, extract the regional texture of the elevation mutation area from the material features, calculate the attention weight of the elevation mutation area and the regional texture, and obtain the material perception result.

[0039] The elevation correction result module is used to determine the high-reflection area in the material feature, extract the reflection intensity of the high-reflection area from the reflection intensity feature, calculate the attention weight of the high-reflection area and the reflection intensity, and obtain the elevation correction result.

[0040] The fusion module is used to perform weighted fusion on the material perception results and the elevation correction results to obtain fusion features.

[0041] An identification module is used to identify hidden danger targets in the transmission corridor through the fusion features.

[0042] Wherein, the system further includes:

[0043] Post-processing module: used for performing elevation filtering and edge optimization on the identification results of the hidden danger targets in the transmission corridor.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above method.

[0045] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0046] The embodiments of the present invention have the following beneficial effects:

[0047] The present invention first collects LiDAR point cloud data and visible light image data of the transmission corridor, and performs feature extraction on the LiDAR point cloud data and visible light image data respectively to obtain the three-dimensional geometric features of the LiDAR point cloud data and the two-dimensional texture features of the visible light image data, wherein the three-dimensional geometric features include elevation features, curvature features and reflection intensity features, and the elevation mutation area is determined based on the three-dimensional geometric features. Such elevation mutation areas will show abnormal vegetation coverage or changes in ground structure, which are key factors affecting the safety of the transmission corridor. Secondly, the two-dimensional texture features of the visible light image data help to further identify high-reflective areas, which may be related to ground reflective materials or power equipment and facilities.

[0048] Furthermore, in the feature dimension, bidirectional cross-modal attention weight calculation is performed on the three-dimensional geometric features of LiDAR point cloud data and the two-dimensional texture features of visible light image data to distinguish and highlight these key features, forming geometry-guided material perception results and texture-guided elevation correction results. Finally, these two results are weightedly fused to generate fusion features. The identification of hidden danger targets in the transmission corridor through fusion features not only improves the recognition accuracy, but also enhances the accurate capture of the boundaries of hidden danger targets, significantly improves the accuracy and efficiency of hidden danger target identification in the transmission corridor, and realizes the accurate identification of elevation mutations and material characteristics of the transmission corridor, thereby locating and dealing with potential safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] in:

[0051] Figure 1 A flowchart of an embodiment of a multi-modal transmission corridor hidden danger target identification method provided by the present invention;

[0052] Figure 2 A flowchart of another embodiment of a multi-modal transmission corridor hidden danger target identification method provided by the present invention;

[0053] Figure 3 A schematic structural diagram of an embodiment of a multi-modal transmission corridor hidden danger target identification system provided by the present invention;

[0054] Figure 4 A schematic structural diagram of an embodiment of the device provided by the present invention;

[0055] Figure 5 This is a schematic structural diagram of an embodiment of the medium provided by the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] like Figure 1 As shown, Figure 1A flowchart of an embodiment of a multi-modal transmission corridor hidden danger target identification method provided by the present invention. A multi-modal transmission corridor hidden danger target identification method, the method comprising:

[0058] S101: Collect LiDAR point cloud data and visible light image data of the transmission corridor.

[0059] For example, the UAV platform is equipped with a visible light camera and a rotating LiDAR. The visible light camera, with parameters of 20 megapixels, a global shutter speed, a frame rate of 30 fps, and a ground resolution of 5 cm, collects visible light image data of the transmission corridor. The rotating LiDAR, with parameters of 0.1° horizontal angular resolution, ±15° vertical field of view, and a point cloud density of ≥200 points / ㎡, collects LiDAR point cloud data of the transmission corridor. Based on the UAV platform's real-time processing architecture, the UAV can promptly eliminate temporary hidden dangers and quickly respond to them, improving dynamic monitoring capabilities.

[0060] Furthermore, the PTP precision clock protocol is used to ensure that the time synchronization error between the LiDAR and the camera is less than 1ms. In addition, the extrinsic parameter matrix is ​​obtained through checkerboard calibration to achieve alignment of point cloud and pixel-level coordinates.

[0061] S102: Extracting three-dimensional geometric features of the LiDAR point cloud data, where the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features.

[0062] For example, the raw LiDAR point cloud data is an unordered set of XYZ coordinates that cannot be directly fed into a standard neural network for processing. Therefore, it is converted into a regular three-dimensional grid through voxelization, dividing the 3D space into a regular grid of cubes (voxels), similar to packing scattered building blocks into the grid of a transparent storage box. Specifically, the LiDAR point cloud data is voxelized to obtain a grid with a number of LiDAR point clouds greater than or equal to a preset threshold. Ultimately, the large number of unordered points is converted into multiple voxels with attributes within the grid, compressing the volume while retaining key geometric information.

[0063] Furthermore, a convolution operation is performed on the LiDAR point cloud data within the grid to obtain intermediate features, including elevation, curvature, and reflection intensity. These intermediate features are then downsampled and upsampled to the original resolution to obtain the 3D geometric features of the LiDAR point cloud data, including elevation, curvature, and reflection intensity.

[0064] S103: Extracting two-dimensional texture features of the visible light image data, where the two-dimensional texture features include material features.

[0065] For example, the original ResNet network contains a large number of convolutional layers. The excessive number of parameters will make it impossible to deploy on the drone side. Therefore, the first-layer convolution kernel is reduced from 7×7 to 3×3 to reduce the computational pressure, and depth-wise separable convolution is used instead of standard convolution to further compress the number of parameters. After the shallow features of the visible light image data are extracted through the first-layer convolution kernel, the deep features are extracted through depth-wise separable convolution.

[0066] Furthermore, the deep features are position-encoded in the X and Y directions, and the position information is added to the deep features to obtain the position-encoded feature map. At the same time, the deep features are processed by 1×1 convolution and Sigmoid activation function to generate the attention weight map.

[0067] Furthermore, the attention weight map is weighted and multiplied with the position-encoded feature map to obtain the two-dimensional texture features of the visible light image data, which include material features.

[0068] S104: Determine the elevation mutation area based on the elevation features and the curvature features, extract the regional texture of the elevation mutation area from the material features, calculate the attention weights of the elevation mutation area and the regional texture, and obtain the material perception result.

[0069] Exemplarily, the three-dimensional geometric features are first projected onto the coordinate system of the two-dimensional texture features using a pre-calibrated extrinsic matrix.

[0070] Furthermore, the elevation difference between adjacent grids is determined based on the elevation feature; adjacent grids whose elevation difference is greater than a second preset threshold are determined as candidate mutation areas; the curvature value of the candidate mutation area is obtained based on the curvature feature; if the curvature value of the candidate mutation area is greater than or equal to a third preset threshold, then a sharp edge exists, and the candidate mutation area is an elevation mutation area, and the regional texture of the elevation area is extracted from the material feature.

[0071] Furthermore, attention weights are calculated for elevation mutation areas and regional textures according to the following formula to obtain geometry-guided material perception results. For example, in elevation mutation areas (such as the top of the iron tower), the weights of features are enhanced:

[0072]

[0073] Among them, F L→R is the material perception result, Q L It is the area of ​​sudden elevation changes. is the transposed amount of the regional texture, V R For regional textures.

[0074] S105: Determine the high-reflection area in the material feature, extract the reflection intensity of the high-reflection area from the reflection intensity feature, calculate the attention weight of the high-reflection area and the reflection intensity, and obtain the elevation correction result.

[0075] Exemplarily, a high-reflection area is determined in the material characteristics, and the reflection intensity of the high-reflection area is determined from the reflection intensity characteristics.

[0076] Furthermore, the attention weights of the highly reflective areas and reflection intensities are calculated according to the following formula to obtain the texture-guided elevation correction results. For example, in the sunlight-reflected area (misjudged as water), the LiDAR elevation confidence of the area is reduced:

[0077]

[0078] Among them, F R→L is the elevation correction result, Q R Highly reflective area. is the transposed value of the reflection intensity, V R is the reflection intensity.

[0079] S106: Perform weighted fusion on the material perception result and the elevation correction result to obtain fusion features.

[0080] Exemplarily, a fusion weight is determined according to the point cloud density of the LiDAR point cloud data; and a weighted fusion is performed on the geometry-guided material perception result and the texture-guided elevation correction result according to the fusion weight to obtain a fusion feature.

[0081] S107: Identify potential hazards in transmission corridors through feature fusion.

[0082] Exemplarily, target detection is performed on the fused features to generate final detection results, which include elevation anomalies and material anomalies of hidden danger targets in the transmission corridor.

[0083] From the above description, it can be seen that the present invention first collects the LiDAR point cloud data and visible light image data of the transmission corridor, performs feature extraction on the LiDAR point cloud data and the visible light image data respectively, obtains the three-dimensional geometric features of the LiDAR point cloud data and the two-dimensional texture features of the visible light image data, wherein the three-dimensional geometric features include elevation features, curvature features and reflection intensity features, and determines the elevation mutation area based on the three-dimensional geometric features. This elevation mutation area will show abnormal vegetation coverage or changes in ground structure, which is a key factor affecting the safety of the transmission corridor. Secondly, the two-dimensional texture features of the visible light image data help to further identify high-reflective areas, which may be related to ground reflective materials or power equipment and facilities.

[0084] Furthermore, in the feature dimension, bidirectional cross-modal attention weight calculation is performed on the three-dimensional geometric features of LiDAR point cloud data and the two-dimensional texture features of visible light image data to distinguish and highlight these key features, forming geometry-guided material perception results and texture-guided elevation correction results. Finally, these two results are weightedly fused to generate fusion features. The identification of hidden danger targets in the transmission corridor through fusion features not only improves the recognition accuracy, but also enhances the accurate capture of the boundaries of hidden danger targets, significantly improves the accuracy and efficiency of hidden danger target identification in the transmission corridor, and realizes the accurate identification of elevation mutations and material characteristics of the transmission corridor, thereby locating and dealing with potential safety hazards.

[0085] like Figure 2 As shown, Figure 2 A flowchart of another embodiment of a multi-modal transmission corridor hidden danger target identification method provided by the present invention. A multi-modal transmission corridor hidden danger target identification method, the method comprising:

[0086] S201: Collect LiDAR point cloud data and visible light image data of the transmission corridor.

[0087] It should be noted that step S201 Figure 1 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0088] S202: Process the LiDAR point cloud data by voxelization to obtain a grid whose number of LiDAR point clouds is greater than or equal to a preset threshold.

[0089] For example, LiDAR point cloud data is converted into a grid through voxelization, dividing the 3D space into a regular grid of cubes (voxels), similar to packing scattered building blocks into the grid of a transparent storage box. Specifically, a processing area (e.g., 200m × 200m × 50m) is first determined based on the drone's flight altitude. A grid is then created within the processing area, with the voxel size set to 0.1m × 0.1m × 0.1m.

[0090] Furthermore, the number and attributes of point clouds within each voxel are counted: if the number of point clouds is less than 5, it is marked as an empty voxel (not involved in subsequent calculations). If the number of point clouds within a voxel is ≥ 5, the height, curvature, and reflection intensity of the point clouds within the grid are calculated as follows:

[0091] Elevation value: Determine the elevation value of a point cloud based on the vertical height (Z coordinate) of a point in the point cloud.

[0092] Curvature Value: For each target point in the point cloud, a KD tree is used to quickly find 26 points within its radius neighborhood, which are used as the domain points. The covariance matrix of the neighborhood points of the current point cloud is calculated and the eigenvalue decomposition is performed on it to obtain the eigenvalues, which reflect the distribution characteristics of the neighborhood points in 3D space. The curvature value of the current point cloud is calculated based on the ratio of the minimum eigenvalue to the sum of the eigenvalues.

[0093] Reflection intensity: The reflection intensity value of each LiDAR point cloud is determined based on the intensity of the echo signal generated by the LiDAR system when it emits a laser pulse and receives the reflected signal from the target.

[0094] S203: Perform a convolution operation on the LiDAR point cloud data in the grid to obtain intermediate features. The LiDAR point cloud data includes height values, curvature values, and reflection intensity.

[0095] For example, a sparse convolutional network is used to obtain the 3D geometric features of LiDAR point cloud data. The sparse convolutional network consists of four layers: a 3×3×3 convolutional layer, a dilated convolutional layer, a maximum pooling layer, and a transposed convolutional layer. First, the LiDAR point cloud data within the grid is convolved using the 3×3×3 convolutional layer and the dilated convolutional layer to obtain intermediate features.

[0096] S204: Downsampling the intermediate features, upsampling the downsampled intermediate features to the original resolution, and obtaining three-dimensional geometric features of the LiDAR point cloud data. The three-dimensional geometric features include elevation features, curvature features, and reflection intensity features.

[0097] Exemplarily, the intermediate features are downsampled through the maximum pooling layer, and the downsampled intermediate features are upsampled to the original resolution through the transposed convolution layer to obtain the three-dimensional geometric features of the LiDAR point cloud data, which include elevation features, curvature features and reflection intensity features.

[0098] S205: Perform a depth-wise separable convolution operation on the visible light image data to obtain deep features.

[0099] For example, a lightweight ResNet-18 network was constructed. The original ResNet network contains a large number of convolutional layers, and its excessive number of parameters makes it impossible to deploy on a drone. Therefore, the first-layer convolution kernel was reduced from 7×7 to 3×3 to reduce computational pressure. Depthwise separable convolution was used instead of standard convolution to further reduce the number of parameters, resulting in a lightweight ResNet-18 network. Deep features were extracted based on the lightweight ResNet-18 network. After the shallow features of the visible light image data were extracted through the first-layer convolution kernel, deep features were extracted through depthwise separable convolution.

[0100] S206: Position encoding is performed on the deep features in the X and Y directions, and the position information is added to the deep features to obtain a feature map after position encoding.

[0101] For example, a CA attention module is introduced to enhance spatial location perception, allowing the network to learn to focus on key locations in the transmission corridor. The CA attention module includes global average pooling, 1×1 convolution, and a sigmoid activation function. Specifically, the deep features along the X-axis are globally average pooled and then fused with the X-direction position information to generate height-dimensional position features. The deep features along the Y-axis are globally average pooled and then fused with the Y-direction position information to generate width-dimensional position features. The height-dimensional position features and width-dimensional position features are then concatenated to generate a position-encoded feature map.

[0102] S207: Perform 1×1 convolution operation and Sigmoid activation function processing on the deep features to generate an attention weight map.

[0103] S208: Perform weighted multiplication on the attention weight map and the position-encoded feature map to obtain a two-dimensional texture feature, which includes a material feature.

[0104] For example, a 1×1 convolution operation and a Sigmoid activation function are performed on the deep features to generate an attention weight map. The attention weight map is weighted multiplied with the position-encoded feature map to obtain a two-dimensional texture feature, which includes material features.

[0105] S209: Determine the elevation difference between adjacent grids according to the elevation feature, and determine the adjacent grids whose elevation difference is greater than a second preset threshold as candidate mutation areas.

[0106] Exemplarily, the elevation difference between adjacent grids is determined according to the elevation feature, and adjacent grids whose elevation difference is greater than a second 2 are determined as candidate mutation areas.

[0107] S210: Obtaining a curvature value of the candidate mutation region according to the curvature feature, and determining that the curvature value of the candidate mutation region is greater than or equal to a third preset threshold, indicating that a sharp edge exists and the candidate mutation region is an elevation mutation region.

[0108] For example, the curvature value of the candidate mutation region is obtained based on the curvature feature. If the curvature value is less than 0.1, the candidate mutation region is a smooth transition region (such as a slope). If the curvature value of the candidate mutation region is greater than or equal to 0.3, there is a sharp edge and the candidate mutation region is an elevation mutation region. The elevation mutation region is used as the query value, as shown in the following formula:

[0109]

[0110] Among them, QL is the elevation mutation area, F L It is a three-dimensional geometric feature.

[0111] Relying solely on elevation mutations will introduce a large number of false positives (such as slopes), and curvature features provide key constraints on geometric shapes.

[0112] S211: Extracting regional texture of elevation mutation area from material features.

[0113] S212: Calculate attention weights for elevation mutation areas and regional textures to obtain material perception results.

[0114] For example, the regional texture of the elevation mutation area is extracted from the material features, and the regional texture is used as the Key value and Value value, as shown in the following formula:

[0115]

[0116] Among them, K R The Key value represents the regional texture, V R The Value value represents the regional texture, F R It is a two-dimensional texture feature.

[0117] Furthermore, the attention weights of the elevation mutation area and regional texture are calculated according to the following formula to obtain the material perception results. For example, in the elevation mutation area (such as the top of the tower), the weight of the feature is enhanced:

[0118]

[0119] Among them, F L→R is the material perception result, Q L It is the area of ​​sudden elevation changes. is the transposed amount of the regional texture, V R For regional textures.

[0120] S213: Determine the high-reflection area in the material feature, extract the reflection intensity of the high-reflection area from the reflection intensity feature, calculate the attention weight of the high-reflection area and the reflection intensity, and obtain the elevation correction result.

[0121] For example, the high-reflective area in the material characteristics is determined and used as the query value, as shown in the following formula:

[0122]

[0123] Among them, Q R For high reflective areas, F R It is a two-dimensional texture feature.

[0124] Furthermore, the reflection intensity of the high-reflection area is determined from the reflection intensity feature, and the reflection intensity is used as the Key value and Value value, as shown in the following formula:

[0125]

[0126] Among them, K L The Key value represents the reflection intensity, V L The Value value represents the reflection intensity, F L It is a three-dimensional geometric feature.

[0127] Furthermore, the attention weights of the highly reflective areas and reflection intensities are calculated according to the following formula to obtain the elevation correction results. For example, in the sunlight reflection area (misjudged as water), the LiDAR elevation confidence of the area is reduced:

[0128]

[0129] Among them, F R→L is the elevation correction result, Q R Highly reflective area. is the transposed value of the reflection intensity, V R is the reflection intensity.

[0130] S214: Determine a fusion weight according to the point cloud density of the LiDAR point cloud data.

[0131] For example, the fusion weight α is dynamically calculated by the point cloud density:

[0132]

[0133] Among them, σ is the Sigmoid function, Density is the point cloud density, and α is the fusion weight.

[0134] S215: Perform weighted fusion on the material perception result and the elevation correction result according to the fusion weight to obtain a fusion feature.

[0135] For example, the geometry-guided material perception result and the texture-guided elevation correction result are weightedly fused according to the fusion weight to obtain the fusion feature, as shown in the following formula:

[0136] F fused =α·F L→R +(1-α)·F R→L ;

[0137] Among them, F fused is the fusion feature, α is the fusion weight, F L→R is the material perception result, F R→L This is the elevation correction result.

[0138] S216: Identify potential hazards in transmission corridors through feature fusion.

[0139] For example, a deep learning network model is trained based on fusion features to predict elevation anomalies and material anomalies of hidden danger targets in the transmission corridor. During the training process, a joint training method is adopted, and the main loss function and auxiliary loss function are set as follows:

[0140] Main loss: Improved boundary sensitive loss, the formula is:

[0141]

[0142] Among them, W b is the boundary area weight matrix (weight within the edge 5 pixels = 3, other areas = 1), P i is the probability that the i-th pixel belongs to the hidden danger target predicted by the model (range [0,1]), G i is the true label (Ground Truth), G i =1 indicates a hidden danger area, G i = 0 indicates background. The main loss function is used to strengthen the loss weight of the boundary area and improve the positioning accuracy of the hidden danger target boundary.

[0143] Auxiliary loss: cross-modal consistency loss (constrained feature space alignment through contrastive learning), the formula is:

[0144]

[0145] Where: is the three-dimensional geometric feature extracted for the i-th sample, is the two-dimensional texture feature extracted for the i-th sample, sim(a,b) is the similarity function, and this method uses cosine similarity by default: τ is a hyperparameter that adjusts the sharpness of the similarity distribution. This method defaults to 0.1, and K is the number of negative samples in a single batch. By using contrastive learning to constrain feature space alignment, we ensure semantic consistency between features extracted by the LiDAR and visible light branches, enhancing the effectiveness of cross-modal information exchange.

[0146] Combining the main loss function and the auxiliary loss function, the deep learning model is optimized from the two perspectives of boundary localization and cross-modal consistency.

[0147] Furthermore, the identification results of potential danger targets in transmission corridors are filtered by elevation and optimized by edges. Specifically, the model's identification results are filtered by elevation threshold (e.g., excluding targets below 1 meter above the ground); and edge optimization is performed using point cloud density information, with Bezier curve interpolation performed on low-density areas.

[0148] As can be seen from the above description, this invention utilizes an improved boundary-sensitive loss and a cross-modal consistency loss to ensure the model's sensitivity to boundaries and elevation anomalies. Finally, through post-processing optimization using elevation threshold filtering and point cloud density interpolation, it generates highly accurate and reliable 3D geographic information, suitable for detailed modeling and target identification in fields such as power inspection and topographic mapping.

[0149] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an embodiment of a multi-modal transmission corridor hidden danger target identification system provided by the present invention. A multi-modal transmission corridor hidden danger target identification system 10 includes:

[0150] The data acquisition module 11 is used to collect LiDAR point cloud data and visible light image data of the transmission corridor.

[0151] The three-dimensional geometric feature extraction module 12 extracts the three-dimensional geometric features of the LiDAR point cloud data. The three-dimensional geometric features include elevation features, curvature features, and reflection intensity features.

[0152] The two-dimensional texture feature extraction module 13 is used to extract two-dimensional texture features of visible light image data, where the two-dimensional texture features include material features.

[0153] The material perception result acquisition module 14 is used to determine the elevation mutation area based on the elevation features and curvature features, extract the regional texture of the elevation mutation area from the material features, calculate the attention weight of the elevation mutation area and the regional texture, and obtain the material perception result.

[0154] The elevation correction result module 15 is used to determine the high-reflection area in the material characteristics, extract the reflection intensity of the high-reflection area from the reflection intensity characteristics, calculate the attention weight of the high-reflection area and the reflection intensity, and obtain the elevation correction result.

[0155] The fusion module 16 is used to perform weighted fusion on the material perception result and the elevation correction result to obtain fusion features.

[0156] The identification module 17 is used to identify potential danger targets in the transmission corridor by fusing features.

[0157] For example, in the data acquisition module 11, LiDAR point cloud data and visible light image data of the transmission corridor are collected. In the 3D geometric feature extraction module 12, the LiDAR point cloud data is voxelized to obtain a grid with a number of LiDAR point clouds greater than or equal to a preset threshold. A convolution operation is performed on the LiDAR point cloud data within the grid to obtain intermediate features, including elevation, curvature, and reflection intensity. The intermediate features are downsampled and upsampled to the original resolution to obtain the 3D geometric features of the LiDAR point cloud data, including elevation, curvature, and reflection intensity. In the two-dimensional texture feature extraction module 13, a depthwise separable convolution operation is performed on the visible light image data to obtain deep features. The deep features are position-encoded in the X and Y directions, and the position information is added to the deep features to obtain a position-encoded feature map. The deep features are subjected to a 1×1 convolution operation and a Sigmoid activation function to generate an attention weight map. The attention weight map is weighted multiplied with the position-encoded feature map to obtain two-dimensional texture features, which include material features. In the material perception result acquisition module 14, the elevation difference between adjacent grids is determined based on the elevation features. Adjacent grids with an elevation difference greater than a second preset threshold are determined to be candidate mutation regions. The curvature value of the candidate mutation region is obtained based on the curvature feature. If the curvature value of the candidate mutation region is greater than or equal to a third preset threshold, a sharp edge is present and the candidate mutation region is an elevation mutation region. Attention weights are calculated for the elevation mutation region and the regional texture to obtain the material perception result. In the elevation correction result module 15, highly reflective areas within the material features are identified, the reflection intensity of these highly reflective areas is extracted from the reflection intensity features, and attention weights are calculated for the highly reflective areas and reflection intensities to obtain the elevation correction results. In the fusion module 16, fusion weights are determined based on the point cloud density of the LiDAR point cloud data. Based on the fusion weights, the material perception results and the elevation correction results are weightedly fused to obtain fused features. In the identification module 17, the fused features are used to identify potential hazards within the transmission corridor.

[0158] A multi-modal transmission corridor hidden danger target identification system 10 also includes a post-processing module (not shown) for performing elevation filtering and edge optimization on the identification results of the transmission corridor hidden danger targets.

[0159] Exemplarily, the recognition results of the model are filtered by an elevation threshold (such as excluding targets below 1 meter above the ground); and the edge is optimized using point cloud density information, and Bezier curve interpolation is performed on low-density areas.

[0160] like Figure 4 As shown, Figure 4The device 20 includes a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 executes the computer program when working to achieve the following. Figure 1 and Figure 2 The method shown.

[0161] The specific technical details of a multimodal transmission corridor hidden danger target identification method implemented when the above-mentioned device 20 executes the computer program have been discussed in detail in the above-mentioned method steps, so they will not be repeated here.

[0162] like Figure 5 As shown, Figure 5 The structure diagram of an embodiment of the medium provided by the present invention. The medium 30 stores at least one computer program 31, which is executed by the processor 22 to implement the following Figure 1 and Figure 2 In one embodiment, the storage medium 30 may be a memory chip, a hard disk, a mobile hard disk, a USB flash drive, an optical disk, or other readable and writable storage tools, or a server.

[0163] Additionally, the processes depicted in the accompanying figures do not necessarily have to be performed in the particular order shown, or sequential order, to achieve desired results. In certain implementations, multitasking and parallel processing are also possible or may be advantageous.

[0164] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer-readable storage medium embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, refer to the description of the method embodiments.

[0165] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification correspond to each other. Therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be repeated here.

[0166] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0167] For the convenience of description, when describing the above device, various units are divided into functions and described separately. Of course, when implementing this specification, the functions of each unit can be implemented in the same one or more software and / or hardware. It should be understood by those skilled in the art that this specification embodiment can be provided as a method, system, or computer program product. Therefore, this specification embodiment can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification embodiment can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0171] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0172] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0173] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0174] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0175] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0176] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are described briefly because they are generally similar to the method embodiments. For relevant parts, refer to the description of the method embodiments.

[0177] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A multi-modal transmission corridor hidden danger target identification method, characterized in that: The method comprises: Collect LiDAR point cloud data and visible light image data of the transmission corridor; Extracting three-dimensional geometric features of the LiDAR point cloud data, wherein the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features; Extracting two-dimensional texture features of the visible light image data, wherein the two-dimensional texture features include material features; determining an elevation mutation region according to the elevation features and the curvature features, extracting a regional texture of the elevation mutation region from the material features, performing attention weight calculation on the elevation mutation region and the regional texture, and obtaining a material perception result; Determine a high-reflection area in the material feature, extract the reflection intensity of the high-reflection area from the reflection intensity feature, perform attention weight calculation on the high-reflection area and the reflection intensity, and obtain an elevation correction result; Performing weighted fusion on the material perception result and the elevation correction result to obtain fusion features; The hidden danger targets in the transmission corridor are identified through the fusion features.

2. A multi-modal transmission corridor hidden danger target identification method according to claim 1, characterized in that: The three-dimensional geometric features of the LiDAR point cloud data are extracted, wherein the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features, and specifically include: Processing the LiDAR point cloud data by voxelization to obtain a grid in which the number of LiDAR point clouds is greater than or equal to a preset threshold; Performing a convolution operation on the LiDAR point cloud data within the grid to obtain intermediate features, wherein the LiDAR point cloud data includes height values, curvature values, and reflection intensity; A downsampling operation is performed on the intermediate features, and the downsampled intermediate features are upsampled to the original resolution to obtain three-dimensional geometric features of the LiDAR point cloud data, where the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features.

3. A multi-modal transmission corridor hidden danger target identification method according to claim 2, characterized in that: The two-dimensional texture features of the visible light image data are extracted, wherein the two-dimensional texture features include material features. Specifically include: Performing a depth-separable convolution operation on the visible light image data to obtain deep features; Position encoding is performed on the deep features in the X and Y directions, and the position information is added to the deep features to obtain a feature map after position encoding; Performing a 1×1 convolution operation and a Sigmoid activation function on the deep features to generate an attention weight map; The attention weight map is weighted and multiplied with the position-encoded feature map to obtain a two-dimensional texture feature, where the two-dimensional texture feature includes a material feature.

4. A multi-modal transmission corridor hidden danger target identification method according to claim 2, characterized in that: Determining the elevation mutation area according to the elevation characteristics and curvature characteristics specifically includes: determining elevation differences between adjacent grids based on the elevation features; Determine adjacent grids whose elevation difference is greater than a second preset threshold as candidate mutation areas; Obtaining a curvature value of the candidate mutation region according to the curvature feature; If it is determined that the curvature value of the candidate mutation region is greater than or equal to a third preset threshold, a sharp edge exists and the candidate mutation region is an elevation mutation region.

5. A multi-modal transmission corridor hidden danger target identification method according to claim 4, characterized in that: The attention weight calculation of the elevation mutation area and the regional texture to obtain the material perception result specifically includes: according to The attention weights of the elevation mutation area and the regional texture are calculated to obtain the material perception results, where F L→R is the material perception result, Q L It is the area of ​​sudden elevation changes. is the transposed amount of the regional texture, V R For regional textures.

6. The multi-modal transmission corridor hidden danger target identification method according to claim 1 is characterized in that: The attention weight calculation of the high-reflective area and the reflection intensity to obtain the elevation correction result specifically includes: according to The attention weight is calculated for the high reflective area and the reflection intensity to obtain the elevation correction result, where F R→L is the elevation correction result, Q R Highly reflective area. is the transposed value of the reflection intensity, V R is the reflection intensity.

7. A multi-modal transmission corridor hidden danger target identification method according to claim 5 or 6, characterized in that: The weighted fusion of the material perception result and the elevation correction result to obtain the fusion feature specifically includes: Determining a fusion weight according to the point cloud density of the LiDAR point cloud data; The material perception result and the elevation correction result are weightedly fused according to the fusion weight to obtain a fusion feature.

8. A multi-modal transmission corridor hidden danger target identification system, characterized by: Said include: Data acquisition module, used to collect LiDAR point cloud data and visible light image data of the transmission corridor; A three-dimensional geometric feature extraction module is used to extract three-dimensional geometric features of the LiDAR point cloud data, wherein the three-dimensional geometric features include elevation features, curvature features, and reflection intensity features; A two-dimensional texture feature extraction module, configured to extract two-dimensional texture features of the visible light image data, wherein the two-dimensional texture features include material features; a material perception result acquisition module, configured to determine an elevation mutation region based on the elevation features and curvature features, extract a regional texture of the elevation mutation region from the material features, calculate an attention weight for the elevation mutation region and the regional texture, and obtain a material perception result; an elevation correction result module, configured to determine a high-reflection area in the material feature, extract the reflection intensity of the high-reflection area from the reflection intensity feature, perform attention weight calculation on the high-reflection area and the reflection intensity, and obtain an elevation correction result; A fusion module, configured to perform weighted fusion on the material perception result and the elevation correction result to obtain fusion features; An identification module is used to identify hidden danger targets in the transmission corridor through the fusion features.

9. The multi-modal transmission corridor hidden danger target identification system according to claim 8, characterized in that: The system further comprises: Post-processing module: used for performing elevation filtering and edge optimization on the identification results of the hidden danger targets in the transmission corridor.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images