A method, system, equipment and medium for three-dimensional modeling of power operations using multi-source information

Abstract

This invention discloses a multi-source information-based 3D modeling method, system, equipment, and medium for power operation, relating to the field of 3D modeling and information fusion technology. It includes acquiring raw point cloud and 2D image data of the power operation site using synchronously calibrated LiDAR and cameras, achieving spatiotemporal alignment through preprocessing, extracting point cloud and image features respectively, inputting them into a feature fusion model, performing association and deep fusion operations at the deep semantic level using a cross-modal alignment mechanism, and performing 3D geometric reconstruction, mesh model optimization, and texture mapping based on the generated fusion features to output a 3D model of the power operation site. This invention utilizes a cross-modal attention mechanism to achieve semantic-level alignment and fusion of geometric and texture features, generating a 3D power operation model with high geometric accuracy, rich semantic information, lightweight structure, and high visual realism, providing efficient and high-fidelity 3D digital assets for digital power grid operation and maintenance.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling and information fusion technology, specifically to a multi-source information power operation 3D modeling method, system, equipment and medium. Background Technology

[0002] With the deepening of smart grid construction, digital power operations have become an inevitable trend in the industry. 3D modeling technology is the core means to realize digital perception and visualization of power operation sites. At present, 3D reconstruction technology based on multi-source sensors has been widely used. By combining high-precision depth information obtained by LiDAR with rich texture information collected by cameras, high-fidelity digital replication of complex power facilities and equipment and their surrounding environment can be achieved. Traditional point cloud and image fusion technology usually focuses on geometric registration of data or simple feature stitching, which to some extent solves the problem of missing information from a single sensor and provides basic data support for power inspection, emergency command and virtual reality interaction.

[0003] However, existing 3D modeling methods for power operations still have shortcomings in terms of in-depth utilization of multi-source information and model refinement. On the one hand, existing technologies mostly adopt simple feature-level concatenation or fusion strategies based on shallow features, lacking an effective cross-modal semantic alignment mechanism. This makes it difficult to establish a precise correlation between point cloud geometric features and image texture features at the deep semantic level, resulting in deviations in the geometric consistency of texture details in the reconstructed model. On the other hand, in the mesh optimization and texture mapping stages, traditional methods often ignore geometric error measurement and dynamic optimization mechanisms based on imaging quality. This results in the generated 3D model having an insufficiently smooth surface mesh when facing complex and delicate structures such as power cables and insulators. The texture mapping is easily distorted by occlusion or lighting effects, failing to meet the stringent requirements of high-precision power operation simulation and analysis. Summary of the Invention

[0004] In view of the above-mentioned existing problems, the present invention provides a multi-source information power operation 3D modeling method, system, device and medium to solve the problems of lack of effective cross-modal semantic alignment mechanism, difficulty in establishing accurate correlation between point cloud geometric features and image texture features at the deep semantic level, and inability to meet the stringent requirements of high-precision power operation simulation and analysis.

[0005] To address the aforementioned technical issues, a multi-source information-based three-dimensional modeling method for power operations is proposed, including: Acquire environmental impact data and real-time power generation data of thermal power units, and monitor the operating status of thermal power units in real time to obtain key parameters such as generator speed, grid frequency, and load demand; construct an environment-load coupling prediction model to obtain predicted electricity demand and unit load demand, and adjust the unit's operating mode based on the comprehensive prediction results; under the adjusted operating mode, adopt multiple combined modes to maximize resource utilization, continuously update the prediction results, and carry out frequency regulation and peak shaving control of thermal power units.

[0006] As a preferred embodiment of the multi-source information power operation three-dimensional modeling method described in this invention, the preprocessing includes performing noise filtering operation based on spatial distribution statistics on the original point cloud data, and sampling operation based on spatial regular grid. Perform lens distortion correction based on camera intrinsic model, illumination consistency adjustment based on image statistical characteristics, and color enhancement based on color space transformation on two-dimensional image data. By utilizing the preset three-dimensional spatial transformation parameters between the lidar and the camera, the processed point cloud data and two-dimensional image data are converted into a unified spatiotemporal coordinate system.

[0007] As a preferred embodiment of the multi-source information power operation three-dimensional modeling method described in this invention, the deep fusion operation includes mapping point cloud features and image features to the same dimension feature space through feature transformation, and normalizing the spatial resolution of the features to form an intermediate feature representation that matches the dimension and resolution. The intermediate feature representation of point cloud features is used as the query source, and the intermediate feature representation of image features is used as the key source. The correlation matrix between the query source and the key source is dynamically calculated using a cross-modal alignment mechanism. Spatial attention weighting is applied to the intermediate feature representations of image features based on the correlation matrix to generate enhanced image features that are semantically and spatially aligned with point cloud features. The intermediate feature representation of point cloud features is combined with enhanced image features along the channel dimension, and the combined features are fused and refined through multi-layer feature aggregation operations to output fused features.

[0008] As a preferred embodiment of the multi-source information power operation three-dimensional modeling method described in this invention, the three-dimensional geometric reconstruction includes taking the fused features as input, decoding and upsampling through the geometric generation network in the three-dimensional reconstruction model, and constructing an initial three-dimensional voxel grid representation of the power operation site scene and facilities. The initial three-dimensional voxel raster representation is subjected to isosurface extraction and transformed into an initial mesh model composed of triangular facets. The initial mesh model is then subjected to iterative optimization. Based on image features and the spatial relationship between the camera and the 3D scene, the projection area of ​​each triangular facet in the optimized mesh model onto the 2D image data is calculated. According to the preset texture quality evaluation standard, the optimal texture source is selected, the texture coordinates are calculated, and the pixel information of the selected texture source is filled into the corresponding texture coordinates to complete the texture mapping.

[0009] As a preferred embodiment of the multi-source information power operation three-dimensional modeling method described in this invention, the noise filtering operation based on spatial distribution statistics includes: for each point in the point cloud, calculating the average distance and standard deviation of all neighboring points within the radius range; when the average distance from the current point to the neighboring points exceeds a specified multiple of the standard deviation, it is determined to be a noise point and filtered out. The sampling operation based on spatial regular grids includes dividing the three-dimensional space into cubic voxel grids with equal side lengths, and for each non-empty voxel grid, using the geometric center coordinates of all points within the grid to represent the current grid, and performing downsampling of the original point cloud.

[0010] As a preferred embodiment of the multi-source information power operation three-dimensional modeling method described in this invention, the point cloud feature extraction model includes: using a voxel-based three-dimensional feature encoding network to divide the point cloud into a regular grid of a two-dimensional plane, with each grid being a support; extracting the three-dimensional coordinates, reflection intensity, and position offset as initial features for all points falling within the same support; encoding these features into high-dimensional point features using a multilayer perceptron; performing max pooling on the high-dimensional point features of all points within each support to obtain support features; and arranging all support features according to the two-dimensional grid position to obtain a pseudo-image feature map. The image feature extraction model includes using a pre-trained deep convolutional network as the backbone network to extract convolutional feature maps of different scales from the input image, and fusing high semantic feature maps and high-resolution feature maps from top to bottom by constructing a feature pyramid network to generate a fused image feature map. The formula for generating the fused image feature map is expressed as: in, To output the value of the feature map at position (i, j), Given the pixel value at position (i+m, j+n) of the input feature map, Let be the weight value of the convolution kernel at position (m, n). For bias terms, The height of the convolution kernel. The width of the convolution kernel. The height of the input feature map, The width of the input feature map. The number of channels in the input feature map. It is the set of real numbers; The cross-modal alignment mechanism includes: transforming the intermediate feature representation of point cloud features into a query vector sequence through a linear projection layer; transforming the intermediate feature representation of image features into a key vector sequence and a value vector sequence through a linear projection layer; calculating an initial attention score matrix; normalizing each row of the matrix using the Softmax function to generate a correlation matrix; and performing matrix multiplication between the correlation matrix and the value vector sequence to obtain enhanced image features. The formula for calculating the initial attention score matrix is ​​expressed as follows: in, For two input sequences and The calculated cross-attention output, To query the input sequence, Input a sequence of key values. For querying the matrix, The key matrix, For value matrices, This is the transpose of the key matrix. Let be the dimension of the key vector. It is a normalized exponential function.

[0011] As a preferred embodiment of the multi-source information power operation three-dimensional modeling method of the present invention, the iterative optimization process includes: extracting the triangular mesh surface from the three-dimensional voxel raster representation using the moving cube algorithm; simplifying the mesh using the edge folding algorithm based on geometric error metric; folding the edge with the smallest geometric shape change by iterative selection; merging the two vertices of the edge into a new vertex; implementing surface smoothing processing based on surface smoothing constraints using the Laplace smoothing algorithm; and iteratively adjusting the position of each vertex in the mesh. The edge folding algorithm formula is expressed as: in, The cost of folding edge e. The edge to be folded. The side length weighting coefficient is used. Let e ​​be the Euclidean distance between the two endpoints of edge e. and Let e ​​be the coordinates of the two vertices of edge e. The weighting coefficient of the angle between the normal vectors. The dot product of the normal vectors of the two vertices. Let p be the new position after smoothing. For smoothing coefficients, Let p be the set of adjacent vertices of vertex p; The completion of texture mapping includes: for a vertex of the mesh model, using the intrinsic and extrinsic parameter matrix obtained by camera calibration, transforming the three-dimensional coordinates of the vertex to the camera coordinate system; calculating the pixel coordinates of the vertex on the corresponding two-dimensional image through the projection model; and measuring the sharpness by calculating the gradient magnitude of the image of the projection area according to the preset texture quality evaluation standard, and measuring the degree of frontal view by calculating the angle between the projection direction and the normal vector of the triangular facet. Texture coordinate calculation involves mapping the projection region of the triangular facet onto the optimal texture source image to a normalized two-dimensional coordinate space through an affine transformation, and assigning two-dimensional texture coordinates to each vertex. The formula for calculating texture coordinates is expressed as: in, The three-dimensional coordinates of the vertex in the camera coordinate system. The three-dimensional coordinates of the vertex in the world coordinate system. Let be a rotation matrix. It is a translation vector. Let be the horizontal pixel coordinates of the vertex in the 2D image. Let be the vertical pixel coordinates of the vertex in the 2D image. For the camera's focal length, This represents the X-coordinate of the vertex in the camera coordinate system. This represents the Y-coordinate of the vertex in the camera coordinate system. This represents the Z-coordinate of the vertex in the camera coordinate system. The x-coordinate of the principal point The ordinate of the main point.

[0012] The beneficial effects of this preferred technical solution are as follows: spatial alignment is achieved by preprocessing multi-source data, semantic representation capability is improved by deeply fusing point cloud and image features through cross-modal mechanism, and high-fidelity mapping is performed by iteratively optimizing the mesh and selecting the texture source, thereby constructing a visually realistic, geometrically accurate and efficient 3D model of the power operation site.

[0013] As a preferred embodiment of the multi-source information power operation three-dimensional modeling system described in this invention, it is characterized by including a multi-source data acquisition module, a data preprocessing module, a dual-modal feature parallel extraction module, a feature deep fusion module, and a three-dimensional reconstruction and texture mapping module.

[0014] The multi-source data acquisition module is used to establish a precise transformation relationship between the two sensor coordinate systems by jointly calibrating the intrinsic and extrinsic parameters of the lidar and the camera, plan and execute scanning and shooting paths adapted to the spatial distribution characteristics of power field equipment, control the sensors to operate at a matching acquisition frequency, and record timestamps.

[0015] The data preprocessing module is used to perform statistical filtering and radius filtering on the laser point cloud to remove outlier noise using a targeted combination of algorithms, and to perform lens distortion correction, adaptive histogram equalization illumination compensation and HSV space color enhancement on the camera image by voxel downsampling to normalize the data density. The processed point cloud and image data are then converted to a unified coordinate system using calibration parameters.

[0016] The dual-modal feature parallel extraction module is used to extract high-level abstract features representing the geometric structure and surface texture of power equipment from preprocessed point cloud and image data, respectively. For point cloud data, a voxel-based encoding network is used to transform the disordered 3D point cloud into a pseudo-image feature map to capture the spatial distribution and shape contour of the equipment. For image data, a deep convolutional network combining a feature pyramid network and an attention mechanism is used to extract and fuse multi-scale image semantic features and detailed texture features.

[0017] The feature deep fusion module is used to map the extracted bimodal features to an interactive dimension, using point cloud features as queries and image features as keys, and dynamically calculates relevance weights using a cross-attention mechanism. Image texture features are attached to the corresponding point cloud geometry, and the fused feature representation is generated by feature addition and convolution.

[0018] The 3D reconstruction and texture mapping module is used to generate an initial voxel model and triangular mesh based on fusion features through 3D convolution and moving cube algorithms, optimize the mesh to obtain a geometric model, and generate texture maps based on the optimized mesh and the original multi-view images through projection calculation and weighted fusion strategies, and integrate semantic tags to output 3D results.

[0019] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a method for three-dimensional modeling of multi-source information power operations.

[0020] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for three-dimensional modeling of multi-source information power operations.

[0021] The beneficial effects of this invention are as follows: By utilizing synchronously calibrated lidar and cameras to acquire and preprocess multi-source data, this invention achieves spatial alignment and quality optimization of data under a unified benchmark, improving the signal-to-noise ratio and geometric consistency of the input data; by employing a voxel-based 3D coding network and feature pyramid network to extract features, and utilizing a cross-modal alignment mechanism for deep fusion computation, it achieves accurate association and complementarity of different modal features at the deep semantic level, significantly enhancing the feature's ability to represent small targets; by decoding the fused features into 3D voxels and converting them into triangular meshes, it achieves the initial construction from abstract features to solid geometry; by using an edge folding algorithm based on geometric error measurement and a Laplacian smoothing algorithm for iterative optimization, it achieves lightweighting and surface smoothing of the model, improving computational efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 The above is a flowchart of a multi-source information power operation three-dimensional modeling method provided in one embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram illustrating feature information alignment for a multi-source information power operation three-dimensional modeling method provided in an embodiment of the present invention.

[0025] Figure 3 The present invention provides a system scheme flowchart for a multi-source information power operation three-dimensional modeling system according to an embodiment of the present invention. Detailed Implementation

[0026] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0027] Example 1, referring to Figure 1 As an embodiment of the present invention, a three-dimensional modeling method for power operations based on multi-source information is provided, comprising: S100: Uses LiDAR to scan the power operation site to obtain raw point cloud data, and uses a camera that is synchronously calibrated with the LiDAR to take pictures of the site to obtain two-dimensional image data. The raw point cloud data and the two-dimensional image data are preprocessed respectively.

[0028] S200: The preprocessed point cloud data is processed by a point cloud feature extraction model to extract point cloud features, and the preprocessed two-dimensional image data is processed by an image feature extraction model to extract image features.

[0029] S300: Input point cloud features and image features into the feature fusion model, associate and align the features at the deep semantic level through a cross-modal alignment mechanism, and perform deep fusion operations to generate fused features.

[0030] S400: Input the fused features into the 3D reconstruction model, perform 3D geometric reconstruction, mesh model optimization and texture mapping based on the fused features, and output a 3D model of the power operation site.

[0031] It should be noted that this invention achieves spatiotemporal alignment of geometric and texture information through synchronous acquisition and preprocessing of multi-source data, deeply fuses bimodal features using a cross-modal attention mechanism to achieve semantic-level association, and constructs a lightweight three-dimensional mesh through geometric generation and iterative optimization to improve processing efficiency. Based on quality evaluation criteria, it intelligently fuses the optimal texture to generate a realistic surface.

[0032] Example 2, refer to Figure 1 and Figure 2 This is a second embodiment of the present invention, which provides a multi-source information-based three-dimensional modeling method for power operations, including: In step S100, acquiring the raw point cloud data includes steps S101~S102: S101: Uses a lidar and a synchronously calibrated camera to collect data at the power operation site; the lidar distance accuracy is ±2cm (within 10m), the angle accuracy is ±0.1° horizontally and ±0.2° vertically, and the scanning frequency is set to 5~20Hz to match the camera frame rate; the camera resolution is 4K (3840×2160), the frame rate is 15-30fps, and the focal length is 50mm.

[0033] S102: During data collection, a grid-like scanning path (spacing ≤3m, overlap rate ≥30%) is used for ground equipment, and a spiral ascending path is used for high-altitude poles / cables, combined with shooting from multiple orthogonal directions (0°, 120°, 240°) to ensure full scene coverage and data complementarity.

[0034] Furthermore, in step S100, the preprocessing includes point cloud data preprocessing, image data preprocessing, and spatiotemporal data alignment, specifically including steps S111~S113: S111: Point cloud data preprocessing includes performing noise filtering operations based on spatial distribution statistics on the raw point cloud data, and sampling operations based on spatial regular grids, to reduce data density and random noise while preserving key geometric structures.

[0035] S112: Image data preprocessing includes performing lens distortion correction based on camera intrinsic parameter model, illumination consistency adjustment based on image statistical characteristics, and color enhancement based on color space transformation on two-dimensional image data to eliminate imaging distortion and improve the contrast and recognizability of image features.

[0036] S113: Data spatiotemporal alignment includes using the three-dimensional spatial transformation parameters obtained by pre-calibration between the lidar and the camera to convert the pre-processed point cloud data and two-dimensional image data into a unified spatiotemporal coordinate system, ensuring that the three-dimensional points in the point cloud and the two-dimensional pixels in the image have a clear spatial correspondence. The conversion formula is expressed as: in, Let the point be the three-dimensional coordinates in the camera coordinate system. Let the coordinates of the point be the three-dimensional coordinates in the lidar coordinate system. Let be a rotation matrix. It is a translation vector.

[0037] It should be noted that, in the embodiments of this application, the point cloud data preprocessing in step S111 includes steps A1 to A3: A1: Statistical filtering includes calculating the average distance (μ) and standard deviation (σ) of the 15 to 20 nearest neighbors for each point in the point cloud. When the distance from a point to the neighborhood center is greater than (μ + 1.5σ ~ 2.0σ), it is considered an outlier noise point and is removed.

[0038] A2: Radius filtering includes setting the search radius to 0.3~0.5 meters. For each point in the point cloud, the number of neighboring points within the radius R is counted. If the number of neighboring points is less than 8~10, it is determined to be a discrete noise point and removed.

[0039] A3: Perform voxel downsampling and divide the filtered point cloud space into a cubic grid with a side length of 0.02 meters; for each non-empty voxel, calculate the arithmetic mean of the coordinates of all points inside to generate representative points, and reduce the point cloud density while retaining the key structure. The grid size can be dynamically adjusted according to 1 / 5 to 1 / 3 of the minimum key structure size of the power equipment.

[0040] In an optional implementation, in step S111, the point cloud data preprocessing further includes setting thresholds for the Z-axis and X / Y-axis based on the approximate spatial distribution of the power equipment, directly cropping the region of interest, removing a large number of background points, and voxelizing the cropped point cloud. Within each voxel, the average density of the points is calculated, and all points within voxels with densities significantly lower than the overall average density are filtered out as outliers, and voxel downsampling is performed.

[0041] In another optional implementation, in step S111, the point cloud data preprocessing may further include, for each point in the point cloud, fitting a local plane or surface using neighboring points, calculating the distance from the point to the fitted surface, setting a distance threshold, filtering out points whose distance exceeds the threshold as noise, and performing adaptive sampling in combination with curvature changes, sampling sparsely in flat areas and densely in high curvature areas.

[0042] Furthermore, in this embodiment of the application, in step S112, the image data preprocessing includes steps B1 to B3: B1: Perform distortion correction. Using the radial and tangential distortion coefficients obtained from camera intrinsic parameter calibration, correct the geometric deformation of the image using the OpenCV cv2.undistort() function based on the inverse mapping model. The function formula is expressed as: in, and For the distortion-free coordinates on the normalized imaging plane, and To account for the actual image coordinates after lens distortion, , and The radial distortion coefficient is... and The tangential distortion coefficient is... The radial distance from the normalized coordinates to the center of the image.

[0043] B2: Apply illumination equalization to the corrected image using a contrast-limited adaptive histogram equalization algorithm. Divide the image into 8×8 pixel blocks for local histogram equalization and limit the contrast enhancement by setting clip limit=2.0. When there are strong shadows in the scene, add gamma correction to optimize the illumination.

[0044] B3: Perform color enhancement by converting the image to the HSV color space. While keeping the hue unchanged, increase the saturation by 10%~20% and the brightness by 5%~15%. Then, convert it back to the RGB color space and perform histogram normalization to complete the color correction and optimization of the image.

[0045] In an optional implementation, in step S112, the image data preprocessing further includes treating the image as a product of illumination and reflection components, estimating the illumination components at different scales using a Gaussian surround function, separating the reflection components from the original image, and stretching and adjusting the reflection components to achieve color enhancement and detail sharpening without changing the illumination distribution assumption, while compensating for uneven illumination to a certain extent.

[0046] In another optional implementation, in step S112, the image data preprocessing may further include, after completing distortion correction, inputting the image into a pre-trained image enhancement neural network, training it with a large number of paired (low-quality to high-quality) power equipment image data, directly learning the mapping relationship from foggy, low-light, low-contrast input images to clear, bright, high-contrast output images, and completing illumination adjustment and color enhancement.

[0047] In this embodiment of the application, step S200, the extraction of point cloud features includes steps S201 to S203: S201: A feature encoding network based on voxel pillars is used to project the point cloud onto the bird's-eye view. The grid is dynamically divided according to the scanning coverage, point cloud density and power scene adaptation coefficient to determine the Pillar side length. The formula for calculating the side length of a Pillar is as follows: in, Let the length of Pillar be the side length. For power scenario adaptation coefficients, The range of devices covered by a single frame scan. This represents the point cloud density.

[0048] S202: For each non-empty Pillar, at most 100 points are randomly sampled. Seven-dimensional features containing three-dimensional coordinates, reflection intensity, and offset relative to the Pillar center are extracted. The seven-dimensional features of each point are mapped to 64 dimensions through a two-layer fully connected network (MLP) using normalization and ReLU activation functions. Max pooling is then performed on the features of all points within the same Pillar to obtain 64-dimensional Pillar-level features.

[0049] S203: Arrange all Pillar features according to the bird's-eye view grid position to generate a pseudo-image feature map of size H×D×64, and further aggregate spatial features through 3 layers of 2D convolution, finally outputting a point cloud pseudo-image feature map of size (H / 2)×(D / 2)×256; where H is the height of the input feature map and D is the width of the input feature map.

[0050] In an optional implementation, in step S200, the extraction of point cloud features further includes operating directly on the point set, constructing hierarchical point set groups through multiple farthest point sampling and ball queries, learning local features within each group using an improved PointNet (multilayer perceptron + symmetric function), and obtaining contextual information features of each point by continuously abstracting from low to high levels, selecting features of key points or upsampling global features as output.

[0051] In another optional implementation, in step S200, the extraction of point cloud features may further include quantizing the point cloud into a high-resolution three-dimensional sparse voxel grid, constructing an encoder-decoder network similar to U-Net using sparse 3D convolutional layers, sparse 3D deconvolutional layers, and skip connections. The encoder part extracts multi-level three-dimensional features by progressively downsampling through sparse convolution; the decoder part recovers the spatial resolution by upsampling through sparse deconvolution, and outputs a feature volume with the same resolution as the input voxel grid, in which each non-empty voxel contains high-dimensional features.

[0052] Furthermore, in this embodiment of the application, in step S200, the extraction of image features includes using an improved ResNet-50 combined with a Feature Pyramid Network (FPN) and an attention mechanism to extract image features, specifically including steps S211~S213: S211: Using a pre-trained ResNet-50 as the backbone network, it processes pre-processed images scaled to 1024×768 pixels and extracts the outputs of four key layers as features at different scales.

[0053] S212: Add a CBAM attention region after Layer4 output, calculate channel attention and spatial attention, extract feature weights by combining average pooling and max pooling with multilayer perceptron and convolution operations, and adjust the feature responses of channel and spatial dimensions sequentially by element-wise multiplication. The formulas for calculating channel attention and spatial attention are expressed as follows: in, For the input feature map, This is a channel attention map. This is a spatial attention map. It is the Sigmoid activation function. It is a multilayer perceptron. For global average pooling, For global max pooling, It is a 7×7 convolutional layer. For channel splicing operations; The output features are represented as follows: in, To output the feature map, This is element-wise multiplication.

[0054] S213: The attention-enhanced deep high semantic features and shallow high resolution features are fused from top to bottom using the FPN structure to generate an image feature map with a size of 256×192×256. The formula for generating the fused image feature map is expressed as: in, To output the value of the feature map at position (i, j), Given the pixel value at position (i+m, j+n) of the input feature map, Let be the weight value of the convolution kernel at position (m, n). For bias terms, The height of the convolution kernel. The width of the convolution kernel. The height of the input feature map, The width of the input feature map. The number of channels in the input feature map. It is the set of real numbers.

[0055] In an optional implementation, in step S200, the extraction of image features further includes using EfficientNet-B3 as the backbone network and a weighted bidirectional feature pyramid network as the multi-scale feature fusion region. BiFPN uses learnable weights to weight the importance of different input features and introduces a top-down and bottom-up bidirectional fusion path, so that the information flow between features of different scales is more complete.

[0056] In another optional implementation, in step S200, the extraction of image features may further include: segmenting the preprocessed image into fixed-size image blocks and linearly embedding them into a one-dimensional sequence; adding learnable classification labels; inputting the entire sequence into a standard Transformer encoder; the Transformer encoder establishes global dependencies between image blocks through a multi-head self-attention mechanism, learns the global context information of the image, and uses the output corresponding to the CLS label as global image features to perform spatial reorganization and obtain a feature map.

[0057] In step S300, generating the fusion feature includes steps S401 to S403: S401: In the feature dimension matching stage, the bilinear interpolation algorithm is used to adjust the point cloud pseudo-image feature map and the image feature map to the same spatial size; S402: In the feature alignment stage based on cross-attention, the point cloud pseudo-image feature map is mapped to a query vector sequence through a linear projection layer, and the image feature map is mapped to a key vector sequence and a value vector sequence. The correlation between the point cloud and the global position of the image is calculated using a scaling dot product model and normalized to attention weights. The enhanced image features that are highly aligned with the point cloud features are output. The formula for calculating the initial attention score matrix is ​​expressed as follows: in, For two input sequences and The calculated cross-attention output, To query the input sequence, Input a sequence of key values. For querying the matrix, The key matrix, For value matrices, This is the transpose of the key matrix. Let be the dimension of the key vector. It is a normalized exponential function.

[0058] S403: In the feature fusion and refinement stage, the obtained enhanced image features and the original point cloud features are added element-wise and then connected by residuals. The result is then concatenated with the original image features along the channel dimension, and channel compression and aggregation are performed through a 1×1 convolutional layer to generate a deep fusion feature map with dimensions of 256×192×256. The residual connection formula is expressed as: in, For the aligned image features, The original point cloud features, These are the initial fusion features after residual connections. This is an element-wise addition operation; The channel compression formula is expressed as: in, This is the spliced ​​feature map. The weights are those of a 1×1 convolution kernel. For bias terms, For the final deep fusion features, For channel indexing.

[0059] Furthermore, in step S400, the three-dimensional model of the power output work site includes steps S401~S403: S401: Perform initial 3D geometric reconstruction, input the deep fusion feature map into the decoding network consisting of 3 layers of 3D convolution to output the voxel feature tensor, and set a high voxel resolution of 0.01 meters. Use the moving cube algorithm to extract the initial triangular mesh model of the power scene to capture the small structure of the equipment.

[0060] S402: Optimize the mesh model by calculating the folding cost of each edge through an edge folding strategy. Iterate the folding of the edge with the minimum cost by comprehensively considering the edge length and the angle between the normal vectors to reduce redundancy. Apply the Laplace smoothing algorithm to iteratively update the vertex coordinates based on the positions of adjacent vertices to achieve surface smoothing. The formula for optimizing the mesh model is expressed as: in, The cost of folding edge e. The edge to be folded. The side length weighting coefficient is used. Let e ​​be the Euclidean distance between the two endpoints of edge e. and Let e ​​be the coordinates of the two vertices of edge e. The weighting coefficient of the angle between the normal vectors. The dot product of the normal vectors of the two vertices. Let p be the new position after smoothing. For smoothing coefficients, Let p be the set of adjacent vertices of vertex p.

[0061] S403: Implement texture mapping, use camera intrinsic and extrinsic parameters to convert the world coordinates of the mesh vertices into pixel coordinates on the synchronized image and normalize them to obtain texture coordinates, and evaluate the projection sharpness and orthographic degree of each view as weights in the multi-view texture fusion stage, perform weighted averaging of the texture colors of the visible image, generate high-quality texture maps without distortion or occlusion, and output a 3D model in .obj or .ply format with integrated device component-level labels. The formula for calculating texture coordinates is expressed as: in, The three-dimensional coordinates of the vertex in the camera coordinate system. The three-dimensional coordinates of the vertex in the world coordinate system. Let be a rotation matrix. It is a translation vector. Let be the horizontal pixel coordinates of the vertex in the 2D image. Let be the vertical pixel coordinates of the vertex in the 2D image. For the camera's focal length, This represents the X-coordinate of the vertex in the camera coordinate system. This represents the Y-coordinate of the vertex in the camera coordinate system. This represents the Z-coordinate of the vertex in the camera coordinate system. The x-coordinate of the principal point The ordinate of the main point.

[0062] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0063] Example 3, referring to Figure 3 This is the third embodiment of the present invention. This embodiment provides a multi-source information power operation three-dimensional modeling system, including a multi-source data acquisition module, a data preprocessing module, a dual-modal feature parallel extraction module, a feature deep fusion module, and a three-dimensional reconstruction and texture mapping module.

[0064] The multi-source data acquisition module is used to establish a precise transformation relationship between the two sensor coordinate systems by jointly calibrating the intrinsic and extrinsic parameters of the lidar and the camera, plan and execute scanning and shooting paths adapted to the spatial distribution characteristics of power field equipment, control the sensors to operate at a matching acquisition frequency, and record timestamps.

[0065] The data preprocessing module is used to perform statistical filtering and radius filtering on the laser point cloud to remove outlier noise using a targeted combination of algorithms, and to perform lens distortion correction, adaptive histogram equalization illumination compensation and HSV space color enhancement on the camera image by voxel downsampling to normalize the data density. The processed point cloud and image data are then converted to a unified coordinate system using calibration parameters.

[0066] The dual-modal feature parallel extraction module is used to extract high-level abstract features representing the geometric structure and surface texture of power equipment from preprocessed point cloud and image data, respectively. For point cloud data, a voxel-based encoding network is used to transform the disordered 3D point cloud into a pseudo-image feature map to capture the spatial distribution and shape contour of the equipment. For image data, a deep convolutional network combining a feature pyramid network and an attention mechanism is used to extract and fuse multi-scale image semantic features and detailed texture features.

[0067] The feature deep fusion module is used to map the extracted bimodal features to an interactive dimension, using point cloud features as queries and image features as keys, and dynamically calculates relevance weights using a cross-attention mechanism. Image texture features are attached to the corresponding point cloud geometry, and the fused feature representation is generated by feature addition and convolution.

[0068] The 3D reconstruction and texture mapping module is used to generate an initial voxel model and triangular mesh based on fusion features through 3D convolution and moving cube algorithms, optimize the mesh to obtain a geometric model, and generate texture maps based on the optimized mesh and the original multi-view images through projection calculation and weighted fusion strategies, and integrate semantic tags to output 3D results.

[0069] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0070] Example 4, the fourth embodiment of the present invention, differs from the previous three embodiments in that: If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0072] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0073] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

Claims

1. A three-dimensional modeling method for power operations based on multi-source information, characterized in that: include, The power operation site is scanned using a lidar to obtain raw point cloud data, and the site is photographed using a camera that is calibrated synchronously with the lidar to obtain two-dimensional image data. The raw point cloud data and the two-dimensional image data are preprocessed respectively. The preprocessed point cloud data is processed using a point cloud feature extraction model to extract point cloud features, and the preprocessed two-dimensional image data is processed using an image feature extraction model to extract image features. Point cloud features and image features are input into the feature fusion model. The features are associated and aligned at the deep semantic level through a cross-modal alignment mechanism, and a deep fusion operation is performed to generate fused features. The fused features are input into the 3D reconstruction model, and 3D geometric reconstruction, mesh model optimization and texture mapping are performed based on the fused features to output a 3D model of the power operation site.

2. The method for three-dimensional modeling of power operations based on multi-source information as described in claim 1, characterized in that: The preprocessing includes performing noise filtering based on spatial distribution statistics on the raw point cloud data, and sampling based on spatial regular grids. Perform lens distortion correction based on camera intrinsic model, illumination consistency adjustment based on image statistical characteristics, and color enhancement based on color space transformation on two-dimensional image data. By utilizing the preset three-dimensional spatial transformation parameters between the lidar and the camera, the processed point cloud data and two-dimensional image data are converted into a unified spatiotemporal coordinate system.

3. The multi-source information power operation three-dimensional modeling method as described in claim 2, characterized in that: The deep fusion operation includes mapping point cloud features and image features to a feature space of the same dimension through feature transformation, and normalizing the spatial resolution of the features to form an intermediate feature representation that matches the dimension and resolution. The intermediate feature representation of point cloud features is used as the query source, and the intermediate feature representation of image features is used as the key value source. The correlation matrix between the query source and the key value source is dynamically calculated using a cross-modal alignment mechanism. Spatial attention weighting is applied to the intermediate feature representations of image features based on the correlation matrix to generate enhanced image features that are semantically and spatially aligned with point cloud features. The intermediate feature representation of point cloud features is combined with enhanced image features along the channel dimension, and the combined features are fused and refined through multi-layer feature aggregation operations to output fused features.

4. The multi-source information power operation three-dimensional modeling method as described in claim 3, characterized in that: The three-dimensional geometric reconstruction includes taking the fused features as input, decoding and upsampling them through the geometric generation network in the three-dimensional reconstruction model, and constructing an initial three-dimensional voxel grid representation of the power operation site scene and facilities and equipment. The initial three-dimensional voxel raster representation is subjected to isosurface extraction and transformed into an initial mesh model composed of triangular facets. The initial mesh model is then subjected to iterative optimization. Based on image features and the spatial relationship between the camera and the 3D scene, the projection area of ​​each triangular facet in the optimized mesh model onto the 2D image data is calculated. According to the preset texture quality evaluation standard, the optimal texture source is selected, the texture coordinates are calculated, and the pixel information of the selected texture source is filled into the corresponding texture coordinates to complete the texture mapping.

5. The multi-source information power operation three-dimensional modeling method as described in claim 4, characterized in that: The noise filtering operation based on spatial distribution statistics includes calculating the average distance and standard deviation of all neighboring points within the radius of each point in the point cloud. If the average distance from the current point to the neighboring points exceeds a specified multiple of the standard deviation, it is determined to be a noise point and filtered out. The sampling operation based on spatial regular grids includes dividing the three-dimensional space into cubic voxel grids with equal side lengths, and for each non-empty voxel grid, using the geometric center coordinates of all points within the grid to represent the current grid, and performing downsampling of the original point cloud.

6. The multi-source information power operation three-dimensional modeling method as described in claim 5, characterized in that: The point cloud feature extraction model includes using a voxel-based three-dimensional feature encoding network to divide the point cloud into a regular grid in a two-dimensional plane, with each grid being a pillar. For all points falling within the same pillar, the three-dimensional coordinates, reflection intensity, and position offset are extracted as initial features. These are then encoded into high-dimensional point features using a multilayer perceptron. Max pooling is performed on the high-dimensional point features of all points within each pillar to obtain pillar features. All pillar features are then arranged according to the two-dimensional grid positions to obtain a pseudo-image feature map. The image feature extraction model includes using a pre-trained deep convolutional network as the backbone network to extract convolutional feature maps of different scales from the input image, and fusing high semantic feature maps and high-resolution feature maps from top to bottom by constructing a feature pyramid network to generate a fused image feature map. The formula for generating the fused image feature map is expressed as: in, To output the value of the feature map at position (i, j), Given the pixel value at position (i+m, j+n) of the input feature map, Let be the weight value of the convolution kernel at position (m, n). For bias terms, The height of the convolution kernel. The width of the convolution kernel. The height of the input feature map, The width of the input feature map. The number of channels in the input feature map. It is the set of real numbers; The cross-modal alignment mechanism includes: transforming the intermediate feature representation of point cloud features into a query vector sequence through a linear projection layer; transforming the intermediate feature representation of image features into a key vector sequence and a value vector sequence through a linear projection layer; calculating an initial attention score matrix; normalizing each row of the matrix using the Softmax function to generate a correlation matrix; and performing matrix multiplication between the correlation matrix and the value vector sequence to obtain enhanced image features. The formula for calculating the initial attention score matrix is ​​expressed as follows: in, For two input sequences and The calculated cross-attention output, To query the input sequence, Input a sequence of key values. For querying the matrix, The key matrix, For value matrices, This is the transpose of the key matrix. Let be the dimension of the key vector. It is a normalized exponential function.

7. The multi-source information power operation three-dimensional modeling method as described in claim 6, characterized in that: The iterative optimization process includes: extracting the triangular mesh surface from the three-dimensional voxel raster representation using the moving cube algorithm; simplifying the mesh using an edge folding algorithm based on geometric error metric; folding the edge with the smallest geometric change by iterative selection; merging the two vertices of the edge into a new vertex; implementing surface smoothing processing based on surface smoothing constraints using the Laplacian smoothing algorithm; and iteratively adjusting the position of each vertex in the mesh. The edge folding algorithm formula is expressed as: in, The cost of folding edge e. The edge to be folded. The side length weighting coefficient is used. Let e ​​be the Euclidean distance between the two endpoints of edge e. and Let e ​​be the coordinates of the two vertices of edge e. The weighting coefficient of the angle between the normal vectors. The dot product of the normal vectors of the two vertices. Let p be the new position after smoothing. For smoothing coefficients, Let p be the set of adjacent vertices of vertex p; The completion of texture mapping includes: for a vertex of the mesh model, using the intrinsic and extrinsic parameter matrix obtained by camera calibration, transforming the three-dimensional coordinates of the vertex to the camera coordinate system; calculating the pixel coordinates of the vertex on the corresponding two-dimensional image through the projection model; and measuring the sharpness by calculating the gradient magnitude of the image of the projection area according to the preset texture quality evaluation standard, and measuring the degree of frontal view by calculating the angle between the projection direction and the normal vector of the triangular facet. Texture coordinate calculation involves mapping the projection region of the triangular facet onto the optimal texture source image to a normalized two-dimensional coordinate space through an affine transformation, and assigning two-dimensional texture coordinates to each vertex. The formula for calculating texture coordinates is expressed as: in, The three-dimensional coordinates of the vertex in the camera coordinate system. The three-dimensional coordinates of the vertex in the world coordinate system. For rotation matrix, It is a translation vector. Let be the horizontal pixel coordinates of the vertex in the 2D image. Let be the vertical pixel coordinates of the vertex in the 2D image. For the camera's focal length, This represents the X-coordinate of the vertex in the camera coordinate system. This represents the Y-coordinate of the vertex in the camera coordinate system. This represents the Z-coordinate of the vertex in the camera coordinate system. The x-coordinate of the principal point The ordinate of the main point.

8. A multi-source information power operation three-dimensional modeling system, using the multi-source information power operation three-dimensional modeling method as described in any one of claims 1 to 7, characterized in that, It includes a multi-source data acquisition module, a data preprocessing module, a dual-modal feature parallel extraction module, a feature deep fusion module, and a 3D reconstruction and texture mapping module; The multi-source data acquisition module is used to establish a precise transformation relationship between the two sensor coordinate systems by jointly calibrating the intrinsic and extrinsic parameters of the lidar and the camera, plan and execute scanning and shooting paths adapted to the spatial distribution characteristics of power field equipment, control the sensors to work at a matching acquisition frequency, and record timestamps. The data preprocessing module is used to perform statistical filtering and radius filtering on the laser point cloud to remove outlier noise using a combination of targeted algorithms, and to perform lens distortion correction, adaptive histogram equalization illumination compensation and HSV space color enhancement on the camera image by voxel downsampling. The processed point cloud and image data are then converted to a unified coordinate system using calibration parameters. The dual-modal feature parallel extraction module is used to extract high-level abstract features representing the geometric structure and surface texture of power equipment from preprocessed point cloud and image data, respectively. For point cloud data, a voxel-based encoding network is used to transform the disordered 3D point cloud into a pseudo-image feature map to capture the spatial distribution and shape contour of the equipment. For image data, a deep convolutional network combining feature pyramid network and attention mechanism is used to extract and fuse multi-scale image semantic features and detailed texture features. The feature deep fusion module is used to map the extracted bimodal features to an interactive dimension, using point cloud features as queries and image features as keys, and dynamically calculates relevance weights using a cross-attention mechanism. It attaches image texture features to the corresponding point cloud geometry, and refines them through feature addition and convolution to generate a fused feature representation. The 3D reconstruction and texture mapping module is used to generate an initial voxel model and triangular mesh based on fusion features through 3D convolution and moving cube algorithms, optimize the mesh to obtain a geometric model, and generate texture maps based on the optimized mesh and the original multi-view images through projection calculation and weighted fusion strategies, and integrate semantic tags to output 3D results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the three-dimensional modeling method for multi-source information power operations as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional modeling method for multi-source information power operations as described in any one of claims 1 to 7.

Images