Three-dimensional Gaussian particle modeling method, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of three-dimensional digital modeling, and discloses a three-dimensional Gaussian particle modeling method, electronic equipment and a readable storage medium. The modeling method comprises the steps of multi-modal data space embedding and fractional order space prior construction, state-guided three-dimensional Gaussian particle initialization and attribute expansion, state-aware projection rendering mechanism design and forward propagation modeling, multi-task particle attribute optimization mechanism under joint supervision and three-dimensional state field inference and output. Fractional order space prior is introduced, so that multi-modal information can be embedded into state particles in a non-local mode, and deep coupling modeling of a structure and an operation state is achieved. A state-aware particle rendering mechanism is introduced, synchronous expression of three attributes of space, color and state is achieved, and the expression ability of three-dimensional state distribution is improved; and a structure-state joint regular optimization strategy is introduced, so that the spatial discontinuity of state expression is effectively inhibited, and the physical consistency and interpretability of a three-dimensional state field are ensured.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional digital modeling technology, and in particular to a three-dimensional Gaussian particle modeling method, electronic device, and readable storage medium. Background Technology

[0002] In recent years, with the development of intelligent inspection of power equipment, higher requirements have been placed on the 3D visualization modeling of equipment status. While traditional point cloud-based modeling methods are relatively mature in geometric representation, their capabilities in modeling and visualizing state information (such as temperature, corrosion level, and aging grade) are limited. With the development of neural rendering, implicit modeling, and particle modeling, 3D Gaussian Splatting (3DGS), as a highly efficient and high-fidelity scene representation method, has gradually gained attention for tasks such as dynamic modeling and lightweight reconstruction. However, most currently available related technologies focus on structural reconstruction and color restoration, lacking the ability to embed operational status, especially in industrial and power scenarios where significant gaps remain.

[0003] Existing 3DGS-related solutions can be broadly categorized into three types: the first type is static or quasi-static particle modeling aimed at geometric restoration; the second type is semantic or dynamic modeling improvements based on the original 3DGS; and the third type attempts to construct multimodal representation structures in other tasks, but its technical solutions are mostly concentrated in high-level semantic spaces rather than geometric or particle levels.

[0004] Currently available 3D Gaussian modeling methods mostly aim at structural reconstruction, semantic representation, or dynamic consistency. Although some progress has been made in static geometric restoration, semantic label projection, and temporal modeling, from the perspective of methodological structure and engineering application, the following main shortcomings still exist: First, the particle structure design is simplistic, generally containing only basic attributes such as spatial position, color, orientation matrix, and covariance. It lacks the dimension of state parameters for task-oriented expansion and cannot express equipment operating status information such as temperature rise, corrosion, and aging. This limitation of particle attribute dimension restricts its applicability in practical scenarios such as industrial equipment inspection and condition diagnosis.

[0005] Second, most existing modeling processes use single-modal RGB images as input, and lack the ability to integrate multi-source information such as thermal infrared, point cloud or status labels at the data fusion level. This makes it impossible to achieve joint modeling and visualization of structural and status information, and in particular, it cannot meet the needs of power systems for operating status distribution and fault symptom prediction.

[0006] Third, in terms of the modeling-driven mechanism, existing methods generally use geometric or semantic loss for particle optimization, without constructing a supervision mechanism aimed at state representation. The particle update process is not guided by state information, making it difficult to form a strategy of "prioritizing modeling of salient state regions", thus affecting the accuracy and completeness of the final modeling results in representing key state regions.

[0007] In summary, existing technologies have not achieved a systematic design for state representation in terms of particle structure, input modes, and modeling mechanisms, and cannot yet meet the actual needs of state modeling and visualization for equipment with spatial state perception capabilities (such as substations, towers, switchgear, and disconnectors in power equipment). Summary of the Invention

[0008] This invention provides a three-dimensional Gaussian particle modeling method to address the problem that existing technologies lack a systematic design oriented towards state representation in terms of particle structure, input modes, and modeling mechanisms, thus failing to meet the practical needs of state modeling and visualization with spatial state perception capabilities. The specific technical solution is as follows: A three-dimensional Gaussian particle modeling method includes the following steps: Step 1, Multimodal Data Spatial Embedding and Fractional-Order Spatial Prior Construction, specifically: Extracting 3D structures from multi-view image sequences to generate sparse point clouds; Aligning thermal infrared data to RGB using extrinsic parameters, and projecting temperature onto the point cloud to form joint feature triples; Introducing a spatial state prior mechanism based on fractional-order kernel functions, constructing spatial propagation operations on the initial state label set to obtain the fused point set; Step 2, State-guided initialization and attribute expansion of 3D Gaussian particles, specifically: modeling particle shapes using anisotropic Gaussian kernels to obtain the covariance matrix; using infrared values, original labels, and fractional priors as inputs, obtaining state vectors through MLP; and constructing a set of nine-tuple particles containing point coordinates, covariance, RGB, state channels, and transparency. Step 3: Design and forward propagation modeling of state-aware projection rendering mechanism. Specifically, this involves: constructing a differentiable projection rendering mechanism that supports the joint output of RGB images and device state visualization diagrams; supervising the alignment of the rendered diagram with the real image, embedding particle parameters and state representations, and establishing a unified 3D-image representation channel. Step 4: Multi-task particle attribute optimization mechanism under joint supervision, specifically: In the structural dimension, the rendered RGB is aligned with the visible light image, and weighted loss is used to supervise color; in the state dimension, alignment supervision with infrared or label images is introduced, and smooth loss is used to fit the rendered state map; inter-particle semantic consistency regularization is added to enhance spatial continuity and state stability. Step 5, the inference and output of the three-dimensional state field, specifically: weighted fusion of all ions in three-dimensional space through particle state characteristics to obtain a continuous three-dimensional state field distribution.

[0009] Preferably, step one specifically includes: Step 1.1: Extract 3D geometric structure information from multi-view image sequences of devices with spatial state awareness capabilities to generate a sparse 3D point cloud set. ; Step 1.2: Obtain the thermal infrared image Transformation matrix using known extrinsic parameters Aligned to the RGB image coordinate system, the temperature information is mapped onto a 3D point cloud through projection, with each spatial point corresponding to an RGB color. Infrared temperature and status labels Three modal features are used to obtain joint feature triples; Step 1.3: Introduce a spatial state prior mechanism based on a fractional kernel function, in the initial state label set. The above constructs a space propagation operation to obtain a fractional-order smooth state field. Each spatial point is represented as containing three-dimensional point cloud coordinates. RGB colors Infrared temperature Status labels and fractional-order smooth state fields quintuple The fused point set is obtained .

[0010] Preferably, in step 1.1, the structured bundle adjustment method is used to jointly register and reconstruct 3D images from multiple frames of RGB images, generating a sparse 3D point cloud set; the structured bundle adjustment method also outputs the camera projection matrix of each frame of the image. .

[0011] Preferably, the infrared temperature in step 1.2 It can be expressed as follows: ; in: Represents perspective projection and rounds it to the image pixel coordinates; Indicates the first Temperature values ​​of corresponding pixels in a frame of thermal infrared image.

[0012] Preferably, the fractional-order smooth state field in step 1.3 It can be expressed as follows: ; ; in: These are the normalization coefficients; The fractional power exponent represents spatial propagation and controls the degree of influence of distant points on the current point; Representing a spatial point The spatial neighborhood.

[0013] Preferably, step two specifically includes: The position of a particle is directly determined by its three-dimensional point cloud coordinates; that is, the particle's center is located at... The spatial shape of the particle is modeled using an anisotropic Gaussian kernel function, and its covariance matrix is ​​denoted as... ; To express directionality, the covariance matrix is ​​decomposed into rotation matrices. With scale matrix The product of: ; Where: scale matrix Scale vector To control the spatial distribution range of particles, it is initialized to be isotropic, i.e. Rotation matrix Quaternion initialization is used, with the initial value set to unit rotation, reflecting the initial lack of direction preference; Define a state vector This is an explicit embedding used to describe the current operating state of the device; the state vector is obtained by MLP mapping with infrared values, original labels, and fractional priors as input, as follows: ; in: State embedding network; The nine-tuple of particles It can be expressed as follows: ; in: This is the transparency parameter for the particles.

[0014] Preferably, step three specifically includes: For any image pixel The particle center is then determined by mapping the ray direction in three-dimensional space back to the camera projection function. Are particles in the vicinity of this line of sight? In pixels Influence factor at location It is approximated by its two-dimensional Gaussian projection in screen space, as follows: ; in: It is a particle Image coordinates under camera projection; It is a three-dimensional Gaussian covariance The two-dimensional covariance matrix obtained by compression under view projection is used to control the diffusion range of particles in screen space; To avoid rendering artifacts, only select Particles participate in rendering calculations, and the color values ​​of image pixels With status visualization values These are defined as the weighted blending results of all particles for that pixel, as follows: ; ; in: Represents the state vector The state visual value parsed from the data; A small constant introduced to prevent numerical instability; For all observed frames Image pixels Execute the rendering process and output ; =1,...,K; All rendered images are supervised and aligned with real images, thereby embedding particle parameters, state representations, and camera poses for joint optimization, establishing a unified 3D-image representation channel.

[0015] Preferably, step four specifically includes: For structural visualization, the rendered RGB image Corresponding real visible light image Perform reconstruction and alignment using weighted loss. Color supervision is performed as follows: ; In terms of state representation, it incorporates infrared maps or state label maps. Alignment supervision, using loss The rendered state heatmap and the real state map are fitted together as follows: ; Introducing regularization terms based on inter-particle semantic consistency To enhance the spatial continuity and state stability of the particle model, the following measures are taken: ; in: Represents particles The state vector; For the first The particle and the first Spatial distance weights between particles; For the state vector of the th The value of the dimension; For the first The covariance matrix of each particle. The target covariance is set; , To adjust the hyperparameters that affect the effect; Total loss function It can be expressed as follows: ; in: , , These are the weighting coefficients.

[0016] The present invention also discloses an electronic device, which includes a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the steps of the three-dimensional Gaussian particle modeling method described above.

[0017] The present invention also discloses a readable storage medium on which a program or instruction is stored, and when the program or instruction is executed by a processor, it implements the steps of the three-dimensional Gaussian particle modeling method described above.

[0018] The effect of applying the technical solution of this invention is: This invention provides a three-dimensional Gaussian particle modeling method, which includes multimodal data spatial embedding and fractional-order spatial prior construction, state-guided three-dimensional Gaussian particle initialization and attribute expansion, state-aware projection rendering mechanism design and forward propagation modeling, a multi-task particle attribute optimization mechanism under joint supervision, and inference and output of the three-dimensional state field. The method introduces fractional-order spatial priors, enabling multimodal information to be embedded into state particles in a non-local manner, achieving deep coupling modeling of structure and operating state. It also introduces a state-aware particle rendering mechanism to achieve simultaneous expression of spatial, color, and state attributes, improving the expressive power of three-dimensional state distribution. Furthermore, it introduces a structure-state joint regularization optimization strategy to effectively suppress spatial discontinuities in state expression, ensuring the physical consistency and interpretability of the three-dimensional state field, accurately restoring the structural details of the equipment surface, and continuously and faithfully expressing temperature rise distribution and abnormal regions in space. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the three-dimensional Gaussian particle modeling method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the three-dimensional Gaussian particle weighted projection rendering mechanism in an embodiment of the present invention.

[0021] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0024] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0025] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0026] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0027] A three-dimensional Gaussian particle modeling method, specifically a three-dimensional Gaussian particle modeling method oriented towards operational state awareness, is detailed in [link to details]. Figure 1 This includes the following steps: Step 1, Multimodal Data Spatial Embedding and Fractional-Order Spatial Prior Construction, specifically: Extracting 3D structures from multi-view image sequences to generate sparse point clouds; Aligning thermal infrared data to RGB using extrinsic parameters, and projecting temperature onto the point cloud to form joint feature triples; Introducing a spatial state prior mechanism based on fractional-order kernel functions, constructing spatial propagation operations on the initial state label set to obtain the fused point set; Step 2, State-guided initialization and attribute expansion of 3D Gaussian particles, specifically: modeling particle shapes using anisotropic Gaussian kernels to obtain the covariance matrix; using infrared values, original labels, and fractional priors as inputs, obtaining state vectors through MLP; and constructing a set of nine-tuple particles containing point coordinates, covariance, RGB, state channels, and transparency. Step 3: Design and forward propagation modeling of state-aware projection rendering mechanism. Specifically, this involves: constructing a differentiable projection rendering mechanism that supports the joint output of RGB images and device state visualization diagrams; supervising the alignment of the rendered diagram with the real image, embedding particle parameters and state representations, and establishing a unified 3D-image representation channel. Step 4: Multi-task particle attribute optimization mechanism under joint supervision, specifically: In the structural dimension, the rendered RGB is aligned with the visible light image, and weighted loss is used to supervise color; in the state dimension, alignment supervision with infrared or label images is introduced, and smooth loss is used to fit the rendered state map; inter-particle semantic consistency regularization is added to enhance spatial continuity and state stability. Step 5, the inference and output of the three-dimensional state field, specifically: weighted fusion of all ions in three-dimensional space through particle state characteristics to obtain a continuous three-dimensional state field distribution.

[0028] In this preferred embodiment, step one specifically includes: Step 1.1: Extract 3D geometric structure information from multi-view image sequences of devices with spatial state perception capabilities (such as substations, towers, switchgear, and disconnectors in power equipment) to generate a sparse 3D point cloud set. Further preferably, the Structure-from-Motion (SfM) adjustment method is used to jointly register and reconstruct 3D images from multiple frames of RGB images, generating a sparse 3D point cloud set, where each spatial point... The structured bundle adjustment method also outputs the camera projection matrix for each frame of the image, representing the location of a local region in space. .

[0029] Step 1.2: To achieve collaborative modeling of 3D structures using multimodal information, thermal infrared images are used... Transformation matrix using known extrinsic parameters Aligned to the RGB image coordinate system, and then the temperature information is further mapped onto the 3D point cloud through projection. Specifically: each spatial point... In the Infrared response under frame image It can be expressed as the following formula: ; in: Represents perspective projection and rounds it to the image pixel coordinates; Indicates the first Temperature values ​​of corresponding pixels in a frame of thermal infrared image.

[0030] RGB colors of an RGB image It can be extracted from the image frame in the same way through projection.

[0031] Ultimately, each spatial point There are three modal features: RGB color Infrared temperature Status labels (Obtained through manual annotation or model inference), resulting in joint feature triples (i.e., constituting joint feature triples).

[0032] Step 1.3: To construct a state modeling capability with spatial continuity, a spatial state prior mechanism based on a fractional-order kernel function is introduced. Specifically, in the initial state label set... The above constructs a space propagation operation to obtain a fractional-order smooth state field. It is expressed by the following formula: ; ; in: These are the normalization coefficients; The fractional power exponent represents spatial propagation and controls the degree of influence of distant points on the current point; Representing a spatial point The spatial neighborhood.

[0033] This operation is essentially a power-law propagation mechanism that not only preserves local state consistency but also introduces global structural correlations, making it particularly suitable for handling long-distance weakly correlated anomalies in power equipment caused by corrosion, thermal expansion, aging, and other factors.

[0034] Each spatial point Represented as containing three-dimensional point cloud coordinates RGB colors Infrared temperature Status labels and fractional-order smooth state fields quintuple The fused point set is obtained .

[0035] The embedding result will serve as the initial input for the next step of particle modeling and state representation optimization, providing a clear and physically relevant basic representation for subsequent modeling.

[0036] In this embodiment, after completing the spatial embedding of multimodal features, the fused point set needs to be... The mapping is performed as a renderable 3D Gaussian particle representation. To this end, this embodiment designs a state-aware particle initialization method, enabling the spatial shape, color representation, and state dimension of each particle to be represented collaboratively. Step two specifically includes: The position of the particle is directly determined by the coordinates of the three-dimensional point cloud. The decision, that is, the particle center is located at The spatial shape of the particle is modeled using an anisotropic Gaussian kernel function, and its covariance matrix is ​​denoted as... ; To express directionality, the covariance matrix is ​​decomposed into rotation matrices. With scale matrix The product of: ; Where: scale matrix Scale vector To control the spatial distribution range of particles, it is initialized to be isotropic, i.e. Rotation matrix Quaternion initialization is used, with the initial value set to unit rotation, reflecting the initial lack of direction preference; The color of the particles is determined by Initialization serves as prior input for subsequently learnable color attributes. To incorporate state information into particle attributes, a state vector is defined. This is an explicit embedding used to describe the current operating state of the device; the state vector is obtained by MLP mapping with infrared values, original labels, and fractional priors as input, as follows: ; in: This is a state embedding network; it can be a two-layer perceptron or a more complex attention structure, used to map state values ​​to state feature vectors. These state vectors are not only used for color modulation in subsequent rendering processes, but also participate in joint optimization, thereby improving the semantic and physical interpretability of particles.

[0037] The nine-tuple of particles It can be expressed as follows: ; in: The transparency parameter of the particles controls their contribution to the final rendering result; initially, it is uniformly set to... This particle structure contains both traditional geometric and color information, as well as state-dimensional features, providing a parameterized framework for subsequent joint optimization based on rendering loss and state supervision.

[0038] In this embodiment, to achieve state-aware rendering of three-dimensional Gaussian particles in image space, a differentiable projection rendering mechanism is constructed, as detailed below. Figure 2 It supports joint output of RGB images and device status visualizations. This mechanism is based on a screen-space Gaussian particle reprojection algorithm, using approximate integral particle contributions to complete forward image synthesis. The rendering process uses the camera projection matrix at a given viewpoint. As input, iterate through the currently visible set of particles. This completes the accumulation of each particle's contribution to the color and state of the screen pixels. Step three specifically includes: For any image pixel The particle center is then determined by mapping the ray direction in three-dimensional space back to the camera projection function. Are particles in the vicinity of this line of sight? In pixels Influence factor at location It is approximated by its two-dimensional Gaussian projection in screen space, as follows: ; in: It is a particle Image coordinates under camera projection; It is a three-dimensional Gaussian covariance The two-dimensional covariance matrix obtained by compression under view projection is used to control the diffusion range of particles in screen space; To avoid rendering artifacts, only select Particles participate in rendering calculations, and the color values ​​of image pixels With status visualization values These are defined as the weighted blending results of all particles for that pixel, as follows: ; ; in: Represents the state vector The state visual value parsed from the data; To prevent numerical instability, a small constant is introduced. The two outputs constitute the state-modulated RGB image and the device state heatmap, respectively. Both are differentiable outputs and participate in subsequent gradient propagation.

[0039] For all observed frames Image pixels Execute the rendering process and output ; =1,...,K; All rendered images are supervised and aligned with real images, thereby embedding particle parameters, state representations, and camera poses for joint optimization, establishing a unified 3D-image representation channel.

[0040] In this preferred embodiment, to fully explore the moderating effect of state information on the expression of structural particles and improve the model's ability to model abnormal features of power equipment (such as hot spots, corrosion, aging, etc.), this embodiment designs a multi-task particle optimization framework, adopts an end-to-end training method, and establishes joint supervision for multiple objectives such as structure restoration, color reconstruction, and state visualization, thereby driving the consistent evolution of particle attributes across multiple semantic dimensions. Step four specifically includes: For structural visualization, the rendered RGB image Corresponding real visible light image Perform reconstruction and alignment using weighted loss. Color supervision is performed as follows: ; In terms of state representation, it incorporates infrared maps or state label maps. Alignment supervision.

[0041] Considering that the state values ​​may have nonlinear scaling relationships, the following approach is adopted. loss The rendered state heatmap and the real state map are fitted together as follows: ; To enhance the spatial continuity and state stability of the particle model, a regularization term based on inter-particle semantic consistency is introduced. ,as follows: ; in: Represents particles The state vector; For the first The particle and the first The spatial distance weights between individual particles are generally set in the form of a Gaussian kernel. For the state vector of the th The value of the dimension is used to estimate the state information entropy to prevent state degradation; For the first The covariance matrix of each particle; The target covariance is set to control particle-scale convergence; , To adjust the hyperparameters that affect the effect.

[0042] This regularization term takes into account the consistency between particle states, information entropy suppression, and the physical interpretability of spatial morphology, thus enabling the particle model obtained from the final training to have strong expressive stability and visual consistency in both geometric and state dimensions.

[0043] Total loss function It can be expressed as follows: ; in: , , The weighting coefficient controls the degree of influence of each task on the overall optimization process.

[0044] In this embodiment, the optimization mechanism is centered on particle properties, allowing the aggregation of supervision signals from different tasks at the particle level. All parameters are learnable variables, continuously updated during training via gradient descent. Because the rendering module is a differentiable structure, the entire process supports end-to-end training and is compatible with existing mainstream neural rendering frameworks (such as 3DGS), achieving efficient, state-aware particle modeling.

[0045] In this preferred embodiment, step five specifically involves: after model training and particle parameter optimization, the present invention utilizes the state-enhanced rendering and projection mechanism established in step three to infer and output the final particle set in three dimensions. In the actual inference stage, the same forward rendering formula as in the training stage is directly used to map the optimized particle structure to an image space from any viewpoint, realizing the visualization of multimodal information and state attributes of the device surface. Simultaneously, through weighted fusion of particle state features in three-dimensional space, a continuous three-dimensional state field distribution can be obtained, providing a high-precision data foundation for subsequent structural health analysis, anomaly detection, and digital twin modeling. The final output includes the optimized particle parameter set, state rendering images from various viewpoints, and a continuous three-dimensional state field. All results are naturally derived based on the aforementioned modeling and inference mechanism, ensuring the accuracy and engineering applicability of the three-dimensional structure and state representation.

[0046] The core of the technical solution applied in this embodiment lies in: ① For the first time, fractional-order spatial priors are introduced into 3D Gaussian particle modeling, enabling multimodal information to be embedded into state particles in a non-local manner (i.e., the particle model can effectively capture long-distance state correlations in 3D space), achieving deep coupling modeling of structure and running state, and breaking through the limitations of existing Gaussian modeling methods based only on local neighborhoods; ② A state-aware particle rendering mechanism is constructed to realize the synchronous expression of spatial, color, and state attributes, improving the expressive power of 3D state distribution; ③ A structure-state joint regularization optimization strategy is proposed to effectively suppress the spatial discontinuity of state expression, and can obtain state field results with stronger physical consistency and higher spatial continuity in 3D space, especially showing better smoothness and interpretability than existing methods in state boundaries and abrupt regions.

[0047] Application examples: Engineering experiments of this invention were conducted using typical equipment such as substation switchgear and busbars. The experiments employed a Basler acA1920-40gc industrial camera and a FLIR E53 thermal infrared imager to acquire multi-view RGB and thermal infrared images of the equipment from different spatial orientations. All sensors underwent extrinsic parameter calibration beforehand to ensure accurate registration of multimodal data in a unified space. To further obtain reference point clouds, a Leica BLK360 3D laser scanner was used to assist in acquiring 3D structural data. After data acquisition, a 3D reconstruction algorithm based on the Structure-from-Motion (SfM) principle was first applied to multiple frames of RGB images for spatial joint registration and sparse point cloud reconstruction, simultaneously obtaining camera attitude parameters. Based on the acquired sparse 3D point cloud, and combined with the extrinsic parameter calibration completed in the experiment, each frame of thermal infrared image was mapped to the corresponding RGB image spatial coordinate system through coordinate transformation. By further utilizing the camera projection matrix, infrared temperature values ​​are assigned to corresponding spatial points in the 3D point cloud, enabling multimodal feature fusion for each point, including RGB color, infrared temperature, and status labels (such as abnormal heating, corrosion, structural loosening, etc.) added based on actual inspection rules.

[0048] To achieve continuous representation of spatial state information, neighborhood relationships are established in the point cloud space. Initial state labels are propagated using a fractional-order kernel function to obtain a fractional-order state field. Each spatial point ultimately contains multimodal features such as spatial coordinates, color, temperature, artificial labels, and fractional-order state priors. Then, a three-dimensional Gaussian particle model is initialized based on the aforementioned point cloud. The particle center is determined by the point cloud spatial coordinates, the covariance parameter is initially an identity matrix, the color attribute is taken from RGB features, and the state attributes are fused and mapped using a multilayer perceptron to include temperature, labels, and fractional-order priors. The particle transparency parameter is set to 0.8.

[0049] All particles are input into the end-to-end optimization framework proposed in this invention, and joint training is performed on an NVIDIA RTX 3090 GPU platform. During training, a multi-objective loss function is incorporated, including a weighted loss between the reconstructed RGB image and the ground truth image. Rendering heatmaps and real infrared images Loss, particle state field space regularization, etc., are used to continuously adjust particle parameters to obtain the optimal structure and state representation. The number of training iterations is set according to the amount of data (e.g., 200,000 steps), and multi-view device images are rendered in batches at each step.

[0050] After model convergence, inference is performed from a new device perspective, outputting a jointly rendered image of the structure and state field. Experimental results show that the final model can accurately reproduce the surface structural details of the device, and spatially continuously and faithfully express the temperature rise distribution and abnormal areas. Using manually labeled high-temperature and corrosion areas as standards, the anomaly detection accuracy of the method of this invention exceeds 91%, and the state field distribution is continuous and the boundaries are natural, which can meet the engineering needs of digital twins, remote diagnostics, etc. The overall inference speed is excellent, which can support real-time field applications.

[0051] This embodiment also provides an electronic device, which includes a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the various steps of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0052] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0053] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described three-dimensional Gaussian particle modeling method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0054] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0055] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A three-dimensional Gaussian particle modeling method, characterized in that, Includes the following steps: Step 1, Multimodal Data Spatial Embedding and Fractional-Order Spatial Prior Construction, specifically: Extracting 3D structures from multi-view image sequences to generate sparse point clouds; Aligning thermal infrared data to RGB using extrinsic parameters, and projecting temperature onto the point cloud to form joint feature triples; Introducing a spatial state prior mechanism based on fractional-order kernel functions, constructing spatial propagation operations on the initial state label set to obtain the fused point set; Step 2, State-guided initialization and attribute expansion of 3D Gaussian particles, specifically: modeling particle shapes using anisotropic Gaussian kernels to obtain the covariance matrix; using infrared values, original labels, and fractional priors as inputs, obtaining state vectors through MLP; and constructing a set of nine-tuple particles containing point coordinates, covariance, RGB, state channels, and transparency. Step 3: Design and forward propagation modeling of state-aware projection rendering mechanism. Specifically, this involves: constructing a differentiable projection rendering mechanism that supports the joint output of RGB images and device state visualization diagrams; supervising the alignment of the rendered diagram with the real image, embedding particle parameters and state representations, and establishing a unified 3D-image representation channel. Step 4: Multi-task particle attribute optimization mechanism under joint supervision, specifically: In the structural dimension, the rendered RGB is aligned with the visible light image, and weighted loss is used to supervise color; in the state dimension, alignment supervision with infrared or label images is introduced, and smooth loss is used to fit the rendered state map; inter-particle semantic consistency regularization is added to enhance spatial continuity and state stability. Step 5, the inference and output of the three-dimensional state field, specifically: weighted fusion of all ions in three-dimensional space through particle state characteristics to obtain a continuous three-dimensional state field distribution.

2. The three-dimensional Gaussian particle modeling method as described in claim 1, characterized in that, Step one specifically includes: Step 1.1: Extract 3D geometric structure information from multi-view image sequences of devices with spatial state awareness capabilities to generate a sparse 3D point cloud set. ; Step 1.2: Obtain the thermal infrared image Transformation matrix using known extrinsic parameters Aligned to the RGB image coordinate system, the temperature information is mapped onto a 3D point cloud through projection, with each spatial point corresponding to an RGB color. Infrared temperature and status labels Three modal features are used to obtain joint feature triples; Step 1.3: Introduce a spatial state prior mechanism based on a fractional kernel function, in the initial state label set. The above constructs a space propagation operation to obtain a fractional-order smooth state field. Each spatial point is represented as containing three-dimensional point cloud coordinates. RGB colors Infrared temperature Status labels and fractional-order smooth state fields quintuple The fused point set is obtained .

3. The three-dimensional Gaussian particle modeling method as described in claim 2, characterized in that, In step 1.1, the structured bundle adjustment method is used to jointly register and reconstruct 3D images from multiple frames of RGB images, generating a sparse 3D point cloud set; The structured bundle adjustment method also outputs the camera projection matrix for each frame of image. .

4. The three-dimensional Gaussian particle modeling method as described in claim 3, characterized in that, Infrared temperature in step 1.2 It can be expressed as follows: ； in: Represents perspective projection and rounds it to the image pixel coordinates; Indicates the first Temperature values ​​of corresponding pixels in a frame of thermal infrared image.

5. The three-dimensional Gaussian particle modeling method as described in claim 2, characterized in that, Fractional smooth state field in step 1.3 It can be expressed as follows: ； ； in: These are the normalization coefficients; The fractional power exponent represents spatial propagation and controls the degree of influence of distant points on the current point; Representing a spatial point The spatial neighborhood.

6. The three-dimensional Gaussian particle modeling method as described in claim 2, characterized in that, Step two specifically includes: The position of the particle is directly determined by the coordinates of the three-dimensional point cloud. The decision, that is, the particle center is located at The spatial shape of the particle is modeled using an anisotropic Gaussian kernel function, and its covariance matrix is ​​denoted as... ; To express directionality, the covariance matrix is ​​decomposed into rotation matrices. With scale matrix The product of: ； Where: scale matrix Scale vector To control the spatial distribution range of particles, it is initialized to be isotropic, i.e. Rotation matrix Quaternion initialization is used, with the initial value set to unit rotation, reflecting the initial lack of direction preference; Define a state vector This is an explicit embedding used to describe the current operating state of the device; the state vector is obtained by MLP mapping with infrared values, original labels, and fractional priors as input, as follows: ； in: State embedding network; The nine-tuple of particles It can be expressed as follows: ； in: This is the transparency parameter for the particles.

7. The three-dimensional Gaussian particle modeling method as described in claim 6, characterized in that, Step three specifically includes: For any image pixel The particle center is then determined by mapping the ray direction in three-dimensional space back to the camera projection function. Are particles in the vicinity of this line of sight? In pixels Influence factor at location It is approximated by its two-dimensional Gaussian projection in screen space, as follows: ； in: It is a particle Image coordinates under camera projection; It is a three-dimensional Gaussian covariance The two-dimensional covariance matrix obtained by compression under view projection is used to control the diffusion range of particles in screen space; To avoid rendering artifacts, only select Particles participate in rendering calculations, and the color values ​​of image pixels With status visualization values These are defined as the weighted blending results of all particles for that pixel, as follows: ； ； in: Represents the state vector The state visual value parsed from the data; A small constant introduced to prevent numerical instability; For all observed frames Image pixels Execute the rendering process and output ; =1,...,K; All rendered images are supervised and aligned with real images, thereby embedding particle parameters, state representations, and camera poses for joint optimization, establishing a unified 3D-image representation channel.

8. The three-dimensional Gaussian particle modeling method as described in claim 7, characterized in that, Step four specifically includes: For structural visualization, the rendered RGB image Corresponding real visible light image Perform reconstruction and alignment using weighted loss. Color supervision is performed as follows: ； In terms of state representation, it incorporates infrared maps or state label maps. Alignment supervision, using loss The rendered state heatmap and the real state map are fitted together as follows: ； Introducing regularization terms based on inter-particle semantic consistency To enhance the spatial continuity and state stability of the particle model, the following measures are taken: ； in: Represents particles The state vector; For the first The particle and the first Spatial distance weights between particles; For the state vector of the th The value of the dimension; For the first The covariance matrix of each particle. The target covariance is set; , To adjust the hyperparameters that affect the effect; Total loss function It can be expressed as follows: ； in: , , These are the weighting coefficients.

9. An electronic device, characterized in that, include: A memory that stores programs or instructions; A processor, configured to implement the steps of the three-dimensional Gaussian particle modeling method as described in any one of claims 1-8 when executing the program or instructions.

10. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the three-dimensional Gaussian particle modeling method as described in any one of claims 1-8.

Images