Substation three-dimensional scene reconstruction method based on neural radiation field

Abstract

The invention discloses a transformer substation three-dimensional scene reconstruction method based on a neural radiation field, and the method comprises the steps: shooting RGB images of a transformer substation at different angles and positions through a camera, recording the position parameters of the camera, then pre-training a transformer substation scene neural radiation field, and constructing a transformer substation three-dimensional simulation scene in an implicit expression manner; a substation three-dimensional simulation scene is rendered into an RGB-D image by specifying a certain visual angle, then three points are selected in an image plane to construct a 3D bounding box, a neural radiation field of a target object is optimized in the bounding box, and finally, a scene neural radiation field and the neural radiation field of the target object are fused to generate a substation three-dimensional reconstruction scene. The invention provides the real and efficient substation three-dimensional scene reconstruction method, and the intelligent operation and maintenance requirements of the substation are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of substation three-dimensional scene reconstruction, and relates to a substation three-dimensional scene reconstruction method based on neural radiation field. Background Art

[0002] With the development of smart grid technology, the demand for substation information models is gradually shifting from two-dimensional to three-dimensional reality. Traditional substation equipment information is presented in two-dimensional form through graphic images such as design drawings, reports, photos, or videos. This lacks the intuitiveness of three-dimensional information display and cannot display realistic information such as the actual size and relative position of equipment. Using 3D reconstruction technology to display substation scenes can help operations and maintenance personnel gain an intuitive understanding of various substation information and is widely used in power industry training.

[0003] Currently, the main method for displaying substation 3D scenes on the market is 3D modeling. First, through the design of file reading and writing, primitive editing, and other functions, the substation 2D plane scene is saved, created, and designed, realizing the design of the 2D scene module. Then, a graphics rendering module is constructed, and the system's 3D model rendering function is realized through the graphics rendering engine. The 3D scene modeling module is implemented using 3dsMax 3D modeling software, combining polygon modeling and patch modeling methods to implement modeling operations. Finally, the designed system is used to implement the 3D scene design of the experimental substation project. However, these methods are affected by the complexity of the scene, resulting in poor scene simulation results, limited operational functions, difficulty in adding new equipment, and the inability to create a relatively realistic substation scene. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing substation three-dimensional scene display, combine the three-dimensional reconstruction technology based on neural radiation field, and construct a three-dimensional reconstruction scene of the substation by fusing the neural radiation field of the substation background scene with the neural radiation field of the target to be added, and display the fused rendered image.

[0005] The present invention is a method for reconstructing a three-dimensional substation scene based on neural radiation fields, aiming to restore realistic multi-view scene images from limited image data. The method includes the following steps: Step 1: Use a camera to capture RGB images of the substation from different viewing angles; Step 2: Pre-train the neural radiation field of the substation scene; Step 3: Render the scene neural radiance field into an RGB-D image; Step 4: Select three points in the image plane and generate a 3D bounding box in the substation scene image; Step 5: Optimize the neural radiation field of the target object within the bounding box; Step 6: Fuse the scene neural radiation field with the target object neural radiation field; Step 7: Construct the fused 3D reconstruction scene of the substation.

[0006] Furthermore, in step 1, a camera is used to capture RGB images of the substation from different perspectives. This can be achieved by capturing the scene from all angles of the substation. Alternatively, photos can be generated frame by frame from the video, while capturing details of each device in the substation, including shielding cabinets, cameras, transformers, etc. Furthermore, the neural radiation field of the substation scene is pre-trained in step 2. The implementation method is: the substation scene image set produced in step 1 and the internal and external parameters corresponding to each image camera are used as input for training to generate the neural radiation field of the substation scene, which can synthesize images from a new perspective of the substation.

[0007] Furthermore, in step 3, the neural radiation field of the scene is rendered into an RGB-D image. The implementation method is as follows: using the neural radiation field of the substation scene obtained in step 2, a new perspective { C 0 , ..., C Nc RGB-D image under {( S 0 , D 0 ), ...,( S Nc , D Nc )}.

[0008] Furthermore, in step 4, three points are selected in the image plane to generate a 3D bounding box in the substation scene image. The implementation method is as follows: first, three points { c 1, c 2, c 3}, and reversely project these three points into the 3D scene to obtain the point { p 1, p 2, p 3}. Using these points, we can construct plane P. Next, p 1- p 2 ray as the x-axis, the z-axis perpendicular to plane P, and the y-axis calculated using the cross product operation to construct the coordinate system of the 3D bounding box. Finally, set the size of each 3D bounding box to { kx d, ky d, kz d}, where d is p 1 and p 2, { kx , ky , kz} are manually set parameters.

[0009] Furthermore, in step 5, the neural radiation field of the target object is optimized within the bounding box by rendering the RGB image of the neural radiation field of the object within the 3D bounding box generated in step 4 { G 0 , ..., G Nc} and opacity map { O 0,…, O Nc}, its implicit representation can be different from the implicit representation of the substation scene. First, from the camera C n Cast a ray and sample the query point within the 3D bounding box; then r i Corresponding RGB

[0010] value G n ( i ) and opacity values O n ( i ) Volume rendering is performed according to formula (1) and formula (2): in δ k and c k Query points q k Volume density and RGB value at ∆ k is the distance between two adjacent query points along the ray. In addition, the opacity value is set to 0, that is, when the ray is aligned with the 3D bounding box ( r i ∉ box) has no intersection or the query point is occluded by the scene foreground ( D box ( i )> D n ( i )), use the original scene content of the substation.

[0011] Furthermore, in step 6, the neural radiation field of the scene and the neural radiation field of the target object are fused.

[0012] To: Use opacity map O n , according to formula (3), the objects and substation scene rendering are seamlessly synthesized to obtain the final output I n .

[0013] Furthermore, in step 7, the fused 3D reconstruction scene of the substation is constructed by: using the final output of step 6 I n , generate a multi-view RGB-D image rendered after the substation is integrated with new equipment, repeat the operation multiple times, add multiple equipment neural radiation fields to the substation background neural radiation field scene, and finally construct a complete fused substation 3D reconstruction scene.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The neural radiation field-based 3D reconstruction method for substations used in this invention differs from previous methods that use 3D modeling to display substations in 3D. Instead, it uses a more realistic 3D reconstruction method that allows for free switching of viewing angles to produce color-rendered images of the real scene, better suited to actual scene requirements.

[0015] 2. Unlike traditional neural radiation field-based 3D reconstruction methods, this invention can freely integrate the neural radiation field of new equipment with the substation's background neural radiation field to generate new 3D objects that coordinate with the substation's 3D reality. This requires not only high-quality neural radiation field generation for the object, but also seamless integration of the generated 3D content into the existing background neural radiation field, meeting the need for flexible adaptation of the 3D display platform for different substations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 : Schematic diagram of the overall operation flow of the present invention.

[0017] Figure 2 : Schematic diagram of 3D bounding box generation in the present invention.

[0018] Figure 3 : Rendering of the fusion effect of background and target object neural radiation fields in the present invention. DETAILED DESCRIPTION

[0019] The present invention will be described in further detail below with reference to the accompanying drawings.

[0020] like Figure 1 As shown in FIG, a method for reconstructing a three-dimensional substation scene based on a neural radiation field has the following steps: Step 1: Use a camera to capture RGB images of the substation from different viewing angles; Step 2: Pre-train the neural radiation field of the substation scene; Step 3: Render the scene neural radiance field into an RGB-D image; Step 4: Select three points in the image plane and generate a 3D bounding box in the substation scene image; Step 5: Optimize the neural radiation field of the target object within the bounding box; Step 6: Fuse the scene neural radiation field with the target object neural radiation field; Step 7: Construct the fused 3D reconstruction scene of the substation.

[0021] Step 1: Use a camera to capture RGB images of the substation from different perspectives: Surround the scene from all angles of the substation, or generate photos frame by frame from the video, while capturing details of each device in the substation, including shielding cabinets, cameras, transformers, etc. Pre-training the neural radiation field of the substation scene in step 2: The substation scene image set produced in step 1 and the internal and external parameters corresponding to each image camera are used as input for training to generate the neural radiation field of the substation scene, which can synthesize images from a new perspective of the substation.

[0022] Step 3: Render the scene neural radiation field into an RGB-D image: Use the substation scene neural radiation field obtained in step 2 to generate a new perspective { C 0 , ..., C Nc RGB-D image under {( S 0 , D 0 ), ...,( S Nc , D Nc )}.

[0023] Step 4 Select three points in the image plane to generate a 3D bounding box in the substation scene image: First, select the three points in the image plane in step 3 { c 1, c 2, c 3}, and reversely project these three points into the 3D scene to obtain the point { p 1, p 2, p 3}. Using these points, we can construct plane P. Next, p 1- p 2 ray as the x-axis, the z-axis perpendicular to plane P, and the y-axis calculated using the cross product operation to construct the coordinate system of the 3D bounding box. Finally, set the size of each 3D bounding box to { kx d, ky d, kz d}, where d is p 1 and p 2, { kx , ky , kz} are manually set parameters, such as Figure 2shown.

[0024] Step 5: Optimize the neural radiation field of the target object within the bounding box: the 3D bounding box generated in step 4 , RGB image of the neural radiance field of the rendered object { G 0 , ..., G Nc} and opacity map { O 0,…, O Nc}, its implicit representation can be different from the implicit representation of the substation scene. First, from the camera C n Cast a ray and sample the query point within the 3D bounding box; then r i Corresponding RGB values G n ( i ) and opacity values O n ( i ) Volume rendering is performed according to formula (1) and formula (2): in δ k and c k Query points q k Volume density and RGB value at ∆ k is the distance between two adjacent query points along the ray. In addition, the opacity value is set to 0, that is, when the ray is aligned with the 3D bounding box ( r i ∉ box) has no intersection or the query point is occluded by the scene foreground ( D box ( i )> D n ( i )), use the original scene content of the substation.

[0025] Step 6: Fusing the scene neural radiation field with the target object neural radiation field: Using opacity map O n , , According to formula (3), objects and substation scene rendering are seamlessly synthesized to obtain the final output I n ,like Figure 3 shown.

[0026] Step 7: Build the fused 3D reconstruction scene of the substation: Using the final output from step 6 I n , generate a multi-view RGB-D image rendered after the substation is integrated with new equipment, repeat the operation multiple times, add multiple equipment neural radiation fields to the substation background neural radiation field scene, and finally construct a complete fused substation 3D reconstruction scene.

[0027] As described above, although the present invention has been shown and described with reference to certain preferred embodiments, it is not to be construed as limiting the invention itself, and various changes in form and details may be made thereto without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for reconstructing a three-dimensional substation scene based on neural radiation field, characterized in that: The following steps are involved: Step 1: Use a camera to capture RGB images of the substation from different viewing angles; Step 2: Pre-train the neural radiation field of the substation scene; Step 3: Render the pre-trained neural radiance field of the substation scene into an RGB-D image; Step 4: Select three points in the image plane and generate a 3D bounding box in the substation scene image; Step 5: Optimize the neural radiation field of the target object within the bounding box; Step 6: Fuse the neural radiation field of the substation scene with the neural radiation field of the target object; Step 7: Construct the fused 3D reconstruction scene of the substation.

2. The method for reconstructing a three-dimensional substation scene based on a neural radiation field according to claim 1, characterized in that: In step 1, a camera is used to capture RGB images of the substation from different perspectives, and the scene is captured from all angles of the substation, or photos are generated frame by frame from the video. At the same time, details of each device in the substation are captured, including shielding cabinets, cameras, and transformers.

3. The method for reconstructing a three-dimensional substation scene based on a neural radiation field according to claim 2, characterized in that: In step 2, the neural radiation field of the substation scene is pre-trained, and the substation scene image set produced in step 1 and the internal and external parameters corresponding to each image camera are used as input for training to generate the neural radiation field of the substation scene and synthesize the image of the substation from a new perspective.

4. The method for reconstructing a three-dimensional substation scene based on a neural radiation field according to claim 3, characterized in that: The step 3 renders the scene neural radiation field into an RGB-D image, and uses the substation scene neural radiation field obtained in step 2 to generate a new perspective { C 0 , ..., C Nc RGB-D image under {( S 0 , D 0 ), ...,( S Nc , D Nc )}.

5. The method for reconstructing a three-dimensional substation scene based on a neural radiation field according to claim 4 is characterized in that: The step 4 selects three points in the image plane to generate a 3D bounding box in the substation scene image. First, the three points in the image plane in step 3 are selected. c 1, c 2, c 3}, and reversely project these three points into the 3D scene to obtain the point { p 1, p 2, p 3}; Use these points to construct plane P; Next, p 1- p 2 ray as the x-axis, with the z-axis perpendicular to plane P, and the y-axis calculated using the cross product operation to construct the coordinate system of the 3D bounding box; finally, set the size of each 3D bounding box to { kx d, ky d, kz d}, where d is p 1 and p 2, { kx , ky , kz } are manually set parameters.

6. The method for reconstructing a three-dimensional substation scene based on a neural radiation field according to claim 5, characterized in that: In step 5, the neural radiation field of the target object is optimized within the bounding box, and the RGB image of the neural radiation field of the object is rendered in the 3D bounding box generated in step 4. G 0 , ..., G Nc } and opacity map { O 0,…, O Nc }, its implicit representation can be different from the implicit representation of the substation scene; first, from the camera C n Cast a ray and sample the query point within the 3D bounding box; then r i Corresponding RGB values G n ( i ) and opacity values O n ( i ) Volume rendering is performed according to formula (1) and formula (2): ， in δ k and c k Query points q k Volume density and RGB value at ∆ k is the distance between two adjacent query points along the ray; in addition, the opacity value is set to 0, that is, when the ray is aligned with the 3D bounding box ( r i ∉ box) has no intersection or the query point is occluded by the scene foreground ( D box ( i ) > D n ( i )), use the original scene content of the substation.

7. The method for reconstructing a three-dimensional substation scene based on neural radiation field according to claim 6, characterized in that: In step 6, the neural radiation field of the scene and the neural radiation field of the target object are fused using the opacity map. O n , according to the formula ， (3) Seamlessly synthesize objects and substation scene rendering to obtain the final output I n。 8. The method for reconstructing a three-dimensional substation scene based on neural radiation field according to claim 7, characterized in that: The fused substation 3D reconstruction scene is constructed in step 7, and the final output of step 6 is used. I n , generate a multi-view RGB-D image rendered after the substation is integrated with new equipment, repeat the operation multiple times, add multiple equipment neural radiation fields to the substation background neural radiation field scene, and finally construct a complete fused substation 3D reconstruction scene.

Images