Dynamic real-time rendering method of large assembly scenes based on 3D Gaussian splashing

Abstract

The present invention discloses a dynamic real-time rendering method for a large assembly scene based on three-dimensional Gaussian splashing. By combining multiple sensors, a three-dimensional Gaussian splashing model for real-time rendering of dynamic scenes is constructed. The method can render any viewing angle of the assembly scene. The three-dimensional Gaussian splashing model includes sensor data alignment, initialization of the three-dimensional Gaussian splashing model, prediction and update of the three-dimensional Gaussian splashing model. In combination with the three-dimensional Gaussian splashing model, a scene space point alignment and predictor are designed to solve the problems of multi-sensor data information redundancy and dynamic real-time rendering difficulties. This method improves the safety and reliability of assembly, can timely discover and handle problems in the assembly process, and helps to promote the rapid and high-quality development of industrial intelligent manufacturing, thereby promoting the transformation of industrial manufacturing intelligence.

Description

Technical Field

[0001] The present invention belongs to the technical field of full-scene perception of industrial assembly manufacturing, and in particular relates to a dynamic real-time rendering method for a large-scale assembly scene based on three-dimensional Gaussian splashing. Background Art

[0002] Large and complex components such as aircraft panels and aircraft skins are important components of major equipment. Ensuring the quality of their assembly process is a key factor in determining the comprehensive performance and safe operation of the equipment. The current monitoring of the assembly process mainly relies on manual monitoring, force or electrical sensor fault diagnosis, and digital twins to determine the accuracy of the assembly process. However, the above methods have the following problems: 1) Manual monitoring is inefficient; 2) The information readability of fault diagnosis through force or electrical sensors is poor and requires high professional knowledge; 3) Digital twins only synchronize data for modeled elements such as robots, actuators, and components to be assembled, while additional factors in the scene, such as people and temporary equipment, cannot obtain accurate real-time feedback. Therefore, how to conduct real-time and effective monitoring of large-scale assembly scenes is a difficulty in the manufacturing of major equipment, and it also plays an irreplaceable basic supporting role in improving the manufacturing quality of major equipment and improving the safety of the assembly process.

[0003] In view of the above problems, multi-source heterogeneous sensors are fused to optimize the three-dimensional reconstruction steps of the assembly process of large and complex parts, and to achieve efficient and accurate dynamic real-time rendering, so that assembly scene monitoring can play an important supporting role in high-quality development such as quality improvement, efficiency increase, cost reduction, and flexible production. At present, there are many monitoring methods for industrial manufacturing assembly scenes, but they are still not enough to meet the requirements of engineering applications such as full scene coverage, friendly interaction, and real-time dynamics. Therefore, how to realize real-time scene monitoring in large-scale assembly manufacturing scenes and provide friendly interaction methods for workers has become an urgent problem to be solved in this field at this stage. Compared with traditional monitoring methods, monitoring the assembly process through visual sensors still has great challenges. Among them, it is difficult for a single camera sensor to achieve reconstruction and rendering. Although a single laser radar can quickly achieve reconstruction, it lacks color information. In addition, there are problems such as large data structure differences, high difficulty in data alignment, and difficulty in information utilization in the process of fusing multiple sensors. Therefore, it is very necessary to design a dynamic real-time rendering method for large assembly scenes to realize the monitoring of the assembly process.

[0004] In recent years, real-time rendering after 3D reconstruction has received increasing attention. Among them, the representation of 3D Gaussian splashing provides a real-time solution for scene rendering. LiDAR and multiple cameras provide geometry and color information for scene rendering during the assembly process. However, there are three difficulties in the current implementation process:

[0005] 1. 3D Gaussian Splatting (3DGS) relies on the camera to optimize parameters from a large number of different perspectives. These perspectives are usually obtained through the movement of the camera itself and cannot handle changes in dynamic scenes. The monitoring of the entire assembly process requires a limited number of cameras to remain stationary, and the scene changes dynamically during the assembly process, which reduces the accuracy of 3D Gaussian Splatting parameter estimation.

[0006] 2. Before the rendering step of 3D Gaussian splash, the Gaussian parameters of the spatial points need to be obtained through deep learning training. The current 3D Gaussian splash technology has excellent real-time rendering performance in static scenes. However, the dynamic assembly process is a data stream, which means that the Gaussian parameters need to be trained for each timestamp in the data stream. Due to the existence of the training process, the real-time rendering of dynamic scenes is difficult to guarantee;

[0007] 3. In large-scale assembly processes, the entire scene is not dynamic, but some elements are in motion. Maintaining the Gaussian parameters of all spatial points in the scene is very redundant and will consume a lot of computing resources and time, which further aggravates the real-time problem. Summary of the invention

[0008] In view of the above technical problems, the present invention provides a dynamic rendering method for a large assembly scene based on three-dimensional Gaussian splashing, which aims to solve the technical problems of the Gaussian splashing rendering method, such as the difficulty in estimating Gaussian parameters at a limited viewing angle, the poor real-time performance of dynamic scene rendering, and the redundant maintenance of Gaussian parameter data.

[0009] The technical solution adopted by the present invention to solve the technical problem is:

[0010] A dynamic real-time rendering method for a large assembly scene based on three-dimensional Gaussian splashing, the method comprising the following steps:

[0011] S100: Build a full-scene perception platform to collect 3D point cloud stream data and multi-view color image stream data in large-scale assembly processes;

[0012] S200: transforming the multi-view color image data and the three-dimensional point cloud data into the world coordinate system, and aligning the point cloud, image pixels and scene space points;

[0013] S300: inputting the first frame of the aligned 3D point cloud stream data into the 3D Gaussian splash model for rendering, defining a loss function between the first frame of the aligned multi-view color flow image and the rendering result of the 3D Gaussian splash model, performing back propagation on the 3D Gaussian splash model to update the model parameters, completing the initialization of the 3D Gaussian splash model, and obtaining the 3D Gaussian parameters;

[0014] S400: Determine whether the current input frame in the aligned three-dimensional point cloud stream data is a key frame. If the current input frame is a key frame, input the key frame into the three-dimensional Gaussian splash model for rendering, and back-propagate the three-dimensional Gaussian splash model through the loss function to update the model parameters. If the current input frame is a non-key frame, design a predictor through a preset time-series state space model, train the predictor, and predict the three-dimensional Gaussian parameters with adjacent frame information to obtain the three-dimensional Gaussian parameters corresponding to the current input frame.

[0015] S500: Render the three-dimensional Gaussian parameters obtained in S400 and the point cloud of the current input frame, and finally realize dynamic real-time rendering of large assembly scenes.

[0016] Preferably, S100 includes:

[0017] S110: Four non-repetitive scanning laser radars are installed and fixed at the four corners of the ceiling beam, with the scanning direction facing the assembly scene, and twelve color cameras are installed and fixed at the four sides of the ceiling beam, and the non-repetitive scanning laser radars and color cameras on the ceiling are kept stationary;

[0018] S120: Four non-repetitive scanning laser radars collect point cloud data of four assembly scenes, and twelve color cameras collect image data of twelve assembly scenes. The point cloud data and image data will be transmitted to the high-performance computing unit in the form of data streams for processing.

[0019] Preferably, S200 includes:

[0020] S210: Obtaining coordinate transformation from any sensor of the full-scene perception platform to the world coordinate system through any annotation tool;

[0021] S220: aligning data from multiple sensors according to timestamps, and rasterizing the space of the assembly scene in a world coordinate system;

[0022] S230: Transform the coordinate system of the data of the four non-repetitive scanning laser radars into the world coordinate system to obtain The scene voxel point cloud in the world coordinate system at all times completes the alignment of the scene space points and the lidar point cloud;

[0023] S240: Aligning scene space points with color image pixels is achieved through rigid body transformation and projection transformation, wherein the rigid body transformation transforms the scene space point coordinates into the camera optical center coordinate system, and the projection transformation transforms the scene space point in the camera optical center coordinate system into pixels on the pixel plane.

[0024] Preferably, in S220, the space of the assembly scene is rasterized in the world coordinate system, specifically:

[0025] Define the assembly scene as a size of The space of voxels is defined as , the resolution of the assembly scene space can be obtained, as shown in formula (1):

[0026] (1)

[0027] S230 is specifically:

[0028] (2)

[0029] in, Indicates that from The point cloud of a LiDAR sensor, represents the coordinates of a point in space, represents the coordinate transformation from any sensor to the world coordinate system, Indicates Sensors, Represents operations involving rotation and translation; spatial point coordinates The following formula (3) should be satisfied:

[0030] (3)

[0031] Calculation Point The voxel center coordinates , calculated as shown in formula (4):

[0032] (4)

[0033] The symbol Indicates rounding up, and the current The voxel coordinates of the four non-repeated scanning lidar data at the moment ,Right now express The voxel point cloud of the scene in the world coordinate system at all times;

[0034] S240 is specifically:

[0035] Calculate the image pixels corresponding to the voxel point cloud, specifically:

[0036] (5)

[0037] (6)

[0038] in, Represents the scene voxel point cloud in the world coordinate system, and its coordinate value is , is the coordinate value of the pixel coordinate system, Indicates The rotation of the sensor to the world coordinate system, Indicates The translation of the sensor to the world coordinate system, Indicates The intrinsic parameters of a sensor can be obtained by calibration in any way or based on the camera parameter information. and The camera is and The focal length in the direction, and They are the imaging plane and the camera coordinate system The value of the intersection of the axes in pixel coordinates;

[0039] Finally, the correspondence between scene space points and image pixels is obtained, and the alignment of scene space points and pixels is achieved.

[0040] Preferably, S300 includes:

[0041] S310: Scene voxel point cloud exist Initialize the three-dimensional Gaussian sphere at the point: First, Perform uniform downsampling to obtain ,by Any point in Initialize the 3D Gaussian sphere at the point as the center , the calculation formula is as follows (7):

[0042] (7)

[0043] in, represents the covariance matrix, which affects the size and shape of the three-dimensional Gaussian sphere;

[0044] Since any three-dimensional Gaussian sphere can be transformed from a standard three-dimensional Gaussian sphere by affine transformation According to the definition of affine transformation, Writing formula (8):

[0045] (8)

[0046] in Indicates rotation, Indicates zoom;

[0047] Since the three-dimensional Gaussian sphere needs to be projected onto the two-dimensional imaging plane to affect the rendering result, according to the image pixels corresponding to the voxel point cloud in S240, any point The three-dimensional Gaussian sphere centered on is projected onto the imaging plane in any camera coordinate system to obtain the projection point The corresponding two-dimensional Gaussian sphere, so the projection point in the camera coordinate system As shown in formula (9), point The corresponding covariance matrix of the two-dimensional Gaussian sphere As shown in formula (10):

[0048] (9)

[0049] (10)

[0050] in Represents the transformation from the world coordinate system to the camera coordinate system, Represents a projection transformation The Jacobian matrix of is used as an affine approximation of the projection transformation. In particular, the result of directly projecting the Gaussian sphere is not necessarily a Gaussian sphere, so it is necessary to pass the projection transformation at the point The position of is approximated using the Jacobian matrix;

[0051] S320: Spherical harmonics are used to describe color information in different directions. The specific expressions of the basis functions of the spherical harmonics are shown in formulas (11)(12)(13)(14):

[0052] (11)

[0053] (12)

[0054] (13)

[0055] (14)

[0056] in represents the order of spherical harmonics, , represents the azimuth of the spherical coordinate system, represents the elevation angle of the spherical coordinate system;

[0057] The color representation in any direction is shown in formula (15):

[0058] (15)

[0059] S330: Define a ray in the assembly scene from a pixel to the optical center , remember the light in Go up to point The probability of not being blocked before is , light at the point The probability density of the colliding particles at , light at the point The color emitted is , so the final color of the pixel can be obtained along the light Integrate as shown in formula (16):

[0060] (16)

[0061] By solving the differential equation, we can get:

[0062] (17)

[0063] in It can be obtained from formula (7) in S310, It can be obtained by formula (15) in S320. Finally, according to The color information of each pixel can be obtained, that is, the rendering result.

[0064] Preferably, the loss function between the aligned first frame of the multi-view color flow image and the rendering result of the three-dimensional Gaussian splash model is defined in S300, specifically:

[0065] Remember 12 color cameras with different viewing angles The 12 images collected at the time are , according to formula (16) in S330, the rendering results corresponding to the viewing angles of the 12 cameras are obtained , respectively using The loss function is constructed by using the norm and structural similarity SSIM, where The loss function is shown in formula (18): The loss function is shown in formula (19):

[0066] (18)

[0067] (19)

[0068] in and Under the sliding window and The mean of and Under the sliding window and The variance of For sliding window and The covariance of and are constants respectively, where the sliding window means that each calculation takes a window of size , and traverse the entire image with this window;

[0069] Finally, the total loss function is obtained As shown in formula (20):

[0070] (20)

[0071] According to the loss function Back-propagation is performed on the three-dimensional Gaussian splash model to optimize the three-dimensional Gaussian parameters.

[0072] Preferably, the method of determining whether the current input frame is a key frame in S400 is as follows:

[0073] S410: determining the motion state of the scene by performing a two-frame difference method on the image; wherein the two-frame difference method refers to performing a difference on the images of adjacent frames and determining whether there is a moving target in the current scene according to the difference result; if the current scene is in a stationary state, the parameters of the three-dimensional Gaussian splash model remain unchanged;

[0074] S420: If the assembly scene is in the assembly operation state, it is necessary to dynamically maintain the parameters of the three-dimensional Gaussian splash model of the assembly scene. Under fixed computing resources, it takes time to train a three-dimensional Gaussian splash model for one frame of the assembly scene. , the time it takes for the laser radar to collect one frame of data is , the time it takes for a color camera to collect one frame of data is , then the minimum time interval for updating a three-dimensional Gaussian splash through training is , satisfying the conditions of formula (21):

[0075] (twenty one)

[0076] Therefore, if the assembly scene is The operation state is entered at the moment, then the dynamic scene A key frame can be considered to be located at time , and at the same time The point cloud and image corresponding to each key frame are shown in formula (22) and formula (23) respectively:

[0077] (twenty two)

[0078] (twenty three)

[0079] The symbol Indicates rounding up. For the The point cloud corresponding to the key frame, No. The image corresponding to each key frame.

[0080] Preferably, predicting the three-dimensional Gaussian parameters in S400 includes the following steps:

[0081] S430: For the key frame, the three-dimensional Gaussian splash model of the assembly scene space point can be updated through training, and for the key frame The assembly scene within the time range is predicted by the preset time series state space model for the three-dimensional Gaussian splash model within the time range: The keyframes are ,but The set of frames in the time range is ,in represents the maximum number of frames that the sensor can collect within this time range. Therefore, the data flow in the dynamic assembly scene can be recorded as the set in formula (24): :

[0082] (twenty four)

[0083] Then the corresponding three-dimensional Gaussian splash model parameters in the data stream can be recorded as the set in formula (25): :

[0084] (25)

[0085] S440: According to the collection , we get the time series of the parameters of the three-dimensional Gaussian splash model. Since the motion in the assembly space is continuous, we assume that the first The parameters of the 3D Gaussian model of the frame are affected by the previous The parameters of the frame are affected, so the relationship expression can be obtained as shown in formula (26):

[0086] (26)

[0087] S450: According to formula (26) in S440, a predictor is designed by a time series state space model , first define the expression of the time series state space model as shown in formula (27):

[0088] (27)

[0089] in , , is the state equation coefficient, Refers to multi-layer perceptron, Implicit function representing the state;

[0090] S460: Then the predictor Training: Construct the data flow in the assembly scenario according to the method described in S430 , the data flow covers any link of the assembly task, and the sample pairs are constructed according to the data flow, as shown in formula (28):

[0091] (28)

[0092] in, Obtained through the training of Gaussian splash model, is the sample pair of the j-th three-dimensional Gaussian splash model, for the data stream All data in the Gaussian splash model are input for training and constructed in the form of Sample pairs form a data set , through the predictor Predicted , the predicted and truth value Calculate the loss function for cosine similarity , as shown in formula (29):

[0093] (29)

[0094] according to For predictors Backpropagation is performed to update the predictor parameters and finally complete the predictor training, according to the trained predictor Make predictions about the parameters of a 3D Gaussian.

[0095] The above-mentioned dynamic real-time rendering method of large assembly scenes based on three-dimensional Gaussian splashing is constructed by combining multiple sensors to construct a three-dimensional Gaussian splashing method for real-time rendering of dynamic scenes. This method can render any perspective of the assembly scene. The model includes sensor data alignment, initialization of the three-dimensional Gaussian splashing model, prediction and update of the three-dimensional Gaussian splashing model. Combined with the three-dimensional Gaussian splashing model, the scene space point alignment and predictor are designed to solve the problem of multi-sensor data information redundancy and dynamic real-time rendering difficulties. This method is expected to improve the safety and reliability of assembly, thereby promoting the transformation of industrial manufacturing intelligence.

[0096] The method proposed in the present invention can perform full-scene dynamic real-time rendering of the assembly process in an industrial manufacturing scenario and complete scene monitoring. The rendered three-dimensional scene can be presented at an image-level display at any angle on the interactive interface, which can increase the safety and reliability of the assembly process, and can promptly discover and handle problems in the assembly process, helping to promote the rapid and high-quality development of industrial intelligent manufacturing. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is a flow chart of a dynamic real-time rendering method of a large assembly scene based on three-dimensional Gaussian splashing in one embodiment of the present invention;

[0098] Figure 2 It is a schematic diagram of a dynamic real-time rendering method of a large assembly scene based on three-dimensional Gaussian splashing in another embodiment of the present invention;

[0099] Figure 3 Schematic diagram of a full-scene perception platform in one embodiment of the present invention, wherein (a) is a side view and (b) is a front view;

[0100] Figure 4 Schematic diagram of a coordinate system in one embodiment of the present invention. DETAILED DESCRIPTION

[0101] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0102] In one embodiment, Figure 1 and Figure 2 As shown, a dynamic real-time rendering method for a large assembly scene based on three-dimensional Gaussian splashing includes the following steps:

[0103] S100: Build a full-scene perception platform to collect 3D point cloud stream data and multi-view color image stream data in large-scale assembly processes;

[0104] S200: transforming the multi-view color image data and the three-dimensional point cloud data into the world coordinate system, and aligning the point cloud, image pixels and scene space points;

[0105] S300: inputting the first frame of the aligned 3D point cloud stream data into the 3D Gaussian splash model for rendering, defining a loss function between the first frame of the aligned multi-view color flow image and the rendering result of the 3D Gaussian splash model, performing back propagation on the 3D Gaussian splash model to update the model parameters, completing the initialization of the 3D Gaussian splash model, and obtaining the 3D Gaussian parameters;

[0106] S400: Determine whether the current input frame in the aligned three-dimensional point cloud stream data is a key frame. If the current input frame is a key frame, input the key frame into the three-dimensional Gaussian splash model for rendering, and back-propagate the three-dimensional Gaussian splash model through the loss function to update the model parameters. If the current input frame is a non-key frame, design a predictor through a preset time-series state space model, train the predictor, and predict the three-dimensional Gaussian parameters with adjacent frame information to obtain the three-dimensional Gaussian parameters corresponding to the current input frame.

[0107] S500: Render the three-dimensional Gaussian parameters obtained in S400 and the point cloud of the current input frame, and finally realize dynamic real-time rendering of large assembly scenes.

[0108] Specifically, the back propagation process in S300 is only performed between multiple cameras in the first frame. When the loss function is less than a certain value, the parameters of the three-dimensional Gaussian model can be considered correct, that is, the initialization of the Gaussian splash model is completed. The state space model in S400 is a mathematical model that describes the behavior of a dynamic system. The model has a fixed mathematical expression. It uses a set of first-order differential equations (continuous time system) or difference equations (discrete time system) to represent the evolution of the internal state of the system, and uses another set of equations to describe the relationship between the system state and output.

[0109] The above-mentioned dynamic real-time rendering method of large assembly scenes based on three-dimensional Gaussian splashing is constructed by combining multiple sensors to construct a three-dimensional Gaussian splashing method for real-time rendering of dynamic scenes. This method can render any perspective of the assembly scene. The model includes sensor data alignment, initialization of the three-dimensional Gaussian splashing model, prediction and update of the three-dimensional Gaussian splashing model. Combined with the three-dimensional Gaussian splashing model, the scene space point alignment and predictor are designed to solve the problem of multi-sensor data information redundancy and dynamic real-time rendering difficulties. This method is expected to improve the safety and reliability of assembly, thereby promoting the transformation of industrial manufacturing intelligence.

[0110] The method proposed in the present invention can perform full-scene dynamic real-time rendering of the assembly process in an industrial manufacturing scenario and complete scene monitoring. The rendered three-dimensional scene can be presented at an image-level display at any angle on the interactive interface, which can increase the safety and reliability of the assembly process, and can promptly discover and handle problems in the assembly process, helping to promote the rapid and high-quality development of industrial intelligent manufacturing.

[0111] In one embodiment, Figure 3 As shown, S100 includes:

[0112] S110: Four non-repetitive scanning laser radars are installed and fixed at the four corners of the ceiling beam, with the scanning direction facing the assembly scene, and twelve color cameras are installed and fixed at the four sides of the ceiling beam, and the non-repetitive scanning laser radars and color cameras on the ceiling are kept stationary;

[0113] S120: Four non-repetitive scanning laser radars collect point cloud data of four assembly scenes, and twelve color cameras collect image data of twelve assembly scenes. The point cloud data and image data will be transmitted to the high-performance computing unit in the form of data streams for processing.

[0114] In one embodiment, S200 includes:

[0115] S210: Obtaining coordinate transformation from any sensor of the full-scene perception platform to the world coordinate system through any annotation tool;

[0116] S220: aligning data from multiple sensors according to timestamps, and rasterizing the space of the assembly scene in a world coordinate system;

[0117] S230: Transform the coordinate system of the data of the four non-repetitive scanning laser radars into the world coordinate system to obtain The scene voxel point cloud in the world coordinate system at all times completes the alignment of the scene space points and the lidar point cloud;

[0118] S240: Aligning scene space points with color image pixels is achieved through rigid body transformation and projection transformation, wherein the rigid body transformation transforms the scene space point coordinates into the camera optical center coordinate system, and the projection transformation transforms the scene space point in the camera optical center coordinate system into pixels on the pixel plane.

[0119] Specifically, in S210, the coordinate systems corresponding to the optical centers of the four non-repetitive scanning laser radars are respectively: ,The coordinate systems corresponding to the optical centers of the twelve RGB color cameras are , the coordinate system corresponding to the optical center of any sensor is taken as the world coordinate. For a more convenient description, the present invention uses Coordinate system as world coordinates . The coordinate transformation from any sensor to the world coordinate system can be obtained through any annotation tool ,in Indicates Sensors, Indicates an operation including rotation and translation. In S220, since the data collected by the perception platform includes multiple sensors of different types, with different data structures, different spatial coordinates, and much redundant information, it is necessary to pre-process the data from multiple sensors. The specific coordinate system relationship in S240 is as follows: Figure 4 shown.

[0120] In one embodiment, in S220, the space of the assembly scene is rasterized in the world coordinate system, specifically:

[0121] Define the assembly scene as a size of The size of the voxel is defined as , the resolution of the assembly scene space can be obtained, as shown in formula (1):

[0122] (1)

[0123] S230 is specifically:

[0124] (2)

[0125] in, Indicates that from The point cloud of a LiDAR sensor, represents the coordinates of a point in space, represents the coordinate transformation from any sensor to the world coordinate system, Indicates Sensors, Represents operations involving rotation and translation; spatial point coordinates The following formula (3) should be satisfied:

[0126] (3)

[0127] Calculation Point The voxel center coordinates , calculated as shown in formula (4):

[0128] (4)

[0129] The symbol Indicates rounding up, and the current The voxel coordinates of the four non-repeated scanning lidar data at the moment ,Right now express The scene voxel point cloud in the world coordinate system at all times;

[0130] S240 is specifically:

[0131] Calculate the image pixels corresponding to the voxel point cloud, specifically:

[0132] (5)

[0133] (6)

[0134] in, Represents the scene voxel point cloud in the world coordinate system, and its coordinate value is , is the coordinate value of the pixel coordinate system, Indicates The rotation of the sensor to the world coordinate system, Indicates The translation of the sensor to the world coordinate system, Indicates The intrinsic parameters of a sensor can be obtained by calibration in any way or based on the camera parameter information. and The camera is and The focal length in the direction, and They are the imaging plane and the camera coordinate system The value of the intersection of the axes in pixel coordinates;

[0135] Finally, the correspondence between scene space points and image pixels is obtained, and the alignment of scene space points and pixels is achieved.

[0136] In one embodiment, S300 includes:

[0137] S310: Scene voxel point cloud exist Initialize the three-dimensional Gaussian sphere at the point: First, Perform uniform downsampling to obtain ,by Any point in Initialize the 3D Gaussian sphere at the point as the center , the calculation formula is as follows (7):

[0138] (7)

[0139] in, represents the covariance matrix, which affects the size and shape of the three-dimensional Gaussian sphere;

[0140] Since any three-dimensional Gaussian sphere can be transformed from a standard three-dimensional Gaussian sphere by affine transformation According to the definition of affine transformation, Writing formula (8):

[0141] (8)

[0142] in Indicates rotation, Indicates zoom;

[0143] Since the three-dimensional Gaussian sphere needs to be projected onto the two-dimensional imaging plane to affect the rendering result, according to the image pixels corresponding to the voxel point cloud in S240, any point The three-dimensional Gaussian sphere centered on is projected onto the imaging plane in any camera coordinate system to obtain the projection point The corresponding two-dimensional Gaussian sphere, so the projection point in the camera coordinate system As shown in formula (9), point The corresponding covariance matrix of the two-dimensional Gaussian sphere As shown in formula (10):

[0144] (9)

[0145] (10)

[0146] in Represents the transformation from the world coordinate system to the camera coordinate system, Represents a projection transformation The Jacobian matrix of is used as an affine approximation of the projection transformation. In particular, the result of directly projecting the Gaussian sphere is not necessarily a Gaussian sphere, so it is necessary to pass the projection transformation at the point The Jacobian matrix is ​​used to perform affine approximation on the position of ; in addition, since formulas (9) and (10) involve dimensionality reduction calculations, and All are homogeneous representations;

[0147] S320: Spherical harmonics are used to describe color information in different directions. The specific expressions of the basis functions of the spherical harmonics are shown in formulas (11)(12)(13)(14):

[0148] (11)

[0149] (12)

[0150] (13)

[0151] (14)

[0152] in represents the order of spherical harmonics, , represents the azimuth of the spherical coordinate system, represents the elevation angle of the spherical coordinate system;

[0153] The color representation in any direction is shown in formula (15):

[0154] (15)

[0155] S330: Define a ray in the assembly scene from a pixel to the optical center , remember the light in Go up to point The probability of not being blocked before is , light at the point The probability density of the colliding particles at , light at the point The color emitted is , so the final color of the pixel can be obtained along the light Integrate as shown in formula (16):

[0156] (16)

[0157] By solving the differential equation, we can get:

[0158] (17)

[0159] in It can be obtained from formula (7) in S310, It can be obtained by formula (15) in S320. Finally, according to The color information of each pixel can be obtained, that is, the rendering result.

[0160] Specifically, in the detailed description, in order to better use mathematical symbols and public expressions, the first frame is defined as t=1.

[0161] In one embodiment, the loss function between the aligned first frame of the multi-view color flow image and the rendering result of the three-dimensional Gaussian splash model is defined in S300, specifically:

[0162] Remember 12 color cameras with different viewing angles The 12 images collected at the time are , according to formula (16) in S330, the rendering results corresponding to the viewing angles of the 12 cameras are obtained , respectively using The loss function is constructed by using the norm and the structural similarity index measure (SSIM), where The loss function is shown in formula (18): The loss function is shown in formula (19):

[0163] (18)

[0164] (19)

[0165] in and Under the sliding window and The mean of and Under the sliding window and The variance of For sliding window and The covariance of and are constants respectively, where the sliding window means that each calculation takes a window of size , and traverse the entire image with this window;

[0166] Finally, the total loss function is obtained As shown in formula (20):

[0167] (20)

[0168] According to the loss function Back-propagation is performed on the three-dimensional Gaussian splash model to optimize the three-dimensional Gaussian parameters.

[0169] In particular, At this moment, 12 color cameras with different viewing angles provided a total of 12 samples for training, and finally completed Initialization of the three-dimensional Gaussian splash model at time t.

[0170] In one embodiment, the method of determining whether the current input frame is a key frame in S400 is as follows:

[0171] S410: determining the motion state of the scene by performing a two-frame difference method on the image; wherein the two-frame difference method refers to performing a difference on the images of adjacent frames and determining whether there is a moving target in the current scene according to the difference result; if the current scene is in a stationary state, the parameters of the three-dimensional Gaussian splash model remain unchanged;

[0172] S420: If the assembly scene is in the assembly operation state, it is necessary to dynamically maintain the parameters of the three-dimensional Gaussian splash model of the assembly scene. Under fixed computing resources, it takes time to train a three-dimensional Gaussian splash model for one frame of the assembly scene. , the time it takes for the laser radar to collect one frame of data is , the time it takes for a color camera to collect one frame of data is , then the minimum time interval for updating a three-dimensional Gaussian splash through training is , satisfying the conditions of formula (21):

[0173] (twenty one)

[0174] Therefore, if the assembly scene is The operation state is entered at the moment, then the dynamic scene A key frame can be considered to be located at time , and at the same time The point cloud and image corresponding to each key frame are shown in formula (22) and formula (23) respectively:

[0175] (twenty two)

[0176] (twenty three)

[0177] The symbol Indicates rounding up. For the The point cloud corresponding to the key frame, No. Furthermore, the acquisition frequency of point clouds and images is not the same, so the data may not be from the exact same moment, and only the nearest point cloud and image can be found based on the timestamp of the nth key frame.

[0178] Specifically, the keyframes are adjusted according to the actual conditions of the sensors and computing devices, and are described as follows with specific numbers:

[0179] Assuming that it takes 1 second for the device to train a Gaussian splash model for one frame, 0.03 seconds for the radar to collect one frame, and 0.015 seconds for the camera to collect one frame, key frames can be defined at 1 second intervals. Training takes 1 second, so it is impossible to train every frame; when the scene enters the working scene at t=8s, 8, 9, 10, 11... are all key frames, that is, is the nth key frame. It is understandable that the key frame can be adjusted at any time according to actual needs. The definition of the key frame is only distinguished relative to the frame that has not been trained with the Gaussian splash model.

[0180] In one embodiment, predicting the three-dimensional Gaussian parameters in S400 includes the following steps:

[0181] S430: For the key frame, the three-dimensional Gaussian splash model of the assembly scene space point can be updated through training, and for the key frame The assembly scene within the time range is predicted by the preset time series state space model for the three-dimensional Gaussian splash model within the time range: The keyframes are ,but The set of frames in the time range is ,in represents the maximum number of frames that the sensor can collect within this time range. Therefore, the data flow in the dynamic assembly scene can be recorded as the set in formula (24): :

[0182] (twenty four)

[0183] Then the corresponding three-dimensional Gaussian splash model parameters in the data stream can be recorded as the set in formula (25): :

[0184] (25)

[0185] S440: According to the collection , we get the time series of the parameters of the three-dimensional Gaussian splash model. Since the motion in the assembly space is continuous, we assume that the first The parameters of the 3D Gaussian model of the frame are affected by the previous The parameters of the frame are affected, so the relationship expression can be obtained as shown in formula (26):

[0186] (26)

[0187] S450: According to formula (26) in S440, a predictor is designed by a time series state space model , first define the expression of the time series state space model as shown in formula (27):

[0188] (27)

[0189] in , , is the state equation coefficient, Refers to multi-layer perceptron, Implicit function representing the state;

[0190] S460: Then the predictor Training: Construct the data flow in the assembly scenario according to the method described in S430 , the data flow covers any link of the assembly task, and the sample pairs are constructed according to the data flow, as shown in formula (28):

[0191] (28)

[0192] in, Obtained through the training of Gaussian splash model, is the sample pair of the j-th three-dimensional Gaussian splash model, for the data stream All data in the Gaussian splash model are input for training and constructed in the form of Sample pairs form a data set , through the predictor Predicted , the predicted and truth value Calculate the loss function for cosine similarity , as shown in formula (29):

[0193] (29)

[0194] according to For predictors Backpropagation is performed to update the predictor parameters and finally complete the predictor training, according to the trained predictor Make predictions about the parameters of a 3D Gaussian.

[0195] In one embodiment, the dynamic real-time rendering of the large assembly scene in S500 includes the following steps:

[0196] Initialize the 3D Gaussian splash model of the assembly scene space point. During the assembly process, if the current frame is a key frame, the 3D Gaussian splash model is trained and the parameters of the 3D Gaussian splash model are updated. If the current frame is not a key frame, the previous frame on the time axis is updated. Frame parameters are fed into the pre-trained predictor In the process, the parameters of the three-dimensional Gaussian splash model predicted in the current frame are updated. The current frame is rendered according to the parameters of the three-dimensional Gaussian splash model of the current frame, and finally dynamic real-time rendering in large assembly scenes is realized.

[0197] The assembly of large parts is an important part of industrial manufacturing. The traditional assembly process usually requires on-site supervision. This method has certain safety hazards due to factors such as the on-site environment and mechanical equipment. In addition, when faced with the assembly process of huge targets such as aircraft sidings, it is difficult for staff to monitor all aspects in a short period of time. At this stage, this method that relies on manpower is inefficient and has certain risk factors. The form of image monitoring has problems such as poor interactive experience and fixed viewing angle. LiDAR has difficulty presenting color information of the assembly scene. Digital twins rely on existing modeling, and there are limitations in the safety and efficiency of assembly scene monitoring. Therefore, the innovative points and beneficial effects of the present invention are as follows:

[0198] ① The present invention proposes a dynamic real-time rendering method for large-scale assembly scenes based on three-dimensional Gaussian splashing. The method reproduces the assembly scene in three dimensions by three-dimensional Gaussian splashing. During the assembly process, elements such as robots, components, and personnel can be monitored in real time, and the workers can be provided with interactive image-level rendering at any perspective to ensure the safety of the assembly process.

[0199] ② In the process of using a large number of sensors, there is redundant scene information, which is not necessary for rendering the entire assembly process and greatly affects the rendering speed. The present invention expresses the scene in voxels, aligns the scene space points with the point cloud data of the laser radar and the image data of the camera, and maintains the three-dimensional Gaussian splash model based on the scene space points instead of maintaining the three-dimensional Gaussian splash model of all the laser radar point cloud data, which greatly reduces the amount of rendering calculations;

[0200] ③ The present invention models the rendering process of the assembly scene. The points in the scene are modeled as three-dimensional Gaussian spheres, and the colors in the scene are modeled as spherical harmonic functions. The process of light reflecting from the surface of the object to the pixel point is regarded as the superposition process of the three-dimensional Gaussian sphere. The color expression of the rendering result is given, and the supervision signal for back propagation during the training of the three-dimensional Gaussian splash model is provided;

[0201] ④ Since the speed of data collected by the sensor is much faster than the speed of training a three-dimensional Gaussian splash model, training the three-dimensional Gaussian splash model for each frame cannot meet the real-time requirements. The present invention designs a key frame judgment branch, first distinguishing between when the assembly scene is stationary and when the assembly scene is moving, and then judging whether the current frame of the assembly scene can meet the time requirements of the three-dimensional Gaussian splash model training to distinguish whether it is a key frame, training the key frame, and predicting the non-key frame; effectively improving the real-time performance;

[0202] ⑤ The present invention regards the three-dimensional Gaussian splash model on the time axis as time series information, and trains the state estimation model as a predictor of the three-dimensional Gaussian splash model by constructing sample pairs. The state estimation model saves the existing information through implicit functions, and the predictor accepts the existing information and outputs the information of the next frame.

[0203] The above is a detailed introduction to the dynamic real-time rendering method of a large assembly scene based on three-dimensional Gaussian splashing provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention, and the description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A dynamic real-time rendering method for a large assembly scene based on three-dimensional Gaussian splashing, characterized in that: The method comprises the following steps: S100: Build a full-scene perception platform to collect 3D point cloud stream data and multi-view color image stream data in large-scale assembly processes; S200: transforming the multi-view color image data and the three-dimensional point cloud data into the world coordinate system, and aligning the point cloud, image pixels and scene space points; S300: input the first frame of the aligned three-dimensional point cloud flow data into the three-dimensional Gaussian splash model for rendering, define the loss function between the first frame of the aligned multi-view color flow image and the rendering result of the three-dimensional Gaussian splash model, perform back propagation on the three-dimensional Gaussian splash model to update the model parameters, complete the initialization of the three-dimensional Gaussian splash model, and obtain the three-dimensional Gaussian parameters; S300 includes: S310: Scene voxel point cloud exist Initialize the three-dimensional Gaussian sphere at the point: First, Perform uniform downsampling to obtain ,by Any point in Initialize the 3D Gaussian sphere at the point as the center , the calculation formula is as follows (7): （7） in, represents the covariance matrix, which affects the size and shape of the three-dimensional Gaussian sphere; Since any three-dimensional Gaussian sphere can be transformed from a standard three-dimensional Gaussian sphere by affine transformation According to the definition of affine transformation, Writing formula (8): （8） in Indicates rotation, Indicates zoom; Since the three-dimensional Gaussian sphere needs to be projected onto the two-dimensional imaging plane to affect the rendering result, according to the image pixels corresponding to the voxel point cloud, any point The three-dimensional Gaussian sphere centered on is projected onto the imaging plane in any camera coordinate system to obtain the projection point The corresponding two-dimensional Gaussian sphere, so the projection point in the camera coordinate system As shown in formula (9), point The corresponding covariance matrix of the two-dimensional Gaussian sphere As shown in formula (10): （9） （10） in Represents the transformation from the world coordinate system to the camera coordinate system, Represents a projection transformation The Jacobian matrix of is used as an affine approximation of the projection transformation. In particular, the result of directly projecting the Gaussian sphere is not necessarily a Gaussian sphere, so it is necessary to pass the projection transformation at the point The position of is approximated using the Jacobian matrix; S320: Spherical harmonics are used to describe color information in different directions. The specific expressions of the basis functions of the spherical harmonics are shown in formulas (11)(12)(13)(14): （11） （12） （13） （14） in represents the order of spherical harmonics, , represents the azimuth of the spherical coordinate system, represents the elevation angle of the spherical coordinate system; The color representation in any direction is shown in formula (15): （15） S330: Define a ray in the assembly scene from a pixel to the optical center , remember the light in Go up to point The probability of not being blocked before is , light at the point The probability density of the colliding particles at , light at the point The color emitted is , so the final color of the pixel can be obtained along the light Integrate as shown in formula (16): （16） By solving the differential equation, we can get: （17） in It can be obtained from formula (7) in S310, It can be obtained by formula (15) in S320. Finally, according to The color information of each pixel can be obtained, that is, the rendering result; In S300, the loss function between the aligned first frame of the multi-view color flow image and the rendering result of the three-dimensional Gaussian splash model is defined as follows: Remember 12 color cameras with different viewing angles The 12 images collected at the time are , according to formula (16) in S330, the rendering results corresponding to the viewing angles of the 12 cameras are obtained , respectively using The loss function is constructed by using the norm and structural similarity SSIM, where The loss function is shown in formula (18): The loss function is shown in formula (19): (18) (19) in and Under the sliding window and The mean of and Under the sliding window and The variance of For sliding window and The covariance of and are constants respectively, where the sliding window means that each calculation takes a window of size , and traverse the entire image with this window; Finally, the total loss function is obtained As shown in formula (20): (20) According to the loss function Back propagation is performed on the three-dimensional Gaussian splash model to optimize the three-dimensional Gaussian parameters; S400: Determine whether the current input frame in the aligned three-dimensional point cloud stream data is a key frame. If the current input frame is a key frame, input the key frame into the three-dimensional Gaussian splash model for rendering, and back-propagate the three-dimensional Gaussian splash model through the loss function to update the model parameters. If the current input frame is a non-key frame, design a predictor through a preset time-series state space model, train the predictor, and predict the three-dimensional Gaussian parameters with adjacent frame information to obtain the three-dimensional Gaussian parameters corresponding to the current input frame. S500: Render the three-dimensional Gaussian parameters obtained in S400 and the point cloud of the current input frame, and finally realize dynamic real-time rendering of large assembly scenes.

2. The method according to claim 1, characterized in that S100 includes: S110: Four non-repetitive scanning laser radars are installed and fixed at the four corners of the ceiling beam, with the scanning direction facing the assembly scene, and twelve color cameras are installed and fixed at the four sides of the ceiling beam, and the non-repetitive scanning laser radars and color cameras on the ceiling are kept stationary; S120: Four non-repetitive scanning laser radars collect point cloud data of four assembly scenes, and twelve color cameras collect image data of twelve assembly scenes. The point cloud data and image data will be transmitted to the high-performance computing unit in the form of data streams for processing.

3. The method according to claim 2, characterized in that S200 includes: S210: Obtaining coordinate transformation from any sensor of the full-scene perception platform to the world coordinate system through any annotation tool; S220: aligning data from multiple sensors according to timestamps, and rasterizing the space of the assembly scene in a world coordinate system; S230: Transform the coordinate system of the data of the four non-repetitive scanning laser radars into the world coordinate system to obtain The scene voxel point cloud in the world coordinate system at all times completes the alignment of the scene space points and the lidar point cloud; S240: Aligning scene space points with color image pixels is achieved through rigid body transformation and projection transformation, wherein the rigid body transformation transforms the scene space point coordinates into the camera optical center coordinate system, and the projection transformation transforms the scene space point in the camera optical center coordinate system into pixels on the pixel plane.

4. The method according to claim 3, characterized in that In S220, the space of the assembly scene is rasterized in the world coordinate system, specifically: Define the assembly scene as a size of The space of voxels is defined as , the resolution of the assembly scene space can be obtained, as shown in formula (1): （1） S230 is specifically: （2） in, Indicates that from The point cloud of a LiDAR sensor, represents the coordinates of a point in space, represents the coordinate transformation from any sensor to the world coordinate system, Indicates Sensors, Represents operations involving rotation and translation; spatial point coordinates The following formula (3) should be satisfied: （3） Calculation Point The voxel center coordinates , calculated as shown in formula (4): （4） The symbol Indicates rounding up, and the current The voxel coordinates of the four non-repeated scanning lidar data at the moment ,Right now express The scene voxel point cloud in the world coordinate system at all times; S240 is specifically: Calculate the image pixels corresponding to the voxel point cloud, specifically: （5） （6） in, Represents the scene voxel point cloud in the world coordinate system, and its coordinate value is , is the coordinate value of the pixel coordinate system, Indicates The rotation of the sensor to the world coordinate system, Indicates The translation of the sensor to the world coordinate system, Indicates The intrinsic parameters of a sensor can be obtained by calibration in any way or based on the camera parameter information. and The camera is and The focal length in the direction, and They are the imaging plane and the camera coordinate system The value of the intersection of the axes in pixel coordinates; Finally, the correspondence between scene space points and image pixels is obtained, and the alignment of scene space points and pixels is achieved.

5. The method according to claim 4, characterized in that The method of determining whether the current input frame is a key frame in S400 is as follows: S410: determining the motion state of the scene by performing a two-frame difference method on the image; wherein the two-frame difference method refers to performing a difference on the images of adjacent frames and determining whether there is a moving target in the current scene according to the difference result; if the current scene is in a stationary state, the parameters of the three-dimensional Gaussian splash model remain unchanged; S420: If the assembly scene is in the assembly operation state, it is necessary to dynamically maintain the parameters of the three-dimensional Gaussian splash model of the assembly scene. Under fixed computing resources, it takes time to train a three-dimensional Gaussian splash model for one frame of the assembly scene. , the time it takes for the laser radar to collect one frame of data is , the time it takes for a color camera to collect one frame of data is , then the minimum time interval for updating a three-dimensional Gaussian splash through training is , satisfying the conditions of formula (21): (21) Therefore, if the assembly scene is The operation state is entered at the moment, then the dynamic scene A key frame can be considered to be located at time , and at the same time The point cloud and image corresponding to each key frame are shown in formula (22) and formula (23) respectively: (22) (23) The symbol Indicates rounding up. For the The point cloud corresponding to the key frame, No. The image corresponding to each key frame.

6. The method according to claim 5, characterized in that The prediction of the three-dimensional Gaussian parameters in S400 includes the following steps: S430: For the key frame, the three-dimensional Gaussian splash model of the assembly scene space point can be updated through training, and for the key frame The assembly scene within the time range is predicted by the preset time series state space model for the three-dimensional Gaussian splash model within the time range: The keyframes are ,but The set of frames in the time range is ,in represents the maximum number of frames that the sensor can collect within this time range. Therefore, the data flow in the dynamic assembly scene can be recorded as the set in formula (24): : (24) Then the corresponding three-dimensional Gaussian splash model parameters in the data stream can be recorded as the set in formula (25): : (25) S440: According to the collection , we get the time series of the parameters of the three-dimensional Gaussian splash model. Since the motion in the assembly space is continuous, we assume that the first The parameters of the 3D Gaussian model of the frame are affected by the previous The parameters of the frame are affected, so the relationship expression can be obtained as shown in formula (26): (26) S450: According to formula (26) in S440, a predictor is designed by a time series state space model , first define the expression of the time series state space model as shown in formula (27): (27) in , , is the state equation coefficient, refers to a multi-layer perceptron, Implicit function representing the state; S460: Then the predictor Training: Construct the data flow in the assembly scenario according to the method described in S430 , the data flow covers any link of the assembly task, and the sample pairs are constructed according to the data flow, as shown in formula (28): (28) in, Obtained through the training of Gaussian splash model, is the sample pair of the j-th three-dimensional Gaussian splash model, for the data stream All data in the Gaussian splash model are input for training and constructed in the form of Sample pairs form a data set , through the predictor Predicted , the predicted and truth value Calculate the loss function for cosine similarity , as shown in formula (29): (29) according to For predictors Backpropagation is performed to update the predictor parameters and finally complete the predictor training, according to the trained predictor Make predictions about the parameters of a 3D Gaussian.

Images