Three-dimensional Gaussian point cloud merging method and system based on common-view optimization

Abstract

The invention relates to a three-dimensional Gaussian point cloud model merging method and system based on common-view optimization, electronic equipment and a storage medium, and relates to a common-view optimization-based three-dimensional Gaussian point cloud model merging method, system, electronic equipment and storage medium. According to the method, images can be collected and recovered according to the influence of segmentation on each training image, and the images are respectively put into a to-be-merged model as complementary extension information for secondary training, so that the model can learn complex image features more efficiently, and the rendering quality is improved; and the low-PSNR image is often located in the boundary region of the sub-scene, so that the secondary training is beneficial to improving the continuity of the joint region between the two blocks.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional modeling, and in particular to a three-dimensional Gaussian point cloud merging method, system, computer device and computer-readable storage medium based on common view optimization. Background Art

[0002] 3D gaussian splatting (3DGS) uses the explicit expression of 3D Gaussian. Based on the properties of Gaussian distribution, it represents 3D point cloud or voxel data as a series of Gaussian distributions, allowing efficient representation and rendering of complex 3D shapes. It is superior to NERF (Neural Radiance Field) in terms of training speed and editability.

[0003] In related technologies, whether NERF or 3DGS, when there are too many scene images, the video memory required for training will increase significantly until it is insufficient; therefore, block training is almost an inevitable operation, and all models are merged after the block training is completed.

[0004] When multiple 3DGS models are merged, noise may be caused by reasons such as lack of observation (such as the back of a wall) and insufficient observation (such as distant views). There is currently no effective solution for efficiently removing noise, seamlessly retaining all details, and improving the quality of the 3D Gaussian in the common view area. Summary of the invention

[0005] The embodiments of the present application provide a three-dimensional Gaussian point cloud model merging method, system, computer device and computer-readable storage medium based on common view optimization, so as to at least solve the problem in the related art that when multiple 3DGS models are merged, there is noise in the merging result due to the existence of unobserved or insufficiently observed feature points.

[0006] In a first aspect, an embodiment of the present application provides a method for merging three-dimensional Gaussian point cloud models based on common view optimization, the method comprising: Determine a first model and a second model that are adjacent to each other, wherein the first model and the second model are obtained by dividing the same initial three-dimensional Gaussian model by a preset segmentation plane; Based on the rendered images and the original images of the first model and the second model, respectively determine a first recycled image set and a second recycled image set having a greater impact on the rendering quality of the first model and the second model; Using the first recycled image set and the second recycled image set as auxiliary information, respectively performing secondary optimization training on the second model and the first model; After the optimization training results are segmented respectively by the preset segmentation plane, the segmentation results are merged to obtain a three-dimensional Gaussian point cloud model merged result.

[0007] In some embodiments, using the first recovered image set and the second recovered image set as auxiliary information, performing secondary optimization training on the second model and the first model respectively includes: Adding the first recycled image set to the training image set corresponding to the second model, performing secondary optimization training on the second model, and obtaining a first intermediate model; The second recycled image set is added to the training image set corresponding to the first model, and the first model is subjected to secondary optimization training to obtain a second intermediate model, wherein the images in the original images that have a greater impact on the rendering quality are deleted from the training image set.

[0008] In some embodiments, after segmenting the optimization training results respectively by the preset segmentation plane, merging the segmentation results to obtain the three-dimensional Gaussian point cloud model merged result includes: According to the preset segmentation plane, the first intermediate model and the second intermediate model are segmented respectively to obtain two segmentation models corresponding to the first intermediate model and the second intermediate model respectively; Selecting a segmentation model located on one side of the preset segmentation plane from the first intermediate model, and selecting a segmentation model located on the other side of the preset segmentation plane from the second intermediate model; By merging the selected segmentation models, a 3D Gaussian point cloud model merging result is obtained.

[0009] In some embodiments, based on the rendered images and the original images of the first model and the second model, determining respectively the first recycled image set and the second recycled image set having a greater impact on the rendering quality of the first model and the second model includes: Acquire initial training images corresponding to the first model and the second model; Using the first model to render the initial training image corresponding to its camera perspective to obtain a first rendering image, and using the second model to render the initial training image corresponding to its camera perspective to obtain a second rendering image; Obtaining a first peak signal-to-noise ratio between an initial training image and a first rendering corresponding to it, and obtaining a second peak signal-to-noise ratio between an initial training image and a rendering corresponding to it in the initial three-dimensional Gaussian model, and if a decrease value of the first peak signal-to-noise ratio compared to the second peak signal-to-noise ratio is greater than a preset threshold, adding the initial training image to the first recovered image set; A third peak signal-to-noise ratio between the initial training image and its corresponding second rendering is obtained, and if a decrease value of the third peak signal-to-noise ratio compared with the second peak signal-to-noise ratio is greater than a preset threshold, the initial training image is added to the second recovered image set.

[0010] In some embodiments, in the process of cutting the initial three-dimensional Gaussian point cloud model by a preset segmentation plane to obtain at least two mutually adjacent first models and second models, the method further comprises: Determine whether there is a 3D Gaussian point that falls outside the segmentation surface; If so, delete the 3D Gaussian points outside the segmentation surface.

[0011] In a second aspect, an embodiment of the present application provides a three-dimensional Gaussian point cloud model optimization training method based on common view optimization, the method comprising: After aligning the initial three-dimensional Gaussian point cloud model with the northeast sky, the initial three-dimensional Gaussian point cloud model is evenly divided into a plurality of point cloud block models by using a preset grid or a preset segmentation algorithm; After the initial optimization training of each point cloud block model, the method described in claim 1 is used to perform multiple rounds of iterative training and merging based on any two adjacent point cloud block models to obtain the optimized training result of the initial three-dimensional Gaussian point cloud model.

[0012] In a third aspect, the embodiment of the present application further provides a three-dimensional Gaussian point cloud model merging system based on common view optimization, the system comprising: a segmentation module, an optimization module and a merging module, wherein: The segmentation module is used to determine a first model and a second model that are adjacent to each other, wherein the first model and the second model are obtained by segmenting the same initial three-dimensional Gaussian model through a preset segmentation plane; The optimization module is used to determine, based on the rendered images and the original images of the first model and the second model, a first recycled image set and a second recycled image set that have a greater impact on the rendering quality of the first model and the second model respectively; and, using the first recovered image set and the second recovered image set as auxiliary information, respectively performing secondary optimization training on the second model and the first model; The merging module is used to segment the optimization training results respectively through the preset segmentation plane, and then merge the segmentation results to obtain a merging result of the three-dimensional Gaussian point cloud model.

[0013] In some embodiments, the optimization module performs secondary optimization training on the second model and the first model respectively, including: Adding the first recycled image set to the training image set corresponding to the second model, performing secondary optimization training on the second model, and obtaining a first intermediate model; The second recycled image set is added to the training image set corresponding to the first model, and the first model is subjected to secondary optimization training to obtain a second intermediate model, wherein the images in the original images that have a greater impact on the rendering quality are deleted from the training image set.

[0014] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0016] Compared with the general technology in the field that directly merges the trained block models after two segmentations, which will cause the area with good effects to deteriorate or even disappear, the three-dimensional Gaussian point cloud model merging method based on common view optimization provided in this embodiment can collect and recover images according to the impact of segmentation on each training image, and put them into another model to be merged as complementary extended information for secondary training, so that the model can better learn these complex image features, thereby improving the rendering quality; and because low PSNR images are often located in the boundary area, secondary training helps to improve the continuity of the junction area between the two blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a schematic top view of a three-dimensional Gaussian point cloud model according to an embodiment of the present application; Figure 2 is a schematic diagram of a combined result obtained by a method in the related art; Figure 3 is a flow chart of a method for merging three-dimensional Gaussian point cloud models based on common view optimization according to an embodiment of the present application; Figure 4 This is a schematic diagram of deleting 3D points based on the probability density ratio; Figure 5 is a schematic diagram of segmenting an initial three-dimensional Gaussian model by a preset segmentation plane according to an embodiment of the present application; Figure 6is a schematic diagram of a first model obtained by segmentation according to an embodiment of the present application; Figure 7 is a schematic diagram of a second model obtained by segmentation according to an embodiment of the present application; Figure 8 is a schematic diagram of comparison effects of rendered images with different peak signal-to-noise ratios according to an embodiment of the present application; Fig. 9 is a schematic diagram of the result of merging two models after segmentation according to an embodiment of the present application; Fig.10 is a flow chart of a three-dimensional Gaussian point cloud model optimization training method based on common view optimization according to an embodiment of the present application; Fig.11 is a structural block diagram of a three-dimensional Gaussian point cloud model merging system based on common view optimization according to an embodiment of the present application; Fig.12 is a structural block diagram of another three-dimensional Gaussian point cloud model merging system based on common view optimization according to an embodiment of the present application; Fig.13 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0019] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0020] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] Figure 1 It is a top view schematic diagram of the three-dimensional Gaussian point cloud model according to the embodiment of the present application. The three-dimensional Gaussian sputtering uses the explicit expression of three-dimensional Gaussian, which is superior to NERF in terms of training speed and editability.

[0023] The three-dimensional Gaussian point cloud model includes: three-dimensional position, covariance matrix (expressed by rotation matrix (quaternion) and numerical expression of three axes), opacity, color and spherical harmonics, among which spherical harmonics is an optional attribute. The three-dimensional Gaussian point cloud model includes multiple 3D points, and each 3D point carries multiple attributes. Multiple such 3D points constitute a three-dimensional Gaussian point cloud model. However, whether it is NERF or 3DGS, when there are too many scene images, the video memory required for training will increase significantly until the video memory is insufficient. Therefore, block training is almost an inevitable operation, and all models are merged after the block training is completed.

[0024] When multiple 3DGS models are merged, there will be noise caused by reasons such as lack of observation (such as the back of the wall) and insufficient observation (such as distant view). Figure 2 is a schematic diagram of the merging result obtained by the method in the related art, such as Figure 2 As shown in the figure, if the point clouds outside the segmentation surface are directly deleted and then merged, the following will occur: Figure 1 The over-deleted area is the double-dashed matrix box area.

[0025] There is currently no effective solution for how to efficiently remove noise, seamlessly retain all details, and improve the quality of 3D Gaussians in the common view area.

[0026] In view of this, the present application provides a three-dimensional Gaussian point cloud model merging method based on common view optimization, which screens out images that have a greater impact on rendering by comparing the impact on rendering effects and the original images after model segmentation, and adds the screened images to another cut model as complementary information for secondary training, so as to achieve the effect of removing noise and retaining the details of the merged junction area.

[0027] Figure 3 is a flow chart of a method for merging three-dimensional Gaussian point cloud models based on common view optimization according to an embodiment of the present application, such as Figure 3 As shown, the process includes the following steps: S301, determining a first model and a second model adjacent to each other, wherein the first model and the second model are obtained by dividing the same initial three-dimensional Gaussian model through a preset segmentation plane; The initial three-dimensional Gaussian model needs to be aligned with the northeast sky, wherein the "northeast sky" (NEU) alignment of the three-dimensional model usually refers to aligning the direction axis (such as X-axis, Y-axis, and Z-axis) of the three-dimensional model with the northeast sky direction in the geographic coordinate system.

[0028] The above alignment is particularly important in the fields of geographic information systems (GIS), virtual reality (VR), augmented reality (AR), etc., because it helps to ensure the directional accuracy of the model in the real world. It should be noted that since the specific implementation means of the northeast sky alignment is a conventional technology in the field and has no core impact on the invention of this solution, it will not be repeated in this embodiment.

[0029] Furthermore, the first model and the second model may be obtained by equally dividing the initial three-dimensional Gaussian model into two models, or by equally dividing the initial three-dimensional Gaussian model into a plurality of models and selecting two of them at random. The preset segmentation plane may be a segmentation plane generated by evenly segmenting the 3D Gaussian model based on a predetermined grid size, or a segmentation plane determined by a graph-based algorithm such as graph_cut.

[0030] Furthermore, since the 3D points in the Gaussian sputtering model are ellipsoidal, if there are 3D points that fall outside the segmentation surface during the segmentation process, the 3D points need to be deleted to reduce data redundancy, clarify the area division, and maintain the consistency of the model. In addition, deletion based on probability density ratio can also be performed. Figure 4 This is a schematic diagram of deleting 3D points based on the probability density ratio, such as Figure 4 As shown in the figure, if the probability distribution of the three-dimensional Gaussian points on one side of the line exceeds a certain proportion, they will be deleted.

[0031] In addition, a more mathematically sound segmentation algorithm based on the paper "2024, A New Split Algorithm for 3D Gaussian Splatting" can also be used to directly prevent the 3D from falling outside the segmentation plane and obtain a model with two clearly divided regions. It should be noted that since the segmentation algorithm in the paper is also a prior art disclosed in the art, the specific steps of how to use the segmentation algorithm to achieve segmentation will not be repeated in this embodiment.

[0032] Finally, the first model obtained after segmentation can be represented by model_seg1, and the second model obtained after segmentation can be represented by model_seg2.

[0033] In an exemplary embodiment, Figure 5 is a schematic diagram of segmenting an initial three-dimensional Gaussian model by a preset segmentation plane according to an embodiment of the present application, Figure 6 is a schematic diagram of a first model obtained by segmentation according to an embodiment of the present application, Figure 7 is a schematic diagram of a second model obtained by segmentation according to an embodiment of the present application, such as Figure 6 and Figure 7 As shown, 3D points outside the segmentation surface are directly deleted.

[0034] S302, based on the rendered images and the original images of the first model and the second model, respectively determining a first recycled image set and a second recycled image set having a greater impact on the rendering quality of the first model and the second model; Specifically, this step includes the following sub-steps: Step 1-Step 3: Step 1, obtaining initial training images corresponding to the first model and the second model; Step 2, using the first model to render the initial training image corresponding to the camera perspective to obtain a first rendering image, and using the second model to render the initial training image corresponding to the camera perspective to obtain a second rendering image; Step 3, obtaining a first peak signal-to-noise ratio between the initial training image and its corresponding first rendering, and obtaining a second peak signal-to-noise ratio between the initial training image and its corresponding rendering in the initial three-dimensional Gaussian model. If the decrease value of the first peak signal-to-noise ratio compared with the second peak signal-to-noise ratio is greater than a preset threshold, the initial training image is added to the first recovered image set; Step 4, obtaining a third peak signal-to-noise ratio between the initial training image and its corresponding second rendering image, and if the decrease value of the third peak signal-to-noise ratio compared with the second peak signal-to-noise ratio is greater than a preset threshold, adding the initial training image to the second recovered image set.

[0035] Specifically, in an exemplary embodiment, if a certain original image is compared with a rendering of the whole model before cutting, PSNR1 =30, and this original image is compared with a rendering of the first model after cutting, PSNR2 =20; If the decrease value of PSNR1 and PSNR3 = 10, which is greater than the preset threshold value 1, it means that the original image has a great impact on the rendering quality of the model, and the image is put into the image recycling set; On the contrary, if the original image is compared with the rendering of the whole model before cutting, PSNR 1 =30, and the original image is compared with the rendering of the first model after cutting, PSNR3 =29.5, and the drop value between PSNR 1 and PSNR3 =0.5 is less than the preset threshold 1, it means that the image has little impact on the rendering quality and the image is still retained.

[0036] Figure 8 is a schematic diagram of the comparison effect of rendering images with different peak signal-to-noise ratios according to an embodiment of the present application, such as Figure 8 As shown, Figure 8 Compared with the original image, the rendered image in the upper middle part is white on both sides, which is a big difference (because the right side falls outside the dividing line and is cut off); Figure 8 The lower middle part shows an example with a smaller PSNR difference, and the overall image restoration is better. Specifically, in step 2, first ensure that the trained block models model_seg1 and model_seg2 have been loaded, use model_seg1 to render each training image, and generate the corresponding rendered image rendered_image1; and use model_seg2 to render each training image, and generate the corresponding rendered image rendered_image2; Among them, the peak signal-to-noise ratio of the rendered image is compared with that of the original image. Optionally, if the peak signal-to-noise ratio decreases by more than a preset threshold value 1 compared with before cutting, the image rendering quality is considered to be poor, and all images with poor rendering quality are divided into a first recycled image set and a second recycled image set according to model resolution; the recycled set of model_seg1 is called image_set1, and the recycled set of model_seg2 is called image_set2.

[0037] It should be noted that the Peak Signal-to-Noise Ratio (PSNR) is a commonly used image quality evaluation indicator, which is particularly suitable for evaluating the effects of image reconstruction or compression. The higher the PSNR value, the higher the similarity between the two images. In the 3D reconstruction scenario of the present embodiment, by comparing the PSNR between the rendered image and the original image in the overall model, and the PSNR between the rendered image and the original image in the segmentation model, the rendering effect of the model can be intuitively evaluated, thereby identifying images that have a greater adverse effect on the rendering effect.

[0038] In addition, if Figure 6 and 7 As shown, Figure 6 The solid black camera in the middle is the image with the largest PSNR drop after deleting the point cloud on the right side of the dividing line, i.e. image_set1. The remaining three-dimensional Gaussian point cloud is pointcloud_seg1, and its corresponding image set is image_seg1. Figure 7 The solid gray camera in the middle is the image whose PSNR drops the most after the second model deletes the point cloud on the left side of the dividing line, that is, image_set2. The remaining three-dimensional Gaussian point cloud is pointcloud_seg2, and its corresponding image set is image_seg2.

[0039] S203, using the first recycled image set and the second recycled image set as auxiliary information, performing secondary optimization training on the second model and the first model respectively, and after segmenting the optimization training results respectively through a preset segmentation surface, merging the segmentation results to obtain a merged result of the first model and the second model.

[0040] Specifically, the specific steps of the secondary optimization training include the following steps Step1-Step2: Step 1, add the first recycled image set to the original training image set corresponding to the second model, and perform secondary optimization training on the second model, that is, add image_set1 to model_seg2 to form model_mixed2, load model2 and perform secondary 3DGS training on it, and then obtain the first intermediate model model_mixed1; Step 2, further, add the second recycled image set to the original training image set corresponding to the first model, perform secondary optimization training on the first model, add image_set2 to model_seg1 to form model_mixed1, load model1 and perform secondary 3DGS training on it to obtain the second intermediate model model_mixed2.

[0041] It should be noted that, in this embodiment, before the above steps, model1 and model2 have completed the initial training optimization, the purpose of which is to improve rendering quality and efficiency, enhance detail retention, and support complex scenes and dynamic effects. This initial optimization training method is also a conventional method in the field, and will not be repeated in this embodiment.

[0042] It is understandable that if the trained block models are directly merged, the model may perform poorly on some images, which are usually unobserved or under-observed, usually located at the boundaries of sub-blocks or contain complex geometry and texture details.

[0043] In this step, by collecting these low PSNR images and adding them to the training sets of other adjacent and complementary models, and using the images in the recycled set as extended information for secondary 3DGS training, the model can better learn these complex image features, thereby improving the rendering quality; and because low PSNR images are often located in boundary areas, secondary training helps to improve the continuity between scenes.

[0044] In addition, after adding image_set to each model, load the already trained models and perform secondary 3DGS training on them. The efficiency of secondary training is very high and can converge quickly.

[0045] After the second training, the two first intermediate models and the second intermediate models need to be re-segmented. After segmentation, the segmentation results are merged to obtain a merged model with better effect, which specifically includes the following steps Step 1-Step 3: Step 1, segmenting the first intermediate model and the second intermediate model according to a preset segmentation surface to obtain two segmentation models respectively; In this step, the preset segmentation surface in step S101 is also used, and two segmentation results can be obtained for the two intermediate models respectively, wherein the first intermediate model located on both sides of the segmentation surface can be marked as sub-model A and sub-model B; the second intermediate model located on both sides of the segmentation surface can be marked as sub-model C and sub-model D; A segmentation model located on one side of the preset segmentation plane is selected from the first intermediate model, and a segmentation model located on the other side of the preset segmentation plane is selected from the second intermediate model; and a merged result of the first model and the second model is obtained by merging the segmentation model on one side of the preset segmentation plane and the segmentation model on the other side of the preset segmentation plane.

[0046] It can be understood that this step, i.e., model_mixed1 retains one side of the segmentation surface, model_mixed2 retains the other side of the segmentation surface, and then merges the two segmented models; that is, select to merge sub-models A and C or merge sub-models B and D to obtain the merged result of a complete and continuous 3D Gaussian spattering point cloud model.

[0047] Fig. 9 is a schematic diagram of the result of merging two models after segmentation according to an embodiment of the present application, such as Fig. 9 As shown, through the solution of the present application, the 3D Gaussian point cloud with poor quality can be deleted, the common view point cloud can be retained and the quality of the common view point cloud can be improved (from top to bottom, the top of the second building is the common view area, the Gaussian points have smaller uncertainty and higher quality); the optimized sub-scene models are effectively merged to construct a continuous and complete 3D reconstruction model. This method can make full use of the advantages of each sub-scene model, seamlessly merge the two models, and ensure that the quality of the intersection area of ​​the two models is higher.

[0048] Through the above steps S301 to S303, compared with the general technology in the field that directly merges the trained block models after two segmentations, which will cause the junction area with good effects to deteriorate or even disappear, the three-dimensional Gaussian point cloud model merging method based on common view optimization provided in this embodiment can collect and recover images according to the impact of segmentation on each training image, and put them into another model to be merged as complementary extended information for secondary training, so that the model can better learn these complex image features, thereby improving the rendering quality; and because low PSNR images are often located in the boundary area of ​​​​the sub-scene, secondary training helps to improve the continuity of the junction area between the two blocks.

[0049] Furthermore, the embodiment of the present application also provides a three-dimensional Gaussian point cloud model optimization training method based on common view optimization. Fig.10 is a flowchart of a three-dimensional Gaussian point cloud model optimization training method based on common view optimization according to an embodiment of the present application, such as Fig.10 As shown, the process includes the following steps S1001 to S1002: S1001, after aligning the initial three-dimensional Gaussian point cloud model to the northeast sky, evenly divide the initial three-dimensional Gaussian point cloud model into a plurality of point cloud blocks through a preset grid; S1002, based on any two adjacent point cloud blocks, multiple rounds of iterative secondary training and merging are performed to obtain an optimized training result of the initial three-dimensional Gaussian point cloud model.

[0050] It can be understood that the process of this step is equivalent to first training and merging the bottom n block models through the above steps S301-S303 to obtain A higher-level merge model; further, The higher-level merged models are trained and merged twice to obtain A higher-level merged model is formed, and the cycle is iterated multiple times until the final overall three-dimensional Gaussian model is obtained.

[0051] This embodiment also provides a three-dimensional Gaussian point cloud model merging system based on common view optimization, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the terms "module", "unit", "subunit", etc. may be a combination of software and / or hardware that implements predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0052] Fig.11 is a structural block diagram of a 3D Gaussian point cloud model merging system based on common view optimization according to an embodiment of the present application, such as Fig.11 As shown, the system includes a segmentation module 10, an optimization module 20 and a merging module 30, wherein: The segmentation module 10 is used to determine a first model and a second model that are adjacent to each other, wherein the first model and the second model are obtained by segmenting the same initial three-dimensional Gaussian model through a preset segmentation plane; The optimization module 20 is used to determine the first recycled image set and the second recycled image set having a greater impact on the rendering quality of the first model and the second model respectively by comparing the rendered images and the original images of the first model and the second model; and, using the first recovered image set and the second recovered image set as auxiliary information, respectively performing secondary optimization training on the second model and the first model; The merging module 30 is used to segment the optimization training results respectively through preset segmentation planes, and then merge the segmentation results to obtain a merging result of a three-dimensional Gaussian point cloud model.

[0053] Through the above system, compared with the general technology in the field that directly merges the trained block models after two segmentations, which will cause the intersection area with poor effect to deteriorate or even disappear, the three-dimensional Gaussian point cloud model merging method based on common view optimization provided in this embodiment can collect and recover images according to the impact of segmentation on each training image, and put them into another model to be merged as complementary extended information for secondary training, so that the model can better learn these complex image features, thereby improving the rendering quality; and because low PSNR images are often located in the boundary area of ​​​​the sub-scene, secondary training helps to improve the continuity of the intersection area between the two blocks.

[0054] also, Fig.12 It is a structural block diagram of another three-dimensional Gaussian point cloud model merging system based on common view optimization according to an embodiment of the present application.

[0055] In one embodiment, Fig.13 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Fig.13 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Fig.13 As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system, the computer program is executed by the processor to implement a three-dimensional Gaussian point cloud model merging method based on common view optimization in augmented reality, and the database is used to store data.

[0056] Those skilled in the art will understand that Fig.13 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0057] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0058] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0059] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional Gaussian point cloud model merging method based on common view optimization, characterized in that: The method comprises: Determining a first model and a second model adjacent to each other, wherein the first model and the second model are obtained by dividing the same initial three-dimensional Gaussian model by a preset segmentation plane; Determining, based on the rendered images and original images of the first model and the second model, a first recycled image set and a second recycled image set that have a greater impact on the rendering quality of the first model and the second model, respectively; Using the first recovered image set and the second recovered image set as auxiliary information, respectively performing secondary optimization training on the second model and the first model; After the optimization training results are segmented respectively by the preset segmentation plane, the segmentation results are merged to obtain a three-dimensional Gaussian point cloud model merging result.

2. The method according to claim 1, characterized in that Using the first recovered image set and the second recovered image set as auxiliary information, performing secondary optimization training on the second model and the first model respectively includes: Adding the first recovered image set to the training image set corresponding to the second model, performing secondary optimization training on the second model to obtain a first intermediate model; The second recovered image set is added to the training image set corresponding to the first model, and the first model is subjected to secondary optimization training to obtain a second intermediate model.

3. The method according to claim 2, characterized in that After the optimized training results are segmented by the preset segmentation planes, the segmentation results are merged to obtain a three-dimensional Gaussian point cloud model merge result, which includes: Segmenting the first intermediate model and the second intermediate model according to the preset segmentation plane to obtain two segmented models corresponding to the first intermediate model and the second intermediate model respectively; Selecting a segmentation model located on one side of the preset segmentation plane from the first intermediate model, and selecting a segmentation model located on the other side of the preset segmentation plane from the second intermediate model; By merging the selected segmentation models, a 3D Gaussian point cloud model merging result is obtained.

4. The method according to claim 1, wherein Based on the rendered images and original images of the first model and the second model, determining respectively that a first recycled image set and a second recycled image set having a significant impact on the rendering quality of the first model and the second model include: Obtaining initial training images corresponding to the first model and the second model; Using the first model to render an initial training image corresponding to its camera perspective to obtain a first rendering image, and using the second model to render the initial training image corresponding to its camera perspective to obtain a second rendering image; Obtaining a first peak signal-to-noise ratio between an initial training image and its corresponding first rendering, and obtaining a second peak signal-to-noise ratio between the initial training image and its corresponding rendering in the initial three-dimensional Gaussian model, and if a decrease in the first peak signal-to-noise ratio compared to the second peak signal-to-noise ratio is greater than a preset threshold, adding the initial training image to the first recovered image set; A third peak signal-to-noise ratio between the initial training image and its corresponding second rendering is obtained, and if a decrease in the third peak signal-to-noise ratio compared to the second peak signal-to-noise ratio is greater than a preset threshold, the initial training image is added to the second recovered image set.

5. The method according to claim 1, wherein In the process of cutting the initial three-dimensional Gaussian point cloud model by a preset segmentation plane to obtain at least two mutually adjacent first and second models, the method further includes: Determine whether there are any 3D Gaussian points falling outside the segmentation surface; If so, delete the 3D Gaussian points outside the segmentation surface.

6. A three-dimensional Gaussian point cloud model optimization training method based on common view optimization, characterized in that: The method comprises: After aligning the initial three-dimensional Gaussian point cloud model with the northeast and west celestial bodies, the initial three-dimensional Gaussian point cloud model is evenly divided into a plurality of point cloud block models by using a preset grid or a preset segmentation algorithm; After the initial optimization training of each point cloud block model, the method described in claim 1 is used to perform multiple rounds of iterative secondary training and merging based on any two adjacent point cloud block models to obtain the optimized training result of the initial three-dimensional Gaussian point cloud model.

7. A three-dimensional Gaussian point cloud model merging system based on common view optimization, characterized in that: The system includes: a segmentation module, an optimization module and a merging module, wherein: The segmentation module is used to determine a first model and a second model that are adjacent to each other, wherein the first model and the second model are obtained by dividing the same initial three-dimensional Gaussian model by a preset segmentation plane; The optimization module is configured to determine, based on the rendered images and original images of the first model and the second model, a first recycled image set and a second recycled image set, respectively, which have a greater impact on the rendering quality of the first model and the second model; and, performing secondary optimization training on the second model and the first model respectively using the first recovered image set and the second recovered image set as auxiliary information; The merging module is used to segment the optimization training results respectively through the preset segmentation plane, and then merge the segmentation results to obtain a merging result of the three-dimensional Gaussian point cloud model.

8. The system according to claim 7, characterized in that The optimization module performs secondary optimization training on the second model and the first model respectively, including: Adding the first recovered image set to the training image set corresponding to the second model, performing secondary optimization training on the second model to obtain a first intermediate model; The second recovered image set is added to the training image set corresponding to the first model, and the first model is subjected to secondary optimization training to obtain a second intermediate model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images