Gaussian scene fusion method and device and electronic equipment
By determining the matching relationship between Gaussian bodies and automatically computing the transformation information, the problem of large manual calculation errors in Gaussian scene fusion is solved, and the accurate and automated fusion of Gaussian scenes is achieved.
Patent Information
- Application Number
- CN202411040686.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-22
AI Technical Summary
The fusion of Gaussian scenarios in the prior art requires manual calculation of the rotation matrix and the translation matrix, which has large errors and is difficult to achieve automation.
By obtaining the Gaussian scenes of multiple Gaussian bodies, the matching relationship between Gaussian bodies is determined, and the transformation information is automatically determined to realize the fusion of Gaussian scenes.
It improves the accuracy of transformed information, realizes the automated fusion of Gaussian scenarios, and reduces the deviation of manual intervention.
Smart Images

Figure CN119006961A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to but is not limited to the field of information technology, and in particular to a Gaussian scene fusion method, device and electronic device. Background Art
[0002] In the related art, when fusing Gaussian scenes, it is necessary to manually calculate the rotation matrix and translation matrix between the two Gaussian scenes, and then align the two Gaussian scenes using the rotation matrix and translation matrix to achieve the fusion of Gaussian scenes. However, this process requires manual operation, has large errors, and is difficult to automate. Summary of the invention
[0003] In view of this, the embodiments of the present application at least provide a Gaussian scene fusion method, device and electronic device.
[0004] The technical solution of the embodiment of the present application is implemented as follows:
[0005] The present application provides a Gaussian scene fusion method, the method comprising:
[0006] Acquire a first Gaussian scene and a second Gaussian scene; wherein the first Gaussian scene includes a plurality of first Gaussian bodies, and the second Gaussian scene includes a plurality of second Gaussian bodies;
[0007] Determining a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies;
[0008] Based on the matching relationship, determining transformation information;
[0009] Based on the transformation information, the first Gaussian scene and the second Gaussian scene are fused to obtain a target Gaussian scene.
[0010] The present application embodiment provides a Gaussian scene fusion device, the device comprising:
[0011] An acquisition module, used to acquire a first Gaussian scene and a second Gaussian scene; wherein the first Gaussian scene includes a plurality of first Gaussian bodies, and the second Gaussian scene includes a plurality of second Gaussian bodies;
[0012] A first determining module, configured to determine a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies;
[0013] A second determination module, configured to determine transformation information based on the matching relationship;
[0014] A fusion module is used to fuse the first Gaussian scene and the second Gaussian scene based on the transformation information to obtain a target Gaussian scene.
[0015] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the method described in any one of the above claims is implemented.
[0016] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0018] Figure 1 A schematic diagram of the implementation process of a Gaussian scene fusion method provided in an embodiment of the present application Figure 1 ;
[0019] Figure 2 A schematic diagram of the implementation process of a Gaussian scene fusion method provided in an embodiment of the present application Figure 2 ;
[0020] Figure 3 A schematic diagram of a block diagram of a method for calculating difference information of a Gaussian scene provided in an embodiment of the present application;
[0021] Figure 4 A schematic block diagram of a Gaussian scene fusion method provided in an embodiment of the present application;
[0022] Figure 5 A schematic diagram of the composition structure of a Gaussian scene fusion device provided in an embodiment of the present application;
[0023] Figure 6 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0025] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0026] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
[0027] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0028] 3D Gaussians Splatting (3DGS) is an alternative to Neural Radiance Field (NeRF) with higher reconstruction quality, training efficiency and rendering efficiency. 3DGS uses a set of Gaussians to represent the scene, which are rendered into the camera internal and external parameters for Gaussian scene fusion.
[0029] In the related art, when fusing Gaussian scenes, it is necessary to manually calculate the rotation matrix and translation matrix between the two Gaussian scenes, and then align the two Gaussian scenes using the rotation matrix and translation matrix to achieve the fusion of Gaussian scenes. However, this process requires manual operation, has large errors, and is difficult to automate.
[0030] The embodiment of the present application provides a method for fusing a Gaussian scene, by obtaining a first Gaussian scene including a plurality of first Gaussian bodies and a second Gaussian scene including a plurality of second Gaussian bodies, determining the matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies, and then fusing the first Gaussian scene and the second Gaussian scene by the transformation information determined by the matching relationship, so as to obtain a target Gaussian scene. In this way, on the one hand, the transformation information is determined by the matching relationship between the Gaussian bodies, and the accuracy of the transformation information is improved; on the other hand, since the transformation information of the present application is automatically determined by the matching relationship, no manual intervention is required, and the automation of the fusion of the Gaussian scene is realized. The method provided by the embodiment of the present application can be performed by an electronic device, wherein the electronic device can be a laptop, a tablet computer, a desktop computer, a set-top box, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other various types of terminals, and can also be implemented as a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.
[0031] Figure 1 A schematic diagram of the implementation process of a Gaussian scene fusion method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes steps S101 to S104, wherein:
[0032] Step S101, obtaining a first Gaussian scene and a second Gaussian scene; wherein the first Gaussian scene includes a plurality of first Gaussian bodies, and the second Gaussian scene includes a plurality of second Gaussian bodies.
[0033] Here, the Gaussian scene is a scene obtained by displaying multiple Gaussian bodies. The Gaussian body is a Gaussian distribution, which is a common continuous probability distribution. In some embodiments, the dimension of the Gaussian body can be three-dimensional, and correspondingly, when the dimension of the Gaussian body is three-dimensional, the corresponding Gaussian scene is also three-dimensional.
[0034] The method for obtaining the Gaussian scene may include but is not limited to: reading the first Gaussian scene and the second Gaussian scene from a database, and receiving the first Gaussian scene and the second Gaussian scene sent by other devices. For example, the database is a warehouse for storing data and has the function of storing Gaussian scenes. During implementation, the first Gaussian scene and the second Gaussian scene can be read from the storage space of the database. For another example, a plurality of Gaussian bodies are stored in the memory of other devices, which has the function of sending the first Gaussian scene and the second Gaussian scene.
[0035] In some embodiments, the first Gaussian scene and the second Gaussian scene are trained Gaussian scenes, and both the first Gaussian body and the second Gaussian body have corresponding feature information. The feature information is used to determine the rendered image features corresponding to the rendered image of the first Gaussian scene or the second Gaussian scene.
[0036] Step S102: determining a matching relationship between a plurality of first Gaussian bodies and a plurality of second Gaussian bodies.
[0037] Here, the method for determining the matching relationship may include, but is not limited to, using a matching identifier, using a twin neural network, and the like. For example, a plurality of first Gaussians and a plurality of second Gaussians include different identifiers, and the matching relationship can be determined based on the identifiers corresponding to the plurality of first Gaussians and the identifiers corresponding to the plurality of second Gaussians. The identifier may be of any suitable type, such as a digital type, a character type, and the like. For example, in the case where the type of the identifier is a digital type, the number 1 may represent a first Gaussian or a second Gaussian. For another example, in the case where the type of the identifier is a character type, X may represent a second Gaussian or a first Gaussian. In some embodiments, the identifiers corresponding to the plurality of first Gaussians included in the first Gaussian scene are all different, the identifiers between the plurality of first Gaussians and the plurality of second Gaussians may be the same, and a matching relationship exists between the first Gaussian and the second Gaussian having the same identifier.
[0038] For another example, the twin neural network is a coupling architecture based on two artificial neural networks, which can perform small sample or single sample learning and is not easily disturbed by erroneous samples. The twin neural network can extract key information from multiple first Gaussian bodies and multiple second Gaussian bodies respectively, and determine the matching relationship between multiple first Gaussian bodies and multiple second Gaussian bodies by comparing the key information.
[0039] In some embodiments, both the first Gaussian body and the second Gaussian body have corresponding characteristic information, and the matching relationship between the multiple first Gaussian bodies and the multiple second Gaussian bodies can be determined through the characteristic information corresponding to the multiple first Gaussian bodies and the characteristic information corresponding to the multiple second Gaussian bodies.
[0040] In some embodiments, there is an overlapping area between the first Gaussian scene and the second Gaussian scene, and a plurality of first Gaussian bodies and a plurality of second Gaussian bodies in the overlapping area have a matching relationship.
[0041] Step S103: determining transformation information based on the matching relationship.
[0042] Here, the transformation information is information used to transform the first Gaussian scene or the second Gaussian scene to achieve the fusion of the first Gaussian scene and the second Gaussian scene. The transformation information may include, but is not limited to: a rotation matrix, a translation matrix, a scaling matrix, etc. A rotation matrix is a matrix that has a Gaussian effect of changing the direction of a Gaussian body but not the size when multiplied by a vector and maintains the chirality. The translation matrix is a matrix that describes the transformation of a Gaussian body moving a certain distance in a specific direction in space. The scaling matrix is a matrix that changes the position, direction, and size of a Gaussian body in two-dimensional or three-dimensional space.
[0043] Methods for determining transformation information may include but are not limited to: using an iterative closest point (ICP) algorithm, using a scale-invariant feature transform (SIFT) algorithm, etc. The ICP algorithm is an iteratively optimized point cloud matching algorithm that can iteratively determine the transformation information between the first Gaussian scene and the second Gaussian scene so as to maximize the overlap between the first Gaussian scene and the second Gaussian scene. The SIFT algorithm is an algorithm in computer vision that is used to detect and describe key points in a Gaussian body, and then determine the transformation information between the first Gaussian scene and the second Gaussian scene based on the key points.
[0044] In some embodiments, the transformation information includes a rotation matrix and a translation matrix, and the three-dimensional point cloud coordinate deviation information between the first and second Gaussian bodies that match can be determined based on the matching relationship, and then the rotation matrix and the translation matrix are determined according to the three-dimensional point cloud coordinate deviation information. The three-dimensional point cloud coordinate deviation information refers to the deviation information between the position of the first Gaussian body that matches and the position of the second Gaussian body.
[0045] Step S104: Based on the transformation information, the first Gaussian scene and the second Gaussian scene are fused to obtain a target Gaussian scene.
[0046] Here, the target Gaussian scene is a Gaussian scene obtained by fusing the first Gaussian scene and the second Gaussian scene. In some embodiments, fusing the first Gaussian scene and the second Gaussian scene may be: based on the transformation information, the position of the first Gaussian scene is transformed so as to achieve the fusion of the first Gaussian scene and the second Gaussian scene. In some embodiments, fusing the first Gaussian scene and the second Gaussian scene may be: based on the transformation information, the position of the second Gaussian scene is transformed so as to achieve the fusion of the first Gaussian scene and the second Gaussian scene.
[0047] In an embodiment of the present application, a first Gaussian scene including a plurality of first Gaussian bodies and a second Gaussian scene including a plurality of second Gaussian bodies are obtained, a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies is determined, and then the first Gaussian scene and the second Gaussian scene are fused through transformation information determined through the matching relationship to obtain a target Gaussian scene. In this way, on the one hand, the transformation information is determined through the matching relationship between the Gaussian bodies, thereby improving the accuracy of the transformation information; on the other hand, since the transformation information of the present application is automatically determined through the matching relationship, no manual intervention is required, thereby realizing the automation of the fusion of Gaussian scenes.
[0048] In some embodiments, the first Gaussian body and the second Gaussian body both have corresponding characteristic information, and step S102 includes step S121, wherein:
[0049] Step S121 : determining a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies based on feature information corresponding to the plurality of first Gaussian bodies and feature information corresponding to the plurality of second Gaussian bodies.
[0050] Here, the characteristic information corresponding to the Gaussian body is used to characterize the inherent properties of the Gaussian body. The characteristic information may include but is not limited to: position information, scale information, rotation information, color information, opacity, etc.
[0051] The position information is information used to represent the position of the Gaussian body. In some embodiments, the method of representing the position information may include, but is not limited to: the center point of the Gaussian body, the mean vector of the Gaussian body, etc.
[0052] The scale information is information used to represent the shape and size of the Gaussian body. In some embodiments, the scale information may be represented by methods including, but not limited to, a covariance matrix, a correlation coefficient matrix, a standard deviation, and the like.
[0053] The rotation information refers to the rotation state of the Gaussian body relative to the standard coordinate system. In some embodiments, the rotation information can be represented by a covariance matrix, wherein the non-diagonal elements of the covariance matrix can represent the rotation of the Gaussian body relative to the standard coordinate axis.
[0054] The color information is information used to represent the color of the Gaussian body. The color information can be any suitable color, for example, red, green, gray, etc. In some embodiments, the Gaussian body can include red, green, and blue coordinate information, and the color information of the Gaussian body can be represented by the red, green, and blue coordinate information. The red, green, and blue coordinate information can be any suitable coordinates, for example, (12, 100, 50), (60, 80, 122), etc.
[0055] Opacity is information used to describe the visibility of the Gaussian volume. Opacity can be any suitable value, for example, 0.5, 0.65, etc.
[0056] In some embodiments, the feature information may be two-dimensional, and the feature information may be used to determine a rendered image feature corresponding to the rendered image of the first Gaussian scene or the second Gaussian scene.
[0057] In some embodiments, the matching relationship between the plurality of first Gaussians and the plurality of second Gaussians can be determined by feature matching based on the feature information corresponding to the plurality of first Gaussians and the feature information corresponding to the plurality of second Gaussians. The feature matching method may include, but is not limited to: using a K-NearestNeighbor (KNN) classification algorithm, searching for a KDTree data structure, etc. The KNN classification algorithm is one of the methods in data mining classification technology. The distance from a first Gaussian to the plurality of second Gaussians can be calculated by the KNN classification algorithm, and then the second Gaussian closest to the first Gaussian can be used as the second Gaussian corresponding to the first Gaussian. KDTree is a tree data structure that stores instance points in a k-dimensional space for rapid retrieval. The matching relationship between the feature information corresponding to different Gaussians can be stored in the KDTree data structure. After determining the feature information corresponding to the plurality of first Gaussians and the feature information corresponding to the plurality of second Gaussians, the matching relationship between the plurality of first Gaussians and the plurality of second Gaussians can be determined by searching the KDTree data structure.
[0058] In some implementations, the matching relationship may be determined by a similarity distance between feature information corresponding to the first Gaussian body and feature information corresponding to the second Gaussian body.
[0059] In an embodiment of the present application, a matching relationship between multiple first Gaussian bodies and multiple second Gaussian bodies is determined based on feature information corresponding to multiple first Gaussian bodies and feature information corresponding to multiple second Gaussian bodies, thereby improving the accuracy of the matching relationship. Moreover, the matching relationship does not require human intervention, thereby reducing the possibility of deviations in manual calculations.
[0060] In some embodiments, the feature information is used to determine a rendered image feature corresponding to the rendered image of the first Gaussian scene or the second Gaussian scene, and step S121 includes step S1211, wherein:
[0061] Step S1211: determining a matching relationship according to a similarity distance between feature information corresponding to the first Gaussian body and feature information corresponding to the second Gaussian body.
[0062] Here, the rendered image of the Gaussian scene refers to an image generated by multiple Gaussian bodies in the Gaussian scene. The rendered image features are used to characterize the attribute information of the rendered image. Since the rendered image is generated by the Gaussian body, the feature information corresponding to the Gaussian body can be used to determine the rendered image features corresponding to the rendered image of the Gaussian scene.
[0063] The similarity distance is used to quantify the similarity between feature information. The similarity distance can be any suitable size, for example, 0.5, 3.3, etc. The method for determining the similarity distance may include but is not limited to: calculating the Euclidean distance, calculating the cosine similarity, etc. The Euclidean distance is a commonly used distance definition, and the true distance between two feature information can be calculated by the Euclidean distance. Cosine similarity can evaluate the similarity distance of two feature information by calculating the cosine value of the angle between them.
[0064] In some embodiments, when determining a matching relationship based on the similarity distance between feature information corresponding to a first Gaussian body and feature information corresponding to a second Gaussian body, the similarity distances between the first Gaussian body and multiple second Gaussian bodies can be calculated respectively, and the second Gaussian body corresponding to the smallest similarity distance is taken as the Gaussian body that matches the first Gaussian body, that is, there is a matching relationship between the first Gaussian body and the second Gaussian body.
[0065] In the implementation manner of the present application, the matching relationship is determined according to the similarity distance between the feature information corresponding to the first Gaussian body and the feature information corresponding to the second Gaussian body, thereby improving the accuracy of the matching relationship.
[0066] In some embodiments, the first Gaussian body and the second Gaussian body both have corresponding three-dimensional point cloud coordinates, the transformation information includes a rotation matrix and a translation matrix, and step S103 includes steps S131 to S132, wherein:
[0067] Step S131 : determining the three-dimensional point cloud coordinate deviation information between the first Gaussian body and the second Gaussian body that match each other based on the matching relationship.
[0068] Here, the three-dimensional point cloud coordinate deviation information refers to the deviation information between the position of the first Gaussian body and the position of the second Gaussian body that match. In some embodiments, the three-dimensional point cloud coordinate deviation information can be determined by point cloud processing software. The point cloud processing software may include but is not limited to: PDAL, CloudCompare, MeshLab, etc.
[0069] In some embodiments, based on the matching relationship, the first Gaussian scene or the second Gaussian scene can be used as a reference Gaussian scene, and the feature information corresponding to the Gaussian body in the reference Gaussian scene can be used as a reference coordinate to calculate the three-dimensional point cloud coordinate deviation information between the Gaussian body in the other Gaussian scene and the Gaussian body in the reference Gaussian scene.
[0070] Step S132, determining a rotation matrix and a translation matrix according to the three-dimensional point cloud coordinate deviation information.
[0071] Here, the method of determining the rotation matrix and the translation matrix may include but is not limited to: using the four-point consensus set algorithm, using the random sampling consensus algorithm (RANdom SAmple Consensus, RANSAC), etc. The four-point consensus set algorithm is a point cloud coarse registration algorithm that can find four coplanar corresponding points based on the three-dimensional point cloud coordinate deviation information, and use these points to calculate the rotation matrix and the translation matrix. RANSAC is a non-deterministic algorithm that uses an iterative method to calculate the rotation matrix and the translation matrix from the three-dimensional point cloud coordinate deviation information.
[0072] In some implementations, the rotation matrix and the translation matrix may be calculated based on the three-dimensional point cloud coordinate deviation information by calculating the squared sum of the Euclidean distances between feature information having a matching relationship.
[0073] In the implementation manner of the present application, transformation information including a rotation matrix and a translation matrix is determined based on the three-dimensional point cloud coordinate deviation information between the matched first Gaussian body and the second Gaussian body, thereby improving the accuracy of the transformation information.
[0074] In some implementations, the first Gaussian scene is obtained through steps S105 to S109, wherein:
[0075] Step S105 , based on the viewing angle parameters of the training sample image, determining an initial rendering image corresponding to the viewing angle parameters in the first initial Gaussian scene; wherein the first initial Gaussian scene includes a plurality of first Gaussian bodies, each of which has corresponding feature information.
[0076] Here, the training sample image refers to an image obtained from the real scene corresponding to the first Gaussian scene. The viewing angle parameters of the training sample image refer to the camera intrinsic parameters of the training sample image and the camera extrinsic parameters of the training sample image. The camera intrinsic parameters are parameters related to the camera's own characteristics, and may include but are not limited to: the focal length, pixel size, etc. of the camera. The camera extrinsic parameters are parameters in the world coordinate system, and may include but are not limited to: the position, rotation direction, viewing angle ray, etc. of the camera. In some embodiments, the training sample image may be an RGB image.
[0077] In some implementations, the viewing angle parameters of the training sample images may be obtained by using Structure from Motion (SFM), which is a technique for obtaining camera parameters by analyzing image sequences and performing three-dimensional reconstruction.
[0078] The first initial Gaussian scene refers to an untrained Gaussian scene. The initial rendered image refers to an image obtained by rendering the untrained Gaussian scene. In some embodiments, there is a large difference between the initial rendered image and the training sample image. Therefore, it is necessary to update the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene at least once, so that the difference between the initial rendered image and the training sample image becomes smaller, thereby obtaining the first Gaussian scene.
[0079] The method for determining the initial rendering image may include but is not limited to: using 3DGS, superimposing two-dimensional projection images, etc. 3DGS is a scene expression method, which uses sputtering to hit a Gaussian body on a two-dimensional plane to achieve fast screen rendering to obtain an initial rendering image. A two-dimensional projection image refers to an image obtained by projecting a Gaussian body under a viewing angle parameter. Multiple first Gaussian bodies of the first initial Gaussian scene are projected to obtain multiple two-dimensional projection images, and then multiple two-dimensional projection images are superimposed to obtain the initial rendering image.
[0080] Step S106: determining an initial rendered image feature corresponding to the initial rendered image based on feature information corresponding to the plurality of first Gaussian volumes.
[0081] Here, the initial rendering image feature is information used to describe the initial rendering image. The method for determining the initial rendering image feature may include but is not limited to: through the Histogram of Oriented Gradient (HOG), through the Local Binary Pattern (LBP), etc. For example, HOG is a feature descriptor used for object detection in computer vision and image processing, and the feature is formed by calculating and counting the gradient direction histogram of the local area of the image. Through HOG, the initial rendering image is divided into small connected areas, and then the gradient or edge direction histogram of each pixel in the connected area is collected. Finally, these histograms are combined to obtain the initial rendering image feature points, and then multiple initial rendering image feature points constitute the initial rendering image feature. For another example, LBP is an operator used to describe the local texture features of an image. Through LBP, each pixel value in the initial rendering image is compared with its neighboring pixel value, the result is saved as a binary number, and the obtained binary bit string is used as the initial rendering image feature.
[0082] In some embodiments, the initial rendered image features include feature values corresponding to multiple target pixel positions in the initial rendered image, and the feature values corresponding to the multiple target pixel positions can be determined based on the pixel values corresponding to the multiple target pixel positions in the two-dimensional projection image, the opacities corresponding to the multiple first Gaussian bodies, and the feature information corresponding to the multiple first Gaussian bodies, thereby determining the initial rendered image features. The pixel value can be any suitable size, for example, 5000, 1200, etc.
[0083] Step S107, obtaining reference image features; wherein the reference image features are obtained by extracting features from the training sample images.
[0084] Here, the reference image feature is information used to describe the training sample image. In some embodiments, the reference image feature may represent a feature corresponding to the reference image on a two-dimensional plane.
[0085] Methods for extracting features from training sample images may include but are not limited to: using a feature extraction model, using LBP, etc. The feature extraction model may be any suitable model, for example, Transformer, DINO v2, etc. Transformer is a model based on a self-attention mechanism and can be used to extract reference image features. DINO v2 is a publicly available pre-trained two-dimensional image feature extraction neural network that can be used to extract reference image features.
[0086] Step S108: determining first difference information between the initial rendering image features and the reference image features.
[0087] Here, the first difference information refers to the difference between the initial rendered image feature and the reference image feature, and is used to characterize the degree of deviation of the initial rendered image relative to the training sample image. The first difference information can be any suitable size, for example, 0.2, 0.5, etc. In some embodiments, the larger the first difference information, the larger the deviation of the initial rendered image relative to the training sample image.
[0088] The method for determining the first difference information may include but is not limited to: calculating the first difference, calculating the weighted value / taking the logarithm / taking the index of the first difference, etc. The first difference refers to the difference between the initial rendering image feature and the reference image feature. For example, the first difference is used as the first difference information.
[0089] In some implementations, the first difference information can be determined by the following formula (1-1): Right now:
[0090]
[0091] Among them, f g is the reference image feature, and f is the initial rendered image feature.
[0092] Step S109: based on the first difference information, updating the feature information corresponding to the plurality of first Gaussian bodies of the first initial Gaussian scene at least once to obtain a first Gaussian scene.
[0093] Here, the updating method of the feature information may include but is not limited to: using a gradient descent method, using a momentum update method, using a Newton momentum method, etc.
[0094] In some embodiments, the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene that has not been updated may be a default value, and at least one update means gradually assigning values to the feature information corresponding to the multiple first Gaussian bodies. The default value may be any appropriate value, for example, 0, 1, etc.
[0095] In some embodiments, it can be determined based on a preset condition whether it is necessary to update the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene. If it is necessary to update the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene, a suitable update algorithm is used to update the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene. After updating the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene, it is determined again whether it is necessary to continue to update the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene based on the training sample image corresponding to the new training sample (which can be the original training sample or another training sample). If it is not necessary to update the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene, the first initial Gaussian scene after the final update is determined as the first Gaussian scene. The preset condition may include but is not limited to satisfying at least one of the number of iterations, the difference information being not greater than a threshold, and the difference information converging.
[0096] For example, the first difference information can be determined based on the initial rendering image features and the reference image features. If the first difference information does not meet the preset conditions, the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene is updated; if the first difference information meets the preset conditions, the updating of the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene is stopped, and the first initial Gaussian scene after the final update is determined as the first Gaussian scene.
[0097] In the implementation mode of the present application, the initial rendered image corresponding to the viewing angle parameter in the first initial Gaussian scene is determined by the viewing angle parameter of the training sample image, and then the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene is updated at least once according to the first difference information between the features of the initial rendered image and the features of the reference image obtained by extracting features from the training sample image, so as to obtain the first Gaussian scene. On the one hand, the initial rendered image is determined according to the viewing angle parameter of the training sample image, which improves the accuracy of the initial rendered image; on the other hand, the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene is updated at least once according to the first difference information, which improves the accuracy of the feature information, thereby ensuring the accuracy of Gaussian scene fusion.
[0098] In some embodiments, the initial rendered image is determined based on two-dimensional projection images of multiple first Gaussian bodies under viewing angle parameters, and the initial rendered image features include feature values corresponding to multiple target pixel positions in the initial rendered image. Step S106 includes steps S161 to S162, wherein:
[0099] Step S161 : determining target parameters corresponding to the plurality of first Gaussian bodies based on pixel values corresponding to the plurality of target pixel positions in the two-dimensional projection image and opacities corresponding to the plurality of first Gaussian bodies.
[0100] Here, the pixel value is a value assigned by a computer when the two-dimensional projection image is digitized, which represents the average brightness information of a small square of the two-dimensional projection image, or the average reflection density information of the small square. The pixel value can be any suitable size, for example, 5000, 1200, etc. The opacity can be any suitable size, for example, 0.5, 0.65, etc. In some embodiments, the target pixel position can be any pixel position in the initial rendering image, and the number of the target pixel positions is not greater than the number of all pixel positions in the initial rendering image.
[0101] In some embodiments, the method for determining the target parameter may include, but is not limited to, calculating the first product, calculating the weighted / logarithmic / index of the first product, etc. The first product refers to the product of the pixel value corresponding to the target pixel position in the two-dimensional projection image and the opacity corresponding to the first Gaussian body. For example, the first product is used as the target parameter.
[0102] Step S162: determining feature values corresponding to a plurality of target pixel positions based on target parameters corresponding to the plurality of first Gaussian bodies and feature information corresponding to the plurality of first Gaussian bodies.
[0103] Here, the characteristic information corresponding to the plurality of first Gaussian bodies may be corrected by using the target parameters corresponding to the plurality of first Gaussian bodies to determine the characteristic values corresponding to the plurality of target pixel positions.
[0104] In some embodiments, the method for determining the eigenvalue corresponding to the target pixel position may include, but is not limited to: calculating the second product, calculating the weighted / logarithmic / index of the second product, etc. The second product refers to the product of the target parameter corresponding to the first Gaussian body and the feature information corresponding to the first Gaussian body. For example, the second product is used as the eigenvalue corresponding to the target pixel position.
[0105] In an embodiment of the present application, the target parameters corresponding to the first Gaussian body are determined by the pixel value corresponding to the target pixel position in the two-dimensional projection image and the opacity corresponding to the first Gaussian body, and then the initial rendered image features including the feature values corresponding to the pixel position are determined according to the target parameters and the feature information, thereby improving the accuracy of the initial rendered image features, so that the first initial Gaussian scene can be trained more accurately according to the initial rendered image features.
[0106] In some embodiments, the viewing angle parameter includes a viewing angle ray, and the plurality of first Gaussian bodies are sequentially arranged along the direction of the viewing angle ray. Step S162 includes steps S1621 to S1622, wherein:
[0107] Step S1621, determining feature components corresponding to the plurality of first Gaussian bodies based on target parameters corresponding to the plurality of first Gaussian bodies, target parameters respectively corresponding to the plurality of preceding first Gaussian bodies before the plurality of first Gaussian bodies, and feature information corresponding to the plurality of first Gaussian bodies.
[0108] Here, the viewing ray refers to a ray that passes through the three-dimensional space and is determined between two points starting from the camera center and a pixel point on the plane where the initial rendered image is located. In some embodiments, the multiple first Gaussian bodies are sequentially sorted along the direction of the viewing ray according to the viewing ray, so that the multiple first Gaussian bodies are sequentially projected according to the sorting, and then the initial rendered image is obtained according to the two-dimensional projection image obtained by the projection. In some embodiments, the number of first Gaussian bodies included in different viewing rays is different.
[0109] The method for determining the characteristic component corresponding to the first Gaussian body may include, but is not limited to: calculating the third product, calculating the weighted / logarithmic / index of the third product, etc. The third product refers to the product of the target parameter corresponding to the first Gaussian body, the target parameters respectively corresponding to the plurality of first Gaussian bodies before the first Gaussian body, and the characteristic information corresponding to the first Gaussian body. For example, the third product is used as the characteristic component corresponding to the first Gaussian body.
[0110] In some implementations, the characteristic component A may be determined by the following formula (1-2), namely:
[0111]
[0112] Among them, f i is the feature information corresponding to the first Gaussian body of the i-th j are the target parameters corresponding to the multiple first-order Gaussian bodies before the i-th first Gaussian body, α i is the target parameter corresponding to the i-th first Gaussian body.
[0113] Step S1622: determining characteristic values corresponding to a plurality of target pixel positions based on characteristic components corresponding to the plurality of first Gaussian volumes.
[0114] Here, the method for determining the eigenvalue corresponding to the pixel position may include but is not limited to: calculating the first sum, calculating the weighted / logarithmic / index of the first sum, etc. The first sum refers to the sum of the eigenvalues corresponding to each first Gaussian volume. For example, the first sum is used as the eigenvalue corresponding to the pixel position.
[0115] In some implementations, the initial rendered image feature f may be determined by the following formula (1-3), namely:
[0116]
[0117] Among them, A i is the i-th eigencomponent, and N is the number of Gaussian volumes traversed along the viewing ray.
[0118] In an embodiment of the present application, the characteristic value corresponding to the pixel position is determined by determining the characteristic component corresponding to the first Gaussian body through the target parameters corresponding to the first Gaussian body, the target parameters corresponding to multiple preceding first Gaussian bodies before the first Gaussian body, and the characteristic information corresponding to the first Gaussian body, thereby improving the accuracy of the initial rendered image features.
[0119] In some implementations, step S109 includes steps S191 to S192, wherein:
[0120] Step S191: determining at least one second difference information based on the initial rendering image and the training sample image.
[0121] Here, the second difference information may be attribute information between the initial rendering image and the training sample image. The attribute information may include but is not limited to: at least one of mean, mathematical expectation, opacity, covariance matrix, etc.
[0122] The second difference information may be of any appropriate size, for example, 0.34, 0.12, etc. In some embodiments, the larger the second difference information is, the larger the deviation of the initial rendered image from the training sample image is.
[0123] The method for determining the second difference information may include, but is not limited to, calculating the second difference, calculating the weighting / logarithm / index of the second difference, etc. The second difference refers to the difference between the attribute information of the initial rendering image feature and the attribute information of the reference image feature. For example, the second difference is used as the second difference information.
[0124] In some embodiments, when calculating the second difference, the attribute information of the rendered image feature and the attribute information of the reference image feature are the same. For example, when the attribute information of the rendered image feature is a covariance matrix, the attribute information of the reference image feature is also a covariance matrix.
[0125] Step S192: Based on the first difference information and at least one second difference information, feature information corresponding to multiple first Gaussian bodies of the first initial Gaussian scene is updated at least once to obtain a first Gaussian scene.
[0126] Here, based on the first difference information and at least one second difference information, the feature information corresponding to multiple first Gaussian bodies of the first initial Gaussian scene is updated at least once, and the process of obtaining the first Gaussian scene is similar to the process of the aforementioned step S109. During implementation, the specific implementation method of the aforementioned step S109 can be referred to.
[0127] In an embodiment of the present application, feature information corresponding to multiple first Gaussian bodies of the first initial Gaussian scene is updated at least once through first difference information and at least one second difference information of the initial rendered image and the training sample image. Since other parameter information of the initial rendered image and the training sample image is added to update the feature information, the accuracy of updating the feature information corresponding to multiple first Gaussian bodies of the first initial Gaussian scene is improved, thereby ensuring the fusion effect of the Gaussian scene.
[0128] The following describes the application of the Gaussian scene fusion method provided in the embodiment of the present application in an actual scene.
[0129] As an alternative to NeRF, 3DGS has higher reconstruction quality and higher training and rendering efficiency. 3DGS uses a set of Gaussian volumes to represent the scene, which are rendered into the camera internal and external parameters for Gaussian scene fusion.
[0130] In the related art, when fusing Gaussian scenes, it is necessary to manually calculate the rotation matrix and translation matrix between the two Gaussian scenes, and then align the two Gaussian scenes using the rotation matrix and translation matrix to achieve the fusion of Gaussian scenes. However, this process requires manual operation, has large errors, and is difficult to automate.
[0131] The embodiment of the present application provides a method for fusing Gaussian scenes, by obtaining a first Gaussian scene including a plurality of first Gaussian bodies and a second Gaussian scene including a plurality of second Gaussian bodies, determining the matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies, and then fusing the first Gaussian scene and the second Gaussian scene through transformation information determined by the matching relationship to obtain a target Gaussian scene. In this way, on the one hand, the transformation information is determined through the matching relationship between the Gaussian bodies, thereby improving the accuracy of the transformation information; on the other hand, since the transformation information of the present application is automatically determined through the matching relationship, no manual intervention is required, thereby realizing the automation of the fusion of Gaussian scenes.
[0132] Figure 2 A schematic diagram of the implementation process of a Gaussian scene fusion method provided in an embodiment of the present application Figure 2 ,like Figure 2 As shown, the method includes steps S201 to S209, wherein:
[0133] Step S201, extracting features from the RGB image (corresponding to the aforementioned training sample image) to obtain reference image features;
[0134] Step S202, determining an initial rendering image corresponding to the viewing angle parameter in the first initial Gaussian scene based on the viewing angle parameter of the RGB image;
[0135] Step S203, determining an initial rendered image feature corresponding to the initial rendered image based on feature information corresponding to the plurality of first Gaussian volumes;
[0136] Step S204, determining a feature loss between the initial rendered image feature and the reference image feature (corresponding to the aforementioned first difference information) and an RGB loss between the initial rendered image and the RGB image (corresponding to the aforementioned second difference information);
[0137] Step S205, based on the feature loss and the RGB loss, updating the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene at least once to obtain a first Gaussian scene;
[0138] Here, the second Gaussian scene can also be obtained through steps S201 to S205.
[0139] Step S206, determining a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies according to a similarity distance between feature information corresponding to the plurality of first Gaussian bodies of the first Gaussian scene and feature information corresponding to the plurality of second Gaussian bodies of the second Gaussian scene;
[0140] Step S207, determining the three-dimensional point cloud coordinate deviation information between the first Gaussian body and the second Gaussian body that match each other based on the matching relationship;
[0141] Step S208, determining a rotation matrix and a translation matrix according to the three-dimensional point cloud coordinate deviation information;
[0142] Step S209: Based on the rotation matrix and the translation matrix, the first Gaussian scene and the second Gaussian scene are fused to obtain a target Gaussian scene.
[0143] Figure 3 A schematic diagram of a block diagram of a method for calculating difference information of a Gaussian scene provided in an embodiment of the present application, such as Figure 3 As shown, where:
[0144] Perform feature extraction on the acquired RGB image 31 to obtain reference image features 32;
[0145] Input the viewing angle parameter 33 into the Gaussian scene 34 to obtain an initial rendered image 35;
[0146] calculating an initial rendered image feature 36 of the initial rendered image 35;
[0147] Determining a feature loss 37 between the initial rendered image features 36 and the reference image features 32;
[0148] An RGB loss 38 between the initial rendered image 35 and the RGB image 31 is determined.
[0149] Figure 4 A block diagram of a Gaussian scene fusion method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, where:
[0150] Perform feature query and matching on feature information 43 corresponding to a plurality of first Gaussian bodies of the first Gaussian scene 41 and feature information 44 corresponding to a plurality of first Gaussian bodies of the second Gaussian scene 42 to obtain a rotation matrix and a translation matrix 45;
[0151] The first Gaussian scene 41 and the second Gaussian scene 42 are fused using the rotation matrix and the translation matrix 45 to obtain a target Gaussian scene 46 .
[0152] Based on the above embodiments, the present application also provides a Gaussian scene fusion device. Figure 5 A schematic diagram of the composition structure of a Gaussian scene fusion device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the fusion device 500 of the Gaussian scene includes an acquisition module 501, a first determination module 502, a second determination module 503 and a fusion module 504, wherein:
[0153] An acquisition module 501 is used to acquire a first Gaussian scene and a second Gaussian scene; wherein the first Gaussian scene includes a plurality of first Gaussian bodies, and the second Gaussian scene includes a plurality of second Gaussian bodies;
[0154] A first determination module 502, configured to determine a matching relationship between a plurality of first Gaussian bodies and a plurality of second Gaussian bodies;
[0155] A second determination module 503, configured to determine transformation information based on the matching relationship;
[0156] The fusion module 504 is used to fuse the first Gaussian scene and the second Gaussian scene based on the transformation information to obtain a target Gaussian scene.
[0157] In some embodiments, both the first Gaussian body and the second Gaussian body have corresponding feature information, and the first determination module 502 is further used to determine the matching relationship between the multiple first Gaussian bodies and the multiple second Gaussian bodies based on the feature information corresponding to the multiple first Gaussian bodies and the feature information corresponding to the multiple second Gaussian bodies.
[0158] In some embodiments, the feature information is used to determine the rendered image features corresponding to the rendered image of the first Gaussian scene or the second Gaussian scene, and the first determination module 502 is further used to determine the matching relationship based on the similarity distance between the feature information corresponding to the first Gaussian body and the feature information corresponding to the second Gaussian body.
[0159] In some embodiments, the first Gaussian body and the second Gaussian body both have corresponding three-dimensional point cloud coordinates, the transformation information includes a rotation matrix and a translation matrix, and the second determination module 503 is further used to determine the three-dimensional point cloud coordinate deviation information between the matching first Gaussian body and the second Gaussian body based on the matching relationship; determine the rotation matrix and the translation matrix according to the three-dimensional point cloud coordinate deviation information.
[0160] In some embodiments, the fusion device 500 of the Gaussian scene also includes a third determination module, a fourth determination module, a second acquisition module, a fifth determination module and an update module, wherein: the third determination module is used to determine the initial rendered image corresponding to the viewing angle parameter in the first initial Gaussian scene based on the viewing angle parameter of the training sample image; wherein the first initial Gaussian scene includes multiple first Gaussian bodies, and the first Gaussian bodies all have corresponding feature information; the fourth determination module is used to determine the initial rendered image features corresponding to the initial rendered image based on the feature information corresponding to the multiple first Gaussian bodies; the second acquisition module is used to obtain reference image features; wherein the reference image features are obtained by feature extraction of the training sample image; the fifth determination module is used to determine the first difference information between the initial rendered image features and the reference image features; the update module is used to update the feature information corresponding to the multiple first Gaussian bodies of the first initial Gaussian scene at least once based on the first difference information to obtain the first Gaussian scene.
[0161] In some embodiments, the initial rendered image is determined based on a two-dimensional projection image of multiple first Gaussian bodies under a viewing angle parameter, and the initial rendered image features include feature values corresponding to multiple target pixel positions in the initial rendered image. The fourth determination module is also used to determine the target parameters corresponding to the multiple first Gaussian bodies based on the pixel values corresponding to the multiple target pixel positions in the two-dimensional projection image and the opacities corresponding to the multiple first Gaussian bodies; determine the feature values corresponding to the multiple target pixel positions based on the target parameters corresponding to the multiple first Gaussian bodies and the feature information corresponding to the multiple first Gaussian bodies.
[0162] In some embodiments, the viewing parameters include a viewing ray, and multiple first Gaussian bodies are arranged in sequence along the direction of the viewing ray. The fourth determination module is also used to determine the characteristic components corresponding to the multiple first Gaussian bodies based on the target parameters corresponding to the multiple first Gaussian bodies, the target parameters corresponding to the multiple preceding first Gaussian bodies before the multiple first Gaussian bodies, and the characteristic information corresponding to the multiple first Gaussian bodies; based on the characteristic components corresponding to the multiple first Gaussian bodies, determine the characteristic values corresponding to the multiple target pixel positions.
[0163] In some embodiments, the updating module is further used to determine at least one second difference information based on the initial rendered image and the training sample image; based on the first difference information and the at least one second difference information, update the feature information corresponding to multiple first Gaussian bodies of the first initial Gaussian scene at least once to obtain a first Gaussian scene.
[0164] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0165] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.
[0166] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.
[0167] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.
[0168] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.
[0169] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.
[0170] It should be noted that Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the electronic device 600 includes: a processor 601, a communication interface 602 and a memory 603, wherein:
[0171] The processor 601 generally controls the overall operation of the electronic device 600 .
[0172] The communication interface 602 enables the electronic device to communicate with other terminals or servers through a network.
[0173] The memory 603 is configured to store instructions and applications executable by the processor 601, and can also cache data to be processed or processed by the processor 601 and each module in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 601, the communication interface 602, and the memory 603 through the bus 604.
[0174] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0175] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.
[0176] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0178] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0179] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0180] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories, magnetic disks or optical disks.
[0181] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0182] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A Gaussian scene fusion method, the method include: Acquire a first Gaussian scene and a second Gaussian scene; wherein the first Gaussian scene includes a plurality of first Gaussian bodies, and the second Gaussian scene includes a plurality of second Gaussian bodies; Determining a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies; Based on the matching relationship, determining transformation information; Based on the transformation information, the first Gaussian scene and the second Gaussian scene are fused to obtain a target Gaussian scene.
2. The method according to claim 1, wherein the first Gaussian body and the second Gaussian body both have corresponding feature information; determining a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies, include: Based on the feature information corresponding to the plurality of first Gaussian bodies and the feature information corresponding to the plurality of second Gaussian bodies, a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies is determined.
3. The method according to claim 2, wherein the feature information is used to determine a rendered image feature corresponding to a rendered image of the first Gaussian scene or the second Gaussian scene, and the matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies is determined based on the feature information corresponding to the plurality of first Gaussian bodies and the feature information corresponding to the plurality of second Gaussian bodies. include: The matching relationship is determined according to a similarity distance between feature information corresponding to the first Gaussian body and feature information corresponding to the second Gaussian body.
4. The method according to claim 1, wherein the first Gaussian body and the second Gaussian body both have corresponding three-dimensional point cloud coordinates, and the transformation information includes a rotation matrix and a translation matrix; The determining of transformation information based on the matching relationship includes: Based on the matching relationship, determining three-dimensional point cloud coordinate deviation information between the first Gaussian body and the second Gaussian body that match each other; The rotation matrix and the translation matrix are determined according to the three-dimensional point cloud coordinate deviation information.
5. The method according to any one of claims 1 to 4, wherein the first Gaussian scene is obtained by: Based on the viewing angle parameters of the training sample image, determining an initial rendered image corresponding to the viewing angle parameters in the first initial Gaussian scene; in, The first initial Gaussian scene includes a plurality of first Gaussian bodies, each of which has corresponding feature information; Determining an initial rendered image feature corresponding to the initial rendered image based on feature information corresponding to the plurality of first Gaussian volumes; Acquire reference image features; wherein the reference image features are obtained by extracting features from the training sample image; Determining first difference information between the initial rendered image feature and the reference image feature; Based on the first difference information, feature information corresponding to a plurality of first Gaussian bodies of the first initial Gaussian scene is updated at least once to obtain the first Gaussian scene.
6. The method according to claim 5, wherein the initial rendered image is determined based on the two-dimensional projection images of the plurality of first Gaussian bodies under the viewing angle parameter, and the initial rendered image features include feature values corresponding to a plurality of target pixel positions in the initial rendered image; The determining, based on the feature information corresponding to the plurality of first Gaussian bodies, the initial rendered image feature corresponding to the initial rendered image comprises: Determining target parameters corresponding to the plurality of first Gaussian bodies based on pixel values corresponding to the plurality of target pixel positions in the two-dimensional projection image and opacities corresponding to the plurality of first Gaussian bodies; Based on the target parameters corresponding to the multiple first Gaussian bodies and the feature information corresponding to the multiple first Gaussian bodies, feature values corresponding to the multiple target pixel positions are determined.
7. The method according to claim 6, wherein the viewing angle parameter comprises a viewing angle ray, and the plurality of first Gaussian bodies are sequentially arranged along the direction of the viewing angle ray; determining the characteristic values corresponding to the plurality of target pixel positions based on the target parameters corresponding to the plurality of first Gaussian bodies and the characteristic information corresponding to the plurality of first Gaussian bodies, include: Determine characteristic components corresponding to the plurality of first Gaussian bodies based on target parameters corresponding to the plurality of first Gaussian bodies, target parameters respectively corresponding to a plurality of preceding first Gaussian bodies before the plurality of first Gaussian bodies, and characteristic information corresponding to the plurality of first Gaussian bodies; Based on the characteristic components corresponding to the multiple first Gaussian volumes, characteristic values corresponding to the multiple target pixel positions are determined.
8. The method according to claim 5, wherein the feature information corresponding to the plurality of first Gaussian bodies of the first initial Gaussian scene is updated at least once based on the first difference information to obtain the first Gaussian scene, include: Determining at least one second difference information based on the initial rendered image and the training sample image; Based on the first difference information and the at least one second difference information, feature information corresponding to a plurality of first Gaussian bodies of the first initial Gaussian scene is updated at least once to obtain the first Gaussian scene.
9. A Gaussian scene fusion device, the device include: An acquisition module, used to acquire a first Gaussian scene and a second Gaussian scene; wherein the first Gaussian scene includes a plurality of first Gaussian bodies, and the second Gaussian scene includes a plurality of second Gaussian bodies; A first determining module, configured to determine a matching relationship between the plurality of first Gaussian bodies and the plurality of second Gaussian bodies; A second determination module, configured to determine transformation information based on the matching relationship; A fusion module is used to fuse the first Gaussian scene and the second Gaussian scene based on the transformation information to obtain a target Gaussian scene.
10. An electronic device comprising a processor and a memory, wherein the memory stores a computer program executable on the processor, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.
Citation Information
Patent Citations
Dynamic 3D target modeling method and device and dynamic 3D object modeling method and device
CN118314254A
Scene three-dimensional reconstruction method based on prior depth and Gaussian sputtering model fusion
CN118351252A
System And Method For Utilizing Enhanced Scene Detection In A Depth Estimation Procedure
US20140064552A1
Method and systems for rendering an image
US20250356572A1
