Semantic-aware 3D gaussian point based scene understanding method and device

Abstract

The application provides a scene understanding method and device based on semantic-aware 3D Gaussian points, and the method comprises the following steps: acquiring picture information of multiple camera perspectives of a same scene; inputting the picture information into a pre-trained scene understanding model to acquire semantic-aware 3D Gaussian points output by the scene understanding model, wherein the semantic-aware 3D Gaussian points are used to determine semantic categories in a 3D scene, and the scene understanding model is obtained based on a semantic presentation loss function and a geometric continuity loss function. The scene understanding method and device based on semantic-aware 3D Gaussian points provided by the application input picture information of multiple camera perspectives of a same scene into a pre-trained scene understanding model, and by adding additional semantic attributes to each 3D Gaussian point to record the semantic category to which the Gaussian point belongs, the accuracy of semantic labels in 3D scene understanding can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a scene understanding method and device based on semantic-aware 3D Gaussian points. BACKGROUND

[0002] The goal of 3D scene understanding is to distinguish different semantic classes in a 3D scene. Considering the powerful 3D scene capability of implicit neural radiance fields (NeRF), the mainstream method of 3D scene understanding is based on NeRF to learn semantic labels, but the training and rendering time of NeRF is very long, and as an implicit representation method it is difficult to be directly used in downstream perception applications.

[0003] Recently, 3D Gaussian has been proposed as a powerful explicit 3D representation for 3D reconstruction, and due to its dependence on fast rasterization rendering, the training and rendering of 3D scene reconstruction can be completed in a very short time. Some methods have preliminarily explored the use of 3D Gaussian to complete 3D scene understanding, but the geometric features of 3D Gaussian have not been fully utilized, resulting in inaccurate semantic labels in 3D scene understanding. SUMMARY

[0004] The present application provides a scene understanding method and device based on semantic-aware 3D Gaussian points to solve the technical problem of inaccurate semantic labels in 3D scene understanding due to the lack of full utilization of the geometric features of 3D Gaussian in the prior art.

[0005] The present application provides a scene understanding method based on semantic-aware 3D Gaussian points, comprising the following steps.

[0006] Obtaining picture information of multiple camera perspectives of the same scene;

[0007] Inputting the picture information into a pre-trained scene understanding model to obtain semantic-aware 3D Gaussian points output by the scene understanding model, the semantic-aware 3D Gaussian points being used to determine semantic classes in a 3D scene, the scene understanding model being trained based on a semantic presentation loss function and a geometric continuity loss function, the semantic presentation loss function being used for pixel space calculation, and the geometric continuity loss function being used for 3D geometric space calculation.

[0008] In some embodiments, the training step of the scene understanding model comprises:

[0009] Determining the basic attributes of each 3D Gaussian point based on the picture information;

[0010] Obtaining a 2D semantic map based on a pre-trained 2D segmentation model, and taking the 2D semantic map as the initial semantic labels of the 3D Gaussian points;

[0011] adding semantic attributes to each 3D Gaussian point, and supervising learning of the semantic attributes based on the initial semantic labels, the semantic rendering loss function, and the geometric continuity loss function.

[0012] In some embodiments, determining the base attributes of each 3D Gaussian point based on the picture information comprises:

[0013] determining camera poses of each picture based on the picture information;

[0014] determining the base attributes of each 3D Gaussian point based on the camera poses and a 3D Gaussian point reconstruction algorithm.

[0015] In some embodiments, the base attributes of each 3D Gaussian point comprise one or more of:

[0016] position;

[0017] color;

[0018] transparency;

[0019] covariance.

[0020] In some embodiments, adding semantic attributes to each 3D Gaussian point further comprises:

[0021] freezing the base attributes of the 3D Gaussian points.

[0022] In some embodiments, supervising learning of the semantic attributes based on the initial semantic labels and the semantic rendering loss function comprises:

[0023] determining a first camera perspective;

[0024] obtaining a semantic segmentation probability map of the first camera perspective based on transparency blending rendering;

[0025] performing L1 loss calculation based on the semantic segmentation probability map and a 2D semantic map of the first camera perspective to obtain a semantic rendering loss result;

[0026] supervising learning of the semantic attributes based on the semantic rendering loss result.

[0027] In some embodiments, supervising learning of the semantic attributes based on the geometric continuity loss function comprises:

[0028] determining a distance matrix based on Euclidean distances between all pairs of 3D Gaussian points;

[0029] determining a cohesion matrix based on the distance matrix;

[0030] determine a first semantic distribution result of each 3D Gaussian point without random walk and a second semantic distribution result of each 3D Gaussian point after random walk based on the affinity matrix;

[0031] supervise the first semantic distribution result and the second semantic distribution result to be consistent based on KL divergence.

[0032] The application further provides a scene understanding device based on semantic-aware 3D Gaussian points, comprising the following modules:

[0033] A first obtaining module is configured to obtain picture information of multiple camera perspectives of a same scene.

[0034] A second obtaining module is configured to input the picture information into a pre-trained scene understanding model to obtain semantic-aware 3D Gaussian points output by the scene understanding model, wherein the semantic-aware 3D Gaussian points are used to determine semantic categories in a 3D scene, and the scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, the semantic presentation loss function is used for pixel space calculation, and the geometric continuity loss function is used for 3D geometric space calculation.

[0035] In some embodiments, the scene understanding device based on semantic-aware 3D Gaussian points further comprises a first training module, and the first training module comprises:

[0036] A first determining sub-module is configured to determine a basic attribute of each 3D Gaussian point based on the picture information.

[0037] A first processing sub-module is configured to obtain a 2D semantic map based on a pre-trained 2D segmentation model and use the 2D semantic map as an initial semantic label of the 3D Gaussian point.

[0038] A first supervising sub-module is configured to add a semantic attribute to each 3D Gaussian point and supervise learning of the semantic attribute based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

[0039] In some embodiments, the first determining sub-module comprises:

[0040] A first determining unit is configured to determine a camera pose of each picture based on the picture information.

[0041] A second determining unit is configured to determine a basic attribute of each 3D Gaussian point based on the camera pose and a 3D Gaussian point reconstruction algorithm.

[0042] In some embodiments, the basic attribute of each 3D Gaussian point comprises one or more of the following:

[0043] Position;

[0044] Color;

[0045] Transparency;

[0046] Covariance.

[0047] In some embodiments, the first supervision submodule further comprises:

[0048] A first processing unit is configured to freeze the base attribute of the 3D Gaussian point.

[0049] In some embodiments, the first supervision submodule comprises:

[0050] A third determination unit is configured to determine a first camera view angle;

[0051] A first acquisition unit is configured to acquire a semantic segmentation probability map of the first camera view angle based on transparency mixed rendering;

[0052] A second acquisition unit is configured to perform L1 loss calculation based on the semantic segmentation probability map and a 2D semantic map of the first camera view angle, and acquire a semantic presentation loss result;

[0053] A first supervision unit is configured to supervise learning of the semantic attribute based on the semantic presentation loss result.

[0054] In some embodiments, the first supervision submodule comprises:

[0055] A fourth determination unit is configured to determine a distance matrix based on the Euclidean distance between each pair of 3D Gaussian points;

[0056] A fifth determination unit is configured to determine an affinity matrix based on the distance matrix;

[0057] A sixth determination unit is configured to determine a first semantic distribution result of each 3D Gaussian point without random walk and a second semantic distribution result after random walk based on the affinity matrix;

[0058] A second supervision unit is configured to supervise consistency of the first semantic distribution result and the second semantic distribution result based on KL divergence.

[0059] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned any one of the scene understanding method based on the semantic-aware 3D Gaussian point.

[0060] The present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the above-mentioned any one of the scene understanding method based on the semantic-aware 3D Gaussian point.

[0061] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the semantic-knowledge-based 3D Gaussian point scene understanding method according to any one of the above.

[0062] The semantic-knowledge-based 3D Gaussian point scene understanding method and device provided by the application can improve the accuracy of semantic labels in 3D scene understanding by adding additional semantic attributes to each 3D Gaussian point to record the semantic category to which the Gaussian point belongs. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0064] Figure 1 is a flowchart of the semantic-knowledge-based 3D Gaussian point scene understanding method provided by the application.

[0065] Figure 2 is a structural schematic diagram of the semantic-knowledge-based 3D Gaussian point scene understanding device provided by the application.

[0066] Figure 3 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the application clearer, the technical solutions in the application will be described clearly and completely below with reference to the drawings in the application. Obviously, the described embodiments are some embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the application.

[0068] Figure 1 is a flowchart of the semantic-knowledge-based 3D Gaussian point scene understanding method provided by the application, as shown in Figure 1 The method comprises the following steps:

[0069] Step 101: Obtain picture information of multiple camera perspectives of the same scene.

[0070] In step 102, the picture information is input into a pre-trained scene understanding model to obtain semantic-aware 3D Gaussian points output by the scene understanding model, the semantic-aware 3D Gaussian points are used to determine semantic categories in a 3D scene, the scene understanding model is trained based on a semantic rendering loss function and a geometric continuity loss function, the semantic rendering loss function is used for pixel space calculation, and the geometric continuity loss function is used for 3D geometric space calculation.

[0071] Specifically, in view of the problems of long training and rendering time and large semantic label noise in the current mainstream 3D scene understanding method, the application can obtain picture information of multiple camera perspectives of the same scene, then input the picture information into a pre-trained scene understanding model, and obtain semantic-aware 3D Gaussian points output according to a semantic rendering loss function and a geometric continuity loss function, the semantic-aware 3D Gaussian points can be used for 3D scene understanding.

[0072] Among them, the semantic rendering loss is calculated in the pixel space, and the geometric continuity loss is calculated in the 3D geometric space.

[0073] The scene understanding method based on semantic-aware 3D Gaussian points provided by the application inputs picture information of multiple camera perspectives of the same scene into a pre-trained scene understanding model, adds an additional semantic attribute to each 3D Gaussian point to record the semantic category to which the Gaussian point belongs, and thus the accuracy of semantic labels in 3D scene understanding can be improved.

[0074] In some embodiments, the training step of the scene understanding model comprises:

[0075] determining the basic attribute of each 3D Gaussian point based on the picture information;

[0076] obtaining a 2D semantic map based on a pre-trained 2D segmentation model, and taking the 2D semantic map as an initial semantic label of the 3D Gaussian point;

[0077] adding a semantic attribute to each 3D Gaussian point, and supervising the learning of the semantic attribute based on the initial semantic label, the semantic rendering loss function and the geometric continuity loss function.

[0078] Specifically, picture information of multiple camera perspectives of the same scene input is acquired, and the pictures are used to fit and reconstruct the 3D scene, and the basic attributes of all 3D Gaussians are optimized, in addition, the pictures are used to obtain a 2D semantic map corresponding to each picture as an initial semantic label through a pre-trained 2D semantic segmentation network, i.e., a 2D segmentation model.

[0079] After the optimization of the basic attributes is completed, the basic attributes are frozen, and then additional semantic attributes are added to each 3D Gaussian, and the semantic attributes are used to represent the semantic category to which the Gaussian belongs.

[0080] Then, the learning of the model is supervised according to the initial semantic label, a semantic presentation loss function and a geometric continuity loss function.

[0081] It is worth mentioning that the semantic segmentation network is a bottom model and can provide technical support for a large number of downstream computer vision related tasks, including scene understanding, image editing, video understanding and the like, and can be applied to automatic driving, medical image understanding and the like.

[0082] The 3D Gaussian-based scene understanding method provided by the application adds additional semantic attributes to each 3D Gaussian to record the semantic category to which the Gaussian belongs, and can be used for new view semantic segmentation map generation, 2D semantic segmentation denoising and downstream semantic editing tasks.

[0083] In some embodiments, the basic attributes of each 3D Gaussian are determined based on the picture information, including:

[0084] The camera pose of each picture is determined based on the picture information.

[0085] The basic attributes of each 3D Gaussian are determined based on the camera pose and a 3D Gaussian reconstruction algorithm.

[0086] Specifically, pictures of multiple perspectives of the same scene input, such as RGB pictures, are acquired, and a structure from motion algorithm is used to obtain the camera estimated pose of each picture.

[0087] The 3D scene is fitted with Gaussians according to an original 3D Gaussian reconstruction algorithm, and the basic attributes such as the position and color of each Gaussian are optimized.

[0088] The 3D Gaussian-based scene understanding method provided by the application optimizes the basic attributes of each 3D Gaussian based on picture information and a 3D Gaussian reconstruction algorithm, and provides a basis for subsequent supervised learning of semantic attributes.

[0089] In some embodiments, the base attribute of each 3D Gaussian point comprises one or more of the following:

[0090] Position;

[0091] Color;

[0092] Transparency;

[0093] Covariance.

[0094] Specifically, the 3D Gaussian point rendering is the same as the triangle rasterization, except that the basic drawing element is changed from a triangle to a Gaussian point.

[0095] Each Gaussian point can be described by the following parameters: position, color, transparency, and covariance.

[0096] Wherein, the position represents the position of the Gaussian point, the color represents the color of the Gaussian point, the transparency represents the transparency of the Gaussian point, and the covariance represents the scaling degree of the Gaussian point.

[0097] The method for scene understanding based on the semantic-aware 3D Gaussian point provided by the application optimizes the base attribute of each 3D Gaussian point according to picture information and a 3D Gaussian reconstruction algorithm, and provides a basis for subsequent supervised learning of semantic attributes.

[0098] In some embodiments, the adding of the semantic attribute to each 3D Gaussian point further comprises:

[0099] Freezing the base attribute of the 3D Gaussian point.

[0100] Specifically, after the base attribute is optimized, it is frozen, and then an additional semantic attribute is added to each 3D Gaussian point, and the semantic attribute is used to represent the semantic category to which the Gaussian point belongs.

[0101] The method for scene understanding based on the semantic-aware 3D Gaussian point provided by the application optimizes the base attribute of each 3D Gaussian point according to picture information and a 3D Gaussian reconstruction algorithm, and provides a basis for subsequent supervised learning of semantic attributes.

[0102] In some embodiments, the learning of the semantic attribute is supervised based on the initial semantic label and the semantic presentation loss function, comprising:

[0103] Determining a first camera perspective;

[0104] Obtaining a semantic segmentation probability map of the first camera perspective based on transparency blending rendering;

[0105] Performing L1 loss calculation based on the semantic segmentation probability map and a 2D semantic map of the first camera perspective to obtain a semantic presentation loss result;

[0106] The learning of the semantic attribute is supervised based on the semantic presentation loss result.

[0107] Specifically, for a certain camera view, after the semantic attribute of a 3D Gaussian point is normalized by softmax, the semantic attribute is rendered by alpha blending according to the camera view to obtain a semantic segmentation probability map under the view, L1 loss calculation is performed on the 2D semantic map under the view, and then the learning of the semantic attribute is supervised according to the calculated semantic presentation loss result.

[0108] The scene understanding method based on the semantic-aware 3D Gaussian point provided by the application can supervise the learning of the semantic attribute according to the initial semantic label and the semantic presentation loss function, improve the accuracy of the semantic attribute of the 3D Gaussian point, and further improve the performance of 3D scene understanding.

[0109] In some embodiments, the learning of the semantic attribute is supervised based on the geometric continuity loss function, including:

[0110] A distance matrix is determined based on the Euclidean distances between all pairs of 3D Gaussian points.

[0111] An affinity matrix is determined based on the distance matrix.

[0112] First semantic distribution results and second semantic distribution results of each 3D Gaussian point after random walk are determined based on the affinity matrix.

[0113] The first semantic distribution results and the second semantic distribution results are supervised to be consistent based on KL divergence.

[0114] Specifically, it is found through observation that most of the Gaussians after 3D reconstruction are covered on the geometric surface of an object, and strong geometric prior is provided, so the semantic attribute distribution of similar Gaussians should be as consistent as possible. The specific method is to calculate the Euclidean distances between all pairs of Gaussians, convert the distance matrix into an affinity matrix, perform random walk on the semantic distribution of each Gaussian through the affinity matrix, and encourage the first semantic distribution results and the second semantic distribution results after random walk to be consistent based on the KL divergence of the geometric continuity loss.

[0115] The scene understanding method based on the semantic-aware 3D Gaussian point provided by the application can supervise the learning of the semantic attribute according to the initial semantic label and the semantic presentation loss function, improve the accuracy of the semantic attribute of the 3D Gaussian point, and further improve the performance of 3D scene understanding.

[0116] Figure 2A structure schematic diagram of a scene understanding device based on semantic-aware 3D Gaussian points is provided in the application, as shown in Figure 2 The scene understanding device based on semantic-aware 3D Gaussian points provided by the application includes a first acquisition module 201 and a second acquisition module 202, wherein:

[0117] The first acquisition module 201 is configured to acquire picture information of multiple camera perspectives of the same scene.

[0118] The second acquisition module 202 is configured to input the picture information into a pre-trained scene understanding model to acquire semantic-aware 3D Gaussian points output by the scene understanding model, wherein the semantic-aware 3D Gaussian points are used to determine semantic categories in a 3D scene, and the scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculation, and the geometric continuity loss function is used for 3D geometric space calculation.

[0119] In some embodiments, the scene understanding device based on semantic-aware 3D Gaussian points further includes a first training module, which includes:

[0120] A first determination sub-module is configured to determine the basic attributes of each 3D Gaussian point based on the picture information.

[0121] A first processing sub-module is configured to acquire a 2D semantic map based on a pre-trained 2D segmentation model, and use the 2D semantic map as an initial semantic label of the 3D Gaussian point.

[0122] A first supervision sub-module is configured to add semantic attributes to each 3D Gaussian point, and supervise the learning of the semantic attributes based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

[0123] In some embodiments, the first determination sub-module includes:

[0124] A first determination unit is configured to determine the camera pose of each picture based on the picture information.

[0125] A second determination unit is configured to determine the basic attributes of each 3D Gaussian point based on the camera pose and a 3D Gaussian point reconstruction algorithm.

[0126] In some embodiments, the basic attributes of each 3D Gaussian point include one or more of the following:

[0127] Position;

[0128] Color;

[0129] Transparency;

[0130] Covariance.

[0131] In some embodiments, the first supervision submodule further includes:

[0132] A first processing unit is configured to freeze the base attribute of the 3D Gaussian point.

[0133] In some embodiments, the first supervision submodule includes:

[0134] A third determining unit is configured to determine a first camera view angle;

[0135] A first obtaining unit is configured to obtain a semantic segmentation probability map of the first camera view angle based on transparency blending rendering;

[0136] A second obtaining unit is configured to perform L1 loss calculation based on the semantic segmentation probability map and a 2D semantic map of the first camera view angle, and obtain a semantic presentation loss result;

[0137] A first supervision unit is configured to supervise learning of the semantic attribute based on the semantic presentation loss result.

[0138] In some embodiments, the first supervision submodule includes:

[0139] A fourth determining unit is configured to determine a distance matrix based on the Euclidean distance between each pair of 3D Gaussian points;

[0140] A fifth determining unit is configured to determine an affinity matrix based on the distance matrix;

[0141] A sixth determining unit is configured to determine a first semantic distribution result of each 3D Gaussian point without random walk and a second semantic distribution result after random walk based on the affinity matrix;

[0142] A second supervision unit is configured to supervise consistency of the first semantic distribution result and the second semantic distribution result based on KL divergence.

[0143] Specifically, the above-mentioned scene understanding device based on the semantic-aware 3D Gaussian point provided by the embodiments of the present application can realize all the method steps realized by the above-mentioned scene understanding method based on the semantic-aware 3D Gaussian point, and can achieve the same technical effects. Therefore, the same parts and beneficial effects of the method embodiments in the embodiments will not be described in detail.

[0144] Figure 3 is the entity structure schematic diagram of the electronic device provided by the present application, such as Figure 3As shown, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 complete mutual communication through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute a semantic-aware 3D Gaussian point-based scene understanding method, which includes:

[0145] Obtaining picture information of multiple camera perspectives of the same scene;

[0146] Inputting the picture information into a pre-trained scene understanding model to obtain a semantic-aware 3D Gaussian point output by the scene understanding model, the semantic-aware 3D Gaussian point being used to determine a semantic category in a 3D scene, the scene understanding model being trained based on a semantic presentation loss function and a geometric continuity loss function, the semantic presentation loss function being used for pixel space calculation, and the geometric continuity loss function being used for 3D geometric space calculation.

[0147] In addition, the logical instruction in the memory 330 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0148] In some embodiments, the training step of the scene understanding model includes:

[0149] Determining a basic attribute of each 3D Gaussian point based on the picture information;

[0150] Obtaining a 2D semantic map based on a pre-trained 2D segmentation model, and taking the 2D semantic map as an initial semantic label of the 3D Gaussian point;

[0151] Adding a semantic attribute to each 3D Gaussian point, and supervising learning of the semantic attribute based on the initial semantic label, the semantic presentation loss function, and the geometric continuity loss function.

[0152] In some embodiments, the determining the base attribute of each 3D Gaussian point based on the picture information comprises:

[0153] determining the camera pose of each picture based on the picture information;

[0154] determining the base attribute of each 3D Gaussian point based on the camera pose and a 3D Gaussian point reconstruction algorithm.

[0155] In some embodiments, the base attribute of each 3D Gaussian point comprises one or more of:

[0156] position;

[0157] color;

[0158] transparency;

[0159] covariance.

[0160] In some embodiments, the adding the semantic attribute to each 3D Gaussian point further comprises:

[0161] freezing the base attribute of the 3D Gaussian point.

[0162] In some embodiments, the supervising the learning of the semantic attribute based on the initial semantic label and the semantic rendering loss function comprises:

[0163] determining a first camera perspective;

[0164] obtaining a semantic segmentation probability map of the first camera perspective based on transparency blending rendering;

[0165] performing L1 loss calculation based on the semantic segmentation probability map and a 2D semantic map of the first camera perspective to obtain a semantic rendering loss result;

[0166] supervising the learning of the semantic attribute based on the semantic rendering loss result.

[0167] In some embodiments, the supervising the learning of the semantic attribute based on the geometric continuity loss function comprises:

[0168] determining a distance matrix based on the Euclidean distance between each pair of 3D Gaussian points;

[0169] determining an affinity matrix based on the distance matrix;

[0170] determining a first semantic distribution result of each 3D Gaussian point without random walk and a second semantic distribution result after random walk based on the affinity matrix;

[0171] The first semantic distribution result and the second semantic distribution result are supervised to be consistent based on KL divergence.

[0172] Specifically, the electronic device provided by the embodiment of the present application can realize all the method steps realized by the method embodiment whose execution subject is the electronic device, and achieve the same technical effects. Here, the same parts and beneficial effects of the method embodiment in the embodiment will not be described in detail.

[0173] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the semantic-aware 3D Gaussian point based scene understanding method provided by the above-mentioned methods, which comprises:

[0174] obtaining picture information of multiple camera perspectives of the same scene;

[0175] inputting the picture information into a pre-trained scene understanding model to obtain semantic-aware 3D Gaussian points output by the scene understanding model, the semantic-aware 3D Gaussian points being used to determine semantic categories in a 3D scene, the scene understanding model being trained based on a semantic presentation loss function and a geometric continuity loss function, the semantic presentation loss function being used for pixel space calculation, and the geometric continuity loss function being used for 3D geometric space calculation.

[0176] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the semantic-aware 3D Gaussian point based scene understanding method provided by the above-mentioned methods, which comprises:

[0177] obtaining picture information of multiple camera perspectives of the same scene;

[0178] inputting the picture information into a pre-trained scene understanding model to obtain semantic-aware 3D Gaussian points output by the scene understanding model, the semantic-aware 3D Gaussian points being used to determine semantic categories in a 3D scene, the scene understanding model being trained based on a semantic presentation loss function and a geometric continuity loss function, the semantic presentation loss function being used for pixel space calculation, and the geometric continuity loss function being used for 3D geometric space calculation.

[0179] The apparatus embodiments described above are merely illustrative, wherein the units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0180] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0181] In addition, it should be noted that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are generally a class, and do not limit the number of objects, for example, the first object can be one or more.

[0182] In the embodiments of the present application, the term "and / or" describes the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0183] In the embodiments of the present application, the term "a plurality of" means two or more, and other quantifiers are similar.

[0184] In the present application, "determining B based on A" means that A is considered as a factor when determining B. It is not limited to "determining B based on A only", but also includes "determining B based on A and C", "determining B based on A, C and E", "determining C based on A, and determining B based on C further", and the like. In addition, it can also include A as a condition for determining B, for example, "when A meets the first condition, determining B using the first method"; for example, "when A meets the second condition, determining B"; for example, "when A meets the third condition, determining B based on the first parameter"; and the like. Of course, A can also be a condition for determining B as a factor, for example, "when A meets the first condition, determining C using the first method, and further determining B based on C"; and the like.

[0185] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A semantic-knowledge-based 3D Gaussian point-based scene understanding method, characterized in that, The method comprises the steps of: acquiring picture information of multiple camera perspectives of the same scene; inputting the picture information into a pre-trained scene understanding model to acquire semantic-aware 3D Gaussian points output by the scene understanding model, the semantic-aware 3D Gaussian points being used to determine semantic categories in a 3D scene, the scene understanding model being trained based on a semantic presentation loss function and a geometric continuity loss function, the semantic presentation loss function being used for pixel space calculation, and the geometric continuity loss function being used for 3D geometric space calculation; the training steps of the scene understanding model comprising: determining basic attributes of each 3D Gaussian point based on the picture information; acquiring a 2D semantic map based on a pre-trained 2D segmentation model and taking the 2D semantic map as initial semantic labels of the 3D Gaussian points; adding semantic attributes to each 3D Gaussian point and supervising learning of the semantic attributes based on the initial semantic labels, the semantic presentation loss function and the geometric continuity loss function; supervising learning of the semantic attributes based on the geometric continuity loss function, comprising: determining a distance matrix based on Euclidean distances between all pairs of 3D Gaussian points; determining an affinity matrix based on the distance matrix; determining a first semantic distribution result of each 3D Gaussian point without random walk and a second semantic distribution result after random walk based on the affinity matrix; supervising consistency of the first semantic distribution result and the second semantic distribution result based on KL divergence. 2.The semantic-knowledge-based 3D Gaussian point based scene understanding method according to claim 1, characterized in that, The step of determining the basic attributes of each 3D Gaussian point based on the picture information comprises: determining camera poses of each picture based on the picture information; determining the basic attributes of each 3D Gaussian point based on the camera poses and a 3D Gaussian point reconstruction algorithm.

3. The semantic-knowledge-based 3D Gaussian point-based scene understanding method according to claim 2, characterized in that, The basic attributes of each 3D Gaussian point comprise one or more of the following: position; color; transparency; covariance.

4. The semantic-knowledge-based 3D Gaussian point-based scene understanding method according to claim 1, characterized in that, The step of adding semantic attributes to each 3D Gaussian point further comprises: freezing the basic attributes of the 3D Gaussian points.

5. The semantic-knowledge-based 3D Gaussian point based scene understanding method according to claim 1, characterized in that, The step of supervising learning of the semantic attributes based on the initial semantic labels and the semantic presentation loss function comprises: determining a first camera perspective; acquiring a semantic segmentation probability map of the first camera perspective based on transparency blending rendering; performing L1 loss calculation based on the semantic segmentation probability map and a 2D semantic map of the first camera perspective to acquire a semantic presentation loss result; supervising learning of the semantic attributes based on the semantic presentation loss result.

6. A semantic-knowledge-based 3D Gaussian point based scene understanding apparatus, characterized by, The method comprises the steps of: a first acquiring module configured to acquire picture information of multiple camera perspectives of the same scene; a second acquiring module configured to input the picture information into a pre-trained scene understanding model to acquire semantic-aware 3D Gaussian points output by the scene understanding model, the semantic-aware 3D Gaussian points being used to determine semantic categories in a 3D scene, the scene understanding model being trained based on a semantic presentation loss function and a geometric continuity loss function, the semantic presentation loss function being used for pixel space calculation, and the geometric continuity loss function being used for 3D geometric space calculation; the training steps of the scene understanding model comprising: determining basic attributes of each 3D Gaussian point based on the picture information; obtaining a 2D semantic map based on a pre-trained 2D segmentation model, and taking the 2D semantic map as an initial semantic label of a 3D Gaussian point; adding a semantic attribute to each 3D Gaussian point, and supervising learning of the semantic attribute based on the initial semantic label, the semantic presentation loss function, and the geometric continuity loss function; supervising learning of the semantic attribute based on the geometric continuity loss function, comprising: determining a distance matrix based on Euclidean distances between all 3D Gaussian points in pairs; determining a closeness matrix based on the distance matrix; determining a first semantic distribution result of each 3D Gaussian point without random walk and a second semantic distribution result after random walk based on the closeness matrix; supervising consistency of the first semantic distribution result and the second semantic distribution result based on KL divergence.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the scene understanding method of the semantic-aware 3D Gaussian point according to any one of claims 1 to 5 when executing the program.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the scene understanding method of the semantic-aware 3D Gaussian point according to any one of claims 1 to 5 when executed by the processor.

Images