Semantic knowable 3D Gaussian point scene understanding method and device

Abstract

The invention provides a scene understanding method and device of a 3D Gaussian point based on semantic knowing. The method comprises the following steps: acquiring picture information of a plurality of camera visual angles in the same scene; the picture information is input into a pre-trained scene understanding model, a semanteme-knowable 3D Gaussian point output by the scene understanding model is obtained, the semanteme-knowable 3D Gaussian point is used for determining a semantic category in a 3D scene, and the scene understanding model is obtained through training based on a semantic presentation loss function and a geometric continuity loss function. According to the scene understanding method and device of the 3D Gaussian points based on semantic knowing provided by the invention, the picture information of multiple camera visual angles of the same scene is input into the pre-trained scene understanding model, and the additional semantic attribute is added to each 3D Gaussian point to record the semantic category to which the Gaussian point belongs; the accuracy of semantic tags can be improved when a 3D scene is understood.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a scene understanding method and device based on semantically knowable 3D Gaussian points. Background Art

[0002] The goal of 3D scene understanding is to distinguish different semantic categories in 3D scenes. Considering the powerful 3D scene capabilities of implicit neural radiance fields (NeRF), mainstream 3D scene understanding methods are based on NeRF to learn semantic labels, but NeRF training and rendering time is very long, and as an implicit representation method, it is difficult to directly use it in downstream perception applications.

[0003] Recently, 3D Gaussian has been proposed as a powerful explicit 3D representation for 3D reconstruction. Since it relies on fast rasterization rendering, the training and rendering of 3D scene reconstruction can be completed in a very short time. Some methods have initially explored the use of 3D Gaussian to complete 3D scene understanding, but they did not fully utilize the geometric features of 3D Gaussian, resulting in inaccurate semantic labels when understanding 3D scenes. Summary of the invention

[0004] The present invention provides a scene understanding method and device based on semantically known 3D Gaussian points, so as to solve the technical problem in the prior art that the geometric features of 3D Gaussian points are not fully utilized, resulting in inaccurate semantic labels during 3D scene understanding.

[0005] The present invention provides a scene understanding method based on semantically known 3D Gaussian points, comprising the following steps.

[0006] Get image information from multiple camera perspectives of the same scene; The image information is input into a pre-trained scene understanding model to obtain semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine semantic categories in a 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculations and the geometric continuity loss function is used for 3D geometric space calculations.

[0007] In some embodiments, the step of training the scene understanding model includes: Determine basic attributes of each 3D Gaussian point based on the image information; Acquire a 2D semantic map based on a pre-trained 2D segmentation model, and use the 2D semantic map as an initial semantic label of a 3D Gaussian point; Semantic attributes are added to each 3D Gaussian point, and the learning of the semantic attributes is supervised based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

[0008] In some embodiments, determining the basic attributes of each 3D Gaussian point based on the image information includes: Determining a camera pose for each image based on the image information; Based on the camera pose and the 3D Gaussian point reconstruction algorithm, basic properties of each 3D Gaussian point are determined.

[0009] In some embodiments, the basic attributes of each 3D Gaussian point include one or more of the following: Location; color; transparency; Covariance.

[0010] In some embodiments, adding semantic attributes to each 3D Gaussian point further includes: Freeze the base properties of the 3D Gaussian points.

[0011] In some embodiments, supervising the learning of the semantic attribute based on the initial semantic label and the semantic presentation loss function includes: Determine the first camera perspective; Acquire a semantic segmentation probability map of the first camera perspective based on transparency mixed rendering; Perform L1 loss calculation based on the semantic segmentation probability map and the 2D semantic map of the first camera perspective to obtain a semantic presentation loss result; The learning of the semantic attributes is supervised based on the semantic presentation loss result.

[0012] In some embodiments, supervising the learning of the semantic attribute based on the geometric continuity loss function includes: Determine the distance matrix based on the Euclidean distance between all 3D Gaussian points; determining an affinity matrix based on the distance matrix; Determine, based on the affinity matrix, a first semantic distribution result of each 3D Gaussian point before random walk and a second semantic distribution result after random walk; Based on KL divergence supervision, the first semantic distribution result and the second semantic distribution result are consistent.

[0013] The present invention also provides a scene understanding device based on semantically known 3D Gaussian points, comprising the following modules: The first acquisition module is used to acquire image information of multiple camera perspectives of the same scene; The second acquisition module is used to input the image information into a pre-trained scene understanding model, and obtain the semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine the semantic categories in the 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculation, and the geometric continuity loss function is used for 3D geometric space calculation.

[0014] In some embodiments, the scene understanding device based on semantically known 3D Gaussian points further includes a first training module, wherein the first training module includes: A first determination submodule, configured to determine basic attributes of each 3D Gaussian point based on the image information; A first processing submodule is used to obtain a 2D semantic map based on a pre-trained 2D segmentation model, and use the 2D semantic map as an initial semantic label of a 3D Gaussian point; The first supervision submodule is used to add semantic attributes to each 3D Gaussian point and supervise the learning of the semantic attributes based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

[0015] In some embodiments, the first determining submodule includes: A first determining unit, configured to determine a camera pose of each picture based on the picture information; The second determining unit is used to determine the basic attributes of each 3D Gaussian point based on the camera posture and the 3D Gaussian point reconstruction algorithm.

[0016] In some embodiments, the basic attributes of each 3D Gaussian point include one or more of the following: Location; color; transparency; Covariance.

[0017] In some embodiments, the first supervision submodule further includes: The first processing unit is used to freeze the basic attributes of the 3D Gaussian points.

[0018] In some embodiments, the first supervision submodule includes: A third determining unit, configured to determine a first camera viewing angle; A first acquisition unit, configured to acquire a semantic segmentation probability map of the first camera perspective based on transparency mixed rendering; A second acquisition unit is used to perform L1 loss calculation based on the semantic segmentation probability map and the 2D semantic map of the first camera perspective to obtain a semantic presentation loss result; The first supervision unit is used to supervise the learning of the semantic attribute based on the semantic presentation loss result.

[0019] In some embodiments, the first supervision submodule includes: A fourth determining unit, configured to determine a distance matrix based on the Euclidean distances between all 3D Gaussian points; a fifth determining unit, configured to determine an affinity matrix based on the distance matrix; A sixth determining unit, configured to determine, based on the affinity matrix, a first semantic distribution result of each 3D Gaussian point before random walk and a second semantic distribution result after random walk; A second supervision unit is configured to supervise, based on KL divergence, that the first semantic distribution result and the second semantic distribution result are consistent.

[0020] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the scene understanding method based on semantically known 3D Gaussian points as described in any one of the above methods is implemented.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described scene understanding methods based on semantically known 3D Gaussian points.

[0022] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned scene understanding methods based on semantically known 3D Gaussian points.

[0023] The scene understanding method and device based on semantically known 3D Gaussian points provided by the present invention input image information of multiple camera perspectives of the same scene into a pre-trained scene understanding model, and by adding additional semantic attributes to each 3D Gaussian point to record the semantic category to which the Gaussian point belongs, the accuracy of semantic labels during 3D scene understanding can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flowchart of the scene understanding method based on semantically known 3D Gaussian points provided by the present invention.

[0026] Figure 2 It is a structural schematic diagram of a scene understanding device based on semantically known 3D Gaussian points provided by the present invention.

[0027] Figure 3 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Figure 1 is a flow chart of a scene understanding method based on semantically known 3D Gaussian points provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain image information of multiple camera perspectives of the same scene.

[0030] Step 102: input the image information into a pre-trained scene understanding model to obtain semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine semantic categories in the 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculations and the geometric continuity loss function is used for 3D geometric space calculations.

[0031] Specifically, the present invention aims at the problems of "too long training rendering time" and "large semantic label noise" existing in the current mainstream 3D scene understanding methods. The present invention can obtain image information of multiple camera perspectives of the same scene, and then input the image information into a pre-trained scene understanding model, and obtain output semantically known 3D Gaussian points according to the semantic rendering loss function and the geometric continuity loss function. The semantically known 3D Gaussian points can be used for 3D scene understanding.

[0032] Among them, semantic rendering loss is calculated in pixel space, while geometric continuity loss is calculated in 3D geometric space.

[0033] The scene understanding method based on semantically known 3D Gaussian points provided by the present invention inputs image information of multiple camera perspectives of the same scene into a pre-trained scene understanding model, and adds additional semantic attributes to each 3D Gaussian point to record the semantic category to which the Gaussian point belongs, thereby improving the accuracy of semantic labels during 3D scene understanding.

[0034] In some embodiments, the step of training the scene understanding model includes: Determine basic attributes of each 3D Gaussian point based on the image information; Acquire a 2D semantic map based on a pre-trained 2D segmentation model, and use the 2D semantic map as an initial semantic label of a 3D Gaussian point; Semantic attributes are added to each 3D Gaussian point, and the learning of the semantic attributes is supervised based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

[0035] Specifically, image information of multiple camera perspectives of the same scene is obtained as input. These images are used to fit and reconstruct the 3D scene and optimize the basic properties of all 3D Gaussian points. In addition, these images are passed through a pre-trained 2D semantic segmentation network, i.e., a 2D segmentation model, to obtain the 2D semantic map corresponding to each image as the initial semantic label.

[0036] After the basic attribute optimization is completed, it is frozen, and then additional semantic attributes are added to each 3D Gaussian point. The semantic attributes are used to characterize the semantic category to which the Gaussian point belongs.

[0037] The model is then supervised based on the initial semantic labels, semantic presentation loss function, and geometric continuity loss function.

[0038] It is worth mentioning that the semantic segmentation network is the underlying model, which can provide technical support for a large number of downstream computer vision-related tasks, including scene understanding, image editing, video understanding, etc. It can be applied to autonomous driving, medical image understanding, etc.

[0039] The scene understanding method based on semantically known 3D Gaussian points provided by the present invention adds additional semantic attributes to each 3D Gaussian point to record the semantic category to which the Gaussian point belongs. It can be used for new perspective semantic segmentation map generation, 2D semantic segmentation denoising, and to facilitate downstream semantic editing tasks.

[0040] In some embodiments, determining the basic attributes of each 3D Gaussian point based on the image information includes: Determining a camera pose for each image based on the image information; Based on the camera pose and the 3D Gaussian point reconstruction algorithm, basic properties of each 3D Gaussian point are determined.

[0041] Specifically, obtain input images of the same scene from multiple perspectives, such as RGB images, and use COLMAP to perform the structure from motion algorithm to obtain the estimated camera pose of each image.

[0042] Gaussians are fitted to the 3D scene according to the original 3D gaussian reconstruction algorithm, and the basic properties of each Gaussian, such as position and color, are optimized.

[0043] The scene understanding method based on semantically known 3D Gaussian points provided by the present invention optimizes the basic attributes of each 3D Gaussian point according to image information and a 3D gaussian reconstruction algorithm, thus providing a basis for the subsequent supervised learning of semantic attributes.

[0044] In some embodiments, the basic attributes of each 3D Gaussian point include one or more of the following: Location; color; transparency; Covariance.

[0045] Specifically, 3D Gaussian stippling is the same as triangle rasterization, except that the basic drawing elements are replaced by Gaussian points instead of triangles.

[0046] Each Gaussian point can be described by the following parameters: position, color, transparency and covariance.

[0047] Among them, the position represents the location of the Gaussian point, the color represents the color of the Gaussian point, the transparency represents the transparency of the Gaussian point, and the covariance represents the scaling degree of the Gaussian point.

[0048] The scene understanding method based on semantically known 3D Gaussian points provided by the present invention optimizes the basic attributes of each 3D Gaussian point according to image information and a 3D gaussian reconstruction algorithm, thus providing a basis for the subsequent supervised learning of semantic attributes.

[0049] In some embodiments, adding semantic attributes to each 3D Gaussian point further includes: Freeze the base properties of the 3D Gaussian points.

[0050] Specifically, after the basic attribute optimization is completed, it is frozen, and then additional semantic attributes are added to each 3D Gaussian point. The semantic attributes are used to characterize the semantic category to which the Gaussian point belongs.

[0051] The scene understanding method based on semantically known 3D Gaussian points provided by the present invention optimizes the basic attributes of each 3D Gaussian point according to image information and a 3D gaussian reconstruction algorithm, thus providing a basis for the subsequent supervised learning of semantic attributes.

[0052] In some embodiments, supervising the learning of the semantic attribute based on the initial semantic label and the semantic presentation loss function includes: Determine the first camera perspective; Acquire a semantic segmentation probability map of the first camera perspective based on transparency mixed rendering; Perform L1 loss calculation based on the semantic segmentation probability map and the 2D semantic map of the first camera perspective to obtain a semantic presentation loss result; The learning of the semantic attributes is supervised based on the semantic presentation loss result.

[0053] Specifically, for a certain camera perspective, the semantic attributes of the 3D Gaussian points are normalized by softmax, and then alpha blending is performed according to the camera perspective to obtain the semantic segmentation probability map under that perspective. The L1 loss is calculated with the 2D semantic map under that perspective, and then the learning of the semantic attributes is supervised based on the calculated semantic rendering loss result.

[0054] The scene understanding method based on semantically knowable 3D Gaussian points provided by the present invention supervises the learning of semantic attributes according to initial semantic labels and semantic presentation loss functions, which can improve the accuracy of the semantic attributes of 3D Gaussian points and further improve the performance of 3D scene understanding.

[0055] In some embodiments, supervising the learning of the semantic attribute based on the geometric continuity loss function includes: Determine the distance matrix based on the Euclidean distance between all 3D Gaussian points; determining an affinity matrix based on the distance matrix; Determine, based on the affinity matrix, a first semantic distribution result of each 3D Gaussian point before random walk and a second semantic distribution result after random walk; Based on KL divergence supervision, the first semantic distribution result and the second semantic distribution result are consistent.

[0056] Specifically, it is observed that most of the Gaussians after 3D reconstruction are covered on the geometric surface of the object, providing a strong geometric prior, so it can be required that the semantic attribute distribution of similar Gaussians should be as consistent as possible. The specific approach is to calculate the Euclidean distance between all Gaussians, convert the distance matrix into an affinity matrix, and randomly walk the semantic distribution of each Gaussian through the affinity matrix. The geometric continuity loss encourages the first semantic distribution result before random walk to be consistent with the second semantic distribution result after random walk based on KL divergence.

[0057] The scene understanding method based on semantically known 3D Gaussian points provided by the present invention can improve the accuracy of the semantic attributes of 3D Gaussian points and thus improve the performance of 3D scene understanding by supervising the learning of semantic attributes according to a geometric continuity loss function.

[0058] Figure 2 is a schematic diagram of the structure of a scene understanding device based on semantically known 3D Gaussian points provided by the present invention, such as Figure 2 As shown, the scene understanding device based on semantically known 3D Gaussian points provided in the embodiment of the present application includes a first acquisition module 201 and a second acquisition module 202, wherein: The first acquisition module 201 is used to acquire image information of multiple camera perspectives of the same scene; The second acquisition module 202 is used to input the image information into a pre-trained scene understanding model, and obtain the semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine the semantic categories in the 3D scene, and the scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculation, and the geometric continuity loss function is used for 3D geometric space calculation.

[0059] In some embodiments, the scene understanding device based on semantically known 3D Gaussian points further includes a first training module, wherein the first training module includes: A first determination submodule, configured to determine basic attributes of each 3D Gaussian point based on the image information; A first processing submodule is used to obtain a 2D semantic map based on a pre-trained 2D segmentation model, and use the 2D semantic map as an initial semantic label of a 3D Gaussian point; The first supervision submodule is used to add semantic attributes to each 3D Gaussian point and supervise the learning of the semantic attributes based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

[0060] In some embodiments, the first determining submodule includes: A first determining unit, configured to determine a camera pose of each picture based on the picture information; The second determining unit is used to determine the basic attributes of each 3D Gaussian point based on the camera posture and the 3D Gaussian point reconstruction algorithm.

[0061] In some embodiments, the basic attributes of each 3D Gaussian point include one or more of the following: Location; color; transparency; Covariance.

[0062] In some embodiments, the first supervision submodule further includes: The first processing unit is used to freeze the basic attributes of the 3D Gaussian points.

[0063] In some embodiments, the first supervision submodule includes: A third determining unit, configured to determine a first camera viewing angle; A first acquisition unit, configured to acquire a semantic segmentation probability map of the first camera perspective based on transparency mixed rendering; A second acquisition unit is used to perform L1 loss calculation based on the semantic segmentation probability map and the 2D semantic map of the first camera perspective to obtain a semantic presentation loss result; The first supervision unit is used to supervise the learning of the semantic attribute based on the semantic presentation loss result.

[0064] In some embodiments, the first supervision submodule includes: A fourth determining unit, configured to determine a distance matrix based on the Euclidean distances between all 3D Gaussian points; a fifth determining unit, configured to determine an affinity matrix based on the distance matrix; A sixth determining unit, configured to determine, based on the affinity matrix, a first semantic distribution result of each 3D Gaussian point before random walk and a second semantic distribution result after random walk; A second supervision unit is configured to supervise, based on KL divergence, that the first semantic distribution result and the second semantic distribution result are consistent.

[0065] Specifically, the scene understanding device based on semantically-knowable 3D Gaussian points provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned scene understanding method embodiment based on semantically-knowable 3D Gaussian points, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0066] Figure 3 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute a scene understanding method based on semantically known 3D Gaussian points, the method comprising: Get image information from multiple camera perspectives of the same scene; The image information is input into a pre-trained scene understanding model to obtain semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine semantic categories in a 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculations and the geometric continuity loss function is used for 3D geometric space calculations.

[0067] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0068] In some embodiments, the step of training the scene understanding model includes: Determine basic attributes of each 3D Gaussian point based on the image information; Acquire a 2D semantic map based on a pre-trained 2D segmentation model, and use the 2D semantic map as an initial semantic label of a 3D Gaussian point; Semantic attributes are added to each 3D Gaussian point, and the learning of the semantic attributes is supervised based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

[0069] In some embodiments, determining the basic attributes of each 3D Gaussian point based on the image information includes: Determining a camera pose for each image based on the image information; Based on the camera pose and the 3D Gaussian point reconstruction algorithm, basic properties of each 3D Gaussian point are determined.

[0070] In some embodiments, the basic attributes of each 3D Gaussian point include one or more of the following: Location; color; transparency; Covariance.

[0071] In some embodiments, adding semantic attributes to each 3D Gaussian point further includes: Freeze the base properties of the 3D Gaussian points.

[0072] In some embodiments, supervising the learning of the semantic attribute based on the initial semantic label and the semantic presentation loss function includes: Determine the first camera perspective; Acquire a semantic segmentation probability map of the first camera perspective based on transparency mixed rendering; Perform L1 loss calculation based on the semantic segmentation probability map and the 2D semantic map of the first camera perspective to obtain a semantic presentation loss result; The learning of the semantic attributes is supervised based on the semantic presentation loss result.

[0073] In some embodiments, supervising the learning of the semantic attribute based on the geometric continuity loss function includes: Determine the distance matrix based on the Euclidean distance between all 3D Gaussian points; determining an affinity matrix based on the distance matrix; Determine, based on the affinity matrix, a first semantic distribution result of each 3D Gaussian point before random walk and a second semantic distribution result after random walk; Based on KL divergence supervision, the first semantic distribution result and the second semantic distribution result are consistent.

[0074] Specifically, the above-mentioned electronic device provided in the embodiment of the present application can implement all the method steps implemented by the method embodiment in which the execution subject is the electronic device, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0075] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the scene understanding method based on semantically known 3D Gaussian points provided by the above methods, the method comprising: Get image information from multiple camera perspectives of the same scene; The image information is input into a pre-trained scene understanding model to obtain semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine semantic categories in a 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculations and the geometric continuity loss function is used for 3D geometric space calculations.

[0076] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the scene understanding method based on semantically known 3D Gaussian points provided by the above methods, the method comprising: Get image information from multiple camera perspectives of the same scene; The image information is input into a pre-trained scene understanding model to obtain semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine semantic categories in a 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculations and the geometric continuity loss function is used for 3D geometric space calculations.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0079] It should also be noted that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0080] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0081] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0082] In this application, "determine B based on A" means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "determine B based on A and C", "determine B based on A, C and E", "determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "when A meets the first condition, use the first method to determine B"; for example, "when A meets the second condition, determine B", etc.; for example, "when A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "when A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A scene understanding method based on semantically known 3D Gaussian points, characterized in that: include: Get image information from multiple camera perspectives of the same scene; The image information is input into a pre-trained scene understanding model to obtain semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine semantic categories in a 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculations and the geometric continuity loss function is used for 3D geometric space calculations.

2. The scene understanding method based on semantically known 3D Gaussian points according to claim 1, characterized in that: The training steps of the scene understanding model include: Determine basic attributes of each 3D Gaussian point based on the image information; Acquire a 2D semantic map based on a pre-trained 2D segmentation model, and use the 2D semantic map as an initial semantic label of a 3D Gaussian point; Semantic attributes are added to each 3D Gaussian point, and the learning of the semantic attributes is supervised based on the initial semantic label, the semantic presentation loss function and the geometric continuity loss function.

3. The scene understanding method based on semantically known 3D Gaussian points according to claim 2, characterized in that: The determining of the basic attributes of each 3D Gaussian point based on the image information includes: Determining a camera pose for each image based on the image information; Based on the camera pose and the 3D Gaussian point reconstruction algorithm, basic properties of each 3D Gaussian point are determined.

4. The scene understanding method based on semantically known 3D Gaussian points according to claim 3, characterized in that: The basic attributes of each 3D Gaussian point include one or more of the following: Location; color; transparency; Covariance.

5. The scene understanding method based on semantically known 3D Gaussian points according to claim 2, characterized in that: The adding of semantic attributes to each 3D Gaussian point further includes: Freeze the base properties of the 3D Gaussian points.

6. The scene understanding method based on semantically known 3D Gaussian points according to claim 2, characterized in that: Supervising the learning of the semantic attribute based on the initial semantic label and the semantic presentation loss function includes: Determine the first camera perspective; Acquire a semantic segmentation probability map of the first camera perspective based on transparency mixed rendering; Perform L1 loss calculation based on the semantic segmentation probability map and the 2D semantic map of the first camera perspective to obtain a semantic presentation loss result; The learning of the semantic attributes is supervised based on the semantic presentation loss result.

7. The scene understanding method based on semantically known 3D Gaussian points according to claim 2, characterized in that: Supervising the learning of the semantic attribute based on the geometric continuity loss function includes: Determine the distance matrix based on the Euclidean distance between all 3D Gaussian points; determining an affinity matrix based on the distance matrix; Determine, based on the affinity matrix, a first semantic distribution result of each 3D Gaussian point before random walk and a second semantic distribution result after random walk; Based on KL divergence supervision, the first semantic distribution result and the second semantic distribution result are consistent.

8. A scene understanding device based on semantically known 3D Gaussian points, characterized in that: include: The first acquisition module is used to acquire image information of multiple camera perspectives of the same scene; The second acquisition module is used to input the image information into a pre-trained scene understanding model, and obtain the semantically known 3D Gaussian points output by the scene understanding model, wherein the semantically known 3D Gaussian points are used to determine the semantic categories in the 3D scene. The scene understanding model is trained based on a semantic presentation loss function and a geometric continuity loss function, wherein the semantic presentation loss function is used for pixel space calculation, and the geometric continuity loss function is used for 3D geometric space calculation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the scene understanding method based on semantically known 3D Gaussian points as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the scene understanding method based on semantically known 3D Gaussian points as claimed in any one of claims 1 to 7 is implemented.

Images