Three-dimensional panoramic scene understanding method and device based on multi-view consistency

Abstract

The embodiments of the present disclosure provide a method and device for understanding a three-dimensional panoramic scene based on multi-perspective consistency, which relates to the field of three-dimensional scene reconstruction. The method includes inputting multi-perspective images of the scene to be reconstructed into a pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each perspective image in the scene to be reconstructed, and performs perspective consistency processing on the segmented geometric features, semantic features and color features, respectively, to obtain perspective-consistent geometric features, semantic features and color features; based on the perspective-consistent geometric features, semantic features and color features, the scene to be reconstructed is reconstructed through the pre-trained three-dimensional panoramic radiation field model to obtain a panoramic image of the scene to be reconstructed. In this way, the problems of semantic ambiguity and geometric structure distortion under multiple perspectives are solved, and the accuracy and consistency of three-dimensional panoramic scene reconstruction and understanding are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional scene reconstruction, and in particular to a three-dimensional panoramic scene understanding method and device based on multi-view consistency. Background Art

[0002] The goal of 3D panoramic scene understanding technology is to reconstruct 3D panoramic scenes and identify object instances and semantic regions in the scene, so as to achieve accurate scene analysis and understanding in various visual technologies and robotic applications. With the development of technologies such as neural radiance field, 3D panoramic scene understanding through multi-view input has gradually become a research hotspot, but the existing solutions for 3D panoramic scene understanding using multi-view input have many shortcomings. For example, these methods usually rely on 2D panoramic segmentation masks for training. These 2D panoramic segmentation masks are inconsistent under different viewpoints, which makes it difficult for the system to maintain semantic information consistency and geometric information consistency under multiple viewpoints during the optimization process, thus affecting the quality of 3D panoramic scene reconstruction. Moreover, the semantic ambiguity and geometric structure distortion caused by this inconsistency are particularly obvious when rendering from a new viewpoint, which will affect the actual application effect of 3D panoramic scenes. Therefore, there is an urgent need for a 3D panoramic scene understanding method that can achieve multi-view semantic consistency and geometric consistency. Summary of the invention

[0003] The present invention provides a method, device, equipment and storage medium for understanding a three-dimensional panoramic scene based on multi-view consistency.

[0004] According to a first aspect of the present disclosure, a method for understanding a three-dimensional panoramic scene based on multi-view consistency is provided. The method comprises:

[0005] Input the multi-view images corresponding to the scene to be reconstructed into a pre-trained 3D panoramic radiation field model, so that the pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each view image in the scene to be reconstructed, and perform perspective consistency processing on the segmented geometric features, semantic features and color features, respectively, to obtain perspective-consistent geometric features, semantic features and color features;

[0006] According to the geometric features, semantic features and color features with consistent viewing angle, the scene to be reconstructed is reconstructed through a pre-trained 3D panoramic radiation field model to obtain a panoramic image corresponding to the scene to be reconstructed.

[0007] In some implementations of the first aspect, the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features, and color features of each perspective image in the scene to be reconstructed, including:

[0008] The pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the images of each view in the scene to be reconstructed and generate a segmentation mask set;

[0009] The segmentation mask set includes the geometric features, semantic labels and color labels of the pixels of each view of the scene to be reconstructed;

[0010] The geometric features of each pixel of the perspective image constitute the geometric features of the corresponding perspective image, the semantic labels of each pixel of the perspective image constitute the semantic features of the corresponding perspective image, and the color labels of each pixel of the perspective image constitute the color features of the corresponding perspective image.

[0011] In some implementations of the first aspect, performing perspective consistency processing on the segmented geometric features includes:

[0012] The scene to be reconstructed is rendered using a volume rendering method to obtain a rendering color label, a rendering semantic label, and a rendering depth value corresponding to each perspective image pixel of the scene to be reconstructed;

[0013] Calculate the pixel matching relationship between the images of each perspective of the scene to be reconstructed according to the dense matching algorithm;

[0014] The geometric features of the pixels of each viewing angle of the scene to be reconstructed are constrained by multi-view projection, and the projection depth values ​​corresponding to the pixels of each viewing angle of the scene to be reconstructed are calculated using the pixel matching relationship; the monocular depth values ​​corresponding to the pixels of each viewing angle of the scene to be reconstructed are calculated according to the monocular depth estimation algorithm;

[0015] According to the projection depth value and the monocular depth value of the corresponding pixel, the rendering depth value of the corresponding pixel is processed for perspective consistency.

[0016] In some possible implementations of the first aspect, multi-view projection constraints are applied to geometric features of pixels of images of each view of the scene to be reconstructed, including:

[0017] The geometric features of the pixels of each perspective image of the scene to be reconstructed are projected onto the images of other perspectives except its own perspective.

[0018] In some implementations of the first aspect, performing perspective consistency processing on the segmented semantic features and color features respectively includes:

[0019] According to the semantic label, color label and pixel matching relationship, the weighted average semantic label and weighted average color label of the image pixels of each perspective of the scene to be reconstructed are calculated respectively by using the probabilistic alignment method;

[0020] According to the weighted average color label, the rendering color label of the corresponding pixel is processed for perspective consistency;

[0021] According to the weighted average semantic label, the rendering semantic label of the corresponding pixel is processed for perspective consistency.

[0022] In some implementations of the first aspect, the three-dimensional panoramic radiation field model is established by the following method:

[0023] Use existing multi-view images corresponding to different scenes as training samples, and existing panoramic images corresponding to different scenes as annotations;

[0024] Inputting the training samples into the three-dimensional panoramic radiation field model to reconstruct each scene in the training samples, so that the three-dimensional panoramic radiation field model outputs panoramic images corresponding to different scenes;

[0025] Calculate the difference between the panoramic images corresponding to different scenes output by the three-dimensional panoramic radiation field model and the existing panoramic images corresponding to different scenes, adjust the model parameters using a gradient descent algorithm according to the difference value, and re-input the adjusted model parameters into the three-dimensional panoramic radiation field model;

[0026] The scene reconstruction, gap value calculation, model parameter adjustment and model parameter re-input are repeated until the loss function converges, and the three-dimensional panoramic radiation field model after the loss function converges is used as the trained three-dimensional panoramic radiation field model.

[0027] According to a second aspect of the present disclosure, a 3D panoramic scene understanding device based on multi-view consistency is provided. The device comprises:

[0028] A perspective consistency processing module is used to input the multi-perspective images corresponding to the scene to be reconstructed into a pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each perspective image in the scene to be reconstructed, and perform perspective consistency processing on the segmented geometric features, semantic features and color features, respectively, to obtain perspective-consistent geometric features, semantic features and color features;

[0029] The scene reconstruction module is used to reconstruct the scene to be reconstructed based on the geometric features, semantic features and color features consistent with the perspective through a pre-trained three-dimensional panoramic radiation field model, and obtain the panoramic image corresponding to the scene to be reconstructed.

[0030] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method described above.

[0031] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described above.

[0032] In the present disclosure, the multi-view images of the scene to be reconstructed are input into a pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each view image in the scene to be reconstructed, and performs perspective consistency processing on the segmented geometric features, semantic features and color features, respectively, to obtain perspective consistent geometric features, semantic features and color features; based on the perspective consistent geometric features, semantic features and color features, the scene to be reconstructed is reconstructed through the pre-trained three-dimensional panoramic radiation field model to obtain a panoramic image of the scene to be reconstructed. In this way, the problems of semantic ambiguity and geometric structure distortion under multiple perspectives are solved, and the accuracy and consistency of three-dimensional panoramic scene reconstruction and understanding are improved.

[0033] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0035] Figure 1 A flowchart of a three-dimensional panoramic scene understanding method based on multi-view consistency provided by an embodiment of the present disclosure is shown;

[0036] Figure 2 A structural diagram of a 3D panoramic scene understanding device based on multi-view consistency provided by an embodiment of the present disclosure is shown;

[0037] Figure 3 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0039] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0040] In response to the problems arising in the background technology, the embodiments of the present disclosure provide a method and device for understanding a three-dimensional panoramic scene based on multi-perspective consistency. The multi-perspective images of the scene to be reconstructed are input into a pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features, and color features of each perspective image in the scene to be reconstructed, and performs perspective consistency processing on the segmented geometric features, semantic features, and color features, respectively, to obtain perspective-consistent geometric features, semantic features, and color features; based on the perspective-consistent geometric features, semantic features, and color features, the scene to be reconstructed is reconstructed through the pre-trained three-dimensional panoramic radiation field model to obtain a panoramic image of the scene to be reconstructed. In this way, the problems of semantic ambiguity and geometric structure distortion under multiple perspectives are solved, and the accuracy and consistency of three-dimensional panoramic scene reconstruction and understanding are improved.

[0041] The following describes in detail the method and device for understanding a three-dimensional panoramic scene based on multi-view consistency provided by the embodiments of the present disclosure through specific embodiments in conjunction with the accompanying drawings.

[0042] Figure 1 A flowchart of a 3D panoramic scene understanding method based on multi-view consistency provided by an embodiment of the present disclosure is shown. The method 100 includes the following steps:

[0043] S110, inputting the multi-view images corresponding to the scene to be reconstructed into a pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each view image in the scene to be reconstructed, and performs perspective consistency processing on the segmented geometric features, semantic features and color features, respectively, to obtain perspective consistent geometric features, semantic features and color features.

[0044] In some embodiments, the pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the geometric features, semantic features, and color features of each perspective image in the scene to be reconstructed, including:

[0045] The pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the images of each view in the scene to be reconstructed and generate a segmentation mask set;

[0046] The segmentation mask set includes the geometric features, semantic labels and color labels of the pixels of each view of the scene to be reconstructed;

[0047] The geometric features of each pixel of the perspective image constitute the geometric features of the corresponding perspective image, the semantic labels of each pixel of the perspective image constitute the semantic features of the corresponding perspective image, and the color labels of each pixel of the perspective image constitute the color features of the corresponding perspective image.

[0048] In some embodiments, the segmented geometric features are subjected to perspective consistency processing, including:

[0049] The scene to be reconstructed is rendered using a volume rendering method to obtain a rendering color label, a rendering semantic label, and a rendering depth value corresponding to each perspective image pixel of the scene to be reconstructed;

[0050] Calculate the pixel matching relationship between the images of each perspective of the scene to be reconstructed according to the dense matching algorithm;

[0051] The geometric features of the pixels of each viewing angle of the scene to be reconstructed are constrained by multi-view projection, and the projection depth values ​​corresponding to the pixels of each viewing angle of the scene to be reconstructed are calculated using the pixel matching relationship; the monocular depth values ​​corresponding to the pixels of each viewing angle of the scene to be reconstructed are calculated according to the monocular depth estimation algorithm;

[0052] According to the projection depth value and the monocular depth value of the corresponding pixel, the rendering depth value of the corresponding pixel is processed for perspective consistency.

[0053] In some embodiments, rendering the scene to be reconstructed using a volume rendering method includes:

[0054] Using a volume rendering method, such as NeRF or 3D GS method, the sample points of each ray in the scene to be reconstructed are rendered.

[0055] In some embodiments, based on the rendered scene to be reconstructed, a convolutional neural network is used to obtain rendering color labels, rendering semantic labels, and rendering depth values ​​corresponding to image pixels of each perspective of the scene to be reconstructed.

[0056] In some embodiments, the pixel matching relationship between the images of each perspective of the scene to be reconstructed includes:

[0057] The positions of the matching pixels corresponding to the images of each perspective of the scene to be reconstructed and the corresponding confidence levels;

[0058] For example, there are N images with different perspectives in the scene to be reconstructed. Perspective images , The Pixels In other perspective imagesm The pixel matching relationship is , Pixel Matching pixels in other view images, is the matching pixel The corresponding confidence level, M Indicates that except The number of images from other perspectives other than the perspective images.

[0059] In some embodiments, the geometric features of the pixels of each viewing angle image of the scene to be reconstructed are subjected to multi-view projection constraints, including:

[0060] The geometric features of the pixels of each perspective image of the scene to be reconstructed are projected onto the images of other perspectives except its own perspective.

[0061] In some embodiments, according to the projected depth value and the monocular depth value of the corresponding pixel, the rendering depth value of the corresponding pixel is processed for perspective consistency, including:

[0062] According to the projection depth value of the image pixel of each perspective of the scene to be reconstructed and the converged depth correlation loss function, the rendering depth value of the corresponding pixel is adjusted to minimize the difference between the rendering depth value and the projection depth value of the corresponding pixel;

[0063] At the same time, according to the monocular depth value of the image pixel of each perspective of the scene to be reconstructed and the converged softened depth loss function, the rendering depth value of the corresponding pixel is adjusted to minimize the gap between the rendering depth value and the monocular depth value of the corresponding pixel;

[0064] So that the adjusted rendering depth value remains consistent at all viewing angles.

[0065] In some embodiments, the depth correlation loss function It is expressed as:

[0066] ;

[0067] in, For the m The projection depth value of the current pixel in the pixel matching relationship, For the m The rendering depth value of the current pixel in the pixel matching relationship.

[0068] In some embodiments, the softening depth loss function It is expressed as:

[0069] ;

[0070] in, is the rendered depth value of the pixel, D is the monocular depth value of the same pixel, Cov represents the covariance function, and Var represents the variance function.

[0071] In some embodiments, the segmented semantic features and color features are respectively subjected to perspective consistency processing, including:

[0072] According to the semantic label, color label and pixel matching relationship, the weighted average semantic label and weighted average color label of the image pixels of each perspective of the scene to be reconstructed are calculated respectively by using the probabilistic alignment method;

[0073] According to the weighted average color label, the rendering color label of the corresponding pixel is processed for perspective consistency;

[0074] According to the weighted average semantic label, the rendering semantic label of the corresponding pixel is processed for perspective consistency.

[0075] In some embodiments, calculating the weighted average semantic label of pixels of each viewing angle image of the scene to be reconstructed includes:

[0076] Semantic probability distribution generated from a 2D panoptic segmentation model and m The pixel matching probability distribution corresponding to the pixel matching relationship , combined with m The semantic label of the current pixel in the pixel matching relationship , use the weighted average semantic label calculation formula to calculate the weighted average semantic label ; Among them, the weighted average semantic label calculation formula is as follows:

[0077] ;

[0078] in, For the m The confidence of the matching pixels in the pixel matching relationship, For the m The confidence of the current pixel in the pixel matching relationship, It is m The matching pixels in the pixel matching relationship.

[0079] In some embodiments, using weighted average semantic labels can avoid the influence of erroneous semantic labels of individual view image pixels on the consistency of overall semantic information, thereby enhancing global semantic consistency.

[0080] In some embodiments, performing perspective consistency processing on the rendered color label of the corresponding pixel according to the weighted average color label includes:

[0081] According to the weighted average color label of the image pixels of each perspective of the scene to be reconstructed and the converged color correlation loss function, the rendered color label of the corresponding pixel is adjusted to minimize the gap between the rendered color label and the weighted average color label of the corresponding pixel, so that the adjusted rendered color label is consistent under each perspective.

[0082] In some embodiments, according to the weighted average semantic label, the rendering semantic label of the corresponding pixel is subjected to perspective consistency processing, including:

[0083] According to the weighted average semantic labels of the pixels of the images of each viewpoint of the scene to be reconstructed and the converged semantic correlation loss function, the rendering semantic labels of the corresponding pixels are adjusted to minimize the gap between the rendering semantic labels and the weighted average semantic labels of the corresponding pixels, so that the adjusted rendering semantic labels are consistent under each viewpoint;

[0084] Furthermore, the semantic relevance loss function It can be expressed as follows:

[0085] ;

[0086] Among them, CE is the cross entropy loss function, For the m The rendering semantic label of the current pixel in the pixel matching relationship, For the m The weighted average semantic label of the current pixel in the pixel matching relationship.

[0087] In some embodiments, the three-dimensional panoramic radiation field model is established by the following method:

[0088] Use existing multi-view images corresponding to different scenes as training samples, and existing panoramic images corresponding to different scenes as annotations;

[0089] Inputting the training samples into the three-dimensional panoramic radiation field model to reconstruct each scene in the training samples, so that the three-dimensional panoramic radiation field model outputs panoramic images corresponding to different scenes;

[0090] Calculate the difference between the panoramic images corresponding to different scenes output by the three-dimensional panoramic radiation field model and the existing panoramic images corresponding to different scenes, adjust the model parameters using a gradient descent algorithm according to the difference value, and re-input the adjusted model parameters into the three-dimensional panoramic radiation field model;

[0091] Repeat scene reconstruction, gap value calculation, model parameter adjustment, and model parameter re-input until the loss function converges, and use the three-dimensional panoramic radiation field model after the loss function converges as the trained three-dimensional panoramic radiation field model;

[0092] Specifically, after the training samples are input into the three-dimensional panoramic radiation field model, each view image in the training samples is segmented to generate a segmentation mask sample set;

[0093] Obtain the geometric features, semantic labels and color labels of image pixels at each viewpoint in different scenes from the segmentation mask sample set;

[0094] Use volume rendering method to render each scene in the training sample, and obtain rendering color labels, rendering semantic labels and rendering depth values ​​corresponding to each perspective image in different scenes;

[0095] Calculate the pixel matching relationship between images of different viewpoints of the same scene according to the dense matching algorithm;

[0096] Multi-view projection constraints are applied to the geometric features of pixels of each viewing angle image in different scenes respectively, and the projection depth values ​​of pixels of each viewing angle image in different scenes are calculated by using the pixel matching relationship between images of different viewing angles of the same scene; the monocular depth values ​​of pixels of each viewing angle image in different scenes are calculated according to the monocular depth estimation algorithm;

[0097] According to the semantic labels and color labels of pixels of each view image in different scenes and the pixel matching relationship between images of different views in the same scene, the weighted average semantic labels and weighted average color labels of pixels of each view image in different scenes are calculated by using the probability alignment method.

[0098] According to the depth correlation loss function and the projected depth values ​​of the image pixels of each viewing angle in different scenes, the rendering depth values ​​of the image pixels of each viewing angle in different scenes are optimized;

[0099] According to the softened depth loss function and the monocular depth values ​​of the image pixels at each perspective in different scenes, the rendering depth values ​​of the image pixels at each perspective in different scenes are optimized;

[0100] According to the color correlation loss function and the weighted average color labels of the image pixels of each perspective in different scenes, the rendering color labels of the image pixels of each perspective in different scenes are optimized;

[0101] Optimize the rendered semantic labels of image pixels at each viewpoint in different scenes based on the semantic relevance loss function and the weighted average semantic labels of image pixels at each viewpoint in different scenes;

[0102] Calculate the difference between the panoramic images corresponding to different scenes output by the three-dimensional panoramic radiation field model and the existing panoramic images corresponding to different scenes, adjust the model parameters using a gradient descent algorithm according to the difference value, and re-input the adjusted model parameters into the three-dimensional panoramic radiation field model;

[0103] Repeat the above steps until all the above loss functions converge.

[0104] S120, reconstructing the scene to be reconstructed by using a pre-trained three-dimensional panoramic radiation field model according to geometric features, semantic features and color features consistent with the viewing angle, and obtaining a panoramic image corresponding to the scene to be reconstructed.

[0105] According to the embodiments of the present disclosure, the multi-view images of the scene to be reconstructed are input into a pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each view image in the scene to be reconstructed, and performs perspective consistency processing on the segmented geometric features, semantic features and color features, respectively, to obtain geometric features, semantic features and color features with consistent perspectives; based on the geometric features, semantic features and color features with consistent perspectives, the scene to be reconstructed is reconstructed through the pre-trained three-dimensional panoramic radiation field model to obtain a panoramic image of the scene to be reconstructed. In this way, the problems of semantic ambiguity and geometric structure distortion under multiple perspectives are solved, and the accuracy and consistency of reconstruction and understanding of three-dimensional panoramic scenes are improved.

[0106] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0107] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0108] Figure 2 The structure diagram of a 3D panoramic scene understanding device based on multi-view consistency provided by an embodiment of the present disclosure is shown. The device 200 includes:

[0109] The perspective consistency processing module 210 is used to input the multi-perspective images corresponding to the scene to be reconstructed into a pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each perspective image in the scene to be reconstructed, and performs perspective consistency processing on the segmented geometric features, semantic features and color features to obtain perspective-consistent geometric features, semantic features and color features.

[0110] In some embodiments, module 210 is specifically configured to:

[0111] The pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the geometric features, semantic features, and color features of each view image in the reconstructed scene, including:

[0112] The pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the images of each view in the scene to be reconstructed and generate a segmentation mask set;

[0113] The segmentation mask set includes the geometric features, semantic labels and color labels of the pixels of each view of the scene to be reconstructed;

[0114] The geometric features of each pixel of the perspective image constitute the geometric features of the corresponding perspective image, the semantic labels of each pixel of the perspective image constitute the semantic features of the corresponding perspective image, and the color labels of each pixel of the perspective image constitute the color features of the corresponding perspective image.

[0115] In some embodiments, module 210 is further configured to:

[0116] The segmented geometric features are processed for perspective consistency, including:

[0117] The scene to be reconstructed is rendered using a volume rendering method to obtain a rendering color label, a rendering semantic label, and a rendering depth value corresponding to each perspective image pixel of the scene to be reconstructed;

[0118] Calculate the pixel matching relationship between the images of each perspective of the scene to be reconstructed according to the dense matching algorithm;

[0119] The geometric features of the pixels of each viewing angle of the scene to be reconstructed are constrained by multi-view projection, and the projection depth values ​​corresponding to the pixels of each viewing angle of the scene to be reconstructed are calculated using the pixel matching relationship; the monocular depth values ​​corresponding to the pixels of each viewing angle of the scene to be reconstructed are calculated according to the monocular depth estimation algorithm;

[0120] According to the projection depth value and the monocular depth value of the corresponding pixel, the rendering depth value of the corresponding pixel is processed for perspective consistency.

[0121] In some embodiments, module 210 is further configured to:

[0122] The geometric features of the pixels of each view image of the scene to be reconstructed are constrained by multi-view projection, including:

[0123] The geometric features of the pixels of each perspective image of the scene to be reconstructed are projected onto the images of other perspectives except its own perspective.

[0124] In some embodiments, module 210 is further configured to:

[0125] The segmented semantic features and color features are processed for perspective consistency, including:

[0126] According to the semantic label, color label and pixel matching relationship, the weighted average semantic label and weighted average color label of the image pixels of each perspective of the scene to be reconstructed are calculated respectively by using the probabilistic alignment method;

[0127] According to the weighted average color label, the rendering color label of the corresponding pixel is processed for perspective consistency;

[0128] According to the weighted average semantic label, the rendering semantic label of the corresponding pixel is processed for perspective consistency.

[0129] In some embodiments, module 210 is further configured to:

[0130] The three-dimensional panoramic radiation field model is established by the following method:

[0131] Use existing multi-view images corresponding to different scenes as training samples, and existing panoramic images corresponding to different scenes as annotations;

[0132] Inputting the training samples into the three-dimensional panoramic radiation field model to reconstruct each scene in the training samples, so that the three-dimensional panoramic radiation field model outputs panoramic images corresponding to different scenes;

[0133] Calculate the difference between the panoramic images corresponding to different scenes output by the three-dimensional panoramic radiation field model and the existing panoramic images corresponding to different scenes, adjust the model parameters using a gradient descent algorithm according to the difference value, and re-input the adjusted model parameters into the three-dimensional panoramic radiation field model;

[0134] The scene reconstruction, gap value calculation, model parameter adjustment and model parameter re-input are repeated until the loss function converges, and the three-dimensional panoramic radiation field model after the loss function converges is used as the trained three-dimensional panoramic radiation field model.

[0135] The scene reconstruction module 220 is used to reconstruct the scene to be reconstructed based on the geometric features, semantic features and color features consistent with the viewing angle through a pre-trained three-dimensional panoramic radiation field model to obtain a panoramic image corresponding to the scene to be reconstructed.

[0136] Understandably, Figure 2 Each module / unit in the illustrated module 200 has the function of implementing each step in the method 100 provided in the embodiment of the present disclosure and can achieve its corresponding technical effect, which will not be described in detail here for the sake of brevity.

[0137] Figure 3A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. The electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 300 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0138] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0139] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] It should be noted that the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in the embodiment of the present disclosure. For the sake of concise description, they are not repeated here.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with the systems and techniques described herein).

[0147] The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0148] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0149] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

Claims

1. A three-dimensional panoramic scene understanding method based on multi-view consistency, characterized in that: include: Input the multi-view images corresponding to the scene to be reconstructed into the pre-trained 3D panoramic radiation field model, so that the pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each view image in the scene to be reconstructed, and perform perspective consistency processing on the segmented geometric features, semantic features and color features, respectively, to obtain perspective consistent geometric features, semantic features and color features; wherein the perspective consistency processing on the segmented geometric features includes: The scene to be reconstructed is rendered using a volume rendering method to obtain rendering color labels, rendering semantic labels and rendering depth values ​​corresponding to pixels of each viewing angle of the scene to be reconstructed; the pixel matching relationship between the images of each viewing angle of the scene to be reconstructed is calculated according to a dense matching algorithm; the geometric features of the pixels of the images of each viewing angle of the scene to be reconstructed are subjected to multi-view projection constraints, and the projection depth values ​​corresponding to the pixels of the images of each viewing angle of the scene to be reconstructed are calculated using the pixel matching relationship; the monocular depth values ​​corresponding to the pixels of the images of each viewing angle of the scene to be reconstructed are calculated according to a monocular depth estimation algorithm; the rendering depth values ​​of the corresponding pixels are adjusted according to the projection depth values ​​of the pixels of the images of each viewing angle of the scene to be reconstructed and a converged depth correlation loss function, and the difference between the rendering depth value and the projection depth value of the corresponding pixel is minimized; the rendering depth values ​​of the corresponding pixels are adjusted according to the monocular depth values ​​of the pixels of the images of each viewing angle of the scene to be reconstructed and a converged softened depth loss function, and the difference between the rendering depth value and the monocular depth value of the corresponding pixel is minimized, so as to ensure that the adjusted rendering depth value remains consistent at each viewing angle; According to the geometric features, semantic features and color features with consistent viewing angle, the scene to be reconstructed is reconstructed through a pre-trained 3D panoramic radiation field model to obtain a panoramic image corresponding to the scene to be reconstructed.

2. The method according to claim 1, characterized in that The pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each perspective image in the scene to be reconstructed, including: The pre-trained 3D panoramic radiation field model uses a 2D panoramic segmentation algorithm to segment the images of each view in the scene to be reconstructed and generate a segmentation mask set; The segmentation mask set includes geometric features, semantic labels and color labels of pixels of each perspective image of the scene to be reconstructed; wherein, The geometric features of each pixel of the perspective image constitute the geometric features of the corresponding perspective image, the semantic labels of each pixel of the perspective image constitute the semantic features of the corresponding perspective image, and the color labels of each pixel of the perspective image constitute the color features of the corresponding perspective image.

3. The method according to claim 2, characterized in that The step of constraining the geometric features of the pixels of each viewing angle image of the scene to be reconstructed by multi-view projection includes: The geometric features of the pixels of each perspective image of the scene to be reconstructed are projected onto the images of other perspectives except its own perspective.

4. The method according to claim 2, characterized in that: The processing of the segmented semantic features and color features for perspective consistency includes: According to the semantic label, color label and pixel matching relationship, a probabilistic alignment method is used to respectively calculate the weighted average semantic label and weighted average color label of the pixels of each viewing angle image of the scene to be reconstructed; According to the weighted average color label, performing perspective consistency processing on the rendering color label of the corresponding pixel; According to the weighted average semantic label, the rendering semantic label of the corresponding pixel is processed for perspective consistency.

5. The method according to claim 1, characterized in that The three-dimensional panoramic radiation field model is established by the following method: Use existing multi-view images corresponding to different scenes as training samples, and existing panoramic images corresponding to different scenes as annotations; Inputting the training samples into a three-dimensional panoramic radiation field model to reconstruct each scene in the training samples so that the three-dimensional panoramic radiation field model outputs panoramic images corresponding to different scenes; Calculating the difference between the panoramic images corresponding to different scenes output by the three-dimensional panoramic radiation field model and the existing panoramic images corresponding to different scenes, adjusting the model parameters using a gradient descent algorithm according to the difference value, and re-inputting the adjusted model parameters into the three-dimensional panoramic radiation field model; The scene reconstruction, gap value calculation, model parameter adjustment and model parameter re-input are repeated until the loss function converges, and the three-dimensional panoramic radiation field model after the loss function converges is used as the trained three-dimensional panoramic radiation field model.

6. A 3D panoramic scene understanding device based on multi-view consistency, characterized in that: include: The perspective consistency processing module is used to input the multi-perspective images corresponding to the scene to be reconstructed into the pre-trained three-dimensional panoramic radiation field model, so that the pre-trained three-dimensional panoramic radiation field model uses a two-dimensional panoramic segmentation algorithm to segment the geometric features, semantic features and color features of each perspective image in the scene to be reconstructed, and respectively perform perspective consistency processing on the segmented geometric features, semantic features and color features to obtain perspective consistent geometric features, semantic features and color features; wherein the perspective consistency processing of the segmented geometric features includes: The scene to be reconstructed is rendered using a volume rendering method to obtain rendering color labels, rendering semantic labels and rendering depth values ​​corresponding to pixels of each viewing angle of the scene to be reconstructed; the pixel matching relationship between the images of each viewing angle of the scene to be reconstructed is calculated according to a dense matching algorithm; the geometric features of the pixels of the images of each viewing angle of the scene to be reconstructed are subjected to multi-view projection constraints, and the projection depth values ​​corresponding to the pixels of the images of each viewing angle of the scene to be reconstructed are calculated using the pixel matching relationship; the monocular depth values ​​corresponding to the pixels of the images of each viewing angle of the scene to be reconstructed are calculated according to a monocular depth estimation algorithm; the rendering depth values ​​of the corresponding pixels are adjusted according to the projection depth values ​​of the pixels of the images of each viewing angle of the scene to be reconstructed and a converged depth correlation loss function, and the difference between the rendering depth value and the projection depth value of the corresponding pixel is minimized; the rendering depth values ​​of the corresponding pixels are adjusted according to the monocular depth values ​​of the pixels of the images of each viewing angle of the scene to be reconstructed and a converged softened depth loss function, and the difference between the rendering depth value and the monocular depth value of the corresponding pixel is minimized, so as to ensure that the adjusted rendering depth value remains consistent at each viewing angle; The scene reconstruction module is used to reconstruct the scene to be reconstructed based on the geometric features, semantic features and color features consistent with the perspective through a pre-trained three-dimensional panoramic radiation field model, and obtain the panoramic image corresponding to the scene to be reconstructed.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

Images