Manhattan layout estimation using geometric and semantic information

Abstract

A plurality of two-dimensional (2D) images of a scene are received. Geometric information and semantic information of each of the plurality of 2D images are determined. The geometric information is indicative of lines and a reference direction of a respective 2D image detected. The semantic information includes classification information of pixels in the respective 2D image. Based on the geometric information and the semantic information of the respective 2D image, a layout estimate associated with the respective 2D image of the scene is determined. Based on a plurality of layout estimates determined to be associated with the plurality of 2D images of the scene, a combined layout estimate associated with the scene is determined. The Manhattan layout associated with the scene is generated based on the combined layout estimate. The Manhattan layout includes at least a three-dimensional (3D) shape of the scene, the three-dimensional (3D) shape including wall surfaces that are orthogonal to each other.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Patent Application No. 17 / 981,156, filed November 4, 2022, entitled “MANHATTAN LAYOUT ESTIMATION USING GEOMETRIC AND SEMANTIC INFORMATION”, which claims priority to U.S. Provisional Application No. 63 / 306,001, filed February 2, 2022, entitled “AMETHOD FOR MANHATTAN LAYOUT ESTIMATION FROM MULTIPLE PANORAMA IMAGES USING GEOMETRY AND SEMANTIC SEGMENTATION INFORMATION”. The entire contents of the earlier application are incorporated herein by reference. Technical Field

[0003] This disclosure describes embodiments that are typically associated with image encoding. Background Technology

[0004] The background description provided herein is for the purpose of presenting the general context of this disclosure. Within the scope described in this background section, neither the work of the currently named inventors nor any aspect of this description that does not qualify as prior art at the time of submission is expressly or implicitly acknowledged as prior art to this invention.

[0005] Various technologies have been developed to capture and represent the world, such as objects in a three-dimensional (3D) space or shape, and the environment within that world. 3D representations of the world enable more immersive forms of interaction and communication. In some examples, 3D shapes are widely used to represent this type of immersive content. Summary of the Invention

[0006] This disclosure provides methods and apparatus for image encoding (e.g., compression and decompression). In some examples, the apparatus for image encoding includes processing circuitry.

[0007] According to one aspect of this disclosure, a method for estimating a Manhattan layout associated with a scene is provided. In this method, a plurality of two-dimensional (2D) images of the scene may be received. Geometric and semantic information of each of the plurality of 2D images may be determined. The geometric information may indicate detected lines and reference directions in the corresponding 2D image. The semantic information may include classification information of pixels in the corresponding 2D image. Based on the geometric and semantic information of the corresponding 2D images, a layout estimate associated with the corresponding 2D images of the scene may be determined. Based on the plurality of determined layout estimates associated with the plurality of 2D images of the scene, a combined layout estimate associated with the scene may be determined. Based on the combined layout estimate, a Manhattan layout associated with the scene may be generated. The Manhattan layout may include at least a three-dimensional (3D) shape of the scene, the three-dimensional (3D) shape including walls orthogonal to each other.

[0008] To determine geometric and semantic information, first geometric information of a first 2D image among multiple 2D images can be extracted. This first geometric information may include at least one of the following: detected lines, a reference orientation of the first 2D image, a ratio between a first distance from the ceiling to the ground and a second distance from the camera to the ground, or the relative pose (e.g., angle or distance) between the first and second 2D images among the multiple 2D images. Pixels of the first 2D image can be labeled to generate first semantic information, wherein the first semantic information may indicate first structural information of the pixels in the first 2D image.

[0009] To determine a layout estimate associated with a corresponding 2D image of the scene, a first layout estimate can be determined from among multiple layout estimates associated with the scene based on first geometric information and first semantic information of the first 2D image. To determine the first layout estimate, it can be determined whether each detected line is a boundary line corresponding to a wall boundary in the scene. The boundary lines among the detected lines can be aligned with a reference orientation of the first 2D image. One of 2D polygon denoising or staircase removal can be used to generate a first polygon indicating the first layout estimate based on the aligned boundary lines.

[0010] To generate the first polygon, multiple incomplete boundary lines in the boundary line can be made complete based on one of the following: (i) estimating multiple incomplete boundary lines based on a combination of ceiling and floor boundary lines in the boundary line, and (ii) connecting pairs of incomplete boundary lines in the multiple incomplete boundary lines. The pairs of incomplete boundary lines are connected by one of the following methods: (i) adding a perpendicular line to the pairs of incomplete boundary lines in response to the pairs being parallel, and (ii) extending at least one of the pairs of incomplete boundary lines such that the intersection of the pairs of incomplete boundary lines lies on the extended pair of incomplete boundary lines.

[0011] To determine a combined layout estimate associated with a scene, a base polygon can be determined by combining a plurality of polygons via a polygon union algorithm. Each polygon of the plurality of polygons can correspond to a respective layout estimate of a plurality of determined layout estimates. A reduced polygon can be determined based on the base polygon. The reduced polygon includes updated edges resulting from updating edges of the base polygon. Using one of 2D polygon denoising or step removal, a final polygon can be determined based on the reduced polygon. The final polygon can correspond to the combined layout estimate associated with the scene.

[0012] To determine a reduced polygon, for an edge of a base polygon, a plurality of candidate edges can be determined based on a plurality of polygons. Each candidate edge of the plurality of candidate edges can correspond to a respective edge of the base polygon. In response to the one or more candidate edges being closer to an original viewpoint location of the plurality of images relative to the respective one or more edges of the base polygon, the reduced polygon updated edges can be generated by replacing the one or more edges of the base polygon with the respective one or more candidate edges.

[0013] In some embodiments, each candidate edge of the plurality of candidate edges can be parallel to a respective edge of the base polygon. A projection overlap between the candidate edge and the respective edge of the base polygon can be greater than a threshold value.

[0014] To determine a combined layout estimate associated with a scene, a set of edges including edges of a final polygon can be determined. Based on the set of edges, a plurality of edge combinations can be generated. A plurality of interior edges of the final polygon can be generated. The plurality of interior edges can be indicated by a plurality of average edges of one or more edge combinations of the set of edges. Each edge combination of the plurality of edge combinations can include a respective number of edges that is greater than a target value. Each average edge of the plurality of average edges can be obtained by averaging edges of a respective one of the one or more edge combinations.

[0015] In some embodiments, the plurality of edge combinations can include a first edge combination. The first edge combination can further include a first edge and a second edge. The first edge and the second edge can be parallel to each other. A distance between the first edge and the second edge can be less than a first threshold value. A projection overlap area between the first edge and the second edge can be greater than a second threshold value.

[0016] To generate a scene-associated Manhattan layout, it can be generated based on one of the following: a triangular mesh triangulated from the combined layout estimate, a quadrilateral mesh quadrangularized from the combined layout estimate, sampled points obtained by sampling one of the triangular mesh and the quadrilateral mesh, or a discrete mesh generated by voxelization based on one of the triangular mesh and the quadrilateral mesh.

[0017] In some embodiments, a Manhattan layout associated with the scene is generated based on a triangular mesh triangulated according to a composite layout estimate. Therefore, to generate the Manhattan layout associated with the scene, the ceiling and floor faces in the scene can be generated by triangulating the composite layout estimate. The walls in the scene can be generated by triangulating the rectangles enclosed by the ceiling and floor boundaries in the scene. The texture of the Manhattan layout associated with the scene can be generated based on ray casting processing.

[0018] According to another aspect of this disclosure, an apparatus is provided. The apparatus includes processing circuitry. The processing circuitry can be configured to perform any method for estimating a Manhattan layout associated with a scene.

[0019] This disclosure also provides a non-transitory computer-readable medium for storing instructions that, when executed by a computer, cause the computer to perform any method for estimating a Manhattan layout associated with a scene. Attached Figure Description

[0020] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0021] Figure 1 These are block diagrams of communication systems in some examples.

[0022] Figure 2 These are block diagrams of some examples of streaming systems.

[0023] Figure 3 These are block diagrams of encoders used to encode point cloud frames in some examples.

[0024] Figure 4 These are block diagrams of decoders used in some examples to decode compressed bitstreams corresponding to point cloud frames.

[0025] Figure 5 These are block diagrams of video decoders in some examples.

[0026] Figure 6 These are block diagrams of video encoders in some examples.

[0027] Figure 7is a block diagram of an encoder for encoding a point cloud frame in some examples.

[0028] Figure 8 is a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame in some examples.

[0029] Figure 9A is an example panoramic image in some examples.

[0030] Figure 9B is an example panoramic image labeled with geometry information in some examples.

[0031] Figure 9C is an example panoramic image labeled with semantic information in some examples.

[0032] Figure 10A is an example estimated room layout in some examples.

[0033] Figure 10B is an example polygon representing an estimated room layout in some examples.

[0034] Figure 10C is an example three-dimensional mesh generated based on polygons in some examples.

[0035] Figure 11 is an overview of a system for estimating a Manhattan layout in some examples.

[0036] Figure 12 is a block diagram of a framework for image processing in some examples.

[0037] Figure 13 is a flowchart outlining a process of estimating a Manhattan layout associated with a scene in some examples.

[0038] Figure 14 is a schematic diagram of a computer system in some examples. DETAILED DESCRIPTION

[0039] Aspects of the disclosure provide technology in the field of three-dimensional (3D) media processing.

[0040] Technological developments in 3D media processing, such as advances in three-dimensional (3D) capture, 3D modeling, and 3D rendering, have facilitated the ubiquitous presence of 3D media content on multiple platforms and devices. For example, a first step of a baby can be taken on one continent, and media technology can allow a grandparent on another continent to watch (and perhaps interact with) and enjoy an immersive experience of being with the baby. According to one aspect of the disclosure, to improve the immersive experience, 3D models have become increasingly complex, and the creation and consumption of 3D models have consumed a large amount of data resources, such as data storage, data transmission resources.

[0041] In some implementations, point clouds and meshes can be used as 3D models to represent immersive content.

[0042] A point cloud can generally refer to a set of points in 3D space, each point having associated attributes such as color, material attributes, texture information, intensity attributes, reflectance attributes, motion-related attributes, modal attributes, and various other attributes. Point clouds can be used to reconstruct an object or a scene as a combination of these points.

[0043] A mesh (also referred to as a mesh model) of an object can include polygons that describe the surface of the object. Each polygon can be defined by the vertices of the polygon in 3D space and information about how to connect the vertices into the polygon. The information about how to connect the vertices is referred to as connectivity information. In some examples, a mesh can also include attributes associated with the vertices, such as color, normal, and the like.

[0044] In some implementations, some encoding tools for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be remeshed to generate a new mesh, and connectivity information for the vertices of the new mesh can be derived (or predefined) for the new mesh. The vertices of the new mesh and the attributes associated with the vertices of the new mesh can be considered as points in a point cloud, and can be compressed using a PCC codec.

[0045] A point cloud can be used to reconstruct an object or a scene as a combination of these points. The points can be captured using multiple cameras, depth sensors, or lidar in various settings, and can consist of thousands, or even billions, of points in order to realistically represent the reconstructed scene or object. A patch can generally refer to a contiguous subset of a surface described by a point cloud. In one example, a patch includes points whose surface normal vectors deviate from each other by less than a threshold amount.

[0046] PCC can be performed according to various schemes, such as a geometry-based scheme referred to as G-PCC, a video coding-based scheme referred to as V-PCC, and the like. According to some aspects of the present disclosure, G-PCC directly encodes 3D geometry and is a purely geometry-based approach with little sharing with video coding, while V-PCC is primarily based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a two-dimensional (2D) mesh (image). A V-PCC scheme can leverage a general-purpose video codec for point cloud compression. A PCC codec (encoder / decoder) in the present disclosure can be a G-PCC codec or a V-PCC codec.

[0047] According to one aspect of the disclosure, a V-PCC scheme can use a video codec to compress the geometry, occupancy, and texture of a point cloud into three separate video sequences. Additional metadata needed to interpret the three video sequences is compressed separately. A small portion of the overall bitstream is metadata, which in one example, can be efficiently encoded / decoded using a software implementation. The majority of the information is handled by the video codec.

[0048] Figure 1 A block diagram of a communication system (100) in some examples is shown. The communication system (100) includes multiple terminal devices that can communicate with each other, e.g., via a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) that are interconnected via the network (150). In Figure 1 In examples, the pair of terminal devices (110) and (120) can perform unidirectional transmission of point cloud data. For example, the terminal device (110) can compress a point cloud (e.g., representing points of a structure) captured by a sensor (105) connected to the terminal device (110). The compressed point cloud can be transmitted to the other terminal device (120) in the form of a bitstream via the network (150), for example. The terminal device (120) can receive the compressed point cloud from the network (150), decompress the bitstream to reconstruct the point cloud, and display the reconstructed point cloud appropriately. Unidirectional data transmission is more common in applications such as media services.

[0049] In examples, Figure 1 The terminal devices (110) and (120) can be shown as servers and personal computers, but the principles of the disclosure can not be limited thereto. Embodiments of the disclosure can be applied to laptop computers, tablet computers, smartphones, gaming terminals, media players, and / or dedicated three-dimensional (3D) devices. The network (150) represents any number of networks that transmit compressed point clouds between the terminal devices (110) and (120). For example, the network (150) can include wired (wired) and / or wireless communication networks. The network (150) can exchange data in circuit- switched and / or packet-switched channels. Representative networks include telecommunication networks, local area networks, wide area networks, the Internet, and the like.

[0050] Figure 2 A block diagram of a streaming media system (200) in some examples is shown. The streaming media system (200) is an application of use of point clouds. The disclosed subject matter can be equally applicable to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, and the like.

[0051] A streaming system (200) can include a capture subsystem (213). The capture subsystem (213) can include a point cloud source (201), such as a Light Detection And Ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component in the form of software that generates uncompressed point clouds, and other components that generate, for example, uncompressed point clouds (202). In one example, the point cloud (202) includes points captured by a 3D camera. The point cloud (202) is shown as a thick line to emphasize the high data volume when compared to the compressed point cloud (204), a bitstream of a compressed point cloud. The compressed point cloud (204) can be generated by an electronic device (220) that includes an encoder (203) coupled to the point cloud source (201). The encoder (203) can include hardware, software, or a combination of both to implement or realize aspects of the disclosed subject matter as described in greater detail below. The compressed point cloud (204), or bitstream of a compressed point cloud (204), can be stored on a streaming server (205) for future use, shown as a thin line to emphasize the lower data volume when compared to the stream of point clouds (202). One or more streaming client subsystems, such as client subsystem (206) and client subsystem (208) in Figure 2

[0052] It is noted that the electronic device (220) and the electronic device (230) can include other components (not shown). For example, the electronic device (220) can include a decoder (not shown), and the electronic device (230) can also include an encoder (not shown).

[0053] In some streaming systems, the compressed point clouds (204), (207), and (209), e.g., bitstreams of compressed point clouds, can be compressed according to certain standards. In some examples, video coding standards are used for compression of point clouds. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and the like.

[0054] Figure 3 ​A block diagram of a V-PCC encoder (300) for encoding point cloud frames according to some embodiments is shown. In some embodiments, the V-PCC encoder (300) can be used in a communication system (100) and a streaming media system (200). For example, an encoder (203) can be configured and operated in a similar manner to the V-PCC encoder (300).

[0055] The V-PCC encoder (300) receives point cloud frames as uncompressed input and generates a bitstream corresponding to the compressed point cloud frames. In some embodiments, the V-PCC encoder (300) can receive point cloud frames from a point cloud source such as a point cloud source (201).

[0056] exist Figure 3 In the example, the V-PCC encoder (300) includes a slice generation module (306), a slice encapsulation module (308), a geometric image generation module (310), a texture image generation module (312), a slice information module (304), an occupancy map module (314), a smoothing module (336), image filling modules (316) and (318), a group expansion module (320), video compression modules (322), (323) and (332), an auxiliary slice information compression module (338), an entropy compression module (334), and a multiplexer (324).

[0057] According to one aspect of this disclosure, a V-PCC encoder (300) converts a 3D point cloud frame into an image-based identifier and some metadata (e.g., occupancy map and piece information). The compressed point cloud is then converted back to a decompressed point cloud using the metadata. In some examples, the V-PCC encoder (300) may convert the 3D point cloud frame into a geometry image, a texture image, and an occupancy map, and then encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. Typically, a geometry image is a 2D image where pixels are filled with geometric values ​​associated with points projected onto the pixels, and pixels filled with geometric values ​​may be referred to as geometry samples. A texture image is a 2D image where pixels are filled with texture values ​​associated with points projected onto the pixels, and pixels filled with texture values ​​may be referred to as texture samples. An occupancy map is a 2D image where pixels are filled with values ​​indicating whether a piece is occupied or not.

[0058] The patch generation module (306) segments the point cloud into a set of patches (e.g., a patch is defined as a continuous subset of surfaces described by the point cloud), which may or may not overlap, such that each patch can be described by a depth domain relative to a plane in 2D space. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while also minimizing reconstruction errors.

[0059] In some examples, the tile information module (304) can collect tile information indicating the size and shape of the tile. In some examples, the tile information can be packed into an image frame, which is then encoded by the auxiliary tile information compression module (338) to generate compressed auxiliary tile information.

[0060] In some examples, the tile packing module (308) is configured to map the extracted tiles onto a two-dimensional (2D) grid while minimizing unused space and ensuring that every MxM (e.g., 16x16) block of the grid is associated with a unique tile. Efficient tile packing can directly impact compression efficiency by minimizing unused space or ensuring temporal consistency.

[0061] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry shape of the point cloud at a given tile location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given tile location. The geometry image generation module (310) and the texture image generation module (312) utilize the 3D to 2D mapping computed in the packing process to store the geometry shape and texture of the point cloud as images. To better handle the case where multiple points are projected to the same sample, each tile is projected onto two images (referred to as layers). In one example, the geometry image is represented by a WxH monochrome frame in YUV420-8bit format. To generate the texture image, the texture generation process utilizes the reconstructed / smoothed geometry shape to compute the color to be associated with the re-sampled points.

[0062] The occupancy map module (314) can generate an occupancy map describing the packing information at each cell. For example, the occupancy image includes a binary map indicating for each cell of the grid whether the cell belongs to empty space or to the point cloud. In one example, the occupancy map uses binary information to describe whether each pixel is packed. In another example, the occupancy map uses binary information to describe for each block of pixels whether the block of pixels is packed.

[0063] The occupancy map generated by the occupancy map module (314) can be compressed using lossless encoding or lossy encoding. When using lossless encoding, the entropy compression module (334) is used to compress the occupancy map. When using lossy encoding, the video compression module (332) is used to compress the occupancy map.

[0064] It should be noted that the tile packing module (308) can leave some space between the 2D tiles packed in the image frame. The image padding modules (316) and (318) can pad the space (referred to as padding) in order to generate image frames suitable for 2D video and image codecs. Image padding is also referred to as background padding, which can pad the unused space with redundant information. In some examples, good background padding minimally increases the bit rate without introducing noticeable coding distortion around the tile boundaries.

[0065] The video compression modules (322), (323), and (332) can encode the 2D images (e.g., the padded geometry images, the padded texture images, and the occupancy map) based on a suitable video coding standard (e.g., HEVC, VVC, etc.). In one example, the video compression modules (322), (323), and (332) are separate components that operate independently. It is noted that in another example, the video compression modules (322), (323), and (332) can be implemented as one component.

[0066] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry image. The smoothed image can be provided to the texture image generation module (312). The texture image generation module (312) can then adjust the generation of the texture image based on the reconstructed geometry image. For example, when the tile shape (e.g., the geometry shape) is slightly distorted during encoding and decoding, the distortion can be considered when generating the texture image to correct the distortion in the tile shape.

[0067] In some embodiments, the group expansion module (320) is configured to pad the pixels around the object boundaries with redundant low-frequency content in order to improve the coding gain as well as the visual quality of the reconstructed point cloud.

[0068] The multiplexer (324) can multiplex the compressed geometry images, the compressed texture images, the compressed occupancy map, the compressed auxiliary tile information into a compressed bitstream.

[0069] Figure 4 A block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame is shown in some examples. In some examples, the V-PCC decoder (400) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate in a similar manner as the V-PCC decoder (400). The V-PCC decoder (400) receives a compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0070] In Figure 4In the example of FIG. 4, the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), occupancy map decompression module (438), auxiliary patch information decompression module (442), geometry reconstruction module (444), smoothing module (446), texture reconstruction module (448), and color smoothing module (452).

[0071] The demultiplexer (432) can receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.

[0072] The video decompression modules (434) and (436) can decode the compressed images according to a suitable standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (434) decodes the compressed texture images and outputs decompressed texture images. The video decompression module (436) decodes the compressed geometry images and outputs decompressed geometry images.

[0073] The occupancy map decompression module (438) can decode the compressed occupancy maps according to a suitable standard (e.g., HEVC, VVC, etc.) and output decompressed occupancy maps.

[0074] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to a suitable standard (e.g., HEVC, VVC, etc.) and output decompressed auxiliary patch information.

[0075] The geometry reconstruction module (444) can receive the decompressed geometry images and generate reconstructed point cloud geometry based on the decompressed occupancy maps and the decompressed auxiliary patch information.

[0076] The smoothing module (446) can smooth inconsistencies at patch boundaries. The smoothing process aims to mitigate potential discontinuities that can appear at patch boundaries due to compression artifacts. In some embodiments, a smoothing filter can be applied to pixels located on patch boundaries to mitigate distortions that can be caused by compression / decompression.

[0077] The texture reconstruction module (448) can determine texture information for points in the point cloud based on the decompressed texture images and the smoothed geometry.

[0078] The color smoothing module (452) can smooth inconsistencies in shading. Non-adjacent patches in 3D space are often packed to be adjacent to each other in 2D video. In some examples, pixel values from non-adjacent patches can be confused by block-based video codecs. The goal of color smoothing is to reduce visible artifacts that appear at patch boundaries.

[0079] Figure 5 A block diagram of a video decoder (510) in some examples is shown. The video decoder (510) can be used in the V-PCC decoder (400). For example, the video decompression modules (434) and (436), occupancy map decompression module (438) can be similarly configured as the video decoder (510).

[0080] The video decoder (510) can include a parser (520) to reconstruct symbols (521) from a compressed image, such as an encoded video sequence. Categories of those symbols include information used to manage operation of the video decoder (510). The parser (520) can parse / entropy-decode the received encoded video sequence. The encoding of the encoded video sequence can be in accordance with a video coding technology or standard, and can follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so forth. The parser (520) can extract from the encoded video sequence, a set of sub-group parameters for use in video decoding of at least one pixel sub-group, based on at least one parameter corresponding to a sub-group. Sub-groups can include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs) and so forth. The parser (520) can also extract information, such as transform coefficients, quantizer parameter values, motion vectors, and so forth, from the encoded video sequence.

[0081] The parser (520) can perform entropy-decoding / parsing operations on the video sequence received from the buffer memory, creating the symbols (521).

[0082] The reconstruction of the symbols (521) can involve a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks), and other factors. Which units are involved, and how, can be controlled by sub-group control information that the parser (520) extracts from the coded video sequence. To conserve space, this sub-group control information flow between the parser (520) and the following units is not depicted.

[0083] Beyond the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In practical implementations, many of these units interact closely with each other, and may, at least partly, be integrated into each other's entity. However, the conceptual subdivision into the following functional units is appropriate for the purpose of describing the disclosed subject matter.

[0084] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantized transform coefficients as symbols (521) and control information, including which transform to use, block size, quantization factor, quantization scaling matrices, etc., from the parser (520). The scaler / inverse transform unit (551) can output a block comprising sample values, which can be input into the aggregator (555).

[0085] In some cases, the output samples of the scaler / inverse transform unit (551) can belong to an intra coded block, i.e., a block that is not using predictive information from previously reconstructed pictures, but can use predictive information from previously reconstructed parts of the current picture. Such predictive information can be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) employs surrounding already reconstructed information extracted from the current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (555) adds, on a per sample basis, the predictive information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0086] In other cases, the output samples of the scaler / inverse transform unit (551) can belong to an inter coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) can access the reference picture memory (557) to fetch samples for prediction. After motion compensation of the fetched samples according to the symbols (521) pertaining to the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551), referred to as residual samples or residual signal in this case, generating the output sample information. The motion compensated prediction unit (553) obtains the addresses of the prediction samples within the reference picture memory (557) can be controlled by motion vectors, and the motion vectors are available to the motion compensated prediction unit (553) in the form of symbols (521), which can have, for example, X, Y, and reference picture components. Motion compensation can also include interpolation of sample values fetched from the reference picture memory (557) when sub-sample precise motion vectors are used, motion vector prediction mechanisms, etc.

[0087] The output samples of the aggregator (555) can be subject to various loop filtering techniques in the loop filter unit (556). Video compression technologies can include in-loop filter technologies that are controlled by parameters included in the coded video sequence (also referred to as coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), however, the video compression technologies can also be responsive to meta-information obtained during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, as well as responsive to previously reconstructed and loop-filtered sample values.

[0088] The output of the loop filter unit (556) can be a stream of samples that can be output to the rendering device and stored in the reference picture memory (557) for use in the inter prediction of future pictures.

[0089] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified (by, for example, the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before commencing the reconstruction of a following coded picture.

[0090] The video decoder (510) can perform decoding operations according to a predetermined video compression technology in a standard, such as ITU-T H.265. In the sense that the coded video sequence adheres to the syntax of the video compression technology or standard, and to level restrictions documented in the video compression technology or standard, the coded video sequence can conform to the syntax specified by the video compression technology or standard in use. In particular, a profile can select certain tools available in the video compression technology or standard, as the tools only a subset of which can be used under the profile. It can also be a requirement that the complexity of the coded video sequence is within bounds as defined by the level of the video compression technology or standard. In some cases, the restrictions by the level can be further restricted by Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0091] Figure 6is a block diagram of a video encoder (603) according to embodiments of the disclosure. The video encoder (603) can be used in the V-PCC encoder (300) for compressing point clouds. In one example, the configurations of the video compression modules (322) and (323) and the video compression module (332) are similar to the encoder (603).

[0092] The video encoder (603) can receive pictures, e.g., padded geometry pictures, padded texture pictures, etc., and generate compressed pictures.

[0093] According to an embodiment, the video encoder (603) can code and compress the pictures of a source video sequence (pictures) into a coded video sequence (compressed pictures) in real time or under any other time constraints as required by the application. Enforcing appropriate coding speed is one function of a controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For clarity, not all coupling has been shown. Parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value of rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, and so on. The controller (650) can be configured to have other suitable functions that pertain to the video encoder (603) optimized for a certain system design.

[0094] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As a simple description, in an example, the coding loop can include a source coder (630) (e.g., to create symbols, e.g., a symbol stream, based on input pictures to be coded, and reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner as a (remote) decoder would create (as any compression between symbols and coded video bitstream is lossless in video compression technologies considered by the disclosed subject matter). The reconstructed sample stream (sample data) is input to the reference picture buffer (634). As the decoding of symbols results in bit-exact results independent of decoder location (local or remote), the content in the reference picture buffer (634) is also bit exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder “sees” as reference picture samples have exactly the same sample values as it would “see” when the decoder would use the predictions during decoding. This principle of reference picture synchronicity is also used in some related arts (as well as the drift that can occur if, for example, channel errors prevent synchronicity from being maintained).

[0095] The operation of the "local" decoder (633) can be the same as, for example, already described above in connection with Figure 5 The "remote" decoder of the video decoder (510) described in detail is the same. However, reference will still be briefly made to Figure 5 When symbols are available and the entropy encoder (645) and parser (520) are able to losslessly encode / decode the symbols into the coded video sequence, the entropy decoding portion of the video decoder (510), including the parser (520), can not be fully implemented in the local decoder (633).

[0096] During operation, in some examples, the source coder (630) can perform motion compensated predictive coding of the input picture data. This motion

[0097] The local video decoder (633) can decode coded video data of pictures that can be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) can advantageously be a lossy process. When the coded video data is decoded at the video decoder (not shown), the reconstructed video sequence can typically be a replica of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that can be performed by the video decoder on reference pictures and can cause reconstructed reference pictures to be stored in the reference picture cache (634). In this manner, the video coder (603) can store copies of reconstructed reference pictures locally that have common content as the reconstructed reference pictures that will be obtained by a far-end video decoder (absent transmission errors). Figure 6

[0098] The predictor (635) can perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, and so on, that can serve as an appropriate prediction reference for the new picture. The predictor (635) can operate on a sample block-by-sample block basis, to find appropriate prediction references. In some cases, as determined by search results obtained by the predictor (635), an input picture can have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0099] The controller (650) can manage coding operations of the source coder (630), including, for example, setting of parameters and subgroup parameters used for encoding the video data.​

[0100] The output of all the above functional units can be entropy encoded in an entropy encoder (645). The entropy encoder (645) losslessly compresses the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, and so on, thereby converting them into an encoded video sequence.

[0101] The controller (650) can manage operation of the video encoder (603). During coding, the controller (650) can assign to each coded picture a certain coded picture type, which can affect the coding techniques that can be applied to the corresponding picture. For example, pictures often can be assigned as one of the following picture types:

[0102] An Intra Picture (I picture) can be one that can be coded and decoded without using any other picture in the sequence as a reference. Some video codecs allow different types of Intra pictures, including, for example Independent Decoder Refresh (“IDR”) pictures. A person of skill in the art is aware of the variants of I pictures and their respective applications and features.

[0103] A predictive picture (P picture) can be one that can be coded and decoded using either intra prediction or inter prediction that uses at most one motion vector and reference index to predict sample values of each block.

[0104] A bi-directionally predictive picture (B picture) can be one that can be coded and decoded using either intra prediction or inter prediction that uses at most two motion vectors and reference indices to predict sample values of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0105] A source picture often can be spatially subdivided into multiple blocks of samples (e.g., 4x4, 8x8, 4x8, or 16x16 samples) and coded block-by-block. These blocks can be predictively coded with reference to other (coded) blocks, determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I picture can be non-predictively coded, or predictively coded with reference to coded blocks of the same picture (spatial prediction or intra prediction). Blocks of a P picture can be predictively coded with reference to one previously coded reference picture, either by spatial prediction or by temporal prediction. Blocks of a B picture can be predictively coded with reference to one or two previously coded reference pictures, either by spatial prediction or by temporal prediction.

[0106] The video encoder (603) can perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T H.265. In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data, therefore, can conform to a syntax specified by the video coding technology or standard being used.

[0107] Video can be in the form of a series of source pictures (images) in temporal sequence. Intra-picture prediction (often simply referred to as intra prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation between pictures (either temporal or other). In an example, a particular picture being encoded / decoded (called the current picture) is partitioned into blocks. A block in the current picture can be coded with a vector called a motion vector when the block is similar to a reference block in a reference picture that has been coded previously and is still buffered. The motion vector points to the reference block in the reference picture, and can have a third dimension identifying the reference picture in case multiple reference pictures are used.

[0108] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture that are both before the current picture in decoding order (but can be past and future, respectively, in display order) in the video. A block in the current picture can be coded with a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block can be predicted from a combination of the first reference block and the second reference block.

[0109] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0110] According to some embodiments of the disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in the unit of blocks. For example, according to the HEVC standard, a picture in a video sequence is partitioned into coding tree units (CTU) for compression, the CTUs in a picture have the same size, such as 64 x 64 pixels, 32 x 32 pixels, or 16 x 16 pixels. In general, a CTU includes three coding tree blocks (CTB), which is one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree- split into one or multiple coding units (CU). For example, a CTU of 64 x 64 pixels can be split into one CU of 64 x 64 pixels, or 4 CUs of 32 x 32 pixels, or 16 CUs of 16 x 16 pixels. In an example, each CU is analyzed to determine a prediction type for the CU, such as an inter prediction type or an intra prediction type. The CU is split into one or more prediction units (PU) depending on the temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB), and two chroma PBs. In an embodiment, a prediction operation in encoding (encoding / decoding) is performed in the unit of a prediction block. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) of pixels, such as 8 x 8 pixels, 16 x 16 pixels, 8 x 16 pixels, 16 x 8 pixels, and / or the like.

[0111] Figure 7 A block diagram of a G-PCC encoder (700) in some examples is shown. The G-PCC encoder (700) can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder (700) can include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic encoding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) to store reconstructed attribute values.

[0112] As shown, an input point cloud (701) can be received at a G-PCC encoder (700). The positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. A duplicate point removal module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate points. An octree encoding module (730) is configured to receive the filtered positions from the duplicate point removal module (712) and perform an octree-based encoding process to generate a sequence of occupancy codes that describe a 3D grid of voxels. The occupancy codes are provided to an arithmetic encoding module (770).

[0113] An attribute transfer module (720) is configured to receive attributes of the input point cloud and, when multiple attribute values are associated with individual voxels, perform an attribute transfer process to determine an attribute value for each voxel. The attribute transfer process can be performed on reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to an attribute prediction module (750). A LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).

[0114] The attribute prediction module (750) processes the points according to an LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for a current point based on reconstructed attributes of a set of neighboring points of the current point stored in a memory (790). A prediction residual can then be obtained based on the original attribute value received from the attribute transfer module (720) and the locally generated attribute prediction. When a candidate index is used in the individual attribute prediction processes, the index corresponding to the selected prediction candidate can be provided to the arithmetic encoding module (770).

[0115] A residual quantization module (760) is configured to receive the prediction residual from the attribute prediction module (750) and perform quantization to generate a quantized residual. The quantized residual is provided to the arithmetic encoding module (770).

[0116] An inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing an inverse operation of the quantization operation performed at the residual quantization module (760). An addition module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and the corresponding attribute prediction from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute prediction, a reconstructed attribute value is generated and stored to the memory (790).

[0117] The arithmetic coding module (770) is configured to receive occupancy codes, candidate indices (if used), quantization residuals (if generated), and other information, and perform entropy coding to further compress the received values ​​or information. Thus, a compressed bitstream (702) carrying compression information can be generated. The bitstream (702) can be sent or otherwise provided to a decoder that decodes the compressed bitstream, or it can be stored in a storage device.

[0118] Figure 8 A block diagram of a G-PCC decoder (800) according to one embodiment is shown. The G-PCC decoder (800) can be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream to generate decoded point cloud data. In one embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0119] As shown in the figure, a compressed bitstream (801) can be received at the arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain the quantization residual (if generated) and occupancy code of the point cloud. The octree decoding module (830) is configured to determine the reconstruction position of the points in the point cloud based on the occupancy code. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstruction positions and determine the LOD-based order. The inverse residual quantization module (820) is configured to generate the reconstruction residual based on the quantization residual received from the arithmetic decoding module (810).

[0120] The attribute prediction module (850) is configured to perform an attribute prediction process to determine the attribute predictions of points based on a LOD-based order. For example, the attribute predictions of the current point can be determined based on the reconstructed attribute values ​​of the current point's neighboring points stored in memory (860). In some examples, the attribute predictions can be combined with the corresponding reconstructed residuals to generate the reconstructed attributes of the current point.

[0121] In one example, the reconstructed attribute sequence generated from the attribute prediction module (850) and the reconstructed position generated from the octree decoding module (830) correspond to the decoded point cloud (802) output from the G-PCC decoder (800). Furthermore, the reconstructed attributes are also stored in memory (860) and can subsequently be used to derive attribute predictions for subsequent points.

[0122] In various embodiments, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) can be implemented in hardware, software, or a combination thereof. For example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) can be implemented with a processing circuitry, such as one or more integrated circuits (ICs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and / or the like, operating with or without software. In another example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) can be implemented as software or firmware including instructions that are stored in a non-volatile (or non-transitory) computer readable storage medium that, when executed by a processing circuitry, such as one or more processors, cause the processing circuitry to perform the functionality of the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800).

[0123] It should be noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein can be included in other decoders or encoders that can have similar or different structures than those shown in FIGS. 7 and 8. Moreover, in various examples, the encoder (700) and the decoder (800) can be included in the same device or in separate devices. Figure 7 and Figure 8 Moreover, in various examples, the encoder (700) and the decoder (800) can be included in the same device or in separate devices.

[0124] The present disclosure includes embodiments related to estimation of Manhattan layouts, including estimating Manhattan layouts of scenes from various panoramas of the scenes. These embodiments can be used to create virtual reality and augmented reality applications, such as virtual tourism. For example, the Manhattan layouts can be estimated from a large number of panorama images with the use of geometric and segmentation information.

[0125] In applications such as robotics, virtual reality, and augmented reality, estimating a layout of a room from images is a common practice. The layout of a room can include the locations, orientations, and heights of the walls of the room relative to a particular reference point. For example, the layout of a room can be depicted using the intersections of the walls, a 3D mesh, or a point cloud. In a Manhattan layout of a room, the walls of the room are perpendicular to each other. Panoramic images can be generated by a camera, such as a panoramic camera. The panoramic images can be used to depict the layout of a room. However, estimating a Manhattan layout of a room by analyzing multiple panoramic images can be challenging. A panorama (or panoramic image) can encapsulate 360-degree information in a scene, and the 360-degree information can contain much more data than a perspective image.

[0126] A Manhattan layout of a room can be estimated by using semantic segmentation and geometric information from information (e.g., pixels) of multiple panoramic views (or panoramic images). Semantic segmentation can be performed using a deep learning algorithm that associates a label or class with each pixel in an image. Semantic segmentation can be used to identify collections of pixels that form different classes.

[0127] Multiple panoramic views (or panoramic images) can be used to estimate a Manhattan layout of a room because a single panoramic view can not provide an accurate representation of the layout of a room. For example, objects in a room can block the wall boundaries of the room, or a room can be very large such that a single panoramic image can not fully capture the room. Geometric information can include two major directions (e.g., X-direction and Z-direction) of a Manhattan layout of a room and line segment information extracted from a panoramic view. However, since geometric information focuses on the geometric content of a room, geometric information can lack semantic information. Therefore, semantic segmentation can be used to provide semantic information of pixels of a panoramic view. Semantic segmentation can refer to labeling of a panoramic view (or panoramic image) to determine a respective class of each pixel of the panoramic view. For example, based on semantic segmentation, a pixel can be labeled as one of a floor, a wall, etc. in a room.

[0128] Figures 9A-9C is an example geometric representation and an example semantic representation of a panoramic view. As shown in Figure 9A , a panoramic view (900) is provided, where the panoramic view (900) can include a scene of a hotel room. In Figure 9B , the panoramic view (900) can be labeled with information of geometric line segments. In Figure 9C , semantic information of pixels of a room can be provided based on semantic segmentation. For example, the semantic information of pixels of a room can indicate a ceiling (902), a floor (904), a sofa (906), a wall (908), etc.

[0129] The layout of a room can be represented in various ways, including a 3D mesh, a boundary line, and a point cloud. In the present disclosure, a 3D mesh and a boundary line of a room can be used to describe the layout of the room. The boundary line can be represented by one or more polygons, and the 3D mesh of the room can be created by the polygons.

[0130] Figures 10A-10C is an exemplary polygonal representation and an exemplary mesh representation of a room. As shown in Figure 10A , an estimated room layout (1000) can be provided. In Figure 10B , a polygonal representation of the estimated room layout (1002) can be provided, in which the corners of the room layout can be labeled with numbers, e.g., 0-7. In Figure 10C , a 3D mesh (1004) can be generated from the polygon (1002). According to Figures 10A-10C , when the layout of a room (e.g., (1000)) is established, the wall faces can be projected into the floor face to obtain the polygon (e.g., (1002)). Conversely, if the polygon and the height of the wall are obtained, the layout and the 3D mesh (e.g., (1004)) can be derived.

[0131] In the present disclosure, the Manhattan layout of a room can be estimated by estimating a polygon (e.g., polygon (1002)) based on the geometric information and the semantic information of a scene (e.g., room layout (1000)). The polygon can be represented by a directed graph G = (v, e), where v is a set of corners of the polygon, and e is a set of edges connecting the set of corners. The position of each corner joint can be defined as a 2D coordinate (x, y) in a 2D space relative to the camera position. Each edge can be represented as a directed line segment (p s , p e ), where p s ∈v is the start point of the edge, and p e ∈v is the end point of the edge. The straight line function of each edge can be represented as ax + by + c = 0, where n = (a, b) is the normal of the straight line satisfying ||n|| = 1, and c is a constant real number.

[0132] Figure 11 An overview of a system (or process) (1100) for estimating the Manhattan layout of a scene (e.g., a room) is shown. As shown in Figure 11As shown, in (1110), an input image can be provided. The input image can include a set of panoramic images (or multiple panoramic images) of a scene. The set of panoramic images can capture the scene from different view positions to more accurately represent the scene. At steps (1120) and (1130) of the process (1100), geometric information of each panoramic image can be extracted, and semantic information of each panoramic image can be determined based on semantic segmentation corresponding to the panoramic image. The geometric information (or geometric factors) can include: (1) lines detected in each panoramic image, (2) major directions (e.g., X and Z directions) of each panoramic image, (3) a ratio between a distance from a ceiling of a room to a floor of the room and a distance from a camera to the floor, and (4) relative poses (e.g., relative positions, angles, or distances) between panoramic images, such as a relative pose between two respective panoramic images. The semantic information can be obtained through semantic segmentation. Semantic segmentation can assign semantic meanings (e.g., floor, door, etc.) to each pixel of a panoramic image. At step (1140), a respective layout of the scene can be estimated based on the geometric information and the semantic segmentation of each panoramic image. At step (1150), the layouts estimated from each panoramic image (or panoramic image) can be combined to generate a final estimate of the room layout. At step (1160), a 3D mesh (or Manhattan layout) of the room can be generated based on the estimated layout (or the final estimate of the room layout).

[0133] To determine a layout estimate associated with the scene (e.g., a room) based on the geometric information and the semantic information of each panoramic image, for example Figure 11 At an example of step (1140), the results of semantic segmentation can be used to determine whether each line segment of a panoramic image (or panoramic image) represents a wall boundary of the room. Assuming a line segment l represents a sequence of points p0, p1,..., p n of a panoramic image, it can be checked whether the neighboring pixels of each point p i contain a wall. If the neighboring pixels of a point p i contain only walls or do not have any wall pixels, the point p i cannot be considered as a boundary point. If the number of boundary points exceeds a certain threshold, for example, 80% of the points are boundaries, the line segment (or line) l can be designated as a boundary (or boundary line).

[0134] The boundary line can be aligned with the major direction (or main direction) of the panoramic image. To align the boundary line with the major direction, each boundary line can be projected onto a horizontal plane (e.g., X-Z plane). Since the two major directions (e.g., X and Z) are perpendicular, an angle between the projected boundary line and the two major directions can be calculated. The projected boundary line can then be rotated around the center of the projected boundary line to make the projected boundary line parallel to the major direction.

[0135] However, objects in the image (or the panoramic image) can block the boundary lines. Thus, one or more boundary lines can be incomplete. In a first approach, a combination of the ceiling boundary lines and the floor boundary lines can be used to estimate the incomplete boundary lines (or complete one or more incomplete boundary lines). For example, each floor line (or floor boundary line) can correspond to a ceiling line (or ceiling boundary line), and a distance between the floor line and the ceiling line can be fixed in the scene. The incomplete boundary line can be projected to the ceiling line and the floor line, respectively. Coordinates of points of the incomplete boundary line on the respective projected boundaries (or projected boundaries) of the ceiling and the floor can be:

[0136] p c,1 ,p c,2 ,p c,3 ,...p c,n

[0137] p f,1 ,p f,2 ,p f,3 ,...p f,n

[0138] Based on a ratio r between the first distance from the ceiling to the floor and the second distance from the camera to the floor, the points of the incomplete boundary line can be scaled (or estimated) to combine the projected points on the ceiling boundary lines and the floor boundary lines as follows in Equation (1):

[0139] where i = 1,..., n Equation (1)

[0140] In a second approach, the projected line segments (e.g., the boundary lines projected onto the horizontal plane) can be connected using a Manhattan layout assumption. As defined in the Manhattan layout assumption, each pair of two connected boundaries can be parallel or perpendicular. Thus, the projected boundary lines can be ordered according to the original spatial coordinates of the boundary lines in the image space (e.g., X-Y-Z space) of the scene. When two lines (or boundary lines) are parallel, a perpendicular line can be added to connect the two lines. When two lines are perpendicular, it can be determined whether an intersection point of the two lines is on both line segments (or lines). In response to the intersection point not being on both lines, the two line segments (or lines) can be extended such that the intersection point can be located on both line segments.

[0141] The polygons can be obtained based on one or a combination of the first and second approaches. In some embodiments, the polygons can be refined using geometric processing methods (e.g., 2D polygon denoising, step removal, etc.). Thus, a plurality of polygons can be obtained based on the panoramic image. Each polygon can be derived from a respective panoramic image and indicate a respective layout estimate of the scene (e.g., a room).

[0142] Various methods can be applied to denoise existing curves (or line segments). In an example, the boundaries of a curve (or line segment) can be fitted to a region with noisy points, and then region thinning can be performed. In an example, a multi-scale analysis such as a Gaussian kernel can be applied. The multi-scale analysis can use an impact probe to preserve sharp points and output a curve as a set of smooth arcs and corners. In another example, Gaussian smoothing can be applied to the noise estimated by local analysis, the Gaussian smoothing having a fixed n number of neighbors, for example n = 30 neighbors.

[0143] Staircase removal can reduce staircase artifacts. Staircase artifacts can be a common artifact that can be observed in many denoising tasks, such as one-dimensional signal denoising, two-dimensional image denoising, and video denoising. Image denoising techniques can flatten one or more regions of an image signal, thereby creating staircase artifacts in the image signal. As a result, the staircase artifacts can manifest as undesired false steps or undesired flat regions in the image signal that are not expected in a smoothly varying image signal in one or more regions of the image signal.

[0144] To generate a layout estimate of a scene based on respective layout estimates from each panoramic image, for example Figure 11 an example of step (1150) in, the polygons derived in a previous step (e.g., step (1140)) can be converted to the same coordinate system based on estimated relative positions between the panoramic images. After step (1140), the converted (or derived) polygons can be represented as poly0, poly1,... poly n To generate a layout estimate of a scene, two independent processes can be employed to obtain the layout, the two independent processes being contour estimation and interior edge estimation.

[0145] In contour estimation, contours of the layout estimate can be determined. Contour estimation can utilize a polygon joint algorithm to convert the polygons poly0, poly1,... poly n together as a baseline polygon poly base .

[0146] The baseline polygon poly base may then be shrunk to form a shrunk polygon poly shrunk . To shrink the baseline polygon poly base , for each edge e base of the baseline polygon poly i a candidate edge E = {e' i , e' i+2 ,... e' i}. The candidate edges E can be selected from the original converted polygons poly0, poly1,... poly n and the candidate edges E satisfy at least one of the following conditions:

[0147] (1) Each edge e' i ∈ E is parallel to e i , and

[0148] (2) The projected overlap percentage of e' i projected to e i is greater than a threshold, e.g., 70%. The projected overlap percentage can be determined by projecting e' i to e i and calculating the fractional overlap of the line segment of e' i projected to the line segment of e i .

[0149] In the candidate edges E = {e' i , e' i+2 ,... e' i}, the candidate edge e' i closer to the original viewpoint position (e.g., the position of the edge in the original panoramic image) can be used to replace e i . Thus, when one or more candidate edges e' i are closer to the original viewpoint position, one or more edges e i can be replaced by the corresponding one or more candidate edges e' i .

[0150] In the formation of the baseline polygon poly base , all edges in the converted polygons poly0, poly1,... poly n are merged together in the baseline polygon poly base , and all edges of the converted polygons can not be considered. By projecting the base polygon poly base into each converted polygon, one or more edges e i can be replaced by the corresponding one or more candidate edges e' i when one or more candidate edges e' i are closer to the original viewpoint position. Thus, the edges e i in the base polygon and the edges in the converted polygons can match in direction, size, position, etc.

[0151] Each edge e base in the baseline polygon poly i can be compared with the corresponding candidate edge e' i . Based on the candidate edge e'i Whether closer to the original viewpoint position, the corresponding edge can be kept or replaced. By replacing one or more edges e base in the baseline polygon poly i , a reduced polygon poly shrunk can be formed.

[0152] By using geometry processing methods, such as 2D polygon denoising, step removal, etc., a final polygon poly final can be further obtained based on the reduced polygon.

[0153] The interior edge estimation can be configured to recover interior edges of the final polygon poly final . To recover the interior edges, all edges of the converted polygons poly0, poly1,... poly final inside the final polygon poly n can be put into a set E'. Then a space-based edge voting strategy can be used to cluster (or group) these edges. For example, when two edges in the final polygon poly final satisfy at least one of the following conditions, the two edges can be grouped into a group:

[0154] (1) The two edges are parallel.

[0155] (2) The distance between the two edges is small enough, e.g., less than a first threshold.

[0156] (3) The post-projection overlap between the two edges is large enough, e.g., greater than a second threshold.

[0157] In addition, if a group of edges contains more than a certain number of edges, the average edge of the group of edges can be calculated to represent the recovered interior edge of the final polygon poly final . Thus, the average edge of the group of edges can be added to the final polygon poly final as a recovered interior edge.

[0158] Based on the estimated polygon (e.g., the final polygon poly final ), various representations can be used to generate the 3D shape (or Manhattan layout) of the room.

[0159] In an embodiment, a triangular mesh can be used to generate the 3D shape of the room. For example, the ceiling face and the floor face of the room can be generated by triangulating the final polygon poly final . The wall faces of the room can be generated by triangulating the rectangle enclosed by the ceiling boundary lines and the floor boundary lines. A ray casting based method can be further applied to generate the texture of the 3D mesh (or 3D shape).

[0160] In the embodiment, by adjusting the final polygon poly final Quadrilateralization allows you to use quadrilaterals to represent the 3D shape of a room.

[0161] In this embodiment, point clouds can be used to represent the 3D shape of a room by sampling points from a triangular or quadrilateral mesh. This can be achieved by modifying the final polygon. final Triangulation is performed to obtain a triangular mesh. This can be achieved by adjusting the final polygon poly. final Quadrilateralization is performed to obtain a quadrilateral mesh.

[0162] In this embodiment, the final polygon can be voxelized. final This allows the generation of the room's 3D shape. Therefore, it's possible to generate the 3D shape of the room by taking the 3D model (e.g., the final polygon)... final This data is converted into volumetric data (e.g., the 3D shape of a room) to create voxels (or 3D shapes).

[0163] This disclosure provides a method for estimating the Manhattan layout of a scene (e.g., a room). The Manhattan layout of the scene can be estimated based on multiple panoramic images of the scene, using geometric and semantic segmentation information associated with the scene.

[0164] In embodiments, main direction, line segments, and semantic segmentation can be used together to estimate the layout of a scene based on a single panoramic view (or panoramic image) of multiple panoramic images.

[0165] In one embodiment, the pose information (e.g., angle or distance) of the panoramic images can be used to combine the layout of each panoramic image into a final layout estimate.

[0166] In an embodiment, by modifying the final polygon (e.g., poly) final Triangulation is performed on the final polygon, and quadrilateralization is performed on the final polygon. final Generate point clouds, or perform point cloud generation on models (e.g., final layout estimation or final polygon poly). final Voxelization is performed so that 3D shapes (or Manhattan layouts) can be generated based on the final room layout.

[0167] In this embodiment, line detection methods can be used to detect line segments in multiple panoramic images. For example, such as... Figure 11 As shown in step (1140), semantic segmentation can assign semantic meaning (e.g., floor, door, etc.) to each pixel of a panoramic image. The result of semantic segmentation can be used to determine whether each line segment of the panoramic image represents the wall boundary of a room.

[0168] In embodiments, the principal directions (e.g., X and Z directions) of the panoramic images can be obtained by analyzing the statistical information of line segments within the panoramic images.

[0169] In embodiments, the semantic segmentation of the panoramic images can be implemented using deep learning based semantic segmentation techniques. For example, the semantic segmentation can be a deep learning algorithm that associates a label or class with each pixel in an image (e.g., a panoramic image of a scene).

[0170] In embodiments, the panoramic pose estimation (e.g., angle or distance) of the panoramic images can be implemented using image alignment techniques. Based on the alignment of two panoramic images, the relative angle or relative distance between the two panoramic images can be determined.

[0171] In embodiments, the ratio between the distance from the ceiling of a scene to the floor of the scene and the distance from the camera to the floor can be computed using the segmentation information.

[0172] Figure 12 is a diagram of a framework (1200) for image processing according to some embodiments of the present disclosure. The framework (1200) includes a video encoder (1210) and a video decoder (1250). The video encoder (1210) encodes input of multiple panoramic images, such as a scene (e.g., a room), into (1205) a bitstream (1245), and the video decoder (1250) decodes the bitstream (1245) to generate a reconstructed 3D shape (1295), e.g., a Manhattan layout of the scene.

[0173] The video encoder (1210) can be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The video decoder (1250) can be any suitable device, such as a computer, a client computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The bitstream (1245) can be transmitted from the video encoder (1210) to the video decoder (1250) via any suitable communication network (not shown).

[0174] In Figure 12In an example, the video encoder (1210) includes a segmentation module (1220), an encoder (1230), and an extraction module (1240) coupled together. The segmentation module (1220) is configured to assign semantic meanings (e.g., floor, door, etc.) to each pixel of a panoramic image associated with a scene. The semantic information of each panoramic image can be transmitted to the encoder (1230) through a bitstream (1225). The extraction module (1240) is configured to extract geometric information of each panoramic image. The geometric information can be transmitted to the encoder (1230) through a bitstream (1227). The encoder (1230) is configured to generate a 3D shape (or Manhattan layout) of the scene based on the geometric information and the semantic information of each panoramic image. For example, the encoder (1230) can generate a respective layout estimate (or polygon) of the scene based on each panoramic image. The layout estimates of the panoramic images can be fused to form a final layout estimate (or final polygon). The 3D shape of the scene can be generated by triangulating the final polygon, quadrangulating the final polygon, generating a point cloud based on the final polygon, or voxelizing the final polygon.

[0175] In Figure 12 an example, the bitstream (1245) is provided to a video decoder (1250). The video decoder (1250) includes a decoder (1260) and a reconstruction module (1290) coupled together as shown in Figure 12 In an example, the decoder (1260) corresponds to the encoder (1230) and can decode the bitstream (1245) encoded by the encoder (1230) and generate decoded information (1265). The decoded information (1265) can be further provided to the reconstruction module (1290). The reconstruction module (1290) can accordingly reconstruct a 3D shape (or Manhattan layout) of the scene (1295) based on the decoded information (1265).

[0176] Figure 13 is a flowchart of an overview process (1300) according to embodiments of the present disclosure. In various embodiments, the process (1300) is performed by processing circuitry. In some embodiments, the process (1300) is implemented in software instructions, and thus when the processing circuitry executes the software instructions, the processing circuitry performs the process (1300). The process starts from (S1301) and proceeds to (S1310).

[0177] At (S1310), a plurality of two-dimensional (2D) images of a scene are received.

[0178] At (S1320), geometric information and semantic information of each of the plurality of 2D images are determined. The geometric information indicates detected lines and a reference direction in the respective 2D image. The semantic information includes classification information of pixels in the respective 2D image.

[0179] At (S1330), based on the geometry information and the semantic information of the respective 2D image, a layout estimate associated with the respective 2D image of the scene is determined.

[0180] At (S1340), based on the plurality of determined layout estimates associated with the plurality of 2D images of the scene, a combined layout estimate associated with the scene is determined.

[0181] At (S1350), a Manhattan layout associated with the scene is generated based on the combined layout estimate. The Manhattan layout includes at least a three-dimensional (3D) shape of the scene, the three-dimensional (3D) shape including walls that are orthogonal to each other.

[0182] To determine the geometry information and the semantic information, first geometry information of a first 2D image of the plurality of 2D images can be extracted. The first geometry information can include at least one of: detected lines, a reference direction of the first 2D image, a ratio between a first distance from a ceiling to a floor and a second distance from a camera to the floor, or a relative pose (e.g., an angle or a distance) between the first 2D image and a second 2D image of the plurality of 2D images. Pixels of the first 2D image can be labeled to generate first semantic information, where the first semantic information can indicate first structure information of the pixels in the first 2D image.

[0183] To determine the layout estimate associated with the respective 2D image of the scene, a first layout estimate of the plurality of determined layout estimates associated with the scene can be determined based on the first geometry information and the first semantic information of the first 2D image. To determine the first layout estimate, it can be determined whether each detected line is a boundary line corresponding to a wall boundary in the scene. The boundary lines of the detected lines can be aligned with the reference direction of the first 2D image. A first polygon indicating the first layout estimate can be generated based on the aligned boundary lines using one of a 2D polygon denoising or a step removal.

[0184] To generate the first polygon, a plurality of incomplete boundary lines of the boundary lines can be made complete based on one of: (i) estimating the plurality of incomplete boundary lines based on a combination of a ceiling boundary line and a floor boundary line of the boundary lines, and (ii) connecting an incomplete boundary line pair of the plurality of incomplete boundary lines. The incomplete boundary line pair can be connected based on one of: (i) adding a perpendicular line to the incomplete boundary line pair in response to the incomplete boundary line pair being parallel, and (ii) extending at least one of the incomplete boundary line pair such that an intersection of the incomplete boundary line pair is located on the extended incomplete boundary line pair.

[0185] To determine the combined layout estimate associated with the scene, a base polygon can be determined by combining the plurality of polygons via a polygon union algorithm. Each polygon of the plurality of polygons can correspond to a respective layout estimate of the plurality of determined layout estimates. A reduced polygon can be determined based on the base polygon. The reduced polygon includes updated edges resulting from updating edges of the base polygon. A final polygon can be determined based on the reduced polygon by using one of a two-dimensional polygon denoising or a step removal. The final polygon can correspond to the combined layout estimate associated with the scene.

[0186] To determine the reduced polygon, for an edge of the base polygon, a plurality of candidate edges can be determined based on the plurality of polygons. Each candidate edge of the plurality of candidate edges can correspond to a respective edge of the base polygon. Responsive to the one or more candidate edges being closer to the original viewpoint location of the plurality of 2D images relative to the respective one or more edges of the base polygon, the updated edges of the reduced polygon can be generated by replacing the one or more edges of the base polygon with the respective one or more candidate edges.

[0187] In some embodiments, each candidate edge of the plurality of candidate edges can be parallel to a respective edge of the base polygon. A projection overlap between the candidate edge and the respective edge of the base polygon can be greater than a threshold value.

[0188] To determine the combined layout estimate associated with the scene, a set of edges including edges of the final polygon can be determined. Based on the set of edges, a plurality of edge combinations can be generated. A plurality of interior edges of the final polygon can be generated. The plurality of interior edges can be indicated by a plurality of average edges of one or more edge combinations of the set of edges. Each edge combination of the plurality of edge combinations can include a respective number of edges that is greater than a target value. Each average edge of the plurality of average edges can be obtained by averaging edges of a respective one of the one or more edge combinations.

[0189] In some embodiments, the plurality of edge combinations can include a first edge combination. The first edge combination can further include a first edge and a second edge. The first edge and the second edge can be parallel to each other. A distance between the first edge and the second edge can be less than a first threshold value. A projection overlap area between the first edge and the second edge can be greater than a second threshold value.

[0190] To generate the Manhattan layout associated with the scene, the Manhattan layout associated with the scene can be generated based on one of: a triangle mesh triangulated from the combined layout estimate, a quadrilateral mesh quadrangulated from the combined layout estimate, a sampling point sampled from one of the triangle mesh and the quadrilateral mesh, or a discrete mesh generated from one of the triangle mesh and the quadrilateral mesh via voxelization.

[0191] In some embodiments, a Manhattan layout associated with the scene is generated based on a triangle mesh that is triangulated from the combined layout estimate. Thus, to generate the Manhattan layout associated with the scene, the ceiling faces and the floor faces in the scene can be generated by triangulating the combined layout estimate. The wall faces in the scene can be generated by triangulating the rectangle enclosed by the ceiling boundary lines and the floor boundary lines in the scene. The texture of the Manhattan layout associated with the scene can be generated based on a ray casting process.

[0192] The techniques described above, can be implemented as computer software using computer readable instructions and physically stored in one or more computer-readable media. For example, Figure 14 A computer system (1400) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0193] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar

[0194] Instructions can be executed by one or more processors, such as one or more central processing units (CPU), graphics processing units (GPU), digital signal processors (DSP), application specific integrated circuits (ASIC), programmable logic devices (PLD), and / or field programmable gate arrays (FPGA).

[0195] Figure 14 The components of computer system (1400) shown in FIG. 14 are exemplary and not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of a computer system (1400).

[0196] Computer system (1400) can include certain human interface input devices. Such a human interface input device can be responsive to

[0197] The input human interface devices can include one or more of (only one of each is shown): keyboard (1401), mouse (1402), touchpad (1403), touchscreen (1410), data glove (not shown), joystick (1405), microphone (1406), scanner (1407), camera (1408).

[0198] The computer system (1400) can also include certain human interface output devices. Such human interface output devices can be stimulating one or more of the human senses, such as sight, touch, taste, and smell / hearing. Such human interface output devices can include tactile output devices (for example, touch screen (1410), data glove, or tactile feedback device for a joystick (1405), but also sonar or ultrasound devices, haptic devices, and printers (not shown)), acoustic output devices (for example, speakers (1409), headphones (not shown)), visual output devices (for example, screens (1410) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without tactile feedback capability, some of which are capable of outputting two dimensional visual output or more than three dimensional output through means such as stereographic output, virtual-reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0199] The computer system (1400) can also include human accessible storage and its associated media such as optical media (including CD / DVD ROM / RW (1420) with CD / DVD etc. media (1421)), thumb-drive (1422), removable hard drive, or solid state drive (1423), legacy magnetic media such as tape and floppy disk (not shown), specialized ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0200] Those skilled in the art should also understand that the term "computer readable medium" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0201] The computer system (1400) can also include an interface (1454) to one or more communication networks (1455). Networks can for example be wireless networks, wire-line networks, optical networks, etc. Networks can further be local networks, wide-area networks, metropolitan area networks, vehicular and industrial networks, real-time networks, delay-tolerant networks, etc. Examples of networks include local area networks such as Ethernet networks, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks to include cable networks, satellite TV, and terrestrial TV, vehicular and industrial to include CANBus, etc. Certain networks commonly require various networking hardware (modems, wireless Hubs, routers, access points, etc.) connected to certain general-purpose data ports or peripheral buses (1449) of the computer system (1400) as an entry point. Other networks are commonly connected to and supported by the internal architecture of the computer system (1400), and can not require additional hardware. The computer system (1400) can communicate over such networks using any of a variety of appropriate protocols (i.e., TCP / IP, SMTP, etc.).

[0202] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (1440) of the computer system (1400).

[0203] The core (1440) can include one or more Central Processing Units (CPU) (1441), Graphics Processing Units (GPU) (1442), specialized programmable processing units in the form of Field Programmable Gate Arrays (FPGA) (1443), hardware accelerators for certain tasks (ACCL) (1444), a graphics adapter (1450), and so forth. These devices, along with Read-only memory (ROM) (1445), Random-access memory (1446), internal mass storage such as internal non-user accessible hard drives, SSDs, and the like, can be connected through a system bus (1448). In some computer systems, the system bus (1448) can be accessible in the form of one or more physical plugs to enable extensions of the system bus (1448) through additional CPUs, GPU, and the like. The peripheral devices can be connected to the system bus (1448) directly, or through a peripheral bus (1449). In one example, the screen (1410) can be connected to the graphics adapter (1450). The peripheral bus’ architecture can be any of a variety of bus architectures, including PCI, USB, and the like.

[0204] CPUs (1441), GPUs (1442), FPGAs (1443), and accelerators (1444) can execute certain instructions that can be combined to constitute the aforementioned computer code. That computer code can be stored in ROM (1445) or RAM (1446). Transitional data can be also stored in RAM (1446), whereas permanent data can be stored for example, in the internal mass storage (1447). Fast storage and retrieval to any of the storage devices can be enabled through the use of caches that can be tightly associated with one or more of the CPU (1441), GPU (1442), mass storage (1447), ROM (1445), RAM (1446), or the like.

[0205] The computer readable media can have computer code thereon for performing various computer-implemented operations. The media and computer code can be those specially designed and constructed for the purposes of the present disclosure, or the media and computer code can be of the kind well known and available to those having skill in the computer software arts.

[0206] As non-limiting examples, the computer system having architecture (1400), and specifically the core (1440), is provided with functionality as a result of processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software embodied in one or more tangible, computer-readable media, such as a mass storage device, and certain of the core’s (1440) memory. Such computer-readable media can be physical storage medium or memory located in a first position, such as close to the core (1440), and to a mass storage device (1447) associated with the user of the computer system. The software, when executed by the processor(s), causes the computer system to provide the described functionality, including the steps of any of the methods that execute on computer system, as well as the specific steps of any of those methods. The software that embodies the methods (or other information processing processes) is stored in memory, such as the core’s (1440) memory, at least at one time. Alternatively, some or all of the software can be stored on the mass storage device (1447), a removable memory cache, or other computer-readable media, including storage local to, or access over a network, to the core (1440).

[0207] While the present disclosure has described a number of exemplary embodiments, there are alterations, modifications, various replacements and various equivalent equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and thus fall within the spirit and scope of the present disclosure.

Claims

1. A method for estimating a Manhattan layout associated with a scene, characterized in that, The method includes: Receive multiple two-dimensional (2D) images of the scene; First geometric information of a first 2D image among multiple 2D images is extracted, wherein the first geometric information includes: detected lines, reference direction of the first 2D image, ratio between a first distance from the ceiling to the ground and a second distance from the camera to the ground, relative pose between the first 2D image and the second 2D image among the multiple 2D images, the first geometric information indicating lines in the detected 2D image and reference direction, the reference direction including the X-axis direction and the Z-axis direction; And to label the pixels of the first 2D image to generate first semantic information, wherein the first semantic information indicates first structural information of the pixels in the first 2D image, and the first semantic information includes classification information of the pixels in the 2D image; Based on the determined multiple layout estimates associated with the multiple 2D images of the scene, a combined layout estimate associated with the scene is determined; Based on the first geometric information and the first semantic information of the first 2D image, a first layout estimate is determined among the plurality of layout estimates associated with the scene. The first layout estimate is determined by: determining whether each of the detected lines is a boundary line corresponding to a wall boundary in the scene; aligning the boundary lines of the detected lines with the reference direction of the first 2D image; and generating a first polygon indicating the first layout estimate based on the aligned boundary lines using either 2D polygon denoising or stair removal, including: making a plurality of incomplete boundary lines complete based on one of: estimating the plurality of incomplete boundary lines based on a combination of ceiling boundary lines and floor boundary lines in the boundary lines; and connecting pairs of incomplete boundary lines based on one of: (i) adding a vertical line to the pairs of incomplete boundary lines in response to the pairs of incomplete boundary lines being parallel, and (ii) extending at least one of the pairs of incomplete boundary lines such that the intersection of the pairs of incomplete boundary lines is located on the extended pairs of incomplete boundary lines. The Manhattan layout associated with the scene is generated based on the combined layout estimation, wherein the Manhattan layout includes at least a three-dimensional (3D) shape of the scene, the three-dimensional (3D) shape including walls that are orthogonal to each other.

2. The method according to claim 1, characterized in that, The determination of the combined layout estimate associated with the scenario also includes: The base polygon is determined by combining multiple polygons via a polygon joint algorithm, wherein each of the multiple polygons corresponds to a corresponding layout estimate in the determined multiple layout estimates; A reduced polygon is determined based on the base polygon, wherein the reduced polygon includes updated edges obtained by updating the edges based on the base polygon; and A final polygon is determined based on the reduced polygon by using either the 2D polygon denoising or the stair removal, wherein the final polygon corresponds to the combined layout estimate associated with the scene.

3. The method according to claim 2, characterized in that, The process of determining the reduced polygon based on the base polygon also includes: For the edges of the base polygon, a plurality of candidate edges are determined based on the plurality of polygons, wherein each of the plurality of candidate edges corresponds to a corresponding edge of the base polygon; and In response to the one or more candidate edges being closer to the original viewpoint position in the plurality of 2D images than the corresponding one or more edges of the base polygon, the updated edges of the reduced polygon are generated by replacing one or more edges of the base polygon with the corresponding one or more candidate edges.

4. The method according to claim 3, characterized in that, The method further includes: Each of the plurality of candidate edges is parallel to the corresponding edge of the base polygon, and The projection overlap between the candidate edge and the corresponding edge of the base polygon is greater than a threshold.

5. The method as described in claim 2, characterized in that, The determination of the combined layout estimate associated with the scenario also includes: Determine the set of edges that includes the edges of the final polygon; Generate multiple edge combinations based on the aforementioned edge set; and Generate multiple internal edges of the final polygon, wherein the final polygon is indicated by multiple average edges of one or more edge combinations of the edge set, each of the one or more edge combinations of the multiple edge combinations including a corresponding number of edges, the number being greater than a target value, and each of the multiple average edges is obtained by averaging the edges of the corresponding edge combinations of the one or more edge combinations.

6. The method according to claim 5, characterized in that, The method further includes: The plurality of edge combinations includes a first edge combination, and The first edge combination also includes a first edge and a second edge, wherein the first edge and the second edge are parallel, the distance between the first edge and the second edge is less than a first threshold, and the projected overlap area between the first edge and the second edge is greater than a second threshold.

7. The method according to claim 1, characterized in that, The process of generating the Manhattan layout associated with the scene based on the combined layout estimation further includes: The Manhattan layout associated with the scene is generated based on one of the following: a triangular mesh triangulated according to the combined layout estimate, a quadrilateral mesh quadrangularized according to the combined layout estimate, sampling points obtained by sampling one of the triangular mesh and the quadrilateral mesh, or a discrete mesh generated by voxelization based on one of the triangular mesh and the quadrilateral mesh.

8. The method according to claim 7, characterized in that, The method further includes: Based on the triangular mesh estimated from the combined layout, the Manhattan layout associated with the scene is generated, and... The generation of the Manhattan layout associated with the scene also includes: The ceiling and floor surfaces in the scene are generated by triangulating the combined layout estimate. The walls in the scene are generated by triangulating the rectangle formed by the ceiling and floor boundaries; and Textures of the Manhattan layout associated with the scene are generated through ray-casting-based processing.

9. An apparatus for estimating a Manhattan layout associated with a scene, characterized in that, include: Memory, used to store program instructions; as well as A processor for invoking program instructions stored in the memory to implement the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 8.

Images