A method, system, electronic device, and storage medium for image plane detection based on wireframe

Abstract

This invention provides a method, system, electronic device, and storage medium for image plane detection based on wireframes. The method involves inputting an image to be detected and its corresponding wireframe segments; rendering all wireframe segments onto a state matrix, where each element indicates the segment information covering that element; merging adjacent endpoints of the wireframe segments using morphological methods; finding and obtaining closed regions based on a region seed filling algorithm; filtering the closed regions based on their pixel variance to obtain the final vectorized plane detection result, which is then output. Compared to existing technologies, this method avoids the jagged edges that may occur during the bitmap segmentation and vectorization process, directly outputting a vectorized plane represented by closed line segments.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method, system, electronic device, and storage medium for image plane detection based on wireframe. Background Technology

[0002] In human life, planar structures are very common geometric structures, such as the exterior surfaces of city buildings, interior walls, floors, ceilings, and the surfaces of tables and chairs. Various planes combine to form the diverse three-dimensional structures in the world. The human brain and visual system have excellent perceptual abilities in understanding these advanced geometric structures, easily resolving geometric information in the three-dimensional world. However, for computers, detecting and recognizing geometric structures in the real world remains a challenging task.

[0003] In recent years, research has focused on enabling computers to detect and recognize geometric structures in the real world. Plane detection is a crucial component of 3D scene analysis and has been a long-standing research focus in computer vision. Plane structure geometry analysis is a key technology for many emerging applications, such as robot navigation, virtual reality, augmented reality, scene understanding, and scene reconstruction. For example, robots need to identify flat surfaces to plan their movement routes or recognize the planar boundaries of a dining table to correctly place items. In augmented reality applications, placing virtual objects on table surfaces, changing floor textures in virtual scenes, or hanging artwork on virtual interior walls all rely on plane detection as a foundation. A fundamental problem in computer vision is developing an algorithm with similar perceptual capabilities to make these applications a reality.

[0004] With the continuous development of deep learning technology, new algorithms and technologies based on convolutional neural networks are constantly emerging, achieving great success in computer vision tasks such as object detection, pose estimation, and instance segmentation. Some scholars have introduced convolutional neural network techniques into image scene structure analysis, achieving some surprising results. The Wireframe detection series of works is a representative example of deep learning-based scene geometry analysis. Wireframes are concise and efficient in representing scene geometry information. They advocate using geometrically meaningful geometric elements in the image, such as points, structural lines, and wireframes, to analyze the image scene structure. However, current related work mainly uses deep convolutional neural networks to detect points and structural lines in images and uses these points and structural lines to analyze the scene structure; there has been no invention that uses closed wireframes to represent surfaces for scene structure analysis. Planes, as important geometric elements in a scene, have not been fully explored and studied by the community. It has been found that representing planes in a scene using wireframes is very direct and effective. Most of the line segments in a wireframe are structural lines in the scene, such as the edges of objects and the intersection lines of house walls. Structural lines are mostly intersection lines between planes, that is, the boundary lines of planes. Therefore, based on wireframes, we can directly obtain vectorized plane results represented by boundary lines, instead of using semantic segmentation or instance segmentation methods to first determine whether a pixel is on a plane, obtain a segmented bitmap, and then vectorize the bitmap. Summary of the Invention

[0005] In view of this, the purpose of this embodiment of the invention is to further transform the method of resolving the image scene structure based on the provided wireframe diagram, that is, the method of resolving the image scene structure using independent loose geometric primitives such as points and structural lines, into the method of resolving the scene structure using closed wireframe polygons composed of planar vertices, and directly outputting a vectorized planar representation.

[0006] A first aspect of the present invention provides a wireframe-based image plane detection method, the method comprising:

[0007] Input the image to be detected, and its corresponding wireframe segment;

[0008] All wireframe segments are rendered into a state matrix with the same resolution as the image to be detected. Each element in the matrix indicates the information of the segment covering the location of that element, which is achieved by setting the element value to the segment ID of the segment covering the element.

[0009] Using morphological methods, the near endpoints of the wireframe segments are merged by first dilating and then eroding the wireframe segments;

[0010] A closed region is obtained by using a region seed filling algorithm; the closed region is composed of multiple line segments.

[0011] The closed region is filtered based on the pixel variance of the region to obtain the final planar detection result, which is then output.

[0012] Furthermore, the morphological method includes: expansion followed by corrosion.

[0013] Furthermore, the filtering of the closed region based on the region's pixel variance includes:

[0014] For a closed region, calculate its pixel mean and variance. If the pixel variance is greater than a given threshold, delete the closed region. If the variance is less than or equal to the given threshold, add the closed region to the output plane set.

[0015] Furthermore, a second aspect of the present invention provides an image plane detection system based on wireframes, the system comprising an input module, a line graph generation module, a morphological operation module, a closed region generation module, and a region filtering module, characterized in that:

[0016] The input module is used to input the image to be detected, as well as its corresponding wireframe segments;

[0017] The line graph generation module renders all wireframe segments into a state matrix with the same resolution as the image to be detected. Each element in the matrix indicates the line information covering the location of that element, which can be achieved by setting the element value to the segment ID of the covered line segment.

[0018] The morphological manipulation module is used to merge adjacent endpoints in the Wireframe segment using morphological methods.

[0019] A closed region generation module is used to find and obtain closed regions based on a region seed filling algorithm; the closed region is composed of multiple line segments;

[0020] The region filtering module is used to filter the closed region based on the pixel variance of the region to obtain the final planar detection image and output it.

[0021] Furthermore, a third aspect of the present invention provides an electronic device comprising: one or more processors, and a memory for storing one or more computer programs; characterized in that the computer programs are configured to be executed by the one or more processors, the programs comprising steps for performing the wireframe-based image plane detection method as described in the first aspect.

[0022] Furthermore, a fourth aspect of the present invention provides a storage medium storing a computer program; the program is loaded and executed by a processor to implement the steps of the wireframe-based image plane detection method as described in the first aspect.

[0023] Compared to existing technologies, firstly, by leveraging the characteristic that wireframes primarily contain scene structure lines, which are typically intersecting planes in an image, this approach enables image plane detection using wireframes as planar boundary lines, without employing semantic segmentation or instance segmentation algorithms to determine whether pixels are on the same plane. This avoids the need for bitmap vectorization to obtain the final vectorized planar representation, thus preventing the jagged edges that may occur during bitmap vectorization. Secondly, it uses the image plane represented by wireframes to parse the scene structure. Compared to using endpoints in wireframes, line segment representation is more advantageous, and the point-line structure is inherent in the closed plane represented by wireframe line segments proposed in this paper. Thirdly, morphological operations are used to merge closely spaced endpoints, resolving the endpoint error problem that may occur in wireframe line segment detection; and filtering the detection plane based on in-plane pixel variance solves the problem of merging multiple planes caused by missing wireframe line segment detection. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of the image plane detection method based on wireframe disclosed in an embodiment of the present invention;

[0026] Figure 2 This is a flowchart illustrating the subjective effect of wireframe-based planar detection as disclosed in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the structure of the Wireframe-based image plane detection system disclosed in an embodiment of the present invention. Detailed Implementation

[0028] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0029] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0032] It should be noted that "multiple" as mentioned in this article refers to two or more.

[0033] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0034] In existing technologies, plane detection algorithms are usually semantic segmentation algorithms or panoptic segmentation algorithms. They typically determine whether each pixel is on a certain plane, and the algorithm results are presented in bitmap form. To achieve vectorized output results, the bitmap needs to be vectorized.

[0035] This implementation example discovers that image plane detection is the detection of polygonal regions in an image, and that polygonal regions are composed of a series of line segments connected end to end. Therefore, the plane detection task can be completed from the bottom up by detecting the line segments. Based on this discovery, this implementation proposes a simple algorithm for image plane detection from wireframe line segments.

[0036] Please see Figure 1 , Figure 1This is a schematic flowchart of an image plane detection method based on a wireframe, as disclosed in an embodiment of the present invention. Figure 1 As shown, the image plane detection method based on wireframe includes:

[0037] S100, Input the image to be detected, and its corresponding wireframe segment.

[0038] Specifically, in this embodiment, step S100 corresponds to the first stage of this embodiment, that is, inputting the image to be detected and its wireframe segments. The wireframe segments of the image can be obtained using existing wireframe detection algorithms. In this example, the FClip algorithm (Xili Dai, Haigang Gong, Shuai Wu, Xiaojun Yuan, Yi Ma. "Fully Convolutional Line Parsing.") is used to obtain the wireframe segments in the image.

[0039] S200: Render all wireframe segments into a state matrix with the same resolution as the image to be detected. Each element in the matrix indicates the information of the segment covering the position of that element, which is achieved by setting the element value to the segment ID of the segment covering the segment.

[0040] Specifically, in this embodiment, step S200 corresponds to the second stage of this embodiment, that is, rendering all detected line segments to a state matrix, i.e., a line graph. The construction of the line graph requires setting the pixel state on the line to the line segment ID.

[0041] Specifically, in this embodiment, the detected wireframe segments are rendered onto a state matrix with the same dimension and resolution as the wireframe segment, and a specific number is used to represent the initial state; in this example, 0 represents the initial state. Each element in the state matrix is ​​sequentially checked to see if it lies on any line segment. Assuming (i,j) lies on the k-th detected line segment, the state of the element at position (i,j) is set to the line segment information of the k-th line segment. In this embodiment, the line segment ID is used as this information, and the line segment's sequence number among all detected line segments is used as its ID. Since the dimension of the state matrix is ​​consistent with the resolution of the line segments and the resolution of the image to be detected, the state matrix can be considered a single-channel image with the same resolution as the image to be detected. The value of each element in the state matrix indicates which line segment covers the element's position; therefore, in this embodiment, the state matrix is ​​referred to as a line graph.

[0042] In practice, we use linear rasterization algorithms such as the basic incremental algorithm (DDA) and the Bresenham algorithm to obtain the set of pixels on a single line segment, and set the state of the pixels in the set to the ID of the line segment. It can be seen that this strategy can effectively determine whether a pixel is on a line, thus achieving the purpose of constructing a line graph.

[0043] S300, use morphological methods to merge adjacent endpoints in the wireframe segment.

[0044] Furthermore, the morphological method includes: expansion followed by corrosion.

[0045] Specifically, in this embodiment, step S300 corresponds to the third stage of this embodiment, namely, the morphological closing operation stage. Since there is a certain error in wireframe segment detection, multiple line segments in the real world intersect at the same vertex, such as a corner vertex. In the detection results, the endpoints may not perfectly coincide. This embodiment uses morphological operations to first dilate and then erode, i.e., closing operations merge endpoints that are close together, to alleviate the problem of line segments not being closed due to endpoint detection errors. For example, there is a real plane in the image with boundary line segments l1 and l2, where the endpoints of l1 are A and B, and the endpoints of line segment l2 are B and C, i.e., line segments l1 and l2 intersect at endpoint B. Assume the real coordinates of point B are (10, 3), and the line segment detected by the model... The coordinates of endpoint B1 corresponding to B are (10, 2), and the detected line segment... The coordinates of endpoint B2, which corresponds to endpoint B, are (10, 4). It can be observed that due to model detection errors, endpoints B1 and B2 do not coincide with the true endpoint B, resulting in the detected line segment... There are gap pixels (10,3) between them, and the presence of these gaps causes the detection line segments to be affected. It cannot form a closed surface. To solve this problem, we need to fill the gap pixel (10,3). Thanks to the fact that the states of pixels (10,2) and (10,4) in the line graph are already represented as the line segment ID to which the pixel belongs, we only need to set the state of pixel (10,3) to the state edge of its neighboring pixel to achieve planar closure.

[0046] S400, a closed region is obtained based on the region seed filling algorithm; the closed region is composed of multiple line segments.

[0047] Specifically, in this embodiment, step S400 corresponds to the fourth stage of this embodiment, namely, the stage of finding closed regions based on the region seed filling algorithm. Multiple closed regions composed of line segments are formed in the line graph. This embodiment uses the region seed algorithm to discover closed regions in the line graph. During the region growth process, this embodiment records the state of the encountered pixels, thus obtaining the line segment information that encloses the closed region. Specifically, using Depth-First Search (DFS) or Breadth-First Search (BFS), starting from any pixel, it traverses the connected regions adjacent to it; this traversal process is the region growth process. Recording the intersection points of the encountered line segments during the region growth process yields the vertices of the closed region. Line segment intersection points can be easily determined through pixel neighborhood. For example, if there are two different line segments, line segment a and line segment b, in the neighborhood of the current pixel, then there is an endpoint in the current plane that is the intersection point of line segment a and line segment b.

[0048] S500: Filter the closed region based on the pixel variance of the region to obtain the final detection plane and output it.

[0049] Filtering the closed region based on the pixel variance of the region includes: for the set of pixels in the closed region, calculating the mean and variance of the corresponding pixels; if the pixel variance is greater than a given threshold, it indicates that the pixel consistency of the closed region is poor, and the closed region can be deleted; if the variance is less than or equal to the given threshold, the closed region is added to the output plane set.

[0050] Specifically, in this embodiment, step S500 corresponds to the fifth stage of this embodiment, namely, the region filtering stage. Because some wireframe segments are missing during wireframe detection, multiple planar regions may be merged into a larger planar region. For example, in the actual annotation, the dividing line between plane A and plane B is line l. If the dividing line l is missing in the wireframe segments, the second stage of this embodiment will merge plane A and plane B into a new plane, which we can denote as C. However, since A and B are different planes, there are certain differences between their pixels, resulting in the pixel variance of plane C being significantly greater than that of plane A and plane B. This embodiment filters candidate planes based on the pixel variance of the region. For the same closed region, this embodiment calculates its pixel mean and variance. If the pixel variance is greater than a given threshold, this embodiment can delete the candidate region. If the variance meets the threshold requirement, the region is added to the output plane set.

[0051] The following is the pseudocode for the planar detection algorithm of the Wireframe in this embodiment.

[0052]

[0053] This embodiment is divided into five stages: The first stage involves inputting the image to be detected and its wireframe segments. The second stage generates a line graph. The third stage performs morphological operations, first dilation and then erosion, merging endpoints that are close together. The fourth stage uses a region seed filling algorithm to find closed regions in the line graph. The fifth stage filters the found closed regions. Figure 2 This is a flowchart illustrating the subjective effect of wireframe-based planar detection in this embodiment.

[0054] This implementation builds upon the work on wireframes, detecting polygonal planes in images using wireframes and using the polygonal plane structure formed by closed wireframes to analyze the scene geometry in the image. This implementation argues that vertices and structure lines alone are insufficient to fully express the 3D structural information contained in an image. Planes represented by polygons formed by closed wireframes can more intuitively and comprehensively describe the 3D scene structure in an image. The geometric information of a polygonal plane can represent a region and its boundaries, while structure lines can only represent the geometric boundaries of objects. Therefore, relatively speaking, planes formed by polygonal wireframes contain richer geometric information, including information from both vertices and structure lines detected by wireframes. The vertices of the closed polygonal faces are the vertices of the structure lines, and the edges of the polygonal faces are the structure lines.

[0055] In summary, this implementation, based on the wireframe approach of using independent, loosely defined geometric primitives such as points and structural lines to parse image scene structure, further transforms this approach into using closed wireframe polygons composed of planar vertices to parse scene structure. Compared to previous wireframe works that only used vertices and structural lines to parse scene structure, the closed wireframe polygon representation is richer in its representation of scene structure semantics. The plane represented by a polygon composed of planar vertices contains not only information about the plane's vertices and boundary structural lines, but also information about the plane's regions. Therefore, this invention argues that using closed wireframe polygons to parse scene structure is more advantageous than using isolated points and structural lines. This invention also directly outputs vectorized results, avoiding the jagged edges that may occur when segmenting bitmaps.

[0056] Furthermore, a second aspect of the present invention provides an image plane detection system based on wireframes, the system comprising an input module 10, a line graph generation module 20, a morphological operation module 30, a closed region detection and generation module 40, and a region filtering module 50, characterized in that:

[0057] Input module 10 is used to input the image to be detected and its corresponding wireframe line segment;

[0058] The line graph generation module 20 renders all wireframe segments onto a state matrix, i.e., a line graph, with the same resolution as the input image; each element in the line graph indicates whether the pixel's location is covered by a line segment.

[0059] Morphological operation module 30 is used to merge adjacent endpoints in the Wireframe segment using morphological methods;

[0060] The closed region generation module 40 is used to find and obtain closed regions based on the region seed filling algorithm; the closed region is composed of multiple line segments.

[0061] The region filtering module 50 is used to filter the closed region based on the pixel variance of the region to obtain the final planar detection result and output it.

[0062] In the morphological operation module 30, the morphological method includes: expansion followed by erosion.

[0063] In the region filtering module 50, the filtering of the closed region based on the pixel variance of the region includes: for the closed region, calculating its pixel mean and variance; if the pixel variance is greater than a given threshold, deleting the closed region; if the variance is less than or equal to the given threshold, adding the closed region to the output plane set.

[0064] Furthermore, embodiments of this application also disclose an electronic device comprising: one or more processors, and a memory for storing one or more computer programs; characterized in that the computer programs are configured to be executed by the one or more processors, and the programs include steps for performing the Wireframe-based image plane detection method described in the first aspect above.

[0065] Furthermore, embodiments of this application also provide a storage medium storing a computer program; the program is loaded and executed by a processor to implement the steps of the wireframe-based image plane detection method described in the first aspect above.

[0066] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0067] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0068] The units described as separate components may or may not be physically separate. As will be appreciated by those skilled in the art, the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0069] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence or as a contribution to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or grid device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting image planes based on wireframes, characterized in that, The method includes: Input the image to be detected, and its wireframe segments; All wireframe segments are rendered into a state matrix with the same resolution as the image to be detected. Each element in the matrix indicates the information of the segment covering the location of that element, which is achieved by setting the element value to the segment ID of the segment covering the segment. Merging of adjacent endpoints in the wireframe segments is performed using morphological methods; these morphological methods include: dilation followed by erosion. A closed region is found and obtained based on a region seed filling algorithm; the closed region is composed of multiple line segments. The closed region is filtered based on the pixel variance of the region to obtain the final planar detection result, which is then output. The filtering of the closed region based on the region's pixel variance includes: For a closed region, calculating its pixel mean and variance; if the pixel variance is greater than a given threshold, deleting the closed region; if the variance is less than or equal to the given threshold, adding the closed region to the output plane set.

2. A wireframe-based image plane detection system, characterized in that, The system includes an input module, a line graph generation module, a morphological operation module, a closed region generation module, and a region filtering module, characterized in that: The input module is used to input the image to be detected, as well as its wireframe segments; The line graph generation module renders all wireframe segments into a state matrix with the same resolution as the image to be detected. Each element in the matrix indicates the information of the line segment covering the position of that element, which is achieved by setting the element value to the line segment ID of the covering line segment. A morphological operation module is used to merge adjacent endpoints in the wireframe segment using morphological methods; the morphological methods include: dilation followed by erosion; A closed region generation module is used to find and obtain closed regions based on a region seed filling algorithm; the closed region is composed of multiple line segments; The region filtering module is used to filter the closed region based on the pixel variance of the region to obtain the final planar detection result and output it. The filtering of the closed region based on the region's pixel variance includes: For a closed region, calculating its pixel mean and variance; if the pixel variance is greater than a given threshold, deleting the closed region; if the variance is less than or equal to the given threshold, adding the closed region to the output plane set.

3. An electronic device, the electronic device comprising: One or more processors, a memory for storing one or more computer programs; characterized in that the computer programs are configured to be executed by the one or more processors, the programs including steps for performing the wireframe-based image plane detection method as described in claim 1.

4. A storage medium storing a computer program; the program being loaded and executed by a processor to implement the steps of the wireframe-based image plane detection method as described in claim 1.

Images