Panoramic three-dimensional instance recovery method and system based on single image, terminal and storage medium

Abstract

The invention discloses a panoramic three-dimensional instance recovery method and system based on a single image, a terminal and a storage medium, and the method comprises the steps: two-dimensional information extraction of a single RGB image, feature dimension raising based on the two-dimensional information, refining of three-dimensional semantic information and instance grouping, and multi-level shielding detection and shielding elimination. And synthesizing the foreground and the background of the panoramic three-dimensional instance so as to obtain a complete three-dimensional scene reconstruction result. According to the method, panoramic three-dimensional reconstruction of an indoor scene in a sheltered environment is realized by utilizing a multi-level foreground and background segmentation and sheltering elimination technology, automatic elimination of a sheltered area can be realized, and an image without sheltering is obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, terminal and computer-readable storage medium for restoring a panoramic three-dimensional instance based on a single image. Background Art

[0002] Panoramic 3D reconstruction of 3D scenes usually requires a lot of preliminary data collection work. However, the main problems faced in this process are: Data requirements for high-quality models: To generate high-quality 3D scenes, it is usually necessary to collect a large amount of comprehensive data. However, achieving this goal may be difficult due to terrain restrictions. Terrain restrictions may involve limited access to geographical areas, inability to obtain data in certain specific locations, or inability to obtain enough detailed information in certain areas.

[0003] Data quality issues: Even if enough data is successfully collected, there may still be occlusion and missing data. Occlusion problems may be caused by the relative positions of objects, which results in some areas not being fully captured. In addition, data incompleteness may be caused by sensor limitations or technical limitations.

[0004] The single-image panoramic 3D instance recovery technology is dedicated to solving such problems. Starting from a single RGB image (RGB image is a kind of image of various colors generated based on different intensities of the three basic colors red, green, and blue), it unifies the panoramic segmentation and generates a 3D instance model with panoramic segmentation results. This technology can use a small number of or a single image at a specific angle to infer the overall 3D scene, effectively reducing the above data collection work. However, the existing single-image panoramic 3D instance recovery method performs poorly in occluded environments. While using depth prediction from a single view, it can only propagate 2D information to the front surface of the camera cone. The reconstruction of the rear area of ​​the front surface after reconstruction is ambiguous and it is difficult to handle images with more complex occlusions.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide a panoramic three-dimensional instance recovery method, system, terminal and computer-readable storage medium based on a single image, aiming to solve the problem that the single-image panoramic three-dimensional instance recovery method in the prior art performs poorly in an occluded environment, and can only propagate two-dimensional information to the front surface of the camera cone while using depth prediction from a single view. The reconstruction of the rear area of ​​the front surface after reconstruction is ambiguous and it is difficult to handle images with more complex occlusion situations.

[0007] To achieve the above object, the present invention provides a method for restoring a panoramic 3D instance based on a single image, the method comprising the following steps: Extracting two-dimensional information of a single RGB image, the two-dimensional information including two-dimensional semantic information, instance center information, depth information, and multi-plane occupancy information; According to the depth information, the two-dimensional semantic information is converted into three-dimensional semantic information, and meaningless areas in the three-dimensional semantic information are extracted according to the multi-plane occupancy information to obtain three-dimensional lifting features; Refining the 3D lifting features, performing 3D reconstruction based on the refined features, and when performing 3D panorama segmentation, grouping instances based on the reconstructed features to obtain 3D instance IDs, and determining whether a single RGB image has occlusions. If there are no occlusions, combining the results of grouping the 3D instances to obtain a 3D scene reconstruction result of a single RGB image under unoccluded conditions; If there is occlusion, the occlusion relationship learning method is used to extract all instances in the foreground part of a single RGB image, and the image restoration technology is used to remove the occlusion, and the multiple prediction results are combined into a complete 3D scene reconstruction result.

[0008] Optionally, in the single-image-based panoramic 3D instance restoration method, the step of extracting 2D information of a single RGB image specifically includes: Given a single RGB image , using a two-dimensional model Extract features to learn 2D prior knowledge and get a single RGB image Two-dimensional information: ; in, , indicating that it contains The two-dimensional semantic information of categories, and Respectively represent the height and width of a single RGB image; , represents depth information; , indicating the prediction Instance center information with corresponding category labels; , represents the predicted multi-plane occupancy information, indicating different depths Whether the plane is occupied by 3D objects or objects.

[0009] Optionally, the panoramic three-dimensional instance restoration method based on a single image, wherein the two-dimensional semantic information is converted into three-dimensional semantic information according to the depth information, and meaningless areas in the three-dimensional semantic information are extracted according to the multi-plane occupancy information to obtain three-dimensional lifting features, specifically includes: Two-dimensional semantic information Convert to 3D semantic information : ; Wherein, the three-dimensional semantic information Covering objects and areas throughout the space; Represents the camera's internal matrix; Indicates location The depth of u and v Represents two dimensions on a two-dimensional image; z Indicates depth, area Indicates free space; Leveraging multi-plane occupancy information Extracting 3D semantic information The meaningless areas in the image are used to obtain three-dimensional lifting features. I 3d , three-dimensional lifting features I 3d 3D semantic information and 3D occupancy information The product of is calculated as: ; ; in, Conv Represents a Conv-BN-ReLU module; Represents element-wise product.

[0010] Optionally, the panoramic 3D instance restoration method based on a single image, wherein the 3D lifting features are refined, 3D reconstruction is performed according to the refined features, and when performing 3D panoramic segmentation, instances are grouped according to the reconstructed features to obtain 3D instance IDs, and it is determined whether there is occlusion in the single RGB image. If there is no occlusion, the results of grouping the 3D instances are combined to obtain the 3D scene reconstruction result of the single RGB image under the condition of no occlusion, specifically including: Leveraging a 3D encoder-decoder model Predict refined 3D semantic information, 3D offset, and 3D occupancy information: ; in, Respectively represent the refined 3D semantic information, 3D offset and 3D occupancy information; When performing 3D reconstruction, the 3D occupancy information is used to guide the Obtain reconstructed 3D semantic information ,pass Get the reconstructed 3D offset ; When performing 3D panoptic segmentation, an instance identifier is assigned to each object’s voxel, and instance grouping is performed using instance center information, reconstructed 3D offset, and reconstructed 3D semantic information; It is determined whether a single RGB image has occlusion. If there is no occlusion, the results of grouping the 3D instances are combined to obtain the 3D scene reconstruction result of a single RGB image under unoccluded conditions.

[0011] Optionally, the method for restoring a panoramic 3D instance based on a single image, wherein the instance grouping is performed using instance center information, reconstructed 3D offset and reconstructed 3D semantic information, specifically comprising: The reconstructed 3D offset and the reconstructed 3D semantic information Convert to three-dimensional offset in multiple planes and corresponding categories k Multi-plane 3D semantic information : ; ; in, Indicates the corresponding category k The reconstructed 3D semantic information; Obtain the corresponding categories under multiple planes k The three-dimensional offset : ; Information by Example Center Index out categories k Instance Center , the category k Instance center and multi-plane category k The three-dimensional offset Combined, the 3D instance centers on multiple planes are grouped to obtain deterministic instance IDs; ; in, Indicates the depth z The predicted instance ID; express k Category j Instances; Indicates j Categories k The 2D instance center of Indicates category k In Depth z Down The three-dimensional offset at Quantity; Indicates category k In Depth z Down The three-dimensional offset at Quantity; Calculate at location Category 3D Instance ID: ; in, Indicates at location Category k The 3D instance ID of the

[0012] Optionally, the panoramic 3D instance restoration method based on a single image, wherein if there is occlusion, an occlusion relationship learning method is used to extract all instances located in the foreground part of a single RGB image, and occlusion culling is performed using image restoration technology, and multiple prediction results are combined into a complete 3D scene reconstruction result, specifically including: Combining the two-dimensional semantic information with the instance center information to obtain two-dimensional instance information, wherein the two-dimensional instance information represents an instance mask of the object, and preliminarily judging whether there is occlusion according to whether the instance mask is tangent; If the instance masks are tangent, obtain an instance boundary line between the objects where the occlusion area may exist, where the instance boundary line is composed of pixel points adjacent to the boundaries of the two objects; The occlusion relationship of objects in a single RGB image is parsed through occlusion relationship learning, and the occlusion relationship is determined based on the relationship between the boundary lines of different instances and the occlusion directions. If there are two or more objects blocking the current area, create a directed graph by comparing the objects in pairs, and obtain the foreground part of the directed graph; After determining the foreground area and the foreground instance ID, perform pixel dilation operation, use occlusion culling or image restoration technology to remove the foreground part, and obtain the newly generated image. ,image For a single RGB image Remove the occluded foreground and restore the result of subsequent objects, and retain the 3D instance content corresponding to the instance ID , if occlusion culling occurs, then Indicates the instance ID part that has been eliminated by occlusion culling. If occlusion culling does not occur, it means that there is no occluded single RGB image at present The corresponding three-dimensional instance content; The 3D instance content corresponding to the instance ID retained by multiple occlusion culling Integrate into a unified target model to obtain a single RGB image The corresponding panoramic 3D instance content represents a 3D scene reconstruction result.

[0013] Optionally, in the single-image-based panoramic three-dimensional instance restoration method, the target model is a three-dimensional model that includes a semantic instance ID contained in the panoramic segmentation and has a restoration degree that meets preset requirements.

[0014] In addition, to achieve the above-mentioned purpose, the present invention further provides a panoramic three-dimensional instance restoration system based on a single image, wherein the panoramic three-dimensional instance restoration system based on a single image comprises: An information extraction module is used to extract two-dimensional information of a single RGB image, wherein the two-dimensional information includes two-dimensional semantic information, instance center information, depth information, and multi-plane occupancy information; A feature dimension enhancement module, used for converting the two-dimensional semantic information into three-dimensional semantic information according to the depth information, extracting meaningless areas in the three-dimensional semantic information according to the multi-plane occupancy information, and obtaining three-dimensional enhancement features; An instance recovery module is used to refine the 3D lifting features, perform 3D reconstruction based on the refined features, and when performing 3D panorama segmentation, group instances based on the reconstructed features to obtain 3D instance IDs, and determine whether a single RGB image has occlusions. If there are no occlusions, the results of grouping the 3D instances are combined to obtain a 3D scene reconstruction result of a single RGB image under unoccluded conditions; The occlusion culling module is used to extract all instances in the foreground part of a single RGB image using the occlusion relationship learning method if there is occlusion, and use image restoration technology to perform occlusion culling, and synthesize multiple prediction results into a complete 3D scene reconstruction result.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a panoramic three-dimensional instance recovery program based on a single image stored in the memory and executable on the processor, wherein the panoramic three-dimensional instance recovery program based on a single image, when executed by the processor, implements the steps of the panoramic three-dimensional instance recovery method based on a single image as described above.

[0016] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a panoramic three-dimensional instance recovery program based on a single image, and when the panoramic three-dimensional instance recovery program based on a single image is executed by a processor, the steps of the panoramic three-dimensional instance recovery method based on a single image as described above are implemented.

[0017] In the present invention, the two-dimensional information of a single RGB image is extracted, and the two-dimensional information includes two-dimensional semantic information, instance center information, depth information and multi-plane occupancy information; according to the depth information, the two-dimensional semantic information is converted into three-dimensional semantic information, and the meaningless area in the three-dimensional semantic information is extracted according to the multi-plane occupancy information to obtain a three-dimensional lifting feature; the three-dimensional lifting feature is refined, and three-dimensional reconstruction is performed according to the refined features. When performing three-dimensional panoramic segmentation, the instances are grouped according to the reconstructed features to obtain a 3D instance ID, and it is determined whether there is occlusion in the single RGB image. If there is no occlusion, the results of grouping the 3D instances are combined to obtain a three-dimensional scene reconstruction result of a single RGB image under the condition of no occlusion; if there is occlusion, the occlusion relationship learning method is used to extract all instances located in the foreground part of the single RGB image, and the image restoration technology is used to perform occlusion culling, and multiple prediction results are combined into a complete three-dimensional scene reconstruction result. The present invention uses multi-level foreground and background segmentation and occlusion culling technology to realize panoramic three-dimensional reconstruction of indoor scenes under occlusion environments, and can realize automatic culling of occluded areas and obtain images after occlusion removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a preferred embodiment of a method for restoring a panoramic three-dimensional instance based on a single image of the present invention; Figure 2 It is a flow chart of a single image panoramic 3D instance restoration process based on multi-level occlusion detection and occlusion culling in a preferred embodiment of the single image based panoramic 3D instance restoration method of the present invention; Figure 3 It is a flowchart of multi-level occlusion detection and occlusion removal in a preferred embodiment of a panoramic three-dimensional instance restoration method based on a single image of the present invention; Figure 4 It is a schematic diagram of a method for representing occlusion direction prediction in a preferred embodiment of a panoramic three-dimensional instance restoration method based on a single image of the present invention; Figure 5 It is a structural diagram of a preferred embodiment of a panoramic three-dimensional instance restoration system based on a single image of the present invention; Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] The method for restoring a panoramic three-dimensional instance based on a single image according to a preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the panoramic 3D instance restoration method based on a single image includes the following steps: Step S10: extracting two-dimensional information of a single RGB image, wherein the two-dimensional information includes two-dimensional semantic information, instance center information, depth information, and multi-plane occupancy information.

[0021] Specifically, given a single RGB image It is very difficult to learn three-dimensional information directly from a single two-dimensional image, so the present invention uses a two-dimensional model Extract features (referring to the parameters learned in each level of the model in deep learning to predict two-dimensional information or the special operating mechanism and rules of the model to correctly identify two-dimensional prior knowledge) to learn rich two-dimensional prior knowledge and obtain a single RGB image Two-dimensional information: ; in, , indicating that it contains The two-dimensional semantic information of categories, and Respectively represent the height and width of a single RGB image; , represents depth information; , indicating the prediction with corresponding category labels ( ) instance center information ( ); , represents the predicted multi-plane occupancy information, which indicates the different depths Whether the plane is occupied by 3D objects or objects.

[0022] Step S20: convert the two-dimensional semantic information into three-dimensional semantic information according to the depth information, extract meaningless areas in the three-dimensional semantic information according to the multi-plane occupancy information, and obtain three-dimensional lifting features.

[0023] Specifically, after extracting the learned two-dimensional prior information (i.e., two-dimensional information), it is necessary to upgrade the two-dimensional prior information to three-dimensional features for subsequent use; first, the two-dimensional semantic information Convert to 3D semantic information ,The 3D semantic information is not limited to the front surface of the object, but can cover objects and areas in the entire space.

[0024] ; Wherein, the three-dimensional semantic information Covering objects and areas throughout the space; Represents the camera's internal matrix; Indicates location The depth of u and v Represents two dimensions on a two-dimensional image; z Indicates depth, area Indicates free space.

[0025] Then, using multi-plane occupancy information Extracting 3D semantic information The meaningless areas in the image are used to obtain three-dimensional lifting features. I 3d (i.e., enhanced 3D features), three-dimensional enhanced features I 3d 3D semantic information and 3D occupancy information The product of is calculated as: ; ; in, Conv Represents a Conv-BN-ReLU module. Conv-BN-ReLU is a commonly used convolutional neural network structure. Its function is to add batch normalization (BN) and rectified linear unit (ReLU) activation functions after the convolution layer to improve the performance and training speed of the model. Represents element-wise product.

[0026] The multi-plane occupancy method can provide shape clues for the occluded area, so the improved features can be used as a good initial value for subsequent 3D optimization, greatly reducing the pressure of 3D model optimization.

[0027] Through this step, the two-dimensional semantic information is promoted to the deterministic channel of the three-dimensional voxel feature by relying on the multi-plane occupancy information and depth information. At this time, the feature contains both the front-view surface feature of the instance and the features of the instance and the background part.

[0028] Step S30, refine the 3D lifting features, perform 3D reconstruction based on the refined features, and when performing 3D panoramic segmentation, group instances based on the reconstructed features to obtain 3D instance IDs, and determine whether there is occlusion in a single RGB image. If there is no occlusion, combine the results of grouping the 3D instances to obtain a 3D scene reconstruction result of a single RGB image under unoccluded conditions.

[0029] Specifically, usually, the 3D lifting features obtained in the previous step are relatively coarse and cannot be directly used for panoramic 3D instance restoration. Therefore, the present invention utilizes a powerful 3D encoder-decoder model Predict refined 3D semantic information, 3D offset, and 3D occupancy information: ; in, They represent the refined 3D semantic information, 3D offset and 3D occupancy information respectively.

[0030] The recovery of panoramic 3D instances requires both 3D reconstruction and 3D panoramic segmentation.

[0031] When performing 3D reconstruction, the 3D occupancy information is used to guide the Obtain reconstructed 3D semantic information ,pass Get the reconstructed 3D offset .

[0032] When performing 3D panoptic segmentation, it is necessary to assign instance identifiers to the voxels of each object, and use the instance center information, the reconstructed 3D offset, and the reconstructed 3D semantic information to group the instances. The specific process is as follows: First, the reconstructed three-dimensional offset and the reconstructed 3D semantic information Convert to three-dimensional offset in multiple planes and corresponding categories k Multi-plane 3D semantic information : ; ; in, Indicates the corresponding category k The reconstructed three-dimensional semantic information; thus obtaining the corresponding categories under multiple planes k The three-dimensional offset : .

[0033] At the same time, you can use the instance center information Index out categories k Instance Center , and finally the category k Instance center and multi-plane category k The three-dimensional offset Combined with the above, the 3D instance centers on multiple planes are grouped to obtain deterministic instance IDs, avoiding the uncertainty of instance ID assignment caused by semantic segmentation alone.

[0034] ; in, Indicates the depth z The predicted instance ID; express k Category j Instances; Indicates j Categories k The 2D instance center of Indicates category k In Depth z Down The three-dimensional offset at Quantity; Indicates category k In Depth z Down The three-dimensional offset at Quantity.

[0035] Calculate at location Category 3D Instance ID: ; in, Indicates at location Category k The 3D instance ID of the

[0036] It is determined whether a single RGB image has occlusion. If there is no occlusion, the results of grouping the 3D instances are combined to obtain the 3D scene reconstruction result of a single RGB image under unoccluded conditions.

[0037] That is, by combining 3D semantic information with the result of grouping 3D instances by the instance grouping module, the result of predicting panoramic 3D scene reconstruction in a single image under unoccluded conditions can be obtained.

[0038] Step S40: If there is occlusion, use the occlusion relationship learning method to extract all instances in the foreground part of the single RGB image, and use image restoration technology to remove the occlusion, and combine multiple prediction results into a complete three-dimensional scene reconstruction result.

[0039] Specifically, for an image, if the number of object elements is small, the entire image can be predicted directly. However, when there are multiple objects occluding each other in the image, the above method will face great challenges. Therefore, the present invention designs a multi-level occlusion detection and occlusion culling method, aiming to maximize the occlusion of each object. Restore to its state before being blocked , so as to realize the panoramic 3D instance recovery more smoothly and unhindered. The specific process is as follows Figure 3 shown.

[0040] First, the two-dimensional semantic information is combined with the instance center information to obtain the two-dimensional instance information, which represents the instance mask of the object. Whether there is occlusion is preliminarily determined based on whether the instance masks are tangent. If there is tangent instance mask, the instance boundary between objects with possible occlusion areas is obtained, and the instance boundary is composed of pixel points adjacent to each other at the boundaries of the two objects. The occlusion relationship of objects in a single RGB image is parsed through occlusion relationship learning, and the occlusion relationship is determined based on the relationship between the instance boundary lines and the occlusion direction. The occlusion direction is represented by a directed arrow, and the arrow is pointing to the left foreground ( Figure 4 fg in the figure), with the background on the right ( Figure 4 bg in), denoted as the direction vector indicating the occlusion direction, expressed as Figure 4 shown.

[0041] If there are two or more objects in the current area that are blocked, the objects are compared with each other to create a directed graph, and the most foreground part in the directed graph is obtained; if there is no part that is tangent to the instance mask, it can be determined that the area is not blocked; on the contrary, according to the position of the dividing line and the direction vector of the blocking direction (the angle represented by each pixel on the curve representing the blocking direction (expressed in radians)), the predicted direction vector can be rotated 90° counterclockwise and compared with the direction vector pointing to the inside of the object from the outer contour of the object. If the two are in the same direction, it means that the object in this area is the foreground of the local position, and if the two are in opposite directions, it is the background. The above part is the judgment method between two objects. If there are two or more objects in the area that are blocked, the objects are compared with each other and then a directed graph is created. Using the transitivity of the directed graph, the most foreground object in the area or the joint mask area of ​​multiple objects can be determined, and the corresponding instance ID can be obtained.

[0042] Through the above steps, it is determined whether the image is blocked. If not, the result generated in step S30 is directly used as the result of panoramic 3D instance restoration; otherwise, after determining the foreground area and the foreground instance ID, a pixel expansion operation is performed, and the foreground part is removed using occlusion culling or image restoration technology to obtain a newly generated image. ,image For a single RGB image Remove the occluded foreground and restore the result of subsequent objects, and retain the 3D instance content corresponding to the instance ID , if occlusion culling occurs, then Indicates the instance ID part that has been eliminated by occlusion culling. If occlusion culling does not occur, it means that there is no occluded single RGB image at present The corresponding three-dimensional instance content then repeats the above steps S10 to S40.

[0043] The 3D instance content corresponding to the instance ID retained by multiple occlusion culling Integrate into a unified target model (the target model is a 3D model that includes the semantic instance IDs contained in the panoptic segmentation and has a restoration degree that meets the preset requirements) to obtain a single RGB image The corresponding panoramic 3D instance content represents a 3D scene reconstruction result.

[0044] The present invention firstly extracts two-dimensional information based on a single RGB image, including two-dimensional semantic information, instance center information, depth information and multi-plane occupancy information; then, relying on the multi-plane occupancy information and depth information, the two-dimensional semantic information is promoted to the deterministic channel of the three-dimensional voxel feature, at which time the feature includes both the front-view surface feature of the instance and the feature of the instance and the background part; then, the three-dimensional semantic information is refined and the foreground and background synthesis stage is entered, and the dense three-dimensional occupancy information, three-dimensional semantic segmentation information and three-dimensional offset in each voxel are reconstructed according to the obtained feature prediction, and the instance serial number is assigned to the three-dimensional voxel occupied by the instance according to the two-dimensional instance center and the three-dimensional offset, and then the occlusion relationship learning method is used to extract all the instances located in the foreground part of the image, and the image restoration technology is used to perform occlusion culling and repeat the above steps for the image after occlusion culling until the instances in the image are not occluded with each other; finally, the multiple prediction results are synthesized into a complete three-dimensional scene reconstruction result.

[0045] The present invention innovatively introduces occlusion detection and occlusion culling technology, based on a bottom-up single-image panoramic 3D reconstruction method, and utilizes an automated occlusion detection and occlusion culling framework to restore the hidden parts of the occluded objects as efficiently as possible, and realizes the panoramic 3D instance recovery under occlusion conditions layer by layer from the outside to the inside according to a multi-plane occupancy grid method, thereby forming a novel panoramic 3D instance recovery method.

[0046] At the same time, the present invention proposes an automated occlusion detection and occlusion removal method, which effectively determines the instance content of the foreground area by using the relationship between the occlusion direction prediction and the potential occlusion area boundary formed by two-dimensional instance segmentation, and combines the powerful application of image restoration technology to achieve automated removal of occlusion areas and obtain an image after occlusion removal.

[0047] Furthermore, if Figure 5 As shown, based on the above-mentioned panoramic 3D instance restoration method based on a single image, the present invention also provides a panoramic 3D instance restoration system based on a single image, wherein the panoramic 3D instance restoration system based on a single image includes: An information extraction module 51 is used to extract two-dimensional information of a single RGB image, wherein the two-dimensional information includes two-dimensional semantic information, instance center information, depth information, and multi-plane occupancy information; A feature dimension enhancement module 52, configured to convert the two-dimensional semantic information into three-dimensional semantic information according to the depth information, and extract meaningless areas in the three-dimensional semantic information according to the multi-plane occupancy information to obtain three-dimensional enhancement features; An instance recovery module 53 is used to refine the 3D lifting features, perform 3D reconstruction based on the refined features, and when performing 3D panorama segmentation, perform instance grouping based on the reconstructed features to obtain a 3D instance ID, and determine whether there is occlusion in a single RGB image. If there is no occlusion, combine the results of grouping the 3D instances to obtain a 3D scene reconstruction result of a single RGB image under the condition of no occlusion; The occlusion culling module 54 is used to extract all instances located in the foreground part of a single RGB image using an occlusion relationship learning method if there is occlusion, and to perform occlusion culling using image restoration technology, and to synthesize multiple prediction results into a complete three-dimensional scene reconstruction result.

[0048] Furthermore, if Figure 6 As shown, based on the above-mentioned method and system for restoring a panoramic three-dimensional instance based on a single image, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0049] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a panoramic three-dimensional instance recovery program 40 based on a single image is stored on the memory 20, and the panoramic three-dimensional instance recovery program 40 based on a single image can be executed by the processor 10, thereby realizing the panoramic three-dimensional instance recovery method based on a single image in the present application.

[0050] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the panoramic three-dimensional instance restoration method based on a single image.

[0051] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The processor 10, the memory 20, and the display 30 of the terminal communicate with each other via a system bus.

[0052] In one embodiment, when the processor 10 executes the single-image-based panoramic 3D instance restoration program 40 in the memory 20 , the steps of the single-image-based panoramic 3D instance restoration method described above are implemented.

[0053] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a panoramic three-dimensional instance recovery program based on a single image, and when the panoramic three-dimensional instance recovery program based on a single image is executed by a processor, the steps of the panoramic three-dimensional instance recovery method based on a single image as described above are implemented.

[0054] In summary, the present invention provides a method, system, terminal and computer-readable storage medium for panoramic three-dimensional instance restoration based on a single image, the method comprising: extracting two-dimensional information of a single RGB image, the two-dimensional information comprising two-dimensional semantic information, instance center information, depth information and multi-plane occupancy information; according to the depth information, converting the two-dimensional semantic information into three-dimensional semantic information, extracting meaningless areas in the three-dimensional semantic information according to the multi-plane occupancy information, and obtaining three-dimensional lifting features; refining the three-dimensional lifting features, performing three-dimensional reconstruction according to the refined features, grouping instances according to the reconstructed features when performing three-dimensional panoramic segmentation, obtaining 3D instance IDs, judging whether there is occlusion in the single RGB image, and if there is no occlusion, combining the results of grouping the 3D instances to obtain a three-dimensional scene reconstruction result of the single RGB image under the condition of no occlusion; if there is occlusion, using an occlusion relationship learning method to extract all instances located in the foreground part of the single RGB image, and using image restoration technology to perform occlusion culling, and synthesizing multiple prediction results into a complete three-dimensional scene reconstruction result. The present invention utilizes multi-level foreground and background segmentation and occlusion culling technology to achieve panoramic three-dimensional reconstruction of indoor scenes under occlusion environments, can achieve automatic culling of occlusion areas, and obtain images after occlusion removal.

[0055] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0056] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0057] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for restoring a panoramic three-dimensional instance based on a single image, characterized in that: The panoramic three-dimensional instance restoration method based on a single image includes: Extracting two-dimensional information of a single RGB image, the two-dimensional information including two-dimensional semantic information, instance center information, depth information, and multi-plane occupancy information; According to the depth information, the two-dimensional semantic information is converted into three-dimensional semantic information, and meaningless areas in the three-dimensional semantic information are extracted according to the multi-plane occupancy information to obtain three-dimensional lifting features; Refining the 3D lifting features, performing 3D reconstruction based on the refined features, and when performing 3D panorama segmentation, grouping instances based on the reconstructed features to obtain 3D instance IDs, and determining whether a single RGB image has occlusions. If there are no occlusions, combining the results of grouping the 3D instances to obtain a 3D scene reconstruction result of a single RGB image under unoccluded conditions; If there is occlusion, the occlusion relationship learning method is used to extract all instances in the foreground part of a single RGB image, and the image restoration technology is used to remove the occlusion, and the multiple prediction results are combined into a complete 3D scene reconstruction result.

2. The method for restoring a panoramic 3D instance based on a single image according to claim 1, characterized in that: The extracting of two-dimensional information of a single RGB image specifically includes: Given a single RGB image , using a two-dimensional model Extract features to learn 2D prior knowledge and get a single RGB image Two-dimensional information: ； in, , indicating that it contains The two-dimensional semantic information of categories, and Respectively represent the height and width of a single RGB image; , represents depth information; , indicating the prediction Instance center information with corresponding category labels; , represents the predicted multi-plane occupancy information, indicating different depths Whether the plane is occupied by 3D objects or objects.

3. The method for restoring a panoramic 3D instance based on a single image according to claim 2, characterized in that: The converting the two-dimensional semantic information into three-dimensional semantic information according to the depth information, and extracting meaningless areas in the three-dimensional semantic information according to the multi-plane occupancy information to obtain three-dimensional lifting features specifically includes: Two-dimensional semantic information Convert to 3D semantic information : ； Wherein, the three-dimensional semantic information Covering objects and areas throughout the space; Represents the camera's internal matrix; Indicates location The depth of u and v Represents two dimensions on a two-dimensional image; z Indicates depth, area Indicates free space; Leveraging multi-plane occupancy information Extracting 3D semantic information The meaningless areas in the image are used to obtain three-dimensional lifting features. I 3d , three-dimensional lifting features I 3d 3D semantic information and 3D occupancy information The product of is calculated as: ； ； in, Conv Represents a Conv-BN-ReLU module; Represents element-wise product.

4. The method for restoring a panoramic 3D instance based on a single image according to claim 3, characterized in that: The three-dimensional lifting features are refined, and three-dimensional reconstruction is performed according to the refined features. When performing three-dimensional panoramic segmentation, instances are grouped according to the reconstructed features to obtain 3D instance IDs, and it is determined whether there is occlusion in a single RGB image. If there is no occlusion, the results of grouping the 3D instances are combined to obtain a three-dimensional scene reconstruction result of a single RGB image under the condition of no occlusion, specifically including: Leveraging a 3D encoder-decoder model Predict refined 3D semantic information, 3D offset, and 3D occupancy information: ； in, Respectively represent the refined 3D semantic information, 3D offset and 3D occupancy information; When performing 3D reconstruction, the 3D occupancy information is used to guide the Obtain reconstructed 3D semantic information ,pass Get the reconstructed 3D offset ; When performing 3D panoptic segmentation, an instance identifier is assigned to each object’s voxel, and instance grouping is performed using instance center information, reconstructed 3D offset, and reconstructed 3D semantic information; It is determined whether a single RGB image has occlusion. If there is no occlusion, the results of grouping the 3D instances are combined to obtain the 3D scene reconstruction result of a single RGB image under unoccluded conditions.

5. The method for restoring a panoramic 3D instance based on a single image according to claim 4, characterized in that: The instance grouping using the instance center information, the reconstructed 3D offset and the reconstructed 3D semantic information specifically includes: The reconstructed 3D offset and the reconstructed 3D semantic information Convert to three-dimensional offset in multiple planes and corresponding categories k Multi-plane 3D semantic information : ； ； in, Indicates the corresponding category k The reconstructed 3D semantic information; Obtain the corresponding categories under multiple planes k The three-dimensional offset : ； Information by Example Center Index out categories k Instance Center , the category k Instance center and multi-plane category k The three-dimensional offset Combined, the 3D instance centers on multiple planes are grouped to obtain deterministic instance IDs; ； in, Indicates the depth z The predicted instance ID; express k Category j Instances; Indicates j Categories k The 2D instance center of Indicates category k In Depth z Down The three-dimensional offset at Quantity; Indicates category k In Depth z Down The three-dimensional offset at Quantity; Calculate at location Category 3D Instance ID: ； in, Indicates at location Category k The 3D instance ID of the 6. The method for restoring a panoramic 3D instance based on a single image according to claim 5, characterized in that: If there is occlusion, the occlusion relationship learning method is used to extract all instances in the foreground part of a single RGB image, and the image restoration technology is used to remove the occlusion, and the multiple prediction results are combined into a complete 3D scene reconstruction result, which specifically includes: Combining the two-dimensional semantic information with the instance center information to obtain two-dimensional instance information, wherein the two-dimensional instance information represents an instance mask of the object, and preliminarily judging whether there is occlusion according to whether the instance mask is tangent; If the instance masks are tangent, obtain an instance boundary line between the objects where the occlusion area may exist, where the instance boundary line is composed of pixel points adjacent to the boundaries of the two objects; The occlusion relationship of objects in a single RGB image is parsed through occlusion relationship learning, and the occlusion relationship is determined based on the relationship between the boundary lines of different instances and the occlusion directions. If there are two or more objects blocking the current area, create a directed graph by comparing the objects in pairs, and obtain the foreground part of the directed graph; After determining the foreground area and the foreground instance ID, perform pixel dilation operation, use occlusion culling or image restoration technology to remove the foreground part, and obtain the newly generated image. ,image For a single RGB image Remove the occluded foreground and restore the result of subsequent objects, and retain the 3D instance content corresponding to the instance ID , if occlusion culling occurs, then Indicates the instance ID part that has been eliminated by occlusion culling. If occlusion culling does not occur, it means that there is no occluded single RGB image at present The corresponding three-dimensional instance content; The 3D instance content corresponding to the instance ID retained by multiple occlusion culling Integrate into a unified target model to obtain a single RGB image The corresponding panoramic 3D instance content represents a 3D scene reconstruction result.

7. The method for restoring a panoramic 3D instance based on a single image according to claim 6, characterized in that: The target model is a three-dimensional model that includes the semantic instance ID contained in the panoramic segmentation and has a restoration degree that meets the preset requirements.

8. A panoramic three-dimensional instance restoration system based on a single image, characterized in that: The panoramic three-dimensional instance restoration system based on a single image includes: An information extraction module is used to extract two-dimensional information of a single RGB image, wherein the two-dimensional information includes two-dimensional semantic information, instance center information, depth information, and multi-plane occupancy information; A feature dimension enhancement module, used for converting the two-dimensional semantic information into three-dimensional semantic information according to the depth information, extracting meaningless areas in the three-dimensional semantic information according to the multi-plane occupancy information, and obtaining three-dimensional enhancement features; An instance recovery module is used to refine the 3D lifting features, perform 3D reconstruction based on the refined features, and when performing 3D panorama segmentation, group instances based on the reconstructed features to obtain 3D instance IDs, and determine whether a single RGB image has occlusions. If there are no occlusions, the results of grouping the 3D instances are combined to obtain a 3D scene reconstruction result of a single RGB image under unoccluded conditions; The occlusion culling module is used to extract all instances in the foreground part of a single RGB image using the occlusion relationship learning method if there is occlusion, and use image restoration technology to perform occlusion culling, and synthesize multiple prediction results into a complete 3D scene reconstruction result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a panoramic three-dimensional instance recovery program based on a single image stored in the memory and executable on the processor. When the panoramic three-dimensional instance recovery program based on a single image is executed by the processor, the steps of the panoramic three-dimensional instance recovery method based on a single image are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a panoramic three-dimensional instance recovery program based on a single image, and when the panoramic three-dimensional instance recovery program based on a single image is executed by a processor, the steps of the panoramic three-dimensional instance recovery method based on a single image are implemented as described in any one of claims 1-7.

Images