Method for conversion of real world to minecraft environment based on three-dimensional semantic occupancy prediction

Abstract

This invention discloses a method for converting a real-world environment to a Minecraft environment based on 3D semantic occupancy prediction. The method includes: obtaining a 3D semantic occupancy grid for each image from an input sequence of labeled RGB images using a monocular semantic occupancy prediction model; fusing multi-view prediction results into a unified global semantic occupancy scene representation using camera parameters; extracting the precise center points of object instances in the scene by calculating local density maps and applying clustering algorithms; mapping the semantic label of each center point to the corresponding Minecraft block type, generating and executing building instructions, thereby automatically reconstructing a high-fidelity, structured, and editable virtual scene in Minecraft. This invention solves the problems of existing real-to-simulation conversion methods relying on CAD models, poor editability, and complex processes, providing an end-to-end, low-cost, and highly scalable scene reconstruction solution, offering an effective technical tool for fields such as simulation, digital twins, etc.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D reconstruction and virtual reality, and in particular to a method for converting real-world scenes into editable Minecraft virtual environments based on 3D semantic occupancy prediction. Background Technology

[0002] Real-world to virtual environment conversion refers to the technology of reconstructing editable virtual scenes based on real-world scene data such as RGB image sequences. Combined with processing methods such as semantic occupancy prediction, it can directly generate structured 3D representations, achieving high-fidelity virtual environments without the need for specialized CAD modeling tools. This enables efficient scene construction in fields such as robot simulation, digital twins, and embodied intelligence research. This technology is also widely used in interactive tasks such as visual language navigation and autonomous driving simulation. Common methods in this field include neural radiation fields, 3D Gaussian splashing, and CAD model-based reconstruction methods.

[0003] Semantic occupancy prediction is a technique that infers the three-dimensional spatial geometry and semantic labels from two-dimensional images, generating voxel-based scene representations where each voxel contains category information. This structured representation provides an important bridge between real-world perception and virtual environment creation, achieving unified geometric and semantic encoding of scenes by mapping visual features to a semantic voxel space. This technology plays a crucial role in applications such as scene understanding, environment modeling, and cross-modal interaction.

[0004] Due to the urgent need for real-to-simulation conversion in agent training and simulation testing, related methods have made significant progress in reconstruction efficiency and visual fidelity. However, most existing conversion methods generate results that lack editability, making it difficult to directly support the customized needs of downstream tasks. Meanwhile, the few techniques that support scene editing struggle to balance reconstruction accuracy, semantic richness, and real-time interactivity, resulting in conversion results that fail to meet the stringent requirements of high-precision, interactive virtual environments in scenarios such as robot navigation and autonomous driving. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an automated, end-to-end method for converting real-world scenes to Minecraft environments based on 3D semantic occupancy prediction. This method utilizes 3D semantic occupancy prediction as an intermediate representation, avoiding the dependency problem of CAD models and directly generating executable construction instructions, achieving efficient and accurate reconstruction.

[0006] The technical solution for achieving the objective of this invention is as follows:

[0007] A method for converting a real-world environment to a Minecraft environment based on 3D semantic occupancy prediction includes the following steps:

[0008] Step 1: Single-view semantic occupancy prediction

[0009] 1a. Input a sequence of RGB images with calibrated camera pose. and its camera internal parameters , Here, we have a set of image sequences. These represent images from different perspectives, where N represents the total number of images, with a value of 30.

[0010] 1b. Monocular prediction part using the pre-trained semantic occupancy prediction model EmbodiedOcc For each image, a prediction is made to obtain the 3D semantic occupancy prediction result for each image. Each voxel is assigned a semantic label selected from C categories. express The 3D semantic occupancy prediction result for the i-th image indicates the i-th... Zhang RGB images, where C represents the total number of predicted semantic categories, following the value of 13 given in EmbodiedOcc;

[0011] The monocular semantic occupancy prediction model ( ) is a monocular branch of EmbodiedOcc, a deep learning model based on semantic voxel prediction, which directly predicts dense semantic occupancy grids in three-dimensional space from two-dimensional image input.

[0012] Step 2: Integrate the single-view semantic occupancy prediction results to obtain the global semantic occupancy prediction result.

[0013] 2a. Input the single-view semantic occupancy prediction result obtained in step 1b. and the camera intrinsic parameters corresponding to the prediction results. and external references ,in express The 3D semantic occupancy prediction result for the i-th image, where N represents the total number of images (30).

[0014] 2b. Using the global prediction part of the pre-trained semantic occupancy prediction model EmbodiedOcc The semantic occupancy prediction results from various perspectives are fused to obtain the global semantic occupancy prediction result for the entire scene. ,in This represents the global semantic occupancy prediction result for the entire scene.

[0015] Step 3: Object instance centroid extraction and clustering

[0016] 3a will predict the global semantic occupancy results Convert to binary occupancy grid Voxels belonging to any object semantic category are marked as 1, and empty semantics are marked as 0.

[0017] 3b. Calculate local density maps by applying 3D convolution on a binary occupancy grid. In the formula It is a uniform convolution kernel with a value of 1. In ) represents the kernel size. Represents a local density map;

[0018] 3c. Set a threshold on the density map Extract potential candidate object center points ,in Represents the candidate set of the object's center point. Represents voxel coordinates, express Local density map at the location;

[0019] 3d. Extracted candidate center points Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is used to cluster applications based on each semantic category, employing Euclidean distance L2 and a distance threshold. By representing the centroid of each cluster as the centroid of that cluster, we obtain the precise set of centroids representing different object instances. , The range of values ​​is ;

[0020] Step 4: Minecraft Command Generation and Scene Building

[0021] 4a. Based on the predicted semantic categories, query the block categories in Minecraft and construct a semantic mapping table;

[0022] 4b. Calculate the center point of each object instance obtained in step 3d. The semantic tags are mapped to Minecraft block categories;

[0023] 4c. Generate the corresponding Minecraft command set based on the center point coordinates and the mapped block type. ;

[0024] 4d. Execute the command set in the Minecraft environment. It completes the automatic reconstruction of the virtual scene.

[0025] Furthermore, in step 1b The size of the prediction result occupied by the single-view semantics is as follows: length, width, and height are... .

[0026] In step 3b It is the kernel size used for 3D convolution. The range of values ​​is .

[0027] The threshold set in step 3c Used to filter densities less than in the density map The voxels, selected here The range of values ​​is .

[0028] In step 4a, the mapping relationship between semantic tags and Minecraft blocks is defined by a predefined, extensible mapping table. This table covers common structural elements in indoor and outdoor scenes, such as walls, floors, and ceilings, as well as furniture objects, such as tables, chairs, and beds.

[0029] This invention provides a low-cost solution for automatically converting real-world images into interactive virtual scenes end-to-end, significantly reducing the workload of manual modeling. The method primarily relies on RGB images and pre-trained models, eliminating the need for expensive specialized scanning equipment or sophisticated CAD model libraries. Furthermore, the resulting Minecraft scene is highly editable, facilitating subsequent modifications, expansions, and interactive design. It also retains object category information through semantic occupancy prediction, which is particularly advantageous for semantic-based agents locating target objects. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the method of the present invention;

[0031] Figure 2 A detailed flowchart of the semantic occupancy prediction module;

[0032] Figure 3 A flowchart for clustering object center points and generating Minecraft commands. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] This invention provides a method for converting real-world scenes to Minecraft environments based on 3D semantic occupancy prediction. The specific steps are as follows:

[0035] Step 1: Single-view semantic occupancy prediction

[0036] 1a. Input a sequence of RGB images with calibrated camera pose. and its camera internal parameters ;

[0037] 1b. Monocular prediction part using the pre-trained semantic occupancy prediction model EmbodiedOcc For each image, a prediction is made to obtain the 3D semantic occupancy prediction result for each image. Each voxel is assigned a semantic label selected from C categories;

[0038] Step 2: Integrate the single-view semantic occupancy prediction results to obtain the global semantic occupancy prediction result.

[0039] 2a. Input the single-view semantic occupancy prediction result obtained in step 1.2. and the corresponding camera internal parameters and external references ;

[0040] 2b. Using the global prediction part of the pre-trained semantic occupancy prediction model EmbodiedOcc The semantic occupancy prediction results from various perspectives are fused to obtain the global semantic occupancy prediction result for the entire scene. ;

[0041] Step 3: Object instance centroid extraction and clustering

[0042] 3a. The global semantic occupancy prediction results Convert to binary occupancy grid Voxels belonging to any object semantic category are marked as 1, and empty semantics are marked as 0.

[0043] 3b. Calculate local density maps by applying 3D convolution on a binary occupancy grid. ,in For uniform convolution kernels, we select [the appropriate kernel type]. This refers to the kernel size;

[0044] 3c. Set a threshold on the density map Extract potential candidate object center points ,in Represents voxel coordinates;

[0045] 3d. Extracted candidate center points Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is used to cluster applications based on each semantic category, employing L2 distance and a distance threshold. By representing the centroid of each cluster as the centroid of that cluster, we obtain the precise set of centroids representing different object instances. , The range of values ​​is ;

[0046] Step 4: Minecraft Command Generation and Scene Building

[0047] 4a. Construct a semantic mapping table by mapping the predicted semantic categories to the block categories in Minecraft;

[0048] 4b. Calculate the center point of each object instance obtained in step 3d. The semantic tags are mapped to Minecraft block categories;

[0049] 4c. Generate the corresponding Minecraft command set based on the center point coordinates and the mapped block type. ;

[0050] 4d. Execute the command set in the Minecraft environment. It completes the automatic reconstruction of the virtual scene.

[0051] Example 1, as Figure 1 As shown

[0052] S100: Multi-view RGB image sequence Corresponding camera intrinsic parameters With external references The input is fed into the semantic occupancy prediction module, where N=30.

[0053] S110: Use the semantic occupancy prediction module (including...) and Given a calibrated RGB image sequence and its corresponding intrinsic and extrinsic parameters, predict the semantic occupancy of each viewpoint in the prediction results. The global semantic prediction result is obtained by integrating the semantic occupancy predictions from various perspectives. .

[0054] S120 According to The semantic category is queried in the block category in Minecraft to construct a semantic mapping table. The semantic mapping table is input simultaneously with the cluster center coordinates and the command generation module. First, binarize Then, convolution is used to obtain candidate center points of the object. Finally, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method was used for clustering to obtain the precise set of centroids. .

[0055] S130: Cluster center coordinate generation instruction, which uses the cluster center coordinates and the instruction generation module, such as... Figure 3 As shown, by inputting the global semantic occupancy prediction result and the semantic mapping table, Minecraft commands are obtained, which are executed in Minecraft to build the reconstructed scene.

[0056] S140: Execute the Minecraft command to obtain the Minecraft scene.

[0057] Table 1 shows an example of semantic mapping representation:

[0058] Table 1. Examples of Semantic Mapping Representations

[0059] Realistic semantic categories Mapping categories in Minecraft empty minecraft:air ceiling minecraft:stripped_acacia_wood floor tmeov:baisehunningtuqupiyunshangmushang wall minecraft:stripped_acacia_wood window tmeov:baiyechuangheise_2x_2kai chair tmeo_ultra:canzhuoyizi[facing=south] bed tmeo_ultra:bedblue_01[facing=north] sofa tmeov:shafabuliao_1x_1[facing=south] table tmeov:chanzhuomuban[facing=south] TVS xianshiqidaiyuping_2guan furniture tmeov:baiseyigui object tmeov:xiaoxingzhuangshi

[0060] Example 2

[0061] like Figure 2 As shown, this embodiment is an example of obtaining a global semantic occupancy prediction result by fusing the single-view semantic occupancy prediction results in step 1 (step 1) and step 2 (step 2).

[0062] S200: Monocular prediction part using the officially trained monocular semantic occupancy prediction model EmbodiedOcc For each image in the image sequence, a prediction is made to obtain the 3D semantic occupancy prediction result for each image. Each voxel is assigned a semantic label from 11 categories (including empty, ceiling, floor, wall, window, chair, bed, sofa, table, TVs, furniture, and object). The dimension is ;

[0063] S210: Global prediction part using the officially trained global semantic occupancy prediction model EmbodiedOcc And the single-view semantic occupancy prediction results obtained from S200 and camera internal reference and external references The semantic occupancy prediction results from various perspectives are fused to obtain the global semantic occupancy prediction result for the entire scene. ;

[0064] Example 3

[0065] like Figure 3 As shown, this embodiment is an example of the cluster center coordinates and instruction generation steps, including:

[0066] S300: Obtain the global semantic prediction result from S110. In the non-zero position, the position of 0 is set to 1, and the position of zero is kept to 0, thus obtaining the binary occupancy network. .

[0067] S310: Using 3D convolutions on binary occupancy networks Calculate the local density map ,in For a uniform convolution kernel, here For density maps Set threshold Perform screening to select those with a density greater than Candidates for the center point of the object, i.e. ,in Represents voxel coordinates.

[0068] S320: Extracted candidate center points Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is performed for each semantic category, using L2 distance and distance threshold. By representing the centroid of each cluster as the centroid of an object, we obtain a precise set of centroids representing different object instances. ,here =0.5.

[0069] S330: The center point of each object instance obtained in step 320 is... The semantic tags are mapped to Minecraft block categories using a semantic mapping table. Based on the center point coordinates and the mapped block category, the corresponding Minecraft command set is generated.

[0070] Example 4

[0071] like Figure 3As shown, this embodiment is the flowchart of the cluster center coordinates and instruction generation steps described in Embodiment 1, including:

[0072] S300: Obtain the global semantic prediction result from S110. In the non-zero position, the position of 0 is set to 1, and the position of zero is kept to 0, thus obtaining the binary occupancy network. .

[0073] S310: Using 3D convolutions on binary occupancy networks Calculate the local density map ,in For a uniform convolution kernel, here For density maps Set threshold Perform screening to select those with a density greater than Candidates for the center point of the object, i.e. ,in Represents voxel coordinates.

[0074] S320: Extracted candidate center points Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is performed for each semantic category, using L2 distance and distance threshold. By representing the centroid of each cluster as the centroid of an object, we obtain a precise set of centroids representing different object instances. ,here =3.

[0075] S330: The center point of each object instance obtained in step 320 is... The semantic tags are mapped to Minecraft block categories using a semantic mapping table. Based on the center point coordinates and the mapped block category, the corresponding Minecraft command set is generated.

Claims

1. A method for converting real-world environments to Minecraft environments based on 3D semantic occupancy prediction, characterized in that, Includes the following steps: Step 1: Single-view semantic occupancy prediction 1a. Input a sequence of RGB images with calibrated camera pose. and its camera internal parameters ; 1b. Monocular prediction part using the pre-trained semantic occupancy prediction model EmbodiedOcc For each image, a prediction is made to obtain the 3D semantic occupancy prediction result for each image. Each voxel is assigned a semantic label selected from C categories; Step 2: Integrate the single-view semantic occupancy prediction results to obtain the global semantic occupancy prediction result. 2a. Input the single-view semantic occupancy prediction result obtained in step 1b. and the corresponding camera internal parameters and external references ; 2b. Using the global prediction part of the pre-trained semantic occupancy prediction model EmbodiedOcc The semantic occupancy prediction results from various perspectives are fused to obtain the global semantic occupancy prediction result for the entire scene. ; Step 3: Object instance centroid extraction and clustering 3a. The global semantic occupancy prediction results Convert to binary occupancy grid Voxels belonging to any object semantic category are marked as 1, and empty semantics are marked as 0. 3b. Calculate local density maps by applying 3D convolution on a binary occupancy grid. In the formula It is a uniform convolution kernel with a value of 1. This refers to the kernel size; 3c. Set a threshold on the density map Extract potential candidate object center points ,in Represents voxel coordinates; 3d. Extracted candidate center points Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is used to cluster applications based on each semantic category, employing L2 distance and a distance threshold. By representing the centroid of each cluster as the centroid of that cluster, we obtain the precise set of centroids representing different object instances. , The range of values ​​is ; Step 4: Minecraft Command Generation and Scene Building 4a. Query the block categories in Minecraft based on the predicted semantic categories and construct a semantic mapping table; 4b. Calculate the center point of each object instance obtained in step 3d. The semantic tags are mapped to Minecraft block categories; 4c. Generate the corresponding Minecraft command set based on the center point coordinates and the mapped block type. ; 4d. Execute the command set in the Minecraft environment. It completes the automatic reconstruction of the virtual scene.

2. The method for converting a real-world environment to a Minecraft environment based on three-dimensional semantic occupancy prediction according to claim 1, characterized in that, In step 1b The size of the prediction result occupied by the single-view semantics is as follows: length, width, and height, with a size of [value missing]. .

3. The method for converting a real-world environment to a Minecraft environment based on three-dimensional semantic occupancy prediction according to claim 1, characterized in that, In step 3b In It is the kernel size used for 3D convolution. The range of values ​​is .

4. The method for converting a real-world environment to a Minecraft environment based on three-dimensional semantic occupancy prediction according to claim 1, characterized in that, The threshold set in step 3c Used to filter densities less than in the density map The voxels.

5. The method for converting a real-world environment to a Minecraft environment based on three-dimensional semantic occupancy prediction according to claim 4, characterized in that, The threshold set by 3c The range of values ​​is .

Images