Method and electronic device for 3D semantic scene reconstruction using regional memory babk

Abstract

This invention proposes a method and electronic device for three-dimensional semantic scene reconstruction. The method includes: acquiring a two-dimensional image; generating multiple symbolic features and multiple voxel features based on the two-dimensional image; adding the symbolic features to a region memory containing multiple key-value pairs; generating a depth map based on the two-dimensional image; generating a reconstruction mask based on the depth map and the symbolic features, containing multiple unseen locations; querying the region memory based on the unseen locations to obtain a first symbolic feature; and updating at least one voxel feature based on the first symbolic feature, and generating multiple three-dimensional scene categories based on the updated voxel feature.

Description

Technical Field

[0001] This disclosure relates to a method and electronic device for reconstructing three-dimensional semantic scenes by using a memory bank to fill in unseen locations. Prior Technology

[0002] With the rapid development of autonomous driving technology, the ability to correctly identify the category of objects in a three-dimensional scene has become one of the core technologies for perception, planning, and decision-making in autonomous driving systems. To achieve this, existing technologies mostly rely on deep learning models to analyze and classify visual data in order to identify environmental elements such as vehicles, pedestrians, and traffic signs in the field of vision ahead.

[0003] However, existing technologies generally suffer from poor recognition performance. The main reason is that traditional models mostly only process the visible area within the Field of View (FOV), lacking effective mechanisms for handling occluded or out-of-view areas. Therefore, when there are vehicles or pedestrians obscured by other objects in the scene, or when important objects are about to enter the field of view, traditional technologies cannot provide sufficient and complete perception information, leading to a decrease in the reliability of autonomous driving decisions. Summary of the Invention

[0004] This disclosure proposes a three-dimensional semantic scene reconstruction method applicable to electronic devices. This three-dimensional semantic scene reconstruction method includes: acquiring a two-dimensional image; generating multiple symbolic features and multiple voxel features based on the two-dimensional image, wherein each symbolic feature is associated with a region; adding the symbolic features to a region memory containing multiple key-value pairs; generating a depth map based on the two-dimensional image; generating a reconstruction mask based on the depth map and the symbolic features, the reconstruction mask containing multiple unseen locations; querying the region memory based on at least one unseen location to obtain a first symbolic feature; and updating at least one voxel feature based on the first symbolic feature, and generating multiple three-dimensional scene categories based on the updated voxel feature.

[0005] In one embodiment of this disclosure, the above-described three-dimensional semantic scene reconstruction method further includes: for each symbol feature, obtaining multiple similar symbol features as keys, and treating the symbol features as values, wherein the keys and values ​​form a new key-value pair to be added to the above-described key-value pair.

[0006] In one embodiment of this disclosure, each of the above-described symbolic features has a position. The step of obtaining similar symbolic features as a key includes: calculating the difference between the positions corresponding to two symbolic features to obtain similar symbolic features.

[0007] In one embodiment of this disclosure, the above-described three-dimensional semantic scene reconstruction method further includes: if the number of key-value pairs is greater than a threshold value, calculating the diversity score and age score of each key-value pair; and deleting at least one key-value pair based on the diversity score and age score.

[0008] In one embodiment of this disclosure, the above-described three-dimensional semantic scene reconstruction method further includes: for each key-value pair, calculating the sum of the cosine similarities between the values ​​of the key-value pair and the values ​​of other key-value pairs as a diversity score.

[0009] In one embodiment of this disclosure, the step of deleting at least one key-value pair based on a diversity score and an age score includes: subtracting the age score from the diversity score to obtain an overall score, and deleting the key-value pair with the lowest overall score.

[0010] In one embodiment of this disclosure, generating a reconstructed mask based on a depth map and symbolic features includes: generating multiple three-dimensional coordinates based on the depth map; projecting the three-dimensional coordinates onto the ground to obtain a visible mask; inverting the visible mask to obtain an invisible mask; performing an expansion procedure based on the position of the symbolic features according to the core to obtain a region mask; and performing element-wise multiplication on the region mask and the invisible mask to obtain a reconstructed mask.

[0011] In one embodiment of this disclosure, the step of querying a region memory based on an unseen location to obtain a first symbol feature includes: using multiple adjacent unseen locations as queries; and comparing the keys in the queries and key-value pairs to obtain corresponding values ​​as the first symbol feature.

[0012] In one embodiment of this disclosure, the step of updating at least one voxel feature based on a first symbol feature includes: obtaining the first voxel feature located at the bottom layer based on adjacent invisible positions; and replacing the first voxel feature with the first symbol feature.

[0013] In one embodiment of this disclosure, generating a 3D scene category based on updated voxel features includes: inputting the updated voxel features into a neural network to obtain a first output; adding the first output to the voxel features to obtain a second output; and inputting the second output into a head to obtain a 3D scene category.

[0014] From another perspective, the embodiments disclosed herein propose an electronic device comprising memory and a processor. The processor is used to execute instructions in the memory to complete the aforementioned three-dimensional semantic scene reconstruction method.

[0015] In the aforementioned electronic device and method, a depth map can be used to find unseen locations, and then a region memory can be queried to find the features of the unseen locations, thereby enabling a more accurate prediction of the scene category.

[0016] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings for detailed explanation. Simple Explanation of the Diagram

[0017] Figure 1 is a block diagram illustrating an electronic device according to one embodiment. Figures 2A and 2B are schematic diagrams illustrating experimental results according to one embodiment. Figure 3 is an architecture diagram illustrating a three-dimensional semantic scene reconstruction method according to one embodiment. Figure 4 is a flowchart illustrating a three-dimensional semantic scene reconstruction method according to one embodiment. Implementation

[0018] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered identical or similar when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application.

[0019] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0020] Figure 1 is a block diagram illustrating an electronic device according to an embodiment. Referring to Figure 1, the electronic device 100 can be a personal computer, notebook computer, server, cloud server, industrial computer, monitoring system, vehicle assistance system, autonomous driving system, or various electronic devices with computing capabilities, etc., and the present invention is not limited thereto. The electronic device 100 includes a processor 110, a memory 120, and an image capturing device 130. The processor 110 is electrically connected to the memory 120 and the image capturing device 130. The processor 110 may include a central processing unit, a graphics processing unit (GPU), a deep-learning processing unit (DPU), a neural network processing unit (NPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). Memory 120 may be random access memory, read-only memory, flash memory, floppy disk, hard disk, optical disk, USB flash drive, magnetic tape, or a database accessible via the Internet. Image capturing device 130 includes a charge-coupled device (CCD) sensor, a complementary metal-oxide-semiconductor (CMOS) sensor, or other suitable photosensitive element. Memory 120 stores multiple instructions, which processor 110 executes to complete a three-dimensional semantic scene reconstruction method.

[0021] Figures 2A and 2B are schematic diagrams illustrating experimental results according to one embodiment. Referring to Figure 2A, in some embodiments, the 3D semantic scene reconstruction method is used for autonomous driving. Two-dimensional image 210 is about a road environment, and after performing this method, a prediction result 220 can be generated. This prediction result 220 contains multiple 3D scene categories, where different colors represent different categories. These categories may include vehicles, bicycles, motorcycles, trucks, pedestrians, roads, parking spaces, sidewalks, other ground surfaces, buildings, fences, green spaces, traffic signs, and other objects, etc., which are not limited to this invention. The conventional technique has poor prediction accuracy for regions 221 and 222 because some objects are invisible. Invisibility can be caused by occlusion or being outside the scene. Referring to Figure 2B, two-dimensional image 230 is also about a road environment, and after the above method, a prediction result 240 can be generated. The conventional technique has poor prediction accuracy for regions 241 and 242. However, the 3D semantic scene reconstruction method proposed in this disclosure can calculate the categories of these invisible regions.

[0022] Figure 3 is an architecture diagram illustrating a three-dimensional semantic scene reconstruction method according to an embodiment. Referring to Figure 3, the three-dimensional semantic scene reconstruction method includes three main parts: the Semantic Scene Completion (SSC) stage 310, the region memory 320, and the re-completion pipeline 330.

[0023] Referring to Figures 1 and 3, the image capturing device 130 first captures a two-dimensional image 311. In this embodiment, the two-dimensional image 311 relates to a road environment, but in other embodiments it could relate to a shopping mall, factory, airport, school, or any other location; this invention is not limited thereto. Next, multiple voxel features 318 and multiple token features 317 are generated based on this two-dimensional image 311. The length of a voxel feature is the same as the length of a token feature. These voxel features 318 are arranged in a three-dimensional matrix to contain features at each location (i.e., voxel) in the three-dimensional space corresponding to the two-dimensional image 311. These voxel features 318 can be used to generate categories in subsequent processes. On the other hand, each token feature 317 is associated with a region in the two-dimensional image 311. These regions can be buildings, roads, grass, vehicles, pedestrians, etc. A region is larger than a voxel but smaller than the entire scene; therefore, the voxel features 318 are more detailed, while the semantics represented by the token features 317 lie between the voxel features 318 and the entire scene.

[0024] Here, any known method can be used to generate voxel features 318 and symbolic features 317. For example, a two-dimensional image 311 can first be input into an encoder 312, which outputs two-dimensional multi-scale features 314 and symbolic features 315. Alternatively, voxel features 313 can be generated from the two-dimensional image 311 using other methods. Next, voxel features 313, two-dimensional multi-scale features 314, and symbolic features 315 are input into a decoder 316 to obtain voxel features 318 and symbolic features 317. In the following mathematical representation, all symbolic features 317 are represented as a set T, and voxel features 318 are represented as... .

[0025] Next, the symbolic feature 317 is added to the region memory 320. This region memory 320 contains multiple key-value pairs 321, where each key consists of multiple symbolic features, and each value contains a single symbolic feature. The region memory 320 is used to retain previously appearing symbolic features, which may contain information about currently invisible areas in the scene. The memory is built at the region level, which offers advantages in terms of computational efficiency and ease of management.

[0026] Specifically, for each of the 317 symbolic features Multiple (e.g., three, but this invention is not limited to) similar symbolic features can be searched as keys, and these three similar symbolic features are represented as a set. And symbolic features Then it serves as the value. Therefore, key-value pairs can be formed. .gather A similar symbol feature Defined as the following mathematical expression 1. [Mathematical Expression 1]

[0027] Where d() represents the distance between two symbolic features. In other words, this involves finding the closest symbolic feature from symbolic features T. Multiple similar symbol features To establish a set Each symbolic feature has a location in three-dimensional space (i.e., the location of a region), for example, symbolic features Having position Symbolic features Having position In some embodiments, the function d() is defined as the following mathematical expression 2. [Mathematical Expression 2]

[0028] In other words, in the above mathematical formulas 1 and 2, the difference between the positions corresponding to two symbolic features is calculated to obtain similar symbolic features. This difference is the Euclidean distance, but in other embodiments, the Manhattan distance can also be used. This invention is not limited to this.

[0029] Calculate the new key-value pair Subsequently, this new key-value pair will be added to the existing key-value pairs 321 in the region memory 320. In some embodiments, after adding a new key-value pair, if the total number of all key-value pairs exceeds a threshold (e.g., 1024), some key-value pairs need to be deleted. Here, a diversity score and an age score are calculated for each key-value pair, and then at least one key-value pair is deleted based on the diversity score and age score. Specifically, the diversity score is used to maintain the diversity of key-value pairs in the region memory 320 to effectively capture regional information across scenes. In some embodiments, the diversity score... The calculation is as shown in the following mathematical formula 3. [Mathematical Expression 3]

[0030] in Let T represent a set, which is the union of existing symbolic features in the region memory 320 and newly generated symbolic features T. , For set The symbolic features in [the context of key-value pairs]. From another perspective, after adding a new key-value pair to an existing key-value pair, the value of that key-value pair will be calculated accordingly. Values ​​paired with multiple other key-value pairs The sum of the cosine similarities between them is used as the diversity score. .

[0031] On the other hand, age scores are used to filter out older information and retain newer information. In some embodiments, age scores... It is initialized to 0, and a number (e.g., 1) is added for each 2D image passed. For the i-th key-value pair, the corresponding age score is represented as... Diversity scores Subtract age score You can get the overall score. As shown in the following mathematical formula 4. Next, one or more key-value pairs with the lowest overall score can be deleted, such that the number of all key-value pairs is less than or equal to the critical value. In other words, the pair with the highest overall score is retained. Multiple key-value pairs (e.g., a total of 1024). [Mathematical Expression 4]

[0032] Furthermore, completing pipeline 330 involves finding the invisible locations in the 2D image 311 and then querying the region memory 320 to update the voxel features 318. Specifically, in step 319, a depth map 331 is first calculated based on the 2D image 311, where the value of each pixel in the depth map 331 represents the depth. Any conventional method can be used to calculate the depth map 331. Next, a reconstruction mask 333 is generated based on the depth map 331 and the notation features 317; this reconstruction mask 333 contains the invisible locations.

[0033] Specifically, the depth map 331 can be projected into three-dimensional space to generate multiple three-dimensional coordinates. This step can be accomplished according to the following mathematical formula 5. [Mathematical Expression 5]

[0034] in , These are the horizontal and vertical coordinates of the camera center, respectively. It is the focal length in the horizontal direction. is the focal length in the vertical direction. u is the horizontal coordinate of a pixel in depth map 331. v is the vertical coordinate of a pixel in depth map 331. z is the numerical value of the pixel located at coordinates (u, v) in depth map 331. This value represents depth. Therefore, a coordinate (u,v) in the depth map can be converted into a three-dimensional coordinate (x,y,z).

[0035] Next, the aforementioned three-dimensional coordinates (x, y, z) are projected onto the ground to obtain a visual mask 332. For example, the Y coordinate, representing height, can be set to 0, meaning the three-dimensional coordinates (x, y, z) are reduced to a two-dimensional coordinate (x, z) on the visual mask. If a pixel is projected from depth map 331 onto the two-dimensional coordinate (x, z), the corresponding pixel value in the visual mask is "1"; conversely, if no pixel is projected from depth map 331 onto the two-dimensional coordinate (x, z), the corresponding pixel value in the visual mask is "0". The visual mask will be represented as follows: If visible mask A pixel with a value of 1 indicates that there is a visible object at the corresponding location; if a pixel has a value of 0, it indicates that the corresponding location is not visible (it may be occluded).

[0036] Next, invert the visible mask. To obtain a non-visual mask, denoted as Specifically, "inverting" here means turning the visible mask... Change the value "1" to "0" and the value "0" to "1". In other words, the invisible mask... The value "1" in the code indicates that the location is not visible.

[0037] Next, an expansion procedure is performed on the position of each symbol feature 317 based on a core to obtain a region mask. This core can be circular, square, or of any shape. For example, the position of the i-th symbol feature 317 is... , at this position Centered on the core, all values ​​within the core area are set to 1. Therefore, the region mask represents the location of all symbol features 317 (with slight expansion).

[0038] Next, we will discuss area masking and invisible masking. Perform element-wise multiplication to obtain the reconstructed mask 333, denoted as... In the reconstruction mask 333, the positions with a value of "1" represent invisible regions with corresponding regional features 317. These positions with a value of "1" are called invisible positions.

[0039] Next, step 334 is executed, using the reconstructed mask 333 and the region memory 320 to update the voxel features 318. Specifically, the region memory 320 is queried based on at least one invisible location to obtain a value in a key-value pair (also known as the first token feature). In this embodiment, three token features are combined into a key, so three adjacent invisible locations (also known as adjacent invisible locations) can be used as queries, which is also represented in Figure 3. Then, compared to this query... The key in key-value pairing 321 is used to find the closest key and obtain the corresponding value as the first symbol feature. In other words, the region memory 320 can be used as a codebook to find matching values. Every three adjacent invisible locations form a query, until all invisible locations have been processed.

[0040] Next, at least one voxel feature 318 is updated based on the matched first symbolic feature. In this embodiment, since the values ​​in the reconstructed mask 333 represent whether objects on the ground are visible, only the voxel features corresponding to the ground are updated. Specifically, based on the aforementioned adjacent invisible locations (i.e., querying...) The voxel feature 318 is obtained from the lowest level of the voxel features (referred to as the first voxel feature). The first voxel feature is then replaced with the first symbol feature to obtain the updated voxel feature 335. In this way, the voxel features located in invisible positions will be updated by information in the region memory 320, which may come from previous scenes.

[0041] Next, multiple 3D scene categories are generated based on the updated voxel features 335. In this embodiment, since the voxel features are updated based on values ​​in the region memory 320, there may be scale inconsistencies. In some implementations, the updated voxel features 335 can be first input into a neural network 336 to obtain a first output 337. This neural network 336 is, for example, an Atrous Spatial Pyramid Pooling (ASPP) model, but this invention is not limited thereto. Next, the first output 337 and the voxel features 318 are added to obtain a second output 338. Finally, the second output 338 is input into a head 340 to obtain a 3D scene category 341. This head 340 is a neural network, for example, containing convolutional layers or fully connected layers.

[0042] Figure 4 is a flowchart illustrating a three-dimensional semantic scene reconstruction method according to an embodiment. Referring to Figure 4, in step 401, a two-dimensional image is acquired, and multiple symbolic features and multiple voxel features are generated based on the two-dimensional image, wherein each symbolic feature is associated with a region. In step 402, these symbolic features are added to a region memory, which contains multiple key-value pairs. In step 403, a depth map is generated based on the two-dimensional image, and a reconstruction mask is generated based on the depth map and the symbolic features. This reconstruction mask contains multiple unseen locations. In step 404, the region memory is queried based on at least one unseen location to obtain a first symbolic feature. In step 405, at least one voxel feature is updated based on the first symbolic feature, and multiple three-dimensional scene categories are generated based on the updated voxel feature. The steps in Figure 4 have been described in detail above and will not be repeated here. It is worth noting that each step in Figure 4 can be implemented as multiple pieces of code or circuits, and the present invention is not limited thereto. Furthermore, the method in Figure 4 can be used in conjunction with the above embodiments or used alone; in other words, other steps can be added between the steps in Figure 4.

[0043] From another perspective, the present invention also proposes a computer program product that can be written in any programming language and / or platform. When this computer program product is loaded into a computer system and executed, the above-described method can be performed.

[0044] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0045] 100: Electronic devices 110: Processor 120: Memory 130: Image capturing device 210, 230, 311: Two-dimensional images 220,240: Prediction Results 221,222,241,242: Area 310: Semantic Scene Completion Stage 312: Encoder 313, 318: Voxel characteristics 314: Two-dimensional multi-scale features 315: Symbolic Features 316: Decoder 317: Symbolic Features 319, 334: Steps 320: Regional Memory 321: Key-value pairing 330: Complete the pipeline again 331: Depth Map 332: Visible Mask 333: Rebuild Mask 335: Updated voxel features 336: Neural Networks 337: First Output 338: Second Output 340: Head 341: 3D Scene Category 401~405: Steps

Claims

1. A three-dimensional semantic scene reconstruction method, applicable to an electronic device, the three-dimensional semantic scene reconstruction method comprising: acquiring a two-dimensional image; generating multiple symbolic features and multiple voxel features based on the two-dimensional image, wherein each of the symbolic features is associated with a region; adding the symbolic features to a region memory, the region memory containing multiple key-value pairs; generating a depth map based on the two-dimensional image, and generating a reconstruction mask based on the depth map and the symbolic features, the reconstruction mask containing multiple unseen locations; querying the region memory based on at least one of the unseen locations to obtain a first symbolic feature; and updating at least one of the voxel features based on the first symbolic feature, and generating multiple three-dimensional scene categories based on the updated voxel features.

2. The three-dimensional semantic scene reconstruction method as described in claim 1 further includes: For each of the symbolic features, multiple similar symbolic features among the symbolic features are taken as a key, and the symbolic feature is taken as a value, wherein the key and the value form a new key-value pair to be added to the key-value pair.

3. The three-dimensional semantic scene reconstruction method as described in claim 2, wherein each of the symbolic features has a position, and the step of obtaining the similar symbolic features among the symbolic features as the key includes: Calculate the difference between the positions corresponding to two of these symbolic features to obtain these similar symbolic features.

4. The three-dimensional semantic scene reconstruction method as described in claim 3 further includes: If the number of these key-value pairs exceeds a threshold, calculate a diversity score and an age score for each of these key-value pairs; and delete one of these key-value pairs based on the diversity score and the age score.

5. The three-dimensional semantic scene reconstruction method as described in claim 4 further includes: For each of these key-value pairs, the sum of the cosine similarities between the values ​​of that key-value pair and the values ​​of the other key-value pairs is calculated as the diversity score.

6. The three-dimensional semantic scene reconstruction method as described in claim 4, wherein the step of deleting one of the key-value pairs based on the diversity score and the age score includes: Subtract the age score from the diversity score to obtain an overall score, and delete the key-value pair with the smallest overall score.

7. The three-dimensional semantic scene reconstruction method as described in claim 1, wherein each of the symbolic features has a location, and the step of generating the reconstruction mask based on the depth map and the symbolic features includes: Multiple 3D coordinates are generated based on this depth map; These three-dimensional coordinates are projected onto a ground surface to obtain a visual mask; The visible mask is inverted to obtain an invisible mask; an expansion procedure is performed on the locations of the symbolic features according to a core to obtain a region mask; and an element-wise multiplication is performed on the region mask and the invisible mask to obtain the reconstructed mask.

8. The three-dimensional semantic scene reconstruction method as described in claim 7, wherein the step of querying the region memory based on at least one of the unseen locations to obtain the first symbolic feature includes: Treat multiple adjacent invisible locations among these invisible locations as a query; And by comparing the query with the keys in the key-value pairs, the corresponding values ​​are obtained as the first symbol feature.

9. The three-dimensional semantic scene reconstruction method as described in claim 8, wherein the step of updating at least one of the voxel features according to the first symbolic feature includes: Based on these adjacent invisible locations, at least one first voxel feature located at the lowest layer of these voxel features is obtained; And to replace the at least one first voxel feature with the first symbol feature.

10. The three-dimensional semantic scene reconstruction method as described in claim 9, wherein generating the three-dimensional scene categories based on the updated voxel features includes: The updated voxel features are input into a neural network to obtain a first output; The first output is added to the voxel features to obtain a second output; and the second output is input to a head to obtain the three-dimensional scene categories.

11. An electronic device comprising: a memory storing a plurality of instructions; and a processor electrically connected to the memory for executing the instructions to perform a plurality of steps: acquiring a two-dimensional image; generating a plurality of symbolic features and a plurality of voxel features based on the two-dimensional image, wherein each of the symbolic features is associated with a region; adding the symbolic features to a region memory containing a plurality of key-value pairs; generating a depth map based on the two-dimensional image; generating a reconstruction mask based on the depth map and the symbolic features, the reconstruction mask containing a plurality of unseen locations; querying the region memory based on at least one of the unseen locations to obtain a first symbolic feature; and updating at least one of the voxel features based on the first symbolic feature, and generating a plurality of three-dimensional scene categories based on the updated voxel features.

12. The electronic device as claimed in claim 11, wherein the steps further include: For each of the symbolic features, multiple similar symbolic features among the symbolic features are taken as a key, and the symbolic feature is taken as a value, wherein the key and the value form a new key-value pair to be added to the key-value pair.

13. The electronic device of claim 12, wherein each of the symbolic features has a location, the step of obtaining the similar symbolic features among the symbolic features as the key includes: Calculate the difference between the positions corresponding to two of these symbolic features to obtain these similar symbolic features.

14. The electronic device as claimed in claim 13, wherein the steps further include: If the number of these key-value pairs exceeds a threshold, calculate a diversity score and an age score for each of these key-value pairs; and delete one of these key-value pairs based on the diversity score and the age score.

15. The electronic device as claimed in claim 14, wherein the steps further include: For each of these key-value pairs, the sum of the cosine similarities between the values ​​of that key-value pair and the values ​​of the other key-value pairs is calculated as the diversity score.

16. The electronic device of claim 14, wherein the step of deleting one of the key-value pairs based on the diversity score and the age score includes: Subtract the age score from the diversity score to obtain an overall score, and delete the key-value pair with the smallest overall score.

17. The electronic device of claim 11, wherein each of the symbolic features has a location, the step of generating the reconstructed mask based on the depth map and the symbolic features includes: Multiple 3D coordinates are generated based on this depth map; These three-dimensional coordinates are projected onto a ground surface to obtain a visual mask; The visible mask is inverted to obtain an invisible mask; an expansion procedure is performed on the locations of the symbolic features according to a core to obtain a region mask; and an element-wise multiplication is performed on the region mask and the invisible mask to obtain the reconstructed mask.

18. The electronic device of claim 17, wherein the step of querying the region memory based on at least one of the invisible locations to obtain the first symbolic feature includes: Treat multiple adjacent invisible locations among these invisible locations as a query; And by comparing the query with the keys in the key-value pairs, the corresponding values ​​are obtained as the first symbol feature.

19. The electronic device of claim 18, wherein the step of updating at least one of the voxel features according to the first notation feature comprises: Based on these adjacent invisible locations, at least one first voxel feature located at the lowest layer of these voxel features is obtained; And to replace the at least one first voxel feature with the first symbol feature.

20. The electronic device as claimed in claim 19, wherein generating the three-dimensional scene categories based on the updated voxel features includes: The updated voxel features are input into a neural network to obtain a first output; The first output is added to the voxel features to obtain a second output; and the second output is input to a head to obtain the three-dimensional scene categories.

Images