Adaptive three-dimensional semantic occupancy prediction method and device for embodied tasks

Abstract

The application discloses a kind of adaptive three-dimensional semantic occupancy prediction methods and devices for embodied task, belong to computer vision, three-dimensional scene understanding, robot perception and artificial intelligence technical field, the method includes: obtaining multi-source perception input;Unified geometry input is obtained according to calibration parameter, and unified geometry point set is obtained;According to task budget, the number of inquiries is determined and inquiry point input is initialized, and inquiry point set is obtained;RGB image set, unified geometry point set and inquiry point set are sent into pre-trained AdaOcc model and forward prediction is carried out, and sparse semantic point set is obtained;Three-dimensional semantic occupancy result is generated according to sparse semantic point set.The present application can improve the adaptability of model to different query budget by controlling reasoning budget, effectively avoid the problem that prediction point coverage is insufficient or precision benefit is not obvious after budget increases, and the present application can also reduce redundant calculation on idle space.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D scene understanding, robot perception and artificial intelligence, and in particular to an adaptive 3D semantic occupancy prediction method and apparatus for embodied tasks. Background Technology

[0002] Occupancy prediction is a core perception task in fields such as robotics, autonomous driving, and embodied intelligence. It aims to divide a 3D space into a voxel grid and predict whether each voxel is occupied, free, or unknown, while also attaching semantic categories (such as "vehicle," "person," "wall," etc.). Currently, occupancy prediction mainly employs the following techniques: I. Three-dimensional semantic occupancy prediction based on dense voxels Existing 3D semantic occupancy prediction methods typically divide the local 3D space into a regular voxel grid and predict the free, occupied, or semantic category for each voxel. Such approaches generally include the following steps: acquiring monocular or multi-view RGB images; extracting 2D features through an image backbone network; upscaling the 2D image features to 3D space or a bird's-eye view space; modeling the spatial context using 3D convolution, Transformer, or multi-plane features; and finally outputting the semantic occupancy result for the complete 3D voxel grid.

[0003] Representative approaches to this type of solution include semantic scene completion, monocular 3D occupancy prediction, multi-view 3D occupancy prediction, and dense voxel prediction for indoor or autonomous driving scenarios. Their advantage is a regular output format, facilitating direct evaluation using standard voxel metrics. However, this type of solution has the following drawbacks: 1. High computational and memory overhead. Dense voxel meshes require allocating storage and computational resources to a large amount of free space, making them unsuitable for robotic platforms with limited computing power.

[0004] 2. Fixed inference budget. Models typically operate at a fixed resolution, a fixed number of voxels, and a fixed network depth, making it difficult to dynamically adjust for the accuracy and latency requirements of different tasks such as navigation and grasping.

[0005] 3. Fixed input format. Most methods are designed around a fixed sensor configuration, such as a fixed number of cameras, a fixed field of view, or a fixed input modality. When migrating to different robotic platforms, they need to be redesigned or retrained.

[0006] 4. Insufficient adaptation to embodied tasks. Although dense voxel results are complete, they still need to take into account real-time performance, sensor deficiencies, viewpoint changes, and downstream planning interfaces in real robotic tasks.

[0007] II. Occupation Prediction Based on Sparse Points, Queries, or 3D Gaussian

[0008] To reduce the overhead of dense voxel prediction, some existing methods employ sparse representations, such as point queries, sparse anchors, or 3D Gaussian representations of occupied regions. These methods typically first set a certain number of query points or spatial anchors, then sample information from image features or 3D features, and predict the 3D position, occupancy confidence, and semantic category corresponding to each query. Finally, the prediction results are converted into a voxel occupancy map.

[0009] Compared to dense voxel methods, this type of technology is lighter, reduces redundant computation in idle spaces, and is more suitable for sparse prediction over larger spatial areas. For example, OPUS proposes using sparse point sets for 3D occupancy prediction, representing occupied areas with a smaller number of predicted points, thereby reducing the computational burden of dense voxel prediction. This sparse point set prediction idea can serve as the basic network background for this invention, but it mainly focuses on sparse occupancy prediction in general or autonomous driving scenarios, and has not yet systematically solved problems such as the unification of heterogeneous geometric inputs, adaptive task budgets, floating point constraints in idle areas during training, and navigation and operation interfaces in embodied robot scenarios. Specifically, this type of technology has the following drawbacks: 1. Point-based prediction lacks sufficient geometric constraints. Commonly used point set reconstruction losses, such as Chamfer Distance, primarily constrain the distance between the predicted point and the target point, and cannot explicitly guarantee that the predicted point is located inside the actual occupied voxel.

[0010] 2. Prone to floating points or inaccurate boundaries. The predicted point may be close to the actual surface but still in free space, leading to instability in the determination of free space boundaries, object outlines, and passable areas.

[0011] 3. Insufficient deployment budget adaptation. While existing sparse point methods reduce computation, they typically lack a unified switching mechanism for query quantity, decoder output depth, and task latency for embodied robot deployment scenarios, making it difficult to adapt to different computing budgets within the same model.

[0012] 4. Insufficient compatibility with multi-source geometric input. Different robots may use pseudo-depth, depth cameras, or LiDAR; without a unified geometric input interface, significant modifications are still required when deploying models to different platforms.

[0013] 5. Insufficient interface between semantic results and embodied tasks. Some methods only focus on offline benchmark output metrics, without fully considering the needs of semantic target localization, cost map construction, and pose generation in navigation, manipulation, and movement operations. Summary of the Invention

[0014] The purpose of this invention is to provide an adaptive three-dimensional semantic occupancy prediction method and apparatus for embodied tasks, so as to at least partially solve the above-mentioned technical problems.

[0015] To solve the above-mentioned technical problems, the present invention provides the following technical solution: An adaptive 3D semantic occupancy prediction method for embodied tasks includes: Step S1: Obtain multi-source sensing input, wherein the multi-source sensing input includes a set of RGB images, calibration parameters corresponding to each image, and at least one geometric input; Step S2: Unify the geometric input according to the calibration parameters to obtain a unified geometric point set; Step S3: Determine the number of queries based on the task budget and initialize the query point input to obtain the query point set; Step S4: Feed the RGB image set, unified geometric point set, and query point set into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set. The AdaOcc model is a learnable neural network, including an encoder and a decoder connected in sequence. The encoder is used to extract image features and geometric features, including an image coding branch and a geometric coding branch. The decoder is used to fuse image features, geometric features, and query points, and outputs sparse semantic point predictions through multi-layer query decoding. Step S5: Generate a three-dimensional semantic occupancy result based on the sparse semantic point set.

[0016] The present invention has the following beneficial effects: 1. This invention unifies geometric inputs of different forms into a point set in a locally occupied coordinate system, and uses this point set simultaneously for query initialization in step S3 and geometric encoding branch input in step S4, so that the same encoder-decoder model can adapt to different geometric input forms such as pseudo-depth, depth camera point cloud and LiDAR point cloud.

[0017] 2. This invention controls the inference budget by adjusting the number of queries and the number of decoder output layers, and improves the model's adaptability to different query budgets through progressive query learning. Fewer query points and fewer decoding layers reduce feature sampling, feature fusion, and intermediate activation storage, thereby reducing latency and GPU memory usage; more query points and more decoding layers provide more comprehensive coverage of occupancy points and iterative correction, thereby improving mIoU, IoU, and geometric details. Since the model has already undergone a gradual increase in query scale during training, when different inference tasks are set with different query numbers in step S3, the problems of insufficient prediction point coverage or insignificant accuracy gains after increasing the budget are less likely to occur.

[0018] 3. This invention uses sparse semantic point prediction and then converts it into voxel or point cloud results, which does not require dense prediction of all three-dimensional spatial units and reduces redundant computation in the idle space. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the adaptive 3D semantic occupancy prediction method for embodied tasks according to the present invention. Figure 2 The principle of the adaptive 3D semantic occupancy prediction method for embodied tasks of the present invention Figure 1 ; Figure 3 The principle of the adaptive 3D semantic occupancy prediction method for embodied tasks of the present invention Figure 2 ; Figure 4 This is a schematic diagram illustrating the multi-view and multi-geometric input adaptive design in this invention; Figure 5 This is a schematic diagram of point optimization including constraint guidance in this invention; Figure 6 This is a schematic diagram illustrating the generation of navigation paths from semantic occupancy results in this invention; Figure 7 This is a schematic diagram illustrating the application of the present invention in navigation, operation, and mobile operation; Figure 8 This is a comparison chart of the three-dimensional semantic occupancy prediction results in this invention; Figure 9 This is a schematic diagram of the structure of the adaptive 3D semantic occupancy prediction device for embodied tasks according to the present invention. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] Terminology definition: AdaOcc: Adaptive 3D Occupancy Prediction.

[0022] Embodied tasks: Tasks performed by robots or intelligent agents in real or simulated 3D environments, such as navigation, manipulation, movement, and target search.

[0023] 3D semantic occupancy: A scene representation method that simultaneously indicates whether a space is occupied and the semantic category of the occupied area in a 3D space.

[0024] RGB image: A visual image containing three channels: red, green, and blue.

[0025] Depth map: Two-dimensional data that records the distance or depth value from image pixels to the camera, which can come from a depth camera or a depth estimation model.

[0026] LiDAR: LiDAR, used to acquire 3D point clouds.

[0027] Point cloud: Data consisting of several three-dimensional points, each of which may contain coordinates, intensity, timestamp, or semantic information.

[0028] Voxel: A spatial unit in a 3D mesh, which can be understood as a small cube or cuboid in space.

[0029] Sparse semantic points: used to represent a small number of three-dimensional points in an occupied region, each point having a semantic category score or semantic feature.

[0030] Query point: An updatable 3D reference point (which may consist of XYZ coordinates and a feature) used in the neural network decoder to actively query and reconstruct the 3D occupied area.

[0031] TPV: Tri-Perspective View, a three-plane view feature that typically includes three orthogonal planes: xy, xz, and yz.

[0032] CPO: Containment-Guided Point Optimization.

[0033] IoU: Intersection over Union, used to evaluate the accuracy of binary occupancy prediction.

[0034] mIoU: mean Intersection over Union, used to evaluate the accuracy of semantic occupancy prediction.

[0035] Open vocabulary occupancy prediction: an occupancy prediction method that allows queries of natural language text to predict the semantic response of three-dimensional occupancy regions.

[0036] This invention addresses the shortcomings of existing 3D semantic occupancy prediction technologies in the deployment of embodied robots, primarily solving the following problems: 1. Different robot platforms may have different numbers of RGB views and different geometric inputs, making it difficult for existing methods to maintain a consistent fit under the same model and process.

[0037] 2. Embossed robots have different requirements for latency, memory, and accuracy in navigation, manipulation, and movement, but existing methods often use fixed query size and fixed decoding depth, which makes it difficult to balance the budgets of multiple tasks.

[0038] 3. Point-based occupation prediction is prone to problems where the predicted point is close to the actual object but still falls into a known empty area, resulting in instability of the floating point and free space boundary.

[0039] 4. Existing occupancy prediction methods are mostly based on offline indicators, and the output results are difficult to be directly used by specific tasks such as navigation, operation, and movement.

[0040] 5. Closed-set semantic categories are difficult to cover the natural language target descriptions in real-world embodied tasks, which limits the scope of use of semantic occupancy results in open-lexical scenarios.

[0041] On the one hand, this invention provides an adaptive 3D semantic occupancy prediction method for embodied tasks, such as... Figure 1 As shown, it includes: Step S1: Obtain multi-source sensing input, wherein the multi-source sensing input includes a set of RGB images, calibration parameters corresponding to each image, and at least one geometric input; Step S2: Unify the geometric input according to the calibration parameters to obtain a unified geometric point set; Step S3: Determine the number of queries based on the task budget and initialize the query point input to obtain the query point set; Step S4: Feed the RGB image set, unified geometric point set, and query point set into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set. The AdaOcc model is a learnable neural network, including an encoder and a decoder connected in sequence. The encoder is used to extract image features and geometric features, including an image coding branch and a geometric coding branch. The decoder is used to fuse image features, geometric features, and query points, and outputs sparse semantic point predictions through multi-layer query decoding. Step S5: Generate a three-dimensional semantic occupancy result based on the sparse semantic point set.

[0042] The adaptive 3D semantic occupancy prediction method for embodied tasks (embodied robot or intelligent agent tasks) provided by this invention unfolds in the order of "first forming input, then calling the model, and finally obtaining the occupancy result": First, step S1 acquires RGB images, calibration parameters, and at least one geometric input; step S2 unifies geometric data from different sources into a local 3D space; then step S3 determines the number of queries based on the task budget and forms query point inputs; then step S4 inputs the RGB image, the unified geometric point set, and the query points into the AdaOcc model composed of an encoder and a decoder to generate sparse semantic points; finally, step S5 converts the sparse semantic points into 3D semantic occupancy results. Based on this workflow, the training phase further provides specific methods to enable the model to possess multi-budget inference, sensor adaptation, and geometric consistency constraint capabilities.

[0043] Overall Architecture

[0044] like Figure 2 As shown, this invention addresses RGB images from different embodied platforms and at least one geometric input. It utilizes the AdaOcc model to output sparse semantic points and further generates a 3D semantic occupancy result, which is then provided to downstream tasks such as navigation, robotic arm manipulation, and mobile manipulation. The geometric input can be a depth map, a point cloud obtained from a depth camera, a LiDAR point cloud, a pseudo-depth estimated from an image, or other geometric observations that can be converted into a 3D point set. Figure 2 The RGB, Depth, and LiDAR areas on the left indicate that different robot platforms (such as humanoid robots, wheeled-arm robots, and quadruped robots) may use different geometric inputs, rather than requiring all platforms to simultaneously possess the aforementioned geometric sensors. The AdaOcc area in the middle represents the process of completing 3D occupancy prediction around the model, where the learnable model mainly consists of encoders and decoders. The 3D occupancy results (i.e., 3D semantic prediction) and task icons on the right indicate that the occupancy results generated by the model output can be reused for different embodied tasks. In this figure, Heterogeneous Robots & Sensors refers to heterogeneous robots and sensors, and the model implements view-adaptive, sensor-adaptive, and budget-adaptive technologies.

[0045] In this invention, the "system flow" and "model structure" are distinguished as follows: Steps S1, S2, S3, S5, and applications A1 to A4 belong to the input processing, inference control, result transformation, and task application flow surrounding the model; Step S4 corresponds to the forward prediction process of the AdaOcc model. The AdaOcc model refers to its learnable neural network portion, mainly composed of an encoder and a decoder. The model receives the RGB image obtained in Step S1, the unified geometric point set obtained in Step S2, and the query points obtained in Step S3, completes the forward computation under a given query quantity constraint, and outputs sparse semantic point predictions. In other words, the steps before Step S3 are used to prepare the model input, Step S4 is the model call itself, and Step S5 is the generation of the model output. The encoder is used to extract image features and geometric features, and is a dual-branch encoder, including an image coding branch and a geometric coding branch. The image coding branch corresponds to... Figure 3 The Image Encoder in the code is used to extract appearance, texture, and semantic context features from one or more RGB views; the geometric encoding branch corresponds to... Figure 3 The 3D Encoder and its three-plane feature representation are used to extract local three-dimensional structural features from a unified set of geometric points. Thus, the encoder solves the problem of feature representation when different RGB viewpoints and different geometric input shapes enter the same model.

[0046] The decoder corresponds to Figure 3 The multi-layer decoder in the model receives image features, geometric features, and query points obtained from the encoder's output, based on the current budget. It then generates sparse semantic points through multi-layer query decoding. Each decoder layer samples and fuses information from the image and geometric features based on the current query location, updates the query features, and outputs the 3D predicted point location and its semantic score. The query quantity control and decoder output layer selection belong to the inference control mechanism: they do not change the basic structure of the encoder and decoder, but rather adjust the inference time, memory usage, and prediction accuracy by changing the number of query points involved in decoding or selecting earlier / later decoder outputs. Therefore, step S3 first determines the query quantity and generates query point inputs outside the model, and step S4 then uses the decoder within the model to complete sparse point prediction under this budget.

[0047] The training phase introduces a supervision objective and CPO inclusion constraints based on the encoder and decoder described above. CPO corresponds to... Figure 3 The Containment Loss area above and Figure 5It is mainly used to improve the geometric consistency of prediction points during the training phase. When used online, the system executes the inference process through steps S1 to S5. In step S4, the model completes the forward prediction through the trained encoder and decoder without having to run the CPO loss calculation.

[0048] Usage / Inference Phase: Online 3D Semantic Occupation Prediction Process of the Trained Model

[0049] The steps of this invention will be described in detail below.

[0050] Step S1: Acquire multi-source sensor input.

[0051] In this step, the robot or embodied agent acquires environmental perception data at the current moment. This perception data includes a set of RGB images (including one or more RGB images), calibration parameters corresponding to each image (including camera intrinsic parameters and camera extrinsic parameters relative to the robot's local or occupied coordinate system), and at least one geometric input. The geometric input can be a pseudo-depth map estimated from RGB images (using a depth model), a depth map or point cloud obtained from a depth camera, a point cloud obtained from LiDAR, or geometric points obtained from other 3D sensors. Understandably, the geometric input can come from pseudo-depth, true depth, LiDAR, ToF, structured light, binocular depth, simulated depth, historical multi-frame reconstructed point clouds, or SLAM maps.

[0052] Thus, after step S1, the system generates a perception data packet with calibration information. This perception data packet includes a set of RGB images, calibration parameters (including camera intrinsic parameters, extrinsic parameters, or calibration relationships between sensors), and a set of original geometric inputs. This perception data packet retains essential information such as "which viewpoint the image comes from, which sensor the geometric points come from, and how coordinates are transformed between the sensors."

[0053] Step S2: Unify the geometric input according to the calibration parameters to obtain a unified geometric point set.

[0054] As an optional embodiment, step S2 includes: Step S21: Based on the calibration parameters, transform the geometric input to the same local occupancy coordinate system; In practice, based on the perception data packet obtained in step S1, the system first converts different geometric sources to the same local occupancy coordinate system.

[0055] If the original geometric input is a depth map, then backprojection is performed on the valid pixels in the depth map. For pixel coordinates... and depth value The camera's intrinsic parameter is focal length. and principal point coordinates The three-dimensional points in the camera coordinate system are:

[0056] The 3D points obtained from the depth map can be represented as homogeneous coordinates. .

[0057] If the input is a LiDAR point cloud or a point cloud from another 3D sensor, it can also be represented as a homogeneous point in that sensor's coordinate system. Based on the extrinsic parameters or calibration relationships in step S1, points in different sensor coordinate systems are uniformly transformed to a locally occupied coordinate system.

[0058] in, The transformation from camera to occupied coordinate system, LiDAR to occupied coordinate system, or other sensor calibration transformations can be performed based on the specific sensor.

[0059] Step S22: After completing the coordinate transformation, perform effective range clipping on all points, retaining only the points located within the locally occupied range, to obtain the unified geometric point set.

[0060] In practice, after the coordinate transformation is completed, the system performs effective range cropping on all points, retaining only those points located within their localized areas. Points within the set yield a unified geometric point set. This step does not change the meaning of the physical coordinates of the points, but rather unifies the geometric points from different sensors into the same local occupancy coordinate system and removes points outside the local occupancy range.

[0061] Thus, through step S2, Figure 3 In the middle, Calibration corresponds to the calibration and transformation process of LiDAR points, and Lift corresponds to the process of backprojecting the depth map into 3D points. Together, they form... Figure 3 The unified geometry input is shown in the Geometry Input area.

[0062] Step S3: Determine the number of queries based on the task budget and initialize the query point input to obtain the query point set.

[0063] In this step, the query point can be generated by geometric point sampling, random sampling, learned embedding, semantic hotspot sampling, deep confidence sampling, historical trajectory sampling, or a combination of the above methods.

[0064] As an optional embodiment, step S3 includes: Step S31: Determine the number of queries based on the current task type, device computing power, real-time requirements, or user-specified configuration; In practice, before inputting data into the AdaOcc model, the system determines the number of queries based on the current task type, device computing power, real-time requirements, or user-specified configuration. This step is equivalent to first determining the "size of the query input sent to the model decoder this time". For example, low-latency navigation tasks can use a smaller... This reduces feature sampling, feature fusion, and point prediction computation in subsequent decoders; fine-grained operation tasks can utilize larger [scales / resources]. To generate more comprehensive semantic point predictions.

[0065] Step S32: Sample a portion of points from the unified geometric point set as geometric guidance query points, and randomly sample a portion of points from the locally occupied range as supplementary query points to obtain the query point set.

[0066] In practice, the system obtains the unified geometric point set from step S2. A subset of points (which can be sampled using various sampling methods in this field, such as the FPS farthest point sampling algorithm) are used as geometry-guided query points, and the results are obtained from a local 3D range. Random sampling (corresponding to) Figure 3 Point Sampling in the model uses a subset of points as supplementary query points. If the geometric query ratio is... The number of queries is Then it is acceptable:

[0067] in, For the set of geometrically guided query points, The set of query points is randomly supplemented. Each query point retains its three-dimensional spatial position as the initial reference position for the decoder. The query features can be initialized as a zero vector or a learnable vector.

[0068] Therefore, a unified geometric point set There is a dual use: the unified geometric point set obtained in step S2 is used as the input of the geometric coding branch in step S4, and also used for the geometric guided query initialization in step S3. This allows the input geometry to not only provide learnable spatial structural features, but also directly affect the spatial distribution of sparse query points.

[0069] This step S3 corresponds to Figure 3 The Adaptive Query Budget area in the model does not perform occupation prediction alone, but rather forms a third type of model input, namely the query point input, before the model's forward prediction.

[0070] Step S4: Feed the RGB image set, unified geometric point set and query point set into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set.

[0071] In specific implementation, based on the RGB image set obtained in step S1 and the unified geometric point set obtained in step S2... The system inputs the above three types of inputs, along with the query point set obtained in step S3, are fed into the AdaOcc model to complete a forward prediction. The AdaOcc model performs this step as a whole, and its learnable structure consists of two parts connected in sequence: an encoder and a decoder. The encoder is used to extract image features and geometric features, including an image coding branch and a geometric coding branch. The decoder is used to fuse image features, geometric features, and query points, and outputs sparse semantic point predictions through multi-layer query decoding.

[0072] First, the encoder's image encoding branch receives the RGB image set (corresponding to) from step S1. Figure 3 The system uses Multi-View Images (MLES) to input images from each viewpoint after normalizing their size and pixels. Figure 3 The Image Encoder in the code generates image features at one or more scales. These image features preserve the scene's appearance texture, object boundaries, and semantic context information. If multiple camera views exist, the image encoding branch extracts image features from each viewpoint separately, preserving their corresponding camera calibration relationships so that these image features can be aligned with the 3D query location in the decoder. Specifically, the image encoder in the image encoding branch can be EfficientNet, ResNet, ViT, SwinTransformer, CLIP visual encoder, DINO series visual encoder, or other basic visual models.

[0073] Secondly, the geometric encoding branch of the encoder is used to first convert the unified geometric point set into a sparse three-dimensional structure that the network can process, to obtain sparse voxel features, and then to encode the sparse voxel features in three dimensions to obtain sparse three-dimensional features. Finally, the sparse three-dimensional features are compressed. In one embodiment, they can be compressed into three orthogonal plane features, where the horizontal plane features represent the scene layout and the two vertical plane features represent the height direction structure.

[0074] Specifically, the encoder's geometric coding branch receives the unified geometric point set obtained in step S2. The model first transforms the point set into a sparse 3D structure that the network can process. For any point... According to the geometrically encoded voxel size Calculate its sparse voxel index:

[0075] in, Let the coordinates of the geometric point be the three-dimensional coordinates in the locally occupied coordinate system. For the minimum coordinates within the local three-dimensional range, For three-dimensional integer voxel indexes, The size of the voxel for geometric coding is the grid resolution used by the geometric coding branch when discretizing the local 3D space into sparse voxels. It is used to convert continuous 3D coordinates into discrete voxel indices. For example, if a voxel size is defined as 0.08m, then... That is 0.08m. The above subtraction, division, and rounding down are all performed according to... , , The three coordinate components are processed separately, that is:

[0076] If multiple geometric points fall into the same sparse voxel, the basic geometric information such as the coordinates or number of points within that voxel can be aggregated to obtain the initial geometric features of that voxel.

[0077] Figure 3 The 3D Encoder in this invention performs 3D encoding on the aforementioned sparse voxel features to obtain sparse 3D features (i.e., local spatial structure features). To reduce memory overhead, this invention further compresses the sparse 3D features into three orthogonal plane features, where the horizontal plane features represent the scene layout and the two vertical plane features represent the height-direction structure. Compared to directly saving complete dense 3D features, the three-plane features can preserve the spatial structure while reducing the computational and storage overhead caused by idle areas. Specifically, the geometric encoding branch can use sparse 3D convolution, PointNet-like networks, Point Transformer, voxel Transformer, BEV encoder, TPV encoder, or multi-scale 3D sparse encoder; the three-plane features can be replaced with BEV features, hexahedral projection features, multi-scale voxel features, octree features, or sparse tensor features.

[0078] Finally, the decoder receives the image features output by the image coding branch, the geometric features output by the geometric coding branch, and the query points (set) obtained in step S3. Based on these inputs, the system sends the query points into... Figure 3The multi-layer decoder in the model consists of several 3D sampling locations generated near each query point in each layer. These sampling locations are projected onto image features through camera calibration to obtain appearance and semantic information, and 3D structural information is also obtained from geometric features. The decoder fuses this information to update the query features and predicts the 3D coordinates and semantic scores of one or more 3D semantic occupancy points. In implementations using binary occupancy branches or open vocabulary branches, occupancy scores or text alignment semantic features can also be output. The predicted points of the current layer can also be used to update the query reference positions of the next layer, allowing the multi-layer decoder to progressively generate sparse semantic point predictions from coarse to fine. Specifically, the decoder can employ a Transformer decoder, a deformable attention decoder, a graph neural network decoder, a recursive refinement decoder, or a hierarchical coarse-to-fine decoder; the output (corresponding to...) Figure 3 The Multi-layer Output can be a semantic point cloud or a feature point cloud (classification head or feature head).

[0079] Thus, through step S4, the RGB image (set), the unified geometric point set, and the query point (set) enter the model together: the RGB image enters the image coding branch, the unified geometric point set enters the geometric coding branch, and the query point enters the decoder; the model finally outputs a sparse semantic point set.

[0080] As an optional embodiment, step S4 may further include the following prior to: Step S3': Determine the target number of decoding layers based on task requirements and device resources; At this point, step S4 is further defined as follows: The RGB image set, unified geometric point set, query point set, and target decoding layer are fed into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set.

[0081] Specifically, before model invocation, the system can determine the target number of decoding layers K based on task requirements and device resources, where K is no greater than the total number of decoder layers L. The system can pre-calculate the inference latency, memory usage, and prediction accuracy corresponding to different query quantities and different numbers of decoding layers on the target device or validation set, forming a budget configuration table. During online inference, if the task is low-latency navigation or target search, a smaller K is selected, and only the first K layers of decoders are executed and the prediction result of the Kth layer is output; if the task is fine-grained operation or high-precision map construction, a larger K or the output of the complete L layers of decoders is selected; if both latency and accuracy constraints exist, the K with the highest accuracy under the conditions of meeting the latency threshold and memory threshold is selected. Since the output of multiple layers of decoders is supervised in the subsequent training phase, the output of earlier decoding layers also has usable occupancy prediction capabilities, and therefore can be used for low-latency settings.

[0082] Step S5: Generate a three-dimensional semantic occupancy result based on the sparse semantic point set.

[0083] As an optional embodiment, step S5 includes: Step S51: Based on the sparse semantic point set, first filter out low occupancy confidence points, and then map the remaining prediction points onto a three-dimensional voxel grid to obtain a three-dimensional semantic occupancy voxel map or semantic point cloud. In practice, based on the sparse semantic point set output in step S4, the system first filters out low-occupancy confidence points, and then maps the remaining prediction points onto a three-dimensional voxel grid (corresponding to...). Figure 3 Voxelization (or voxelization in the model) yields a 3D semantic occupancy voxel map or semantic point cloud. Let the minimum coordinates of the local 3D extent be... voxel size is The predicted point coordinates are The corresponding voxel index is:

[0084] in, To predict the coordinates of a 3D point, This is the 3D voxel index corresponding to the predicted point. This formula also indicates that the grid number is calculated separately for each of the three coordinate components, rather than for a single scalar.

[0085] Step S52: If multiple prediction points fall into the same voxel, select several points with higher confidence based on occupancy confidence and semantic confidence, and average or weighted average their semantic scores. The final semantic category of the voxel is determined by the aggregated semantic score, and the occupancy confidence is obtained by aggregating the confidence scores of prediction points falling into that voxel.

[0086] thus, Figure 3The multi-layer output on the right is converted into a robot-readable 3D semantic occupancy representation. This step S5 converts sparse semantic points or voxel semantic maps into a spatial representation that can be directly used for navigation, manipulation, and movement operations, instead of just outputting offline benchmark metrics.

[0087] Thus, by following steps S1 to S5, the embodied adaptive reasoning mechanism can be achieved. Specifically, when steps S1 to S5 are repeatedly executed on different robot platforms or in different embodied tasks, the system can adjust the reasoning method according to sensor configuration and task resource constraints.

[0088] The first type is sensor adaptation. Since step S2 has unified different geometric inputs such as pseudo-depth, depth camera or LiDAR into a point set interface, the query initialization in step S3, the model encoding and decoding in step S4 and the occupation output in step S5 can all be run on the unified interface.

[0089] The second type is performance-adaptive. This is due to the number of queries in step S3. Adjustable; the number of decoder layers or output layers (i.e., the aforementioned target decoding layer number K) in step S4 is selectable, allowing the system to switch between computational cost and prediction accuracy in different application scenarios. Smaller This reduces the number of query points entering the decoder, thereby reducing computations across feature sampling, feature fusion, and point prediction; fewer decoder layers reduce the number of consecutive refinements and intermediate activation storage, thus reducing inference latency and GPU memory usage. Conversely, a larger... More or more decoder layers can generate more comprehensive semantic point predictions and perform more thorough iterative corrections, thereby improving geometric detail and semantic integrity. For example... Figure 4 As shown, the same model can run under conditions such as RGB plus depth points, RGB plus LiDAR points, and different numbers of camera viewpoints, and produce 3D occupancy output.

[0090] Training phase: Used to train the AdaOcc model described above.

[0091] T1: Construct a supervision signal based on the training data.

[0092] Preferably, the AdaOcc model is trained using training samples with supervised signals. Specifically, to obtain a model capable of predicting sparse semantic points in step S4 and used in conjunction with the inference process of steps S1 to S5, samples with real semantic occupancy labels are used during the training phase. Each training sample includes an RGB image, geometric input, and calibration parameters in the same form as in step S1, along with real semantic occupancy voxel labels. The system divides real semantic occupancy voxels into occupied voxels and known empty voxels; the center point of the occupied voxel and its semantic category constitute a point-level supervised objective, used to supervise the sparse semantic points output by the model in step S4.

[0093] T2: Supervised training of multi-layer decoders.

[0094] After obtaining the point-level supervision target in T1, the training process can apply supervision to the output of each layer of the decoder in step S4. Specifically, the system first performs point set matching or nearest neighbor assignment between the predicted points and the real occupied points, and then calculates the point set reconstruction loss and semantic classification loss. The point set reconstruction loss is used to ensure that the predicted points cover the real occupied areas, and the semantic classification loss is used to ensure that the predicted points have the correct semantic categories. Since each layer of the decoder is supervised, the output of intermediate layers can also form usable 3D occupancy results. Therefore, even when fewer decoder layer outputs are selected in step S4 and invoked as a low-latency setting, the model can still provide effective output.

[0095] T3: Progressive Query Learning

[0096] Due to the number of queries in step S3 The computational cost of the decoder in step S4 and the output details in step S5 are directly affected. Therefore, this invention preferably adopts progressive query learning during the training phase. This training design is one of the important technical features of this invention in achieving multi-query budget adaptability. If the model is always trained with a fixed number of queries, the model mainly adapts to a single query scale. When fewer or more query points are set in step S3 in different inference tasks, the number of query points, the spatial distribution of query points, and the scale of feature interactions received by the decoder will all change, which can easily lead to insufficient prediction coverage, unstable semantic point distribution, or the inability of high query budgets to be fully converted into accuracy gains.

[0097] To address the aforementioned issues, this invention does not merely change the number of queries during the inference phase, but rather employs progressive query learning during the training phase, allowing the model to gradually experience a growing scale of query input. Specifically, progressive query learning can utilize linear growth, piecewise growth, training with random query numbers, dynamic allocation based on task complexity, allocation based on the number of viewpoints, or allocation based on semantic uncertainty. In the initial training phase, fewer query points are used, allowing the model to learn coarse-grained geometric and semantic structures first, avoiding the initial dispersion of supervision by a large number of sparse query points. Subsequently, the number of queries is gradually increased with each training epoch, enabling the model to gradually learn finer-grained occupancy predictions and adapt to the decoder feature fusion process under different query numbers. (Training...) The number of queries used in a round can be represented as:

[0098] in, This represents the maximum number of queries. This represents the initial number of queries. To begin increasing the number of training rounds, The interval for the increase in the number of queries. This represents the number of queries added each time.

[0099] It should be noted that progressive query learning is not a regular training hyperparameter, but a training design used to adapt the model to different query input sizes during the training phase; this design supports the ability to set different query budgets before the model is invoked for different inference tasks.

[0100] This mechanism allows the same model to learn decoding behavior under different query budgets during training. This enables different query counts to be set before different inference tasks begin via step S3, without needing to train a separate model for each query count. Compared to simply changing the number of query points in deployment or inference configuration while keeping the query size fixed during training, this training design improves model usability under low query budgets and accuracy utilization under high query budgets.

[0101] T4: Training with changes in the number of viewpoints.

[0102] Since the input for step S1 may come from different numbers of camera views, it is preferable to randomly vary the number of input views during training. Specifically, in each training iteration, the system can randomly retain... Each camera viewpoint is used, and only the RGB images and camera calibration information corresponding to these preserved viewpoints are used. For geometric input, the system does not need to randomly switch geometric modalities in each training iteration. Instead, based on the visible range of the current preserved RGB viewpoints, it synchronously generates, filters, or matches geometric inputs such as depth map backprojection points, pseudo-depth points, depth camera point clouds, or LiDAR point clouds, ensuring that the set of geometric points entering step S2 is consistent with the current RGB viewpoint set in terms of spatial visible range. Regardless of the original shape of the geometric input, the system performs coordinate transformation and range clipping according to the unified geometric interface of step S2. If the number of preserved viewpoints is... This allows the current query budget to be related to the number of viewpoints:

[0103] This design allows the model to see multiple numbers and combinations of viewpoints during the training phase, and synchronizes the corresponding geometric input with the current RGB visibility range, thereby improving the model's adaptability to different camera configurations and different geometric input shapes.

[0104] T5: Point optimization with constraint guidance.

[0105] The inventors discovered that traditional point set reconstruction loss typically only constrains the distance between the predicted point and the actual occupied point, and cannot explicitly guarantee that the predicted point is located inside the actual occupied voxel. Therefore, even if the predicted point is close to the target point, it may still fall into a known empty area, forming a floating point or boundary error. Existing point-based occupation prediction methods generally do not identify and specifically constrain such predicted points that are "located in a known empty area but close to the object surface".

[0106] To address this problem, this invention proposes a constraint-guided point optimization (CPO). For example... Figure 5 As shown in the figure, the red area in the upper right corner represents outliers that are prone to occur in traditional point-based prediction, and the CPO module in the lower left corner represents the constraint that our invention (Ours) imposes on the predicted points in the empty region by occupying voxel boxes. This mechanism first determines whether the predicted point falls within a known empty region in the effective supervision region; if so, it calculates the distance from the predicted point to the nearest real occupied voxel box and constrains it back to the effective occupied region using a loss function; if not, no inclusion constraint is imposed on the predicted point, and the predicted point still participates in regular training according to the semantic classification loss and point set reconstruction loss.

[0107] Let the set of real occupied voxel centers be... voxel size is The predicted point is The distance from the predicted point to the nearest occupied voxel box is:

[0108] Let the set of prediction points that fall within the effective monitoring region and are located in the known free region be . Then the constraint loss can be simplified as:

[0109] in, Describes the Smooth L1 loss function. This represents a non-negative boundary bias used to maintain constraints near the boundary. In actual training, this loss can be applied to one or more decoder layers. This loss only constrains predicted points in known empty regions and does not suppress correct predicted points in effectively occupied regions.

[0110] Understandably, the distance metric used to constrain predicted points in CPO is not limited to the distance from the point to the actual occupied voxel box; other metrics that characterize the spatial deviation between the predicted point and the nearest actual occupied region can also be used. For example, the distance from the predicted point to the center of the nearest actual occupied voxel, the distance from the predicted point to the surface of the nearest actual occupied voxel, the distance based on the signed distance field, the distance based on the nearest neighbor occupied voxel, or the occupancy probability constraint obtained from differentiable grid sampling can be used. The common effect of these alternative methods is to impose a constraint on anomalous predicted points that fall into known empty regions, oriented towards the actual occupied region.

[0111] T6: Overall training objective.

[0112] The optimal training objectives for the AdaOcc model include semantic classification loss, point set reconstruction loss, and inclusion constraint loss in T5:

[0113] in, Indicates the number of decoder layers. Indicates the interlayer attenuation coefficient. Indicates the first Layer semantic classification loss, Indicates the first Layer point set reconstruction loss, This indicates that the loss includes constraint. , and Preset weight coefficients are used to control the relative importance of semantic classification loss, point set reconstruction loss, and inclusion constraint loss in the overall training objective. During training, these weight coefficients can be set or adjusted based on semantic accuracy, geometric reconstruction quality, and floating point suppression effect on the validation set. It is important to emphasize that... It is not a simple superposition of ordinary training objectives, but a special constraint proposed to address the previously insufficiently addressed problem in point-based occupation prediction: "the predicted point falls into a known empty area".

[0114] Understandably, semantic loss can be achieved using cross-entropy, Focal Loss, Dice Loss, or class reweighting loss; point reconstruction loss can be achieved using Chamfer Distance, DCD, EMD, or Smooth L1 matching loss.

[0115] Embossed Task Application Phase

[0116] A1: Navigation application.

[0117] In other words, step S5 may include the following: Step S6: Apply navigation.

[0118] In specific implementation, based on the three-dimensional semantic occupancy results (specifically, the three-dimensional semantic occupancy point cloud or voxel map obtained in step S5 of the usage phase), the ground or passable categories are projected onto a two-dimensional or 2.5D map as passable areas, and non-ground occupancy points are designated as obstacles. These obstacles are then expanded according to the robot's radius and safety distance. Path planning is then performed based on the semantic target location. For example... Figure 6 As shown, the semantic occupation result can be converted into a cost map, and an executable path is generated between the starting point and the target point using algorithms such as A-Star Algorithm.

[0119] A2: Operation and application.

[0120] In other words, step S5 may include the following: Step S6': Perform the operation and application.

[0121] In practice, based on the three-dimensional semantic occupancy results (specifically, the semantic point cloud obtained in step S5), the system extracts the target object point set and container point set for operational use. For example, in the task of grasping a bottle and placing it in a basket, the bottle point set and basket point set can be extracted based on semantic tags or open vocabulary queries. The bottle point set is input into the grasping planner to generate the robotic arm's grasping pose; the geometric center of the basket point set can be used as the placement position.

[0122] A3: Mobile operating application.

[0123] In other words, step S5 may include the following: Step S6'': Perform mobile operation application.

[0124] In practice, based on the navigation capabilities of A1 and the operational capabilities of A2, the mobile operation task can form a continuous execution process. The robot first uses the semantic occupancy results obtained from long-distance observation to determine the target area, and moves to the vicinity of the target according to the path generated by A1; after reaching the vicinity of the target, the robot executes steps S1 to S5 again to obtain the local semantic occupancy point cloud, and generates the grasping pose and placement position according to A2. Figure 7 It showcases real-world examples of embodied applications in navigation, manipulation, and mobile manipulation, with the top section representing navigation, the middle section representing manipulation, and the bottom section representing mobile manipulation.

[0125] A4: Optional application of open lexical semantic occupation.

[0126] In other words, step S5 may include the following: Step S6''': Apply natural language target navigation, semantic target search, or open category operation tasks.

[0127] In an optional implementation, the decoder in step S4, in addition to predicting the binary occupancy score or closed-set semantic category, also predicts semantic features aligned with the text semantic space. After generating the 3D semantic occupancy points in step S5, if the user inputs a natural language query, the system encodes the text into a semantic vector and calculates the similarity with the semantic features of the 3D occupancy points. The system activates relevant 3D regions based on the matching score for natural language target navigation, semantic target search, and open-category operation tasks.

[0128] Understandably, embodied tasks, in addition to navigation, operation, and mobility, can also be used for inspection, target search, indoor service robots, assistive robots, AR / VR scene understanding, and autonomous driving perception.

[0129] The following are supplementary explanations for each of the attached figures.

[0130] Figure 2 This is a schematic diagram of the overall invention. The diagram corresponds to the overall architecture, showing how an RGB image and at least one geometric input are processed by the AdaOcc model to obtain sparse semantic points, and further generate 3D semantic occupancy results to serve embodied tasks such as navigation, manipulation, and movement. The geometric input can take different forms, such as depth maps, pseudo-depth, depth camera point clouds, or LiDAR point clouds. The input area on the left corresponds to steps S1 and S2, the model body in the middle AdaOcc area corresponds to step S4, the query budget before model invocation corresponds to step S3, the occupancy result generation after model invocation corresponds to step S5, and the task area on the right corresponds to applications A1 to A3, illustrating the overall closed loop of the invention from sensor input to embodied task output.

[0131] Figure 3This is a flowchart of the system and model of the present invention. The diagram corresponds to the core processes of the usage and training phases, showing input acquisition, geometric input unification, image encoding, 3D encoding, adaptive query budget, multi-layer query decoding, constraint loss, and multi-layer output. Specifically, Image Encoder, 3D Encoder, and Multi-layer Decoder correspond to the encoder and decoder in the model described in this invention; Geometry Input, Calibration, Lift, Adaptive Query Budget, Multi-layer Output, and Containment Loss correspond to external input processing, inference control, output transformation, or training constraint processes, respectively. In the diagram, Geometry Input corresponds to the geometric input acquired in step S1; Calibration and Lift correspond to step S2; Adaptive Query Budget corresponds to steps S3 and T3; Image Encoder, 3D Encoder, three-plane features, and Multi-layer Decoder correspond to steps S4 and T2; Multi-layer Output corresponds to step S5; and Containment Loss corresponds to T5 and T6.

[0132] Figure 4 This diagram illustrates multi-view and multi-geometric input adaptation. It corresponds to the embodied adaptive inference mechanism and T4, demonstrating that the same model can adapt to different sensor combinations and different numbers of viewpoints. Different input combinations in the diagram correspond to the multi-source sensing inputs in step S1, which are uniformly entered into the geometric point set interface in step S2. The different output results on the right illustrate that sensor adaptation is not merely about input compatibility, but rather about generating usable 3D semantic occupancy results under different input conditions.

[0133] Figure 5 This diagram illustrates point optimization with constraint guidance in the CPO (Constraint-Oriented Point Optimization) module. Corresponding to T5 and T6, this diagram illustrates the training-side innovation of this invention. The red outliers area in the diagram corresponds to the problem of "predicted points near the object surface but still falling into known empty regions." The CPO module corresponds to the constraint-guided loss, which calculates the distance from the predicted point to the nearest truly occupied voxel box and constrains outlier predicted points located in empty regions back to the effectively occupied regions. This diagram addresses technical issues regarding geometric consistency and floating points.

[0134] Figure 6This diagram illustrates the generation of a navigation path from the semantic occupancy result. Corresponding to A1, it explains how the 3D semantic occupancy result output in step S5 is converted into a cost map and path planning result usable for navigation. The traversable area, obstacle area, starting point, target point, and planned path in the diagram correspond to the ground / obstacle projection, obstacle inflation, semantic target localization, and A-satellite path generation steps in A1, respectively.

[0135] Figure 7 This diagram illustrates the application of the present invention (AdaOcc) in navigation, manipulation, and movement. Figures A1 to A3 illustrate that the output of the invention is not an offline evaluation result, but rather a spatial interface that can serve as a real-world embodied task. The navigation example in the diagram corresponds to semantic target localization and path execution; the manipulation example corresponds to target object point set extraction, grasp pose generation, and placement position calculation; and the movement operation example corresponds to a combined process of navigation followed by manipulation.

[0136] Figure 8 This is a comparison chart of the 3D semantic occupancy prediction results. The chart illustrates the experimental results and qualitatively demonstrates the improvements of this invention (AdaOcc) over existing methods (GT (Ground Truth), SplatSSC) in scene structural integrity, object boundary, and semantic category prediction.

[0137] In summary, the present invention has at least the following technical effects: 1. In response to the technical problem that "different robot platforms may have different numbers of RGB views and different geometric inputs, and existing methods are difficult to maintain uniformity and adaptability under the same model and process", this invention unifies the geometric inputs of different forms into a point set under the local occupied coordinate system, and uses this point set for both the query initialization in step S3 and the geometric encoding branch input in step S4, so that the same encoder-decoder model can adapt to different geometric input forms such as pseudo depth, depth camera point cloud and LiDAR point cloud.

[0138] 2. Addressing the technical problem that "embodied robots have different requirements for latency, memory, and accuracy in navigation, manipulation, and movement, but existing methods often use fixed query sizes and fixed decoding depths, making it difficult to accommodate various task budgets," this invention controls the inference budget by adjusting the number of queries and the number of decoder output layers. Furthermore, it enhances the adaptability of the same model to different query budgets through progressive query learning in step S3, enabling the same encoder-decoder model to switch between different task requirements such as low-latency navigation and high-precision manipulation. Fewer query points and fewer decoding layers reduce feature sampling, feature fusion, and intermediate activation storage, thereby reducing latency and memory usage. More query points and more decoding layers provide more comprehensive occupancy point coverage and iterative correction, thereby improving mIoU, IoU, and geometric details. Since the model has already undergone a gradual increase in query size during training, when different query numbers are set for different inference tasks through step S3, the problems of insufficient prediction point coverage or insignificant accuracy gains after increasing the budget are less likely to occur.

[0139] 3. In response to the technical problem that “point-based occupancy prediction is prone to the problem that the predicted point is close to the real object but still falls into the known free area, resulting in the instability of the floating point and the free space boundary”, the CPO loss of this invention explicitly constrains the predicted point falling into the known free area to move into the occupancy voxel, which can reduce the number of floating points and improve the free space boundary.

[0140] 4. In response to the technical problem that "existing occupancy prediction methods are mostly based on offline indicators, and the output results are difficult to be directly used by embodied tasks such as navigation, operation and mobility operations", the semantic point cloud or voxel map output by this invention can be converted into target localization results, cost map, path planning input, grasping pose generation input and placement position estimation input, thus making it suitable for navigation, operation and mobility operation tasks.

[0141] 5. In response to the technical problem that "closed-set semantic categories are difficult to cover the natural language target description in real embodied tasks, which limits the scope of use of semantic occupancy results in open-vocabulary scenarios," this invention can optionally output text alignment semantic features, enabling robots to query target regions in three-dimensional space based on natural language, thereby expanding the scope of use of closed-set semantic occupancy prediction.

[0142] 6. This invention uses sparse semantic point prediction and then converts it into voxel or point cloud results, which does not require dense prediction of all three-dimensional spatial units and reduces redundant computation in the idle space.

[0143] 7. The experimental results are significant. In specific experiments, the proposed method achieves better mIoU and IoU than existing methods on Occ-ScanNet and Occ-ScanNet-mini, and demonstrates its usability in real robot navigation, manipulation, and mobility tasks.

[0144] To illustrate the effectiveness of this invention compared to existing methods, some key experimental results are listed below. Table 1 shows the improvement in semantic occupancy prediction accuracy achieved by this invention, and Table 2 shows that this invention can achieve adaptive inference under different computational budgets by adjusting the number of queries and the number of decoder output layers.

[0145] like Figure 8 As shown, the three-dimensional semantic occupancy results output by this invention have more stable performance in terms of indoor layout, object structural integrity and boundary continuity compared with existing methods, and can serve as an intuitive supplement to the quantitative accuracy improvement in Table 1. Figure 8 Together with Table 1, this addresses technical questions regarding the prediction of quality and geometric boundary stability, in which Figure 8 To provide a visual representation, Table 1 provides quantitative indicators.

[0146] Table 1. Accuracy comparison between the present invention and existing methods on the 3D semantic occupancy prediction task.

[0147] As shown in Table 1, on the same dataset, the present invention achieves higher mIoU and IoU compared to existing superior methods, indicating that the present invention can improve semantic category discrimination ability and overall occupancy prediction quality.

[0148] Table 2. Adaptive inference performance of the present invention under different query counts and decoder output layers.

[0149] As shown in Table 2, this invention can strike a balance between inference time, memory usage, and prediction accuracy by adjusting the number of queries or selecting different decoder layer outputs. Specifically, using a 3-layer decoder results in lower inference time and memory usage compared to a 6-layer decoder, but correspondingly lower mIoU and IoU; using a full 6-layer decoder increases computational cost but provides higher prediction accuracy. Therefore, this invention is suitable for deployment on embodied robot platforms with varying computing power and task accuracy requirements.

[0150] On the other hand, corresponding to the above method, the present invention also provides an adaptive three-dimensional semantic occupancy prediction device for embodied tasks, such as... Figure 9 As shown, it includes: The acquisition module 11 is used to acquire multi-source sensing input, wherein the multi-source sensing input includes a set of RGB images, calibration parameters corresponding to each image, and at least one geometric input; Unification module 12 is used to unify the geometric input according to the calibration parameters to obtain a unified geometric point set; Initialization module 13 is used to determine the number of queries based on the task budget and initialize the query point input to obtain the query point set; The prediction module 14 is used to feed the RGB image set, the unified geometric point set, and the query point set into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set. The AdaOcc model is a learnable neural network, which includes an encoder and a decoder connected in sequence. The encoder is used to extract image features and geometric features, including an image coding branch and a geometric coding branch. The decoder is used to fuse image features, geometric features, and query points, and output sparse semantic point predictions through multi-layer query decoding. The generation module 15 is used to generate a three-dimensional semantic occupancy result based on the sparse semantic point set.

[0151] The apparatus of this embodiment can be used to perform Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0152] Preferably, the unified module 12 includes: A transformation unit is used to transform the geometric input to the same local occupancy coordinate system according to the calibration parameters; The clipping unit is used to clip the effective range of all points after the coordinate transformation, retaining only the points located within the locally occupied range, thus obtaining the unified geometric point set.

[0153] Preferably, the initialization module 13 includes: The determination unit is used to determine the number of queries based on the current task type, device computing power, real-time requirements, or user-specified configuration. The sampling unit is used to sample a portion of points from the unified geometric point set as geometrically guided query points, and randomly sample a portion of points from the locally occupied range as supplementary query points, to obtain the query point set.

[0154] Preferably, the device further includes: The determination module is used to determine the target number of decoding layers based on task requirements and device resources; At this point, the prediction module 14 is further used to feed the RGB image set, unified geometric point set, query point set and target decoding layer into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set.

[0155] Preferably, the generation module 15 includes: The mapping unit is used to first filter out low-occupancy confidence points based on the sparse semantic point set, and then map the remaining prediction points onto a three-dimensional voxel grid to obtain a three-dimensional semantic occupancy voxel map or semantic point cloud. The averaging unit is used to select several points with higher confidence based on occupancy confidence and semantic confidence if multiple prediction points fall into the same voxel, and then average or weighted average their semantic scores. The final semantic category of the voxel is determined by the aggregated semantic score, and the occupancy confidence is obtained by aggregating the confidence scores of the prediction points falling into that voxel.

[0156] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, features defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0157] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0158] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0159] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An adaptive 3D semantic occupancy prediction method for embodied tasks, characterized in that, include: Acquire multi-source sensing input, wherein the multi-source sensing input includes a set of RGB images, calibration parameters corresponding to each image, and at least one geometric input; The geometric input is unified according to the calibration parameters to obtain a unified geometric point set; The number of queries is determined based on the task budget, and the query point input is initialized to obtain the query point set; The RGB image set, unified geometric point set, and query point set are fed into a pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set. The AdaOcc model is a learnable neural network, which includes an encoder and a decoder connected in sequence. The encoder is used to extract image features and geometric features, including an image coding branch and a geometric coding branch. The decoder is used to fuse image features, geometric features, and query points, and outputs sparse semantic point predictions through multi-layer query decoding. A three-dimensional semantic occupancy result is generated based on the sparse semantic point set.

2. The method according to claim 1, characterized in that, The RGB image set includes one or more RGB images; And / or, the calibration parameters include camera intrinsic parameters and camera extrinsic parameters relative to the robot's local coordinate system or occupied coordinate system; And / or, the geometric input is a pseudo-depth map estimated from an RGB image, a depth map or point cloud obtained from a depth camera, a point cloud obtained from LiDAR, or geometric points obtained from other 3D sensors.

3. The method according to claim 1, characterized in that, The step of unifying the geometric input according to the calibration parameters to obtain a unified geometric point set includes: Based on the calibration parameters, the geometric input is transformed to the same local occupancy coordinate system; After completing the coordinate transformation, all points are clipped within their effective range, retaining only those points located within their local occupied range, thus obtaining the unified geometric point set.

4. The method according to claim 1, characterized in that, The process of determining the number of queries based on the task budget and initializing the query point inputs yields a query point set, including: The number of queries is determined based on the current task type, device computing power, real-time requirements, or user-specified configuration. The query point set is obtained by sampling a portion of points from the unified geometric point set as geometrically guided query points, and randomly sampling a portion of points from the locally occupied range as supplementary query points.

5. The method according to claim 1, characterized in that, The step of feeding the RGB image set, unified geometric point set, and query point set into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set, before which the following is also included: Determine the target number of decoding layers based on task requirements and equipment resources; The step of feeding the RGB image set, unified geometric point set, and query point set into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set is further as follows: The RGB image set, unified geometric point set, query point set, and target decoding layer are fed into the pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set.

6. The method according to claim 1, characterized in that, The geometric encoding branch of the encoder is used to first convert the unified geometric point set into a sparse three-dimensional structure that the network can process to obtain sparse voxel features, then encode the sparse voxel features in three dimensions to obtain sparse three-dimensional features, and finally compress the sparse three-dimensional features. And / or, in each layer of the decoder, each query point generates several 3D sampling positions near its current position. These sampling positions are projected onto image features through camera calibration to obtain appearance and semantic information, and 3D structural information is obtained from geometric features. After fusing the above information, the decoder updates the query features and predicts the 3D coordinates and semantic scores of one or more 3D semantic occupant points. The predicted points of the current layer are also used to update the query reference positions of the next layer, so that the multi-layer decoder gradually generates sparse semantic point predictions from coarse to fine.

7. The method according to claim 1, characterized in that, The step of generating a three-dimensional semantic occupancy result based on the sparse semantic point set includes: Based on the sparse semantic point set, low-occupancy confidence points are first filtered out, and then the remaining prediction points are mapped onto a three-dimensional voxel grid to obtain a three-dimensional semantic occupancy voxel map or semantic point cloud. If multiple prediction points fall into the same voxel, then several points with higher confidence are selected based on the occupancy confidence and semantic confidence, and their semantic scores are averaged or weighted averaged. The final semantic category of the voxel is determined by the aggregated semantic score, and the occupancy confidence is obtained by aggregating the confidence scores of the prediction points falling into that voxel.

8. The method according to claim 1, characterized in that, The AdaOcc model is trained using training samples with a supervision signal. And / or, during the training of the AdaOcc model, supervision is applied to the output of each layer of the decoder; And / or, the training of the AdaOcc model employs progressive query learning, i.e.: in the initial training phase, fewer query points are used to allow the model to learn coarse-grained geometric and semantic structures first, avoiding the initial dispersion of supervision by a large number of sparse query points; subsequently, the number of queries is gradually increased according to the training epochs, allowing the model to gradually learn finer-grained occupancy predictions and adapt to the decoder feature fusion process under different query numbers, wherein the training of the first... The number of queries used in the round is expressed as: ,in, This represents the maximum number of queries. This represents the initial number of queries. To begin increasing the number of training rounds, The interval for the increase in the number of queries. This represents the number of queries added each time. And / or, during the training of the AdaOcc model, the number of input camera viewpoints is randomly varied, that is, in each training iteration, the number of viewpoints is randomly retained. Each camera viewpoint is used, and only the RGB images and camera calibration information corresponding to these reserved views are used, making the current query budget related to the number of views: ,in, For the first The number of queries used in the round; And / or, during the training of the AdaOcc model, point optimization with constraints is adopted, that is, it is determined whether the predicted point falls in the known free area of ​​the effective supervision area. If so, the distance from the predicted point to the nearest real occupied voxel box is calculated, and it is constrained back to the effective occupied area through the loss function. And / or, the overall training objective of the AdaOcc model includes semantic classification loss, point set reconstruction loss, and inclusion constraint loss: ,in, Indicates the number of decoder layers. Indicates the interlayer attenuation coefficient. Indicates the first Layer semantic classification loss, Indicates the first Layer point set reconstruction loss, This indicates that the loss includes constraint. , and These are preset weighting coefficients.

9. The method according to any one of claims 1-8, characterized in that, The step of generating a three-dimensional semantic occupancy result based on the sparse semantic point set then includes: The navigation application involves projecting the ground or passable categories onto a two-dimensional or 2.5D map as passable areas based on the three-dimensional semantic occupancy results, treating non-ground occupancy points as obstacles, expanding the obstacles according to the robot radius and safety distance, and then performing path planning based on the semantic target location. Alternatively, operations can be performed, namely, extracting the target object point set and container point set based on the three-dimensional semantic occupancy results for operational use.

10. An adaptive 3D semantic occupancy prediction device for embodied tasks, characterized in that, include: The acquisition module is used to acquire multi-source sensing input, wherein the multi-source sensing input includes a set of RGB images, calibration parameters corresponding to each image, and at least one geometric input; A unified module is used to unify the geometric input according to the calibration parameters to obtain a unified geometric point set; The initialization module is used to determine the number of queries based on the task budget and initialize the query point inputs to obtain the query point set. The prediction module is used to feed the RGB image set, unified geometric point set, and query point set into a pre-trained AdaOcc model for forward prediction to obtain a sparse semantic point set. The AdaOcc model is a learnable neural network, including an encoder and a decoder connected in sequence. The encoder is used to extract image features and geometric features, including an image coding branch and a geometric coding branch. The decoder is used to fuse image features, geometric features, and query points, and output sparse semantic point predictions through multi-layer query decoding. The generation module is used to generate a three-dimensional semantic occupancy result based on the sparse semantic point set.

Images