Visual perception method and device based on neural radiance field geometry prior, and medium

Abstract

The application discloses a visual perception method and device based on neural radiance field geometric prior, and a medium, and relates to the field of artificial intelligence, and comprises the following steps: acquiring multi-view historical videos of a fixed monitoring scene and calibrating and pose estimation, and training a neural radiance field model according to the obtained parameters; further, acquiring a current frame in a monocular video stream of a target scene, inputting the model to render a first depth map and a first normal map; taking the current frame as a sample, taking the first depth map and the first normal map as a supervision signal, and training a lightweight geometric estimation network; further, in an online stage, inputting a current frame in a real-time video stream collected by a monocular camera on an embodied intelligent device into the trained lightweight geometric estimation network to obtain a second depth map and a second normal map as geometric prior for device perception decision. According to the application, high-precision dense geometric information can be acquired only by relying on ordinary monocular videos, and a laser radar is not needed, so that the hardware cost and deployment threshold are reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a visual perception method, device and medium based on the geometric prior of neural radiation fields. Background Technology

[0002] In the fields of computer vision and robot perception, recovering the three-dimensional geometry of a scene from two-dimensional images captured by a monocular camera—that is, depth estimation and surface normal estimation—is a fundamental challenge supporting high-level tasks such as object recognition, spatial relationship understanding, and autonomous navigation. Because monocular images lack parallax information, there is a significant information gap between their two-dimensional pixel grid and three-dimensional physical space. This makes it impossible to directly measure the geometric depth in a single frame of an image; instead, it must rely on additional prior information or statistical assumptions for inference.

[0003] In existing technologies, obtaining such geometric information typically relies on two approaches. First, active or passive depth sensors such as LiDAR, structured light, or stereo vision systems are used. These sensors can directly output depth measurements of the scene. However, LiDAR point clouds suffer from sparsity and are prone to noise on highly reflective or distant surfaces. Stereo vision is limited by matching failures in sparse texture regions. More importantly, the hardware costs of these dedicated sensors are high, and deployment conditions are demanding, making them difficult to popularize in large-scale industrial scenarios such as fixed monitoring. Second, deep learning-based monocular depth estimation networks are employed. These networks are trained on large-scale public datasets to learn an end-to-end mapping function from a single color image to a depth map. However, training these networks heavily relies on massive amounts of image samples with ground truth depth annotations. Obtaining high-precision, dense ground truth depth data itself depends on the aforementioned expensive sensors or complex acquisition devices. Furthermore, there are significant domain differences between public datasets and specific application scenarios, making it difficult for general-purpose models to meet the stringent requirements of industrial applications in specific fixed scenarios.

[0004] The inventors of this application have discovered that, due to the high hardware cost and poor versatility of existing solutions, high-precision depth sensors, and the limitations of learning-based monocular estimation methods in terms of expensive and cumbersome ground truth acquisition processes and insufficient scene generalization capabilities, the aforementioned dual bottlenecks have resulted in monocular vision systems being unable to efficiently and cost-effectively acquire high-precision dense geometric information in industrial-grade fixed monitoring scenarios. Summary of the Invention

[0005] This invention provides a visual perception method, device, and medium based on neural radiation field geometric priors. The technical problem it aims to solve is: how to provide an effective solution for obtaining high-precision dense geometric information from monocular videos captured by ordinary cameras.

[0006] In a first aspect, the present invention provides a visual perception method based on the geometric prior of neural radiation fields, comprising: A multi-view historical video of a fixed monitoring scene is acquired, and camera calibration and pose estimation are performed on the multi-view historical video to obtain camera calibration parameters and pose estimation results. Based on the camera calibration parameters and pose estimation results, a neural radiation field model is trained. The neural radiation field model can render a color image and the corresponding depth map and surface normal map from any input viewpoint. The current frame image in the monocular video stream of the target scene is acquired; the current frame image is input into the neural radiation field model, and the neural radiation field model renders a first depth map and a first normal map from the viewpoint corresponding to the current frame image; the current frame image is used as an input sample, and the first depth map and the first normal map are used as supervision signals to train a lightweight geometry estimation network, which is used to predict the depth map and normal map from the monocular image; During the online phase, the frame images to be processed are acquired from the real-time video stream captured by the monocular camera on the embodied intelligent device; the frame images to be processed are input into the trained lightweight geometric estimation network to obtain the predicted second depth map and second normal map, which serve as the geometric prior input for the embodied intelligent device to perform environmental perception and interaction decision-making.

[0007] Optionally, acquiring multi-view historical video of a fixed monitoring scene includes: Acquire raw multi-view video data in a fixed monitoring scenario. The raw multi-view video data is collected synchronously by multiple fixed cameras or by a single movable camera at multiple different spatial locations. The original multi-view video data is time-stamp aligned, and the image frames from different perspectives at the same time are established to establish a perspective correspondence, resulting in a time-synchronized set of multi-view image frames. For each frame in the time-synchronized multi-view image frame set, key frame filtering is performed, the amount of image content change between adjacent frames is calculated, and image frames whose image content change exceeds a preset change threshold are retained as key frames, while image frames whose image content change does not exceed the preset change threshold are discarded. All the retained keyframes are organized according to their respective viewpoints and acquisition times to obtain the multi-view historical video.

[0008] Optionally, training the neural radiation field model based on the camera calibration parameters and the pose estimation results includes: The initial neural radiation field model is obtained by training based on the camera calibration parameters and the pose estimation results. Obtain the complete network structure and complete network parameters of the initial neural radiation field model; A network pruning method is used to evaluate the importance of neuron connections in the complete network structure. Neuron connections with importance metrics below a preset pruning threshold are removed from the complete network structure to obtain a sparse network structure. The floating-point network parameters in the sparse network structure are mapped to low-bit-width integer parameters using a parameter quantization method to obtain compressed network parameters. The sparse network structure and the compressed network parameters are combined to form a compressed neural radiation field model, which is then used as the neural radiation field model.

[0009] Optionally, the step of rendering a first depth map and a first normal map from the viewpoint corresponding to the current frame image by the neural radiation field model includes: Obtain the camera pose parameters corresponding to the current frame image, and input the camera pose parameters as the target viewpoint into the neural radiation field model; The neural radiation field model performs dense sampling along each line of sight from the target viewpoint to obtain the spatial coordinates and line of sight direction of multiple sampling points on each line of sight; The spatial coordinates and line-of-sight direction of the multiple sampling points are input into the implicit function of the neural radiation field model, and the implicit function outputs the volume density value and color value of the multiple sampling points. Based on the volume density values ​​of the multiple sampling points, the color value is integrated using volume rendering to obtain the color image corresponding to the current frame image; The depth of the sampling points is expected to be calculated based on the volume density values ​​of the multiple sampling points to obtain the first depth map; The spatial gradient of the volume density field is calculated based on the volume density values ​​of the multiple sampling points, and the spatial gradient is normalized to obtain the first normal map.

[0010] Optionally, training a lightweight geometry estimation network by using the current frame image as input samples and the first depth map and the first normal map as supervision signals includes: S21, sample a training image from the monocular video stream of the target scene, input the training image into the lightweight geometric estimation network, and obtain a predicted depth map and a predicted normal map; S22, the same frame of the training image is input into the neural radiation field model, and the neural radiation field model renders the training depth map and training normal map from the viewpoint corresponding to the training image. S23, perform invalid value detection on the training depth map, mark pixels with a depth value of zero or exceeding a preset depth range as invalid pixels, and exclude pixels in the predicted depth map that are at the same position as the invalid pixels from the depth loss calculation. S24, calculate the depth value difference of each valid pixel between the predicted depth map after excluding invalid pixels and the training depth map, and take the average of the sum of the squares of the depth value differences of all valid pixels to obtain the depth loss value; S25, calculate the angle between the normal direction vector of each pixel between the predicted normal map and the training normal map, and take the sum of the squares of the angles between the normal direction vectors of all pixels and then take the average value to obtain the normal loss value. S26, the depth loss value and the normal loss value are weighted and summed to obtain the total loss value; S27, with minimizing the total loss as the optimization objective, the network parameters of the lightweight geometric estimation network are updated using the backpropagation algorithm; S28. Repeat steps S21-S27 until the total loss value is lower than the preset convergence threshold or the number of iterations reaches the preset maximum number of iterations.

[0011] Optionally, the step of inputting the frame image to be processed into the trained lightweight geometry estimation network to obtain the predicted second depth map and second normal map includes: The frame image to be processed is input into the feature extraction encoder of the lightweight geometric estimation network. The feature extraction encoder performs multi-level downsampling convolution on the frame image to be processed to obtain a multi-scale feature map. The multi-scale feature map is input into the depth decoder of the lightweight geometric estimation network. The depth decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map and regresses the depth value pixel by pixel to obtain the second depth map. The same multi-scale feature map is input into the normal decoder of the lightweight geometry estimation network. The normal decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map and regresses the three-dimensional normal direction vector pixel by pixel to obtain a preliminary normal map. The preliminary normal map is post-processed to obtain the second normal map; The upsampling path of the depth decoder is independent of the upsampling path of the normal decoder, and the depth decoder and the normal decoder share the multi-scale feature map output by the feature extraction encoder.

[0012] Optionally, the post-processing of the preliminary normal map to obtain the second normal map includes: Normalize the normal direction vector of each pixel in the preliminary normal map by unit length so that the L2 norm of the normal direction vector of each pixel is equal to 1. A sign consistency check is performed on the normal direction vector of each pixel in the preliminary normal map after unit length normalization to detect whether the normal direction vector points to the visible surface of the scene; if the normal direction vector is detected to point to the invisible surface of the scene, the three components of the normal direction vector are simultaneously inverted to obtain the second normal map.

[0013] Optionally, after using the predicted second depth map and second normal map as geometric prior inputs for the embodied intelligent device to perform environmental perception and interaction decisions, the method further includes: Obtain the two-dimensional bounding box of the target object in the frame image to be processed; Based on the second depth map, the depth values ​​of all pixels within the two-dimensional bounding box are extracted, and the median of the extracted depth values ​​is taken to obtain the center depth value of the target object. Based on the second normal map, the normal direction vectors of all pixels within the two-dimensional bounding box are extracted. A clustering algorithm is used to cluster all the extracted normal direction vectors, and the cluster center direction vector of the largest cluster is determined as the main surface orientation vector of the target object. Based on the center depth value and the main surface orientation vector, the position and orientation of the target object in the three-dimensional environment of the embodied intelligent device are determined, and the position and orientation are used as the decision basis for the embodied intelligent device to perform grasping or obstacle avoidance operations.

[0014] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0015] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0016] This invention provides a visual perception method, device, and medium based on neural radiation field geometric priors. In an offline phase, multi-view historical video of a fixed monitoring scene is acquired. A neural radiation field model is trained using camera calibration and pose estimation results, enabling the model to render color images and corresponding depth and surface normal maps from any input viewpoint. Using this model as a geometry teacher, for each frame of the monocular video stream of the target scene, the current image is input to the model to render a first depth map and a first normal map. This rendering result is then used as a supervisory signal to train a lightweight geometry estimation network, enabling the network to predict depth and normal maps from monocular images. This allows for the acquisition of high-precision, dense geometric ground truth without relying on active depth sensors such as LiDAR. In the online phase, only the frames to be processed from the real-time video stream captured by the monocular camera on the embedded intelligent device need to be input into the trained lightweight geometry estimation network. The network then predicts the second depth map and second normal map frame by frame, serving as geometric prior inputs for environmental perception and interactive decision-making. The entire inference process is completed by the lightweight network, meeting real-time requirements. Therefore, high-precision dense geometric information can be obtained by relying solely on monocular video captured by ordinary cameras, effectively solving the problems of high hardware costs caused by relying on expensive sensors in traditional solutions and insufficient accuracy of general models in specific scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a visual perception method based on geometric priors of neural radiation fields, provided for an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0024] Please see Figure 1 This invention provides a visual perception method based on neural radiation field geometric priors. This method trains a neural radiation field model offline as a geometric teacher, which provides self-supervised training signals to a lightweight geometric estimation network. This enables the trained network to predict high-precision depth maps and normal maps from monocular images in real time during the online phase, thus providing reliable geometric prior information for environmental perception and interactive decision-making in embodied intelligent devices. Specifically, the method includes the following steps: S1. Acquire multi-view historical videos of a fixed monitoring scene, perform camera calibration and pose estimation on the multi-view historical videos, and obtain camera calibration parameters and pose estimation results; based on the camera calibration parameters and pose estimation results, train a neural radiation field model, which can render a color image and corresponding depth map and surface normal map from any input viewpoint.

[0025] In specific implementation, multi-view historical videos of fixed monitoring scenes are acquired. These multi-view historical videos refer to a collection of image sequences acquired from multiple different perspectives within the same fixed scene. Furthermore, for fixed monitoring scenes, such as factory workshops, intersections, or warehousing and logistics areas, the background geometry is relatively stable, providing favorable conditions for offline 3D reconstruction. Furthermore, when acquiring multi-view historical videos, multiple fixed cameras can be used for simultaneous acquisition, or a single movable camera can be used to acquire data at different times and locations. Furthermore, after acquisition, camera calibration and pose estimation are performed on the multi-view historical videos to obtain camera calibration parameters and pose estimation results. Camera calibration is used to determine the camera's intrinsic parameter matrix and distortion coefficients. The intrinsic parameter matrix includes parameters such as focal length and principal point coordinates. The distortion coefficients include radial distortion coefficients and tangential distortion coefficients, used to correct image deformations generated during lens manufacturing and installation. Furthermore, pose estimation is used to determine the extrinsic parameter matrix corresponding to each frame of the image, i.e., the camera's rotation matrix and translation vector in 3D space. Camera calibration and pose estimation can employ calibration board methods based on feature point matching, such as the Zhang Zhengyou calibration method. This method is based on the projection relationship of corner points on a planar checkerboard calibration board under different viewpoints, and jointly estimates intrinsic and extrinsic parameters by solving the homography matrix. Alternatively, a self-calibration method based on motion reconstruction structure can be used, which simultaneously solves for camera parameters and 3D point cloud by extracting and matching feature points between images. This method does not require a calibration board and is suitable for acquisition environments without calibration conditions. This invention does not specifically limit the scope of this method.

[0026] Further, based on the camera calibration parameters and the pose estimation results, a neural radiation field model is trained. This neural radiation field model is optimized for the current fixed monitoring scene. It employs an implicit neural representation method to encode the scene's three-dimensional geometric structure and appearance information into the network parameters of a multilayer perceptron. Further, the training process of the neural radiation field model is based on volume rendering principles: for each training image frame, starting from the camera's viewpoint corresponding to that image, dense sampling is performed along each line of sight. The spatial coordinates of the sampling points and the direction of the line of sight are input into the multilayer perceptron to obtain the volume density and color values ​​of each sampling point. Then, a volume rendering integral is used to obtain the rendered image under that viewpoint. The pixel value difference between the rendered image and the real image is calculated as the loss, and the network parameters are updated via backpropagation. Further, the basic form of the volume rendering integral is: along the line of sight, the color values ​​of each sampling point are weighted and accumulated according to their volume density and cumulative transmittance. The cumulative transmittance represents the probability of light propagating from a distance in the scene to the current sampling point, and it is calculated by subtracting the cumulative effect of the volume density values ​​of all sampling points in front from 1. Furthermore, after multiple rounds of iterative training, the neural radiation field model can render a color image from any input viewpoint, along with the corresponding depth map and surface normal map. In the depth map, the grayscale value of each pixel represents the distance from the corresponding spatial point to the camera. In the surface normal map, the 3D vector of each pixel represents the orientation direction of the surface where the corresponding spatial point is located, and the three components of this 3D vector correspond to the components of the normal along the three orthogonal axes in the camera coordinate system. Furthermore, the neural radiation field model employs a compact network structure design during training and can further reduce its size after training through compression techniques such as pruning and quantization to adapt to edge devices with limited storage and computing resources.

[0027] In some preferred embodiments, acquiring multi-view historical video of a fixed monitoring scene includes: acquiring raw multi-view video data of the fixed monitoring scene, wherein the raw multi-view video data is simultaneously collected by multiple fixed cameras or by a single movable camera at multiple different spatial locations; performing timestamp alignment processing on the raw multi-view video data, establishing a viewpoint correspondence between image frames from different perspectives at the same time, and obtaining a time-synchronized set of multi-view image frames; performing keyframe filtering on each image frame in the time-synchronized set of multi-view image frames, calculating the amount of image content change between adjacent frames, retaining image frames whose image content change exceeds a preset change threshold as keyframes, and discarding image frames whose image content change does not exceed the preset change threshold; and organizing all the retained keyframes according to their respective corresponding viewpoints and acquisition times to obtain the multi-view historical video.

[0028] In this specific implementation, the original video data is acquired from multiple fixed cameras or a single movable camera, and preprocessing is performed on timestamp alignment and keyframe filtering to provide high-quality, low-redundancy multi-view image data for the subsequent training of the neural radiation field model. The specific implementation method is as follows: Acquire raw multi-view video data in a fixed monitoring scenario. This raw multi-view video data can come from two different acquisition methods. In the first acquisition method, the raw multi-view video data is simultaneously acquired by multiple fixed cameras arranged at different locations in physical space. Each camera has a different optical center and overlapping fields of view. Specifically, in a typical industrial monitoring scenario, multiple cameras can be installed at different locations within the scene, such as one camera at each of the four corners of a rectangular workshop. This ensures that each camera covers a different angle of the scene, and that the fields of view of adjacent cameras overlap, facilitating feature point matching and triangulation during subsequent 3D reconstruction. This method is typically used in fixed industrial monitoring scenarios, where each camera is fixedly installed at a pre-marked position, and all cameras simultaneously acquire scene images at the same time. Further, in the second acquisition method, the raw multi-view video data is acquired by a single movable camera at multiple different spatial locations. That is, the same camera acquires a video segment at different times when it is located at multiple different spatial locations, and these multiple spatial locations have translational or rotational relationships. For example, a camera can be mounted on the end effector of a mobile cart or robotic arm, controlling the cart or arm to move along a preset trajectory within a scene, capturing video clips at different positions and angles. This method is suitable for scenes where multiple camera deployments are not feasible; by moving the camera to capture images at different angles and distances, image data covering multiple perspectives of the scene can still be obtained. The core of both acquisition methods lies in obtaining dense perspective sampling covering the complete geometry of the scene, enabling subsequent neural radiation field models to learn the three-dimensional geometric representation of the scene from sufficient perspective information.

[0029] Furthermore, the original multi-view video data undergoes timestamp alignment. When the original data comes from multiple fixed cameras, due to potential clock discrepancies between different cameras, timestamp alignment of the image frames captured by each camera is necessary. Specifically, the timestamp alignment method is as follows: all image frames captured by the cameras are sorted according to their respective timestamps, and the multiple frames with the smallest timestamp difference are selected as the set of multi-view image frames at the same time. For example, when the timestamps of two cameras differ within a preset tolerance range, these two image frames are considered as image frames corresponding to the same viewpoint captured at the same time. For multiple video segments captured by a single movable camera, since the system clock of the same camera is consistent, alignment can be directly performed according to the recording time order. After timestamp alignment, a viewpoint correspondence is established between image frames from different viewpoints at the same time, resulting in a time-synchronized multi-view image frame set. This time-synchronized multi-view image frame set is the basic data for subsequent keyframe selection and neural radiation field model training.

[0030] Furthermore, keyframe filtering is performed on each frame in the time-synchronized multi-view image frame set. The purpose of keyframe filtering is to remove redundant frames with highly similar content, reduce the number of samples for subsequent neural radiation field model training, and improve training efficiency. The specific filtering method is as follows: calculate the amount of image content change between adjacent frames. This amount of image content change can be measured using multiple methods. The first method is the mean square error of the pixel-by-pixel difference between adjacent frames. This method is simple to calculate but easily affected by changes in illumination. The second method is the structural similarity index, which comprehensively considers the similarity of brightness, contrast, and structure, and is more sensitive to visual changes perceived by the human eye. The third method is the optical flow field amplitude, which calculates the average amplitude of the motion vectors of each pixel between adjacent frames. This reflects the degree of motion of objects in the scene and has good adaptability to dynamic scenes. Furthermore, image frames whose image content change exceeds a preset change threshold are retained as keyframes, while image frames whose image content change does not exceed the preset change threshold are discarded. The preset change threshold can be pre-set according to the dynamic level of the scene. For example, when the scene is a static industrial workshop, the threshold can be set lower to retain more detailed frames; further, when the scene is a dynamic traffic intersection, the threshold can be set higher to remove a large number of similar frames. Furthermore, all retained keyframes are organized according to their corresponding viewpoints and acquisition times to obtain the multi-view historical video. This organization method ensures that the image frames of each viewpoint are arranged in chronological order of acquisition time, and that different viewpoints are aligned in the time dimension, facilitating the subsequent neural radiation field model to learn the three-dimensional geometric structure and appearance information of the scene from the multi-view images.

[0031] This embodiment provides a low-cost, easily deployable multi-view video acquisition solution for applications with only ordinary cameras and no expensive sensors such as LiDAR, by acquiring raw multi-view video data from multiple fixed cameras or a single movable camera. Furthermore, timestamp alignment ensures the consistency of image frames from different perspectives in the temporal dimension, avoiding mismatches between perspectives caused by clock skew, and providing data quality assurance for the accurate training of the neural radiation field model. Furthermore, keyframe filtering effectively eliminates redundant frames with highly similar content by calculating the amount of image content change between adjacent frames and retaining frames with significant changes. This significantly reduces the amount of training data while preserving the integrity of scene geometric information, thus improving the training efficiency of the neural radiation field model.

[0032] In some preferred embodiments, training the neural radiation field model based on the camera calibration parameters and the pose estimation results includes: training based on the camera calibration parameters and the pose estimation results to obtain an initial neural radiation field model; obtaining the complete network structure and complete network parameters of the initial neural radiation field model; using a network pruning method to evaluate the importance of neuron connections in the complete network structure, removing neuron connections with importance metrics below a preset pruning threshold from the complete network structure to obtain a sparse network structure; using a parameter quantization method to map the floating-point network parameters in the sparse network structure to low-bit-width integer parameters to obtain compressed network parameters; combining the sparse network structure and the compressed network parameters to form a compressed neural radiation field model, and using the compressed neural radiation field model as the neural radiation field model.

[0033] In specific implementation, this embodiment integrates the training and compression of the neural radiation field model into the same process. After training is completed, a compact compressed model is obtained, which solves the problems of the original neural radiation field model having a large number of parameters, slow inference speed, and difficulty in deployment on edge devices with limited storage and computing resources. The specific implementation method is as follows: The initial neural radiation field model is obtained through training based on the camera calibration parameters and pose estimation results. The neural radiation field model is a 3D scene modeling method based on implicit neural representation. Its core idea is to encode the geometric structure and appearance information of the 3D scene into the network parameters of a multilayer perceptron. The initial neural radiation field model employs a compact network structure design during training, for example, by reducing the hidden layer dimension of the implicit function to decrease the number of model parameters. The training process uses the reconstruction loss of multi-view images as a supervision signal, optimizing the network parameters by minimizing the pixel value difference between the rendered image and the real image. Specifically, for each training frame, the neural radiation field model starts from the camera viewpoint corresponding to that image, performs dense sampling along each line of sight, inputs the spatial coordinates of the sampling points and the line of sight direction into the multilayer perceptron, obtains the volume density value and color value of each sampling point, and then obtains the rendered image under that viewpoint through volume rendering integration. The difference between the rendered image and the real image is calculated and backpropagated to update the network parameters. Furthermore, during training, positional encoding techniques can be used to map the input coordinates to a higher-dimensional space to enhance the model's ability to represent high-frequency details. Furthermore, the basic principle of positional encoding is to transform the components of the input coordinates through sine and cosine functions of different frequencies, so that the network can more easily learn the spatial features that change at high frequencies.

[0034] Furthermore, the complete network structure and parameters of the initial neural radiation field model are obtained. The complete network structure includes the number of layers in the multilayer perceptron, the number of neurons in each layer, and the connection relationships between layers. The complete network parameters include the connection weights and bias values ​​between neurons in each layer.

[0035] Furthermore, a network pruning method is employed to evaluate the importance of neuron connections in the complete network structure. Neuron connections with importance metrics below a preset pruning threshold are removed from the complete network structure, resulting in a sparse network structure. Network pruning is a technique for compressing neural networks by removing redundant connections. Its core idea is to evaluate the contribution of each neuron connection to the network output and remove connections with lower contributions to reduce the number of model parameters. Further, the importance evaluation can be based on weight magnitude, where connections with larger absolute weight values ​​are considered more important because they have a greater impact on the activation values ​​of subsequent neurons during forward propagation. This method is computationally simple and requires no additional forward or backward propagation calculations. Alternatively, a gradient-based method can be used, evaluating the importance of each connection by calculating the gradient of the loss function with respect to each connection. A larger gradient indicates a higher sensitivity of the connection to the loss function, and removing the connection will significantly increase the loss function. This method requires additional backpropagation calculations, but the evaluation results are more accurate. Furthermore, after pruning, the number of parameters in the sparse network structure is significantly reduced, allowing 50%-80% of the original network parameters to be removed while keeping the model accuracy within an acceptable range.

[0036] Furthermore, a parameter quantization method is employed to map the floating-point network parameters in the sparse network structure to low-bit-width integer parameters, resulting in compressed network parameters. Parameter quantization is the process of mapping parameter values ​​represented by high-precision floating-point numbers to low-precision integer representations. The quantized network parameters occupy significantly less storage space, and the quantized model can use integer arithmetic instructions during inference, resulting in higher computational efficiency and lower power consumption compared to floating-point arithmetic. Furthermore, the quantization method can employ uniform quantization, mapping floating-point numbers to the integer range according to a linear mapping relationship; or it can employ non-uniform quantization, allocating different quantization step sizes in different numerical ranges based on the parameter distribution, using finer quantization step sizes in densely distributed parameter regions to reduce quantization errors. Furthermore, the sparse network structure and the compressed network parameters are combined to form a compressed neural radiation field model, which is used as the neural radiation field model for subsequent rendering operations.

[0037] In this embodiment, the network pruning method significantly reduces the number of parameters in the sparse network structure and substantially reduces the model storage size by removing redundant neuron connections whose importance metric is below a preset threshold. Furthermore, the parameter quantization method maps floating-point network parameters to low-bit-width integer parameters, further compressing the model size. Simultaneously, quantized integer operations offer higher inference speed and lower power consumption compared to floating-point operations. Moreover, the combined use of pruning and quantization compression methods can compress the model size to a smaller proportion of the original size while maintaining model accuracy. This allows the trained neural radiation field model to be deployed on edge devices with limited storage and computing resources, providing a lightweight geometric teacher model for subsequent online inference stages.

[0038] S2, acquire the current frame image in the monocular video stream of the target scene; input the current frame image into the neural radiation field model, and have the neural radiation field model render a first depth map and a first normal map from the viewpoint corresponding to the current frame image; use the current frame image as an input sample, and the first depth map and the first normal map as supervision signals to train a lightweight geometric estimation network, which is used to predict the depth map and normal map from the monocular image.

[0039] In specific implementation, the current frame image in the monocular video stream of the target scene is acquired. The current frame image is each frame of color image captured from the monocular camera of the target scene during the training phase. The monocular camera can be an existing surveillance camera in a fixed monitoring scene, without the need for additional deployment of dedicated sensors. The current frame image is input into the neural radiation field model, which renders a first depth map and a first normal map from the viewpoint corresponding to the current frame image. Further, the specific rendering process is as follows: the camera pose parameters corresponding to the current frame image are acquired, and these camera pose parameters are input into the neural radiation field model as the target viewpoint; the neural radiation field model performs dense sampling along each line of sight of the target viewpoint to acquire the spatial coordinates and volume density values ​​of each sampling point; based on the principle of volume rendering integration, the depth values ​​of each sampling point are weighted and expected to obtain the depth value of each pixel, forming the first depth map; based on the spatial gradient direction of the volume density field, the surface normal direction corresponding to each pixel is calculated to form the first normal map.

[0040] Furthermore, the current frame image is used as the input sample, and the first depth map and the first normal map are used as supervision signals to train a lightweight geometry estimation network. The lightweight geometry estimation network is a convolutional neural network with few parameters and fast inference speed, used to predict the depth map and normal map from a monocular image. The number of parameters in the lightweight geometry estimation network is significantly smaller than that of the neural radiation field model, typically less than one-tenth of the latter, enabling real-time inference on edge devices with limited computing resources. Furthermore, the network can employ a lightweight backbone network based on an encoder-decoder architecture, such as MobileNet or EfficientNet-Lite, as the encoder, combined with a decoder structure that integrates multi-level upsampling and skip connections. Furthermore, during training, the current frame image is used as input, and the first depth map and the first normal map rendered by the neural radiation field model are used as ground truth values; the network parameters are updated by minimizing the difference between the predicted and ground truth values. Specifically, the mean squared error between the predicted depth value and the ground truth depth value is calculated as the depth loss, and the angle error between the predicted normal vector and the ground truth normal vector is calculated as the normal loss. The weighted sum of the depth loss and the normal loss is used as the total loss, and the backpropagation algorithm is used to update the network parameters. Furthermore, this training method, which uses the rendering results of the neural radiation field model as a supervision signal, falls under the category of self-supervised learning because the ground truth labels required for training are automatically generated by the neural radiation field model, eliminating the need for manual annotation.

[0041] In some preferred embodiments, the step of rendering a first depth map and a first normal map from the viewpoint corresponding to the current frame image by the neural radiation field model includes: obtaining camera pose parameters corresponding to the current frame image, and inputting the camera pose parameters as a target viewpoint into the neural radiation field model; the neural radiation field model performs dense sampling along each line of sight of the target viewpoint to obtain the spatial coordinates and line of sight of multiple sampling points on each line of sight; inputting the spatial coordinates and line of sight of the multiple sampling points into an implicit function of the neural radiation field model, the implicit function outputting the volume density value and color value of the multiple sampling points; performing volume rendering integral on the color value based on the volume density value of the multiple sampling points to obtain a color image corresponding to the current frame image; performing expectation calculation on the depth of the sampling points based on the volume density value of the multiple sampling points to obtain the first depth map; calculating the spatial gradient of the volume density field based on the volume density value of the multiple sampling points, and normalizing the spatial gradient to obtain the first normal map.

[0042] In specific implementation, the camera pose parameters corresponding to the current frame image are obtained, and these parameters are input into the neural radiation field model as the target viewpoint. The camera pose parameters include the camera's spatial position coordinates and orientation, typically represented as a rotation matrix and translation vector, or as a quaternion plus a translation vector. This parameter determines the virtual camera's observation position and direction during rendering. In specific implementation, the camera pose parameters can be obtained by reading the metadata of the current frame image, or estimated in real time using visual odometry or a simultaneous localization and mapping (SLT / Map) system; this invention does not specifically limit this. Visual odometry is a technique that recursively estimates the camera's motion trajectory by matching feature points between consecutive image frames; the SLT / Map system is a technique that simultaneously estimates the device's own position and orientation in an unknown environment and constructs a 3D map of the environment. Through loop closure detection and global optimization, it can eliminate accumulated errors and provide more accurate globally consistent pose estimation results.

[0043] Furthermore, the neural radiation field model performs dense sampling along each line of sight from the target viewpoint, obtaining the spatial coordinates and line-of-sight direction of multiple sampling points on each line of sight. Specifically, for each pixel on the target viewpoint imaging plane, a line of sight is formed from the camera's optical center through the pixel's center. The neural radiation field model selects multiple sampling points along the path of this line of sight at a preset sampling interval, with each sampling point corresponding to a spatial coordinate. The line-of-sight direction is a unit vector in three-dimensional space, used to indicate the viewing direction. The number of sampling points and the sampling interval can be configured according to the complexity of the scene and computational resources; for example, a denser sampling strategy can be used in areas with complex scene geometry. Furthermore, for static scenes, the coordinates of the sampling points for each line of sight can be pre-calculated and cached to accelerate subsequent multiple rendering processes.

[0044] Furthermore, the spatial coordinates and viewing direction of the multiple sampling points are input into the implicit function of the neural radiation field model. The implicit function outputs the volume density and color values ​​of the multiple sampling points. The implicit function is a multilayer perceptron network whose input is a concatenated vector of spatial coordinates and viewing direction, and whose output is the volume density and color values ​​at the corresponding locations. The volume density value represents the degree to which the spatial point obstructs light propagation; a larger value indicates that the point is closer to the scene surface, and the probability of light being reflected or absorbed at that location is higher. The color value represents the apparent color of the spatial point under different viewing directions. Since many materials in the real world have anisotropic reflective properties, the same spatial point may appear different colors when viewed from different directions. Therefore, the prediction of the color value requires the viewing direction as an additional input.

[0045] Further, a volume rendering integral is performed on the color value based on the volume density values ​​of the multiple sampling points to obtain the color image corresponding to the current frame image. Volume rendering integral is the core operation of the neural radiation field model. Its principle is to weightedly accumulate the color values ​​of the sampling points along the viewing direction. The weight is determined by the volume density value of the sampling point and the cumulative transmittance of the sampling points preceding it. Specifically, cumulative transmittance represents the probability that light can propagate from a distance in the scene to the current sampling point without being blocked by an intermediate medium or surface. Under discrete sampling conditions, the volume rendering integral is approximated as a weighted sum of the color values ​​of the sampling points. The weight of each point is equal to its volume density value multiplied by the product of the transmittance of all sampling points preceding it. Sampling points with larger volume density values ​​contribute more to the final pixel color, while sampling points preceding it with large volume density values ​​will obscure the contribution of subsequent sampling points to the final color.

[0046] Further, the depth of the sampling points is expected based on the volume density values ​​of the multiple sampling points to obtain the first depth map. The depth expectation calculation is also based on the principle of volume rendering integral, weighting the depth value of each sampling point according to its contribution weight to obtain the expected depth value corresponding to the pixel. This expected depth value is the depth value of the pixel position in the first depth map. Due to the use of the expectation calculation method, the rendered depth map has sub-pixel precision smoothness and is robust to noise. Further, the spatial gradient of the volume density field is calculated based on the volume density values ​​of the multiple sampling points, and the spatial gradient is normalized to obtain the first normal map. The calculation of the surface gradient direction is based on the implicit surface representation of the scene, obtained by normalizing the spatial gradient of the volume density field. Specifically, for each pixel, based on the spatial coordinates and volume density values ​​of each sampling point on the line of sight corresponding to the pixel, the spatial gradient direction of the volume density field at the surface position corresponding to the pixel is calculated, and the spatial gradient is normalized. The spatial gradient direction is the direction perpendicular to the local surface, which is the normal direction of the pixel. Furthermore, the normal direction vector of each pixel is filled into the corresponding pixel position to obtain the first normal map. Since the three components of the normal direction vector may be positive or negative, the normal vector can be encoded by mapping from the [-1,1] interval to the [0,1] interval during storage, and then reversed to restore it when used.

[0047] This embodiment fully discloses the internal mechanism of a neural radiation field model for simultaneously generating color images, depth maps, and normal maps through a complete processing chain, including dense ray sampling, implicit function inference, volume rendering integration, depth expectation calculation, and surface gradient calculation. Furthermore, since the volume rendering integration and depth expectation calculation are performed at multiple sampling points along each viewpoint, the rendered depth map has dense pixel-level depth values, with a resolution consistent with the input image, avoiding the sparse and uneven point cloud problem common in traditional LiDAR systems. Moreover, the calculation of surface gradient directions is based on the spatial gradient of the volume density field, enabling the rendered normal map to accurately reflect the orientation of each visible surface in the scene, providing high-quality geometric supervision signals for subsequent training of the student network.

[0048] In some preferred embodiments, the step of using the current frame image as input sample and the first depth map and the first normal map as supervision signals to train the lightweight geometry estimation network includes: S21, sampling a training image from the monocular video stream of the target scene, inputting the training image into the lightweight geometry estimation network to obtain a predicted depth map and a predicted normal map; S22, inputting the same training image frame into the neural radiation field model, and having the neural radiation field model render the training depth map and training normal map from the viewpoint corresponding to the training image; S23, performing invalid value detection on the training depth map, marking pixels with a depth value of zero or exceeding a preset depth range as invalid pixels, and excluding pixels in the predicted depth map that are at the same position as the invalid pixels from the depth loss calculation; S2 4. Calculate the depth value difference between the predicted depth map and the training depth map for each valid pixel after excluding invalid pixels. Sum the squares of the depth value differences of all valid pixels and then take the average to obtain the depth loss value. S25. Calculate the angle between the normal direction vectors of each pixel between the predicted normal map and the training normal map. Sum the squares of the angles between the normal direction vectors of all pixels and then take the average to obtain the normal loss value. S26. Sum the depth loss value and the normal loss value with weights to obtain the total loss value. S27. Update the network parameters of the lightweight geometric estimation network using the backpropagation algorithm with the optimization objective of minimizing the total loss value. S28. Repeat steps S21-S27 until the total loss value is lower than the preset convergence threshold or the number of iterations reaches the preset maximum number of iterations.

[0049] In this specific implementation, the depth loss and normal loss are calculated separately and then weighted and summed to achieve joint optimization of multiple tasks. Furthermore, invalid pixel filtering addresses the problem of holes and invalid regions in the training ground value. The specific implementation method is as follows: Step S21: Sample a training image from the monocular video stream of the target scene and input the training image into the lightweight geometry estimation network to obtain a predicted depth map and a predicted normal map. During the training phase, the lightweight geometry estimation network iteratively updates its parameters using optimization algorithms such as stochastic gradient descent. In the specific forward inference process, the network performs multi-level convolution and downsampling operations on the input training image to extract features. The downsampling operation can use convolutional layers with a stride of two or max-pooling layers. With each downsampling stage, the spatial resolution of the feature map is halved while the number of channels is doubled, gradually extracting higher-level semantic features. Then, upsampling and skip connections restore the image resolution. The upsampling operation can use transposed convolution or bilinear interpolation followed by a convolutional layer to regress the depth value and normal direction vector pixel by pixel. The predicted depth map and predicted normal map obtained in this step are the output results under the current network parameters.

[0050] Further, in step S22, the same frame of the training image is input into the neural radiation field model, which then renders a training depth map and a training normal map from the viewpoint corresponding to the training image. This step is the same as the method of rendering the first depth map and the first normal map in Embodiment 4, the only difference being the input training image. The training depth map and training normal map obtained in this step serve as supervision signals, which are the geometric ground truth corresponding to the current training image. Since the neural radiation field model is a 3D geometric representation of the scene learned from multi-view historical videos, its rendering results have advantages in terms of viewpoint continuity and geometric consistency, and can provide smoother and denser depth values ​​than traditional sensors.

[0051] Further, in step S23, invalid value detection is performed on the training depth map. Pixels with depth values ​​of zero or exceeding a preset depth range are marked as invalid pixels, and pixels in the predicted depth map that are at the same position as the invalid pixels are excluded from the depth loss calculation. This invalid value detection aims to remove invalid regions from the neural radiation field model rendering results caused by scene occlusion, poor lighting conditions, or non-convergence of depth estimation. For example, in the sky or distant background areas of a scene, the neural radiation field model may not reliably estimate depth values; the depth values ​​in these areas are typically zero or abnormally large. The preset depth range can be pre-set according to the actual physical depth range of the scene, for example, setting a minimum and maximum depth value; depth values ​​exceeding this range are considered invalid. By excluding pixels at the same position in the predicted depth map, the negative impact of invalid ground truth values ​​on the training process is avoided. This invalid pixel filtering mechanism plays a crucial role in the quality control of training data and the purity of loss calculation.

[0052] Further, in step S24, the depth value difference for each valid pixel between the predicted depth map (excluding invalid pixels) and the training depth map is calculated. The sum of squares of the depth value differences for all valid pixels is then averaged to obtain the depth loss value. The depth loss value can be calculated using the mean squared error loss function, which imposes a larger penalty on larger depth errors, prompting the network to prioritize correcting significant depth prediction biases. This depth loss value measures the degree of difference between the predicted depth map and the ground truth depth map; a smaller depth loss value indicates higher prediction accuracy.

[0053] Further, in step S25, the angle between the normal direction vector of each pixel between the predicted normal map and the training normal map is calculated. The sum of the squares of the angles between the normal direction vectors of all pixels is then averaged to obtain the normal loss value. The angle between two unit normal vectors can be calculated using the vector dot product. Specifically, the dot product of two unit vectors is equal to the cosine of their angle. The inverse cosine of the dot product result yields the angle in radians. The smaller the angle, the more consistent the predicted normal direction is with the true normal direction. This normal loss value is used to measure the accuracy of the predicted normal map.

[0054] Further, in step S26, the depth loss value and the normal loss value are weighted and summed to obtain the total loss value. Specifically, the weighted summation is performed by multiplying the depth loss value by a first weight coefficient and the normal loss value by a second weight coefficient, then adding them together. The first and second weight coefficients are used to balance the relative importance of the depth and normal tasks in joint training. The principle for setting the weight coefficients is to ensure that the two loss terms are comparable in magnitude, preventing one task's loss from becoming too large and dominating the training direction.

[0055] Further, in step S27, with minimizing the total loss as the optimization objective, the network parameters of the lightweight geometric estimation network are updated using the backpropagation algorithm. The backpropagation algorithm calculates the gradient of the total loss with respect to the parameters of each layer of the network, and updates the parameter values ​​according to the direction of gradient descent, so that the total loss value decreases in the next iteration. The backpropagation algorithm is a standard optimization method in deep learning training, which calculates the gradient layer by layer and backpropagates it using the chain rule.

[0056] Further, in step S28, steps S21 to S27 are repeated until the total loss value is lower than a preset convergence threshold or the number of iterations reaches a preset maximum number of iterations. This iterative process ensures that the network parameters of the lightweight geometry estimation network are fully optimized, so that its predicted depth map and normal map gradually approach the geometric truth rendered by the neural radiation field model. The preset convergence threshold and the preset maximum number of iterations can be configured according to specific scenarios and computing resources.

[0057] This embodiment calculates and weights the depth loss and normal loss separately, enabling the lightweight geometry estimation network to simultaneously optimize both depth prediction and normal prediction tasks during training. This avoids the problem of a single task's loss dominating the training direction and causing performance degradation in the other task. Furthermore, the invalid pixel detection mechanism identifies zero values ​​or invalid pixels exceeding a preset depth range in the training depth map, excluding the corresponding pixels in the predicted depth map from the loss calculation. This makes the training signal cleaner and prevents invalid ground truth values ​​from misleading the network training. Moreover, the trained lightweight geometry estimation network can achieve high-precision end-to-end prediction from monocular images to depth and normal maps in specific target scenes, with prediction accuracy exceeding that of models trained on general public datasets in such specific scenes.

[0058] S3, In the online phase, acquire the frame image to be processed from the real-time video stream captured by the monocular camera on the embodied intelligent device; input the frame image to be processed into the trained lightweight geometric estimation network to obtain the predicted second depth map and second normal map, which serve as the geometric prior input for the embodied intelligent device to perform environmental perception and interaction decision-making.

[0059] In specific implementation, during the online phase, the image of the frame to be processed is acquired from the real-time video stream captured by the monocular camera on the embodied intelligent device. The embodied intelligent device refers to an intelligent system with a physical entity capable of performing perception, decision-making, and actions in a three-dimensional environment, such as an autonomous mobile robot, a robotic arm operating platform, or an autonomous vehicle. The monocular camera installed on this device captures environmental images in real time, forming a real-time video stream. Further, the image of the frame to be processed is input into the trained lightweight geometric estimation network. This network performs forward inference calculations on the input image and outputs a predicted second depth map and a second normal map. The second depth map and the second normal map serve as geometric prior inputs for the embodied intelligent device to perform environmental perception and interactive decision-making, supporting high-level visual tasks such as object recognition, spatial relationship understanding, grasping operations, and obstacle avoidance operations.

[0060] This embodiment uses an offline-trained neural radiation field model as a geometry teacher. This model renders high-precision depth and normal maps from any viewpoint as self-supervised signals to train a lightweight geometry estimation network. Furthermore, because the neural radiation field model can learn the dense 3D geometric structure of a scene from multi-view historical videos, the accuracy of its rendered depth and normal maps is higher than the sparse measurements output by traditional sensors such as LiDAR, without requiring additional hardware costs. Moreover, the trained student network can predict high-precision geometric information in real-time with only monocular image input during the online phase, and its inference speed meets the real-time requirements of embodied intelligent devices. Thus, ordinary camera video can drive high-precision geometric perception, significantly reducing the hardware deployment cost and technical barriers to 3D environment perception.

[0061] In some preferred embodiments, the step of inputting the frame image to be processed into the trained lightweight geometry estimation network to obtain a predicted second depth map and a second normal map includes: inputting the frame image to be processed into the feature extraction encoder of the lightweight geometry estimation network, wherein the feature extraction encoder performs multi-level downsampling convolution on the frame image to be processed to obtain a multi-scale feature map; inputting the multi-scale feature map into the depth decoder of the lightweight geometry estimation network, wherein the depth decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map, and regresses depth values ​​pixel by pixel to obtain the second depth map; inputting the same multi-scale feature map into the normal decoder of the lightweight geometry estimation network, wherein the normal decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map, and regresses three-dimensional normal direction vectors pixel by pixel to obtain a preliminary normal map; and post-processing the preliminary normal map to obtain the second normal map; wherein the upsampling path of the depth decoder and the upsampling path of the normal decoder are independent of each other, and the depth decoder and the normal decoder share the multi-scale feature map output by the feature extraction encoder.

[0062] In practice, this embodiment achieves real-time inference of depth maps and normal maps through a shared encoder and dual independent decoder architecture, ensuring both computational efficiency and prediction accuracy for both tasks. The specific implementation is as follows: The frame image to be processed is input into the feature extraction encoder of the lightweight geometric estimation network. The feature extraction encoder performs multi-level downsampling convolution on the frame image to obtain multi-scale feature maps. The feature extraction encoder adopts a lightweight backbone network architecture, such as MobileNetV2 or EfficientNet-Lite, which gradually reduces the spatial resolution of the input image and increases the number of channels through multiple convolutional and pooling layers, achieving a balance between high-level semantic information and spatial detail information. Furthermore, the core design of MobileNetV2 is depthwise separable convolution, which decomposes the standard convolution into two steps: depthwise convolution and pointwise convolution, which can significantly reduce the amount of computation. Furthermore, the core design of EfficientNet-Lite is a composite scaling method, which achieves the optimal accuracy-efficiency balance by simultaneously adjusting the network depth, width, and input resolution. Specifically, the first stage of the encoder performs preliminary convolution on the input image and outputs a high-resolution feature map, which retains more spatial location information. As the number of stages increases, the resolution of the feature map is halved at each stage, the number of channels is doubled at each stage, and the receptive field gradually increases, enabling the capture of a wider range of contextual information. Furthermore, during the multi-level downsampling process, each level of the encoder outputs an intermediate feature map. These feature maps at different scales constitute a multi-scale feature map set, providing multi-level visual information for the subsequent decoder.

[0063] Further, the multi-scale feature map is input into the depth decoder of the lightweight geometric estimation network. The depth decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map, regressing depth values ​​pixel by pixel to obtain the second depth map. Further, the depth decoder adopts an upsampling structure symmetrical to the encoder, gradually restoring the spatial resolution of the image through transposed convolution or upsampling interpolation operations. Further, during each upsampling stage, the depth decoder performs skip connection fusion with the corresponding level's output feature map from the encoder, that is, by feature concatenation or element-wise addition, transferring the detailed information retained by the encoder to the decoder to restore the fine spatial structure required for high-resolution output. Further, this skip connection mechanism effectively alleviates the problem of spatial detail information being lost step-by-step during downsampling in deep convolutional networks, enabling the output image to maintain the high-resolution structure of the input image. Finally, the depth decoder regresses a depth value for each pixel in the output layer, forming a second depth map with the same resolution as the input image.

[0064] Furthermore, the same multi-scale feature map is input into the normal decoder of the lightweight geometry estimation network. The normal decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map, regressing the three-dimensional normal direction vector pixel by pixel to obtain a preliminary normal map. Further, the upsampling structure and skip connection fusion mechanism of the normal decoder are similar to those of the depth decoder, but its output layer regresses three values ​​for each pixel, corresponding to the three orthogonal components of the normal direction vector in three-dimensional space. Further, since the normal direction vector is a unit vector, its three components should satisfy the constraint that the sum of squares is 1. However, the original output of the neural network usually does not satisfy this constraint, therefore, post-processing of the preliminary normal map is required.

[0065] Further, the preliminary normal map is post-processed to obtain the second normal map. The post-processing includes two steps: unit length normalization and sign consistency check.

[0066] Furthermore, the upsampling path of the depth decoder and the upsampling path of the normal decoder are independent of each other, and the depth decoder and the normal decoder share the multi-scale feature map output by the feature extraction encoder. Sharing the encoder allows both tasks to share the underlying visual feature extraction computation, significantly reducing the overall number of parameters and inference computation of the network. Furthermore, the independence of the two decoders allows each task to learn its own dedicated high-level representation and regression mapping according to its own characteristics, avoiding inter-task interference that might result from forced sharing of high-level features. Furthermore, in specific implementations, the depth decoder and the normal decoder can receive gradient updates from their respective tasks during training without interfering with each other.

[0067] This embodiment achieves parallel inference for depth estimation and normal estimation within the same network using a network architecture with a shared encoder and dual independent decoders. The shared encoder allows both tasks to share the underlying visual feature extraction computation, avoiding the need to train separate feature extraction networks for each task. This significantly reduces the overall number of parameters and inference computation, enabling the lightweight geometry estimation network to run in real-time on resource-constrained embodied intelligent devices. Furthermore, the independent nature of the dual decoders allows the depth regression and normal regression tasks to learn dedicated high-level representations based on their respective characteristics. The depth decoder focuses on accurate spatial distance estimation, while the normal decoder focuses on precise surface orientation determination, avoiding inter-task interference that might result from forced sharing of high-level features. Furthermore, the multi-level upsampling and skip connection fusion mechanism transmits the spatial detail information preserved in each layer of the encoder to the decoder, ensuring that the predicted depth and normal maps maintain the high-resolution spatial structure of the input image and preventing detail loss during the upsampling process.

[0068] In some preferred embodiments, the post-processing of the preliminary normal map to obtain the second normal map includes: normalizing the normal direction vector of each pixel in the preliminary normal map by unit length, so that the L2 norm of the normal direction vector of each pixel is equal to 1; performing a sign consistency check on the normal direction vector of each pixel in the unit-length normalized preliminary normal map to detect whether the normal direction vector points to the visible surface side of the scene; if the normal direction vector is detected to point to the invisible surface side of the scene, then inverting the three components of the normal direction vector simultaneously to obtain the second normal map.

[0069] In specific implementation, this embodiment solves the physical meaning problem of non-unique magnitude and inconsistent orientation of the normal direction vector output by the neural network through a dual processing mechanism of unit length normalization and sign consistency check, thereby improving the geometric and physical correctness of the normal map. The specific implementation method is as follows.

[0070] The normal direction vector of each pixel in the preliminary normal map is normalized by a unit length, making the L2 norm of the normal direction vector of each pixel equal to 1. When a neural network regresses the 3D normal direction vector pixel by pixel, its output layer typically uses a linear activation function. Therefore, the magnitudes of the three output components are not constrained, and the length of the combined vector may be greater than or less than 1, which does not conform to the physical definition of a normal direction vector: a normal must be a unit vector. Further, the unit length normalization process scales the 3D vector of each pixel to a unit length by dividing it by its own L2 norm. The normalized normal direction vector satisfies the unit vector constraint of a normal, providing physically consistent input data for subsequent geometric calculations such as surface smoothness analysis, lighting calculations, and object pose estimation.

[0071] Further, a sign consistency check is performed on the normal direction vector of each pixel in the preliminary normal map after unit length normalization to detect whether the normal direction vector points to the visible surface of the scene. In the implicit representation of the 3D scene, the sign of the normal direction, i.e., whether the normal points to the outside or inside of the scene, is calculable during the rendering process. Specifically, for each pixel, the spatial gradient direction of the volume density field at that location is calculated based on the spatial position and viewing direction of the pixel; the spatial gradient direction is the direction of the surface normal. Further, if the orientation of the normal direction vector predicted by the neural network is consistent with the spatial gradient direction, it indicates that the normal points to the visible surface of the scene; further, if the two directions are opposite, it indicates that the normal points to the inside of the scene. If the normal direction vector is detected to point to the invisible surface of the scene, the three components of the normal direction vector are simultaneously inverted to obtain the second normal map. Further, the sign inversion operation rotates the normal direction by 180 degrees, making it face outwards from the scene, thus correctly representing the orientation of the visible surface in a physical sense. Furthermore, if the normal direction vector is detected to be pointing to the visible surface, its original direction is maintained. The sign consistency check can be implemented using a vector dot product, that is, calculating the dot product between the predicted normal vector and the spatial gradient direction vector. If the dot product is less than 0, it indicates that the directions are opposite and need to be reversed.

[0072] This embodiment uses unit-length normalization to uniformly scale the 3D vectors of arbitrary length output by the neural network to a unit length, ensuring that the normal direction vector strictly satisfies the mathematical definition of a unit vector. This provides physically accurate input data for subsequent geometric calculations based on the normal vector. Furthermore, a sign consistency check automatically corrects inconsistent normal vector orientations by comparing the predicted normal direction with the spatial gradient direction of the volume density field, ensuring that the normal direction of each pixel points to the visible surface of the scene, avoiding geometric semantic errors caused by inconsistent normal orientations. Moreover, the dual post-processing mechanism significantly improves the geometric and physical accuracy and practicality of the normal map without increasing the network training burden, enabling the predicted second normal map to be directly used for 3D environmental perception and interactive decision-making in embodied intelligent devices.

[0073] In some preferred embodiments, after using the predicted second depth map and second normal map as geometric prior inputs for the embodied intelligent device to perform environmental perception and interaction decisions, the method further includes: obtaining a two-dimensional bounding box of the target object in the frame image to be processed; based on the second depth map, extracting the depth values ​​of all pixels within the two-dimensional bounding box, taking the median of the extracted depth values ​​to obtain the center depth value of the target object; based on the second normal map, extracting the normal direction vectors of all pixels within the two-dimensional bounding box, using a clustering algorithm to cluster all extracted normal direction vectors, and determining the cluster center direction vector of the largest cluster as the main surface orientation vector of the target object; based on the center depth value and the main surface orientation vector, determining the position and orientation of the target object in the three-dimensional environment where the embodied intelligent device is located, and using the position and orientation as the decision basis for the embodied intelligent device to perform grasping or obstacle avoidance operations.

[0074] In specific implementation, this embodiment extracts the center depth value and principal surface orientation vector of the target object from the depth map and normal map, transforming abstract geometric priors into specific three-dimensional spatial position and attitude information. This provides a direct decision-making basis for the grasping and obstacle avoidance operations of the embodied intelligent device. The specific implementation method is as follows: Obtain the two-dimensional bounding box of the target object in the frame image to be processed. The two-dimensional bounding box is a rectangular box that identifies and locates the object of interest in the current frame image by an object detection algorithm, and is represented in the form of upper left and lower right corner coordinates or center point coordinates plus width and height. The object detection algorithm can adopt a two-stage detector based on deep learning, such as Faster R-CNN, which first generates candidate regions and then classifies and regresses the candidate regions; or it can adopt a single-stage detector, such as YOLO or SSD, which outputs the category and bounding box of all targets simultaneously through a single forward inference, and has a faster inference speed. The category of the target object can be various predefined object types. For example, in an industrial grasping scenario, the target object can be a workpiece, part, or package; in an autonomous driving scenario, the target object can be a vehicle, pedestrian, or obstacle.

[0075] Furthermore, based on the second depth map, the depth values ​​of all pixels within the two-dimensional bounding box are extracted, and the median of the extracted depth values ​​is taken to obtain the center depth value of the target object. Since the two-dimensional bounding box of the target object may contain some background pixels or outlier depth values ​​from the object surface, directly taking the average value may be affected by extreme values ​​and deviate from the true depth. Using the median as the statistical measure of the center depth value has better robustness and can effectively suppress the influence of outliers, obtaining a representative depth estimate of the central region of the target object. This center depth value represents the distance of the target object relative to the camera and is the basis for subsequent three-dimensional spatial position determination.

[0076] Further, based on the second normal map, the normal direction vectors of all pixels within the two-dimensional bounding box are extracted. A clustering algorithm is then used to cluster all extracted normal direction vectors, and the cluster center direction vector of the largest cluster is determined as the main surface orientation vector of the target object. The clustering algorithm can be a density-based clustering algorithm such as DBSCAN (density-based clustering algorithm), which can automatically determine the number of clusters and identify outliers, suitable for scenarios with uneven normal direction distribution; or a partition-based clustering algorithm such as K-means, which requires pre-specifying the number of clusters, has higher computational efficiency, and is suitable for scenarios with relatively uniform normal direction distribution. The clustering algorithm groups the normal direction vectors of multiple pixels within the two-dimensional bounding box according to directional similarity. Furthermore, since the visible surface of the target object typically presents itself in space as a combination of one or more approximate planes, normal direction vectors on the same plane should point in similar directions. Clustering algorithms group normals with similar directions into the same cluster. The largest cluster corresponds to the main visible surface of the target object, and its cluster center direction vector is obtained by averaging and normalizing all normal direction vectors within the cluster; this is the main surface orientation vector. The main surface orientation vector describes the orientation direction of the main visible surface of the target object in three-dimensional space.

[0077] Furthermore, based on the center depth value and the main surface orientation vector, the position and orientation of the target object in the 3D environment of the embodied intelligent device are determined. Specifically, by combining the camera's intrinsic parameter matrix and the pixel coordinates of the target object within the 2D bounding box, the 3D spatial position of the target object in the camera coordinate system can be calculated, and the center depth value is the depth component of that position. The main surface orientation vector serves as the orientation information of the target object, describing its orientation direction in 3D space. The position and orientation are used as the decision-making basis for the embodied intelligent device to perform grasping or obstacle avoidance operations. In grasping operations, the robotic arm needs to determine the target point of the end effector based on the 3D position of the target object, and determine the grasping angle and approach direction of the end effector based on the main surface orientation. In obstacle avoidance operations, the mobile robot needs to determine the spatial area occupied by the obstacle based on its position, and determine its possible movement trend based on its orientation, in order to plan a safe avoidance path.

[0078] This embodiment extracts the median depth value within the 2D bounding box from the second depth map as the center depth value. Leveraging the robustness of the median statistic to outliers, it effectively suppresses depth value fluctuations caused by abnormal reflections from background pixels or object surfaces within the bounding box, resulting in an accurate depth estimate of the target object. Furthermore, by extracting the normal direction vector from the second normal map and performing cluster analysis, the cluster center direction vector of the largest cluster is determined as the main surface orientation vector, accurately reflecting the orientation direction of the main visible surfaces of the target object and providing an intuitive geometric basis for grasping posture planning. Thus, this embodiment transforms the predicted second depth map and second normal map from abstract pixel-level geometric information into object-level position and posture decision-making basis that can be directly used by embodied intelligent devices, achieving a complete technical closed loop from perception to decision-making.

[0079] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0080] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0081] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0082] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to execute a visual perception method based on the geometric prior of neural radiation fields.

[0083] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0084] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a visual perception method based on the geometric prior of neural radiation fields.

[0085] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0086] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a visual perception method based on neural radiation field geometric prior provided in any of the above method embodiments.

[0087] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0088] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0089] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of a visual perception method based on neural radiation field geometric priors provided in any of the above-described method embodiments.

[0090] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0091] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0092] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0093] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0095] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0096] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0097] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual perception method based on geometric priors of neural radiation fields, characterized in that, include: Acquire multi-view historical videos of a fixed monitoring scene, perform camera calibration and pose estimation on the multi-view historical videos, and obtain camera calibration parameters and pose estimation results; Based on the camera calibration parameters and the pose estimation results, a neural radiation field model is trained. The neural radiation field model can render a color image and the corresponding depth map and surface normal map from any input viewpoint. The current frame image in the monocular video stream of the target scene is acquired; the current frame image is input into the neural radiation field model, and the neural radiation field model renders a first depth map and a first normal map from the viewpoint corresponding to the current frame image; the current frame image is used as an input sample, and the first depth map and the first normal map are used as supervision signals to train a lightweight geometry estimation network, which is used to predict the depth map and normal map from the monocular image; During the online phase, the images of frames to be processed are acquired from the real-time video stream captured by the monocular camera on the embodied smart device; The image of the frame to be processed is input into the trained lightweight geometric estimation network to obtain a predicted second depth map and a second normal map, which serve as the geometric prior input for the embodied intelligent device to perform environmental perception and interaction decision-making.

2. The visual perception method based on neural radiation field geometric prior according to claim 1, characterized in that, The acquisition of multi-view historical video of a fixed monitoring scene includes: Acquire raw multi-view video data in a fixed monitoring scenario. The raw multi-view video data is collected synchronously by multiple fixed cameras or by a single movable camera at multiple different spatial locations. The original multi-view video data is time-stamp aligned, and the image frames from different perspectives at the same time are established to establish a perspective correspondence, resulting in a time-synchronized set of multi-view image frames. For each frame in the time-synchronized multi-view image frame set, key frame filtering is performed, the amount of image content change between adjacent frames is calculated, and image frames whose image content change exceeds a preset change threshold are retained as key frames, while image frames whose image content change does not exceed the preset change threshold are discarded. All the retained keyframes are organized according to their respective viewpoints and acquisition times to obtain the multi-view historical video.

3. The visual perception method based on neural radiation field geometric prior according to claim 1, characterized in that, The step of training the neural radiation field model based on the camera calibration parameters and the pose estimation results includes: The initial neural radiation field model is obtained by training based on the camera calibration parameters and the pose estimation results. Obtain the complete network structure and complete network parameters of the initial neural radiation field model; A network pruning method is used to evaluate the importance of neuron connections in the complete network structure. Neuron connections with importance metrics below a preset pruning threshold are removed from the complete network structure to obtain a sparse network structure. The floating-point network parameters in the sparse network structure are mapped to low-bit-width integer parameters using a parameter quantization method to obtain compressed network parameters. The sparse network structure and the compressed network parameters are combined to form a compressed neural radiation field model, which is then used as the neural radiation field model.

4. The visual perception method based on neural radiation field geometric prior according to claim 1, characterized in that, The rendering of the first depth map and the first normal map from the viewpoint corresponding to the current frame image by the neural radiation field model includes: Obtain the camera pose parameters corresponding to the current frame image, and input the camera pose parameters as the target viewpoint into the neural radiation field model; The neural radiation field model performs dense sampling along each line of sight from the target viewpoint to obtain the spatial coordinates and line of sight direction of multiple sampling points on each line of sight; The spatial coordinates and line-of-sight direction of the multiple sampling points are input into the implicit function of the neural radiation field model, and the implicit function outputs the volume density value and color value of the multiple sampling points. Based on the volume density values ​​of the multiple sampling points, the color value is integrated using volume rendering to obtain the color image corresponding to the current frame image; The depth of the sampling points is expected to be calculated based on the volume density values ​​of the multiple sampling points to obtain the first depth map; The spatial gradient of the volume density field is calculated based on the volume density values ​​of the multiple sampling points, and the spatial gradient is normalized to obtain the first normal map.

5. The visual perception method based on neural radiation field geometric prior according to claim 1, characterized in that, The step of training a lightweight geometry estimation network by using the current frame image as input samples and the first depth map and the first normal map as supervision signals includes: S21, sample a training image from the monocular video stream of the target scene, input the training image into the lightweight geometric estimation network, and obtain a predicted depth map and a predicted normal map; S22, the same frame of the training image is input into the neural radiation field model, and the neural radiation field model renders the training depth map and training normal map from the viewpoint corresponding to the training image. S23, perform invalid value detection on the training depth map, mark pixels with a depth value of zero or exceeding a preset depth range as invalid pixels, and exclude pixels in the predicted depth map that are at the same position as the invalid pixels from the depth loss calculation. S24, calculate the depth value difference of each valid pixel between the predicted depth map after excluding invalid pixels and the training depth map, and take the average of the sum of the squares of the depth value differences of all valid pixels to obtain the depth loss value; S25, calculate the angle between the normal direction vector of each pixel between the predicted normal map and the training normal map, and take the sum of the squares of the angles between the normal direction vectors of all pixels and then take the average value to obtain the normal loss value. S26, the depth loss value and the normal loss value are weighted and summed to obtain the total loss value; S27, with minimizing the total loss as the optimization objective, the network parameters of the lightweight geometric estimation network are updated using the backpropagation algorithm; S28. Repeat steps S21-S27 until the total loss value is lower than the preset convergence threshold or the number of iterations reaches the preset maximum number of iterations.

6. The visual perception method based on neural radiation field geometric prior according to claim 1, characterized in that, The step of inputting the frame image to be processed into the trained lightweight geometry estimation network to obtain the predicted second depth map and second normal map includes: The frame image to be processed is input into the feature extraction encoder of the lightweight geometric estimation network. The feature extraction encoder performs multi-level downsampling convolution on the frame image to be processed to obtain a multi-scale feature map. The multi-scale feature map is input into the depth decoder of the lightweight geometric estimation network. The depth decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map and regresses the depth value pixel by pixel to obtain the second depth map. The same multi-scale feature map is input into the normal decoder of the lightweight geometry estimation network. The normal decoder performs multi-level upsampling and skip connection fusion on the multi-scale feature map and regresses the three-dimensional normal direction vector pixel by pixel to obtain a preliminary normal map. The preliminary normal map is post-processed to obtain the second normal map; The upsampling path of the depth decoder is independent of the upsampling path of the normal decoder, and the depth decoder and the normal decoder share the multi-scale feature map output by the feature extraction encoder.

7. The visual perception method based on neural radiation field geometric prior according to claim 6, characterized in that, The post-processing of the preliminary normal map to obtain the second normal map includes: Normalize the normal direction vector of each pixel in the preliminary normal map by unit length so that the L2 norm of the normal direction vector of each pixel is equal to 1. A sign consistency check is performed on the normal direction vector of each pixel in the preliminary normal map after unit length normalization to detect whether the normal direction vector points to the visible surface of the scene; if the normal direction vector is detected to point to the invisible surface of the scene, the three components of the normal direction vector are simultaneously inverted to obtain the second normal map.

8. The visual perception method based on neural radiation field geometric prior according to claim 1, characterized in that, After using the predicted second depth map and second normal map as geometric prior inputs for the embodied intelligent device to perform environmental perception and interaction decisions, the method further includes: Obtain the two-dimensional bounding box of the target object in the frame image to be processed; Based on the second depth map, the depth values ​​of all pixels within the two-dimensional bounding box are extracted, and the median of the extracted depth values ​​is taken to obtain the center depth value of the target object. Based on the second normal map, the normal direction vectors of all pixels within the two-dimensional bounding box are extracted. A clustering algorithm is used to cluster all the extracted normal direction vectors, and the cluster center direction vector of the largest cluster is determined as the main surface orientation vector of the target object. Based on the center depth value and the main surface orientation vector, the position and orientation of the target object in the three-dimensional environment of the embodied intelligent device are determined, and the position and orientation are used as the decision basis for the embodied intelligent device to perform grasping or obstacle avoidance operations.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images