Methods, systems, devices, and storage media for generating visual environments based on neural radiation fields.

Abstract

This application relates to the field of UAV control technology, providing a method, system, device, and storage medium for generating visual environments based on neural radiation fields, solving the problem of poor visual environment generation effects. The method includes: semantically segmenting a corrected image to generate image data with semantic labels, which are used to label the categories of objects in an outdoor scene; updating a neural radiation field model based on the semantically labeled image data and mapping the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud; identifying non-critical regions in multi-view images, performing redundant voxel clipping on the non-critical regions to generate clipped semantic point clouds; and using the clipped semantic point clouds to drive the neural radiation field model to generate arbitrary viewpoint visuals of an outdoor scene. This technical solution achieves lightweight semantic modeling and efficient free-viewpoint rendering of outdoor visual environments, improving the efficiency and quality of generating new perspectives in complex environments.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a method and system for generating visual environment based on neural radiation fields. Background Technology

[0002] In applications such as UAV aerial surveying, stringent requirements are placed on efficient 3D reconstruction and real-time scene generation for large-scale outdoor scenes. This necessitates large-scale reconstruction capabilities, fully restoring the geometric structure and semantic information of kilometer-scale scenes, while effectively overcoming lighting variations, object occlusion, and dynamic interference in the actual environment. Furthermore, 3D semantic consistency must be achieved, ensuring that different types of objects, such as buildings, roads, and vegetation, maintain consistent semantic label recognition from any viewing angle. Moreover, under limited computing power, a precise balance must be struck between computational efficiency and output quality, guaranteeing reconstruction accuracy while meeting real-time rendering performance requirements, and avoiding memory overload issues.

[0003] The existing solution first decouples the dynamic scene by splitting the entire scene into two parts: "objects" and "background." Each object instance is modeled using an independent small multilayer perceptron, combined with its 3D bounding box and radiation field parameters. Second, it employs a multimodal data fusion strategy, utilizing existing mature algorithms to pre-acquire camera pose, object motion trajectory, and 2D semantic segmentation map. The weight parameters of the multilayer perceptron are jointly optimized by fusing self-supervised and pseudo-supervised learning. Finally, an implicit scene representation is constructed, outputting comprehensive information including voxel density, radiosity, and semantic labels, thereby supporting key tasks such as new perspective synthesis and 3D scene editing, and is particularly suitable for real-world road environment scenarios.

[0004] However, existing solutions lead to a linear increase in model parameters when the number of instances is large in large-scale urban scenes, increasing computational burden and memory consumption. The solution relies heavily on dense multi-angle image input for training to ensure reconstruction integrity. When the input viewpoint is sparse, insufficient geometric constraints can easily lead to semantic label mapping errors and false artifacts in non-critical areas. In addition, the solution lacks the use of prior knowledge of scene structure and does not introduce a spatial redundancy judgment mechanism, which results in equal allocation of computational resources to areas far from the viewpoint center or with low visual contribution, causing invalid voxel accumulation and thus limiting real-time rendering efficiency. Summary of the Invention

[0005] This application provides a visual environment generation method and system based on neural radiation fields to solve the problems in the prior art, such as insufficient model scalability leading to a surge in model parameters, reliance on dense viewpoint input causing semantic mapping errors and artifacts in sparse scenes, and lack of scene structure priors causing invalid voxel accumulation that limits real-time rendering efficiency, ultimately resulting in poor visual environment generation effects.

[0006] In a first aspect, this application provides a method for generating visual environments based on neural radiation fields, including:

[0007] Acquire multi-view images of an outdoor scene captured by a wide-angle lens, perform multi-camera synchronous calibration on the multi-view images, and generate a corrected image;

[0008] The corrected image is semantically segmented to generate image data with semantic labels, which are used to label the categories of objects in the outdoor scene;

[0009] In a cascaded rendering computer cluster, the neural radiation field model is updated based on the image data with semantic labels, and the semantic labels are mapped to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud.

[0010] Based on the semantic point cloud, non-critical regions in multi-view images are identified, and redundant voxel clipping is performed on the non-critical regions to generate a clipped semantic point cloud.

[0011] The cropped semantic point cloud is used to drive the neural radiation field model to generate an arbitrary view of an outdoor scene.

[0012] Optionally, the step of using the cropped semantic point cloud to drive the neural radiation field model to generate an arbitrary viewpoint of the outdoor scene includes:

[0013] Define the virtual camera parameters for the target viewpoint, including the virtual camera center position, pose, and intrinsic parameter matrix;

[0014] Based on the virtual camera parameters, rays are emitted from the center of the virtual camera along the direction of each pixel to form a ray set.

[0015] For each ray in the ray set, adaptive sampling along the ray direction is driven by the cropped semantic point cloud to generate a sampling point sequence;

[0016] For each sampling point in the sampling point sequence, the color attribute and density attribute of each sampling point are obtained from the voxel feature space of the neural radiation field model;

[0017] The color and density attributes of each sampling point are integrated along the ray direction to generate the final color value of each pixel.

[0018] The final color values ​​of all the pixels are aggregated to form the arbitrary viewpoint of the outdoor scene.

[0019] Optionally, the step of generating a sampling point sequence by using the cropped semantic point cloud to drive adaptive sampling along the ray direction for each ray in the ray set includes:

[0020] Based on the spatial distribution of the cropped semantic point cloud, the ray is divided into multiple initial sampling intervals along the ray direction, and an initial sampling point set is generated in each initial sampling interval according to a uniform step size.

[0021] Within the neighborhood of the initial sampling point set, a dynamic sampling step size is calculated based on the voxel density distribution of the cropped semantic point cloud and the spatial continuity of the semantic labels, and candidate sampling points are generated around the initial sampling points according to the dynamic sampling step size.

[0022] The candidate sampling points are subjected to semantic label consistency verification. When the similarity between the predicted semantic label of the candidate sampling point in the voxel feature space of the neural radiation field and the label of the semantic point cloud of the nearest initial sampling point is less than a preset similarity, a preset distance is extended in the ray direction with the candidate sampling point as the center to generate a label difference region; or, when the similarity is greater than the preset similarity, the candidate sampling point is retained.

[0023] Insert encrypted sampling points along the ray direction of the label difference region;

[0024] The initial set of sampling points, the retained candidate sampling points, and the encrypted sampling points are merged to generate a sampling point sequence along the ray direction.

[0025] Optionally, the step of identifying non-critical regions in multi-view images based on the semantic point cloud, and performing redundant voxel clipping on the non-critical regions to generate a clipped semantic point cloud includes:

[0026] Based on the object categories in the semantic tags, the frequency of each object category in the spatial region is counted, and the region where the frequency of all physical categories is lower than a preset frequency threshold is designated as the marked region.

[0027] Merge adjacent marked regions that meet a preset continuity condition to generate a non-critical region. The preset continuity condition includes: the overall similarity of adjacent marked regions exceeds a preset similarity threshold.

[0028] Locate the set of voxels corresponding to the non-critical region in the semantic point cloud, and remove voxels whose density values ​​are below the density threshold.

[0029] The retained voxels are recombined with voxels in the semantic point cloud that do not belong to the non-critical regions to generate a cropped semantic point cloud.

[0030] Optionally, merging adjacent marked regions that meet a preset continuity condition to generate a non-critical region includes:

[0031] Each marked region is abstracted as a graph node. If the Euclidean distance between two adjacent marked regions is less than a preset spatial distance threshold, an undirected edge is established between the graph nodes to form an adjacency graph.

[0032] For each undirected edge in the adjacency graph, calculate semantic label similarity, boundary geometric similarity, and density distribution similarity. Then, weight the semantic label similarity, boundary geometric similarity, and density distribution similarity to obtain the corresponding comprehensive similarity.

[0033] Iteratively merge adjacent marked regions whose overall similarity is greater than a preset similarity threshold to update the adjacency graph until the overall similarity between all adjacent marked regions in the adjacency graph is lower than the preset similarity threshold to obtain candidate regions;

[0034] Calculate the semantic entropy of the candidate regions, and select candidate regions whose semantic entropy is less than a preset semantic entropy threshold as valid merged regions;

[0035] A filtering process is performed on the effective merged region to generate non-critical regions. The filtering process includes at least one of the following: area threshold filtering and aspect ratio constraint.

[0036] Optionally, the step of updating the neural radiation field model based on the image data with semantic labels, and mapping the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud, includes:

[0037] Extract the geometric and semantic features of each voxel from the image data with semantic labels;

[0038] The parameters of the neural radiation field model are updated by jointly optimizing the geometric and semantic features.

[0039] In the voxel feature space of the neural radiation field model, the semantic labels are mapped to the corresponding voxel positions;

[0040] By fusing the spatial coordinates of the voxel positions with the semantic labels, a semantic point cloud is constructed.

[0041] Optionally, updating the parameters of the neural radiation field model by jointly optimizing the geometric and semantic features includes:

[0042] Calculate the matching error between the geometric features and the corresponding three-dimensional geometric representation in the neural radiation field model, and generate a geometric loss.

[0043] Calculate the consistency error between the semantic features and the semantic labels to generate semantic loss;

[0044] The geometric loss and semantic loss are jointly optimized to generate a joint optimized loss.

[0045] The parameters of the neural radiation field model are updated based on the joint optimization loss.

[0046] Secondly, this application provides a visual environment generation system based on neural radiation fields, comprising:

[0047] The acquisition module is used to acquire multi-view images of outdoor scenes captured by a wide-angle lens, perform multi-camera synchronous calibration on the multi-view images, and generate a corrected image.

[0048] The generation module is used to perform semantic segmentation on the corrected image to generate image data with semantic labels, which are used to label the categories of objects in the outdoor scene;

[0049] The module is used to update the neural radiation field model based on the image data with semantic labels in a cascaded rendering computer cluster, and to map the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud.

[0050] The recognition module is used to identify non-critical regions in multi-view images based on the semantic point cloud, perform redundant voxel clipping on the non-critical regions, and generate a clipped semantic point cloud.

[0051] The driving module is used to drive the neural radiation field model using the cropped semantic point cloud to generate an arbitrary view of the outdoor scene.

[0052] Thirdly, this application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are to be invoked and executed by the processing component to implement a visual environment generation method based on neural radiation fields as described in any of the first aspects.

[0053] Fourthly, this application provides a computer storage medium storing a computer program, which, when executed by a computer, implements a visual environment generation method based on neural radiation fields as described in any of the first aspects.

[0054] This application provides a method for generating a visual environment based on a neural radiation field. The method includes: acquiring multi-view images of an outdoor scene captured by a wide-angle lens; performing multi-camera synchronous calibration on the multi-view images to generate corrected images; performing semantic segmentation on the corrected images to generate image data with semantic labels, whereby the semantic labels are used to label the categories of objects in the outdoor scene; updating a neural radiation field model based on the image data with semantic labels in a cascaded rendering computer cluster, and mapping the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud; identifying non-critical regions in the multi-view images based on the semantic point cloud, performing redundant voxel clipping on the non-critical regions to generate a clipped semantic point cloud; and using the clipped semantic point cloud to drive the neural radiation field model to generate an arbitrary-view view of the outdoor scene, thereby improving the generation effect of the visual environment.

[0055] The beneficial effects of this application are:

[0056] This application employs multi-camera synchronous calibration, utilizing wide-angle lens multi-view images for precise spatial and temporal alignment, effectively eliminating lens distortion and shooting sequence deviations, laying a geometric consistency foundation for subsequent processing. Building upon this, semantic segmentation technology enhances data representation, precisely labeling object categories at the pixel level, elevating visual information to the semantic level, thus laying the foundation for structured scene understanding. Subsequently, a semantic neural radiation field is constructed, using cascaded computing capabilities to fuse two-dimensional semantic label information into a three-dimensional voxel space, forming a point cloud model that simultaneously contains geometric shape, radiation degree, and semantic attributes, achieving implicit semantic representation of the scene. Furthermore, based on semantic labels, non-critical regions with low visual contribution are identified, and redundant invalid voxels are dynamically pruned, effectively reducing the overall complexity of the model. Finally, the simplified semantic point cloud drives the rendering pipeline, ensuring the reconstruction accuracy of key regions while improving the synthesis speed of new perspective images, meeting the application requirements of real-time interaction.

[0057] Furthermore, firstly, a set of rays covering all pixel directions is emitted based on the position, pose, and internal parameters of the virtual camera from the target viewpoint. Then, semantically driven adaptive sampling is implemented, dividing the initial interval along each ray direction and generating initial sampling points with uniform step sizes. Next, candidate sampling points are generated within the neighborhood of the initial points based on the spatial distribution and semantic continuity of the semantic point cloud. Then, semantic verification and expansion are performed: when the similarity between the predicted semantic label of a candidate point and the label of the nearest initial point is lower than a set threshold, a label difference region is generated by expanding a preset distance around that point; if the similarity reaches or exceeds the threshold, the candidate point is retained. Next, encrypted sampling points are inserted along the ray direction in the identified label difference region. The initial points, retained candidate points, and encrypted points are merged to form the final sampling point sequence. Finally, the color and density attributes of the sampling points in the sequence are queried, and pixel color values ​​for any viewpoint are generated by accumulating along the ray integral. This process achieves semantically guided sampling optimization, which dynamically adjusts the sampling density based on semantic information, automatically encrypts sampling in areas rich in geometric features, and sparsely samples in flat areas, effectively reducing invalid sampling points. At the same time, it enhances boundary accuracy by accurately identifying and encrypting semantically abrupt regions through semantic consistency verification, eliminating the blur artifacts at object contours in traditional methods. Finally, it optimizes the reallocation of computing resources, focusing limited computing power on key areas with significant semantic differences, avoiding the waste of resources in uniform sampling, and accelerating the ray integration process while maintaining rendering quality, meeting the needs of high frame rate real-time scene generation.

[0058] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A flowchart illustrating a visual environment generation method based on neural radiation fields, provided for embodiments of this application;

[0061] Figure 2 A schematic diagram of the structure of a visual environment generation system based on neural radiation fields provided in an embodiment of this application;

[0062] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0063] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0064] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 11, 12, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] To address the problems of poor visual environment generation in existing technologies, such as insufficient model scalability leading to a surge in model parameters, reliance on dense viewpoint input causing semantic mapping errors and artifacts in sparse scenes, and lack of scene structure priors resulting in invalid voxel accumulation limiting real-time rendering efficiency, this application provides a visual environment generation method based on neural radiation fields. This method employs the following concept: eliminating distortion and temporal deviations through wide-angle lens multi-view image acquisition and multi-camera spatiotemporal synchronous calibration to establish a geometric consistency foundation; using semantic segmentation technology to perform pixel-level object category labeling on the corrected images, elevating them to a semantic information layer to support structured scene analysis; integrating semantic labels and neural radiation field voxel feature space in a cascaded computing architecture to construct a 3D point cloud model with geometric, radiometric, and semantic attributes; identifying non-critical regions based on semantic importance and dynamically pruning redundant voxels to compress model complexity; and finally driving the neural radiation field rendering pipeline with a simplified semantic point cloud to achieve high-precision real-time visual environment generation from any viewpoint in large outdoor scenes, optimizing computational resource allocation efficiency while ensuring the reconstruction quality of critical areas.

[0067] Figure 1 A flowchart illustrating a visual environment generation method based on neural radiation fields provided in this application embodiment is shown below. Figure 1 As shown, the method includes:

[0068] S11. Acquire multi-view images of an outdoor scene captured by a wide-angle lens, perform multi-camera synchronous calibration on the multi-view images, and generate corrected images.

[0069] Wide-angle lenses refer to optical devices with a focal length shorter than that of standard lenses, used to capture wide-field images of large outdoor scenes, including fisheye lenses or ultra-wide-angle lenses. Outdoor scenes refer to open spaces in natural or urban environments, including static objects such as buildings, roads, and vegetation, as well as dynamic factors such as changes in lighting. Multi-view images refer to a collection of images of the same scene simultaneously captured from different angles by multiple spatially distributed cameras. Multi-camera synchronous calibration refers to the process of calculating the relative pose and temporal deviations between cameras based on a common calibration object, used to eliminate lens distortion and shooting time differences. Corrected images refer to multi-view images that have undergone geometric distortion correction and spatiotemporal alignment processing, possessing pixel-level coordinate consistency.

[0070] In this embodiment, firstly, multi-view images are simultaneously acquired by a wide-angle lens array deployed in an outdoor scene. Secondly, multi-camera synchronous calibration technology is used to perform spatiotemporal alignment processing on the images, specifically including lens distortion correction and timing deviation compensation based on a checkerboard calibration board, and finally, a corrected image is generated.

[0071] S12. Perform semantic segmentation on the corrected image to generate image data with semantic labels. The semantic labels are used to label the categories of objects in the outdoor scene.

[0072] Semantic segmentation refers to a technique that uses deep learning models to classify images at the pixel level, used to identify object category boundaries. Semantic labels are discrete classification identifiers assigned to each pixel, used to label the specific category attributes of objects in the image. Image data refers to a digital image carrier containing a pixel value matrix and associated metadata.

[0073] In this embodiment, the calibration image is first input into a pre-trained semantic segmentation network. Then, the image content is analyzed pixel by pixel through a convolutional neural network to identify and label the category attributes of objects such as buildings, roads, and vegetation, and finally, image data with semantic labels is generated.

[0074] S13. In the cascaded rendering computer cluster, the neural radiation field model is updated based on the image data with semantic labels, and the semantic labels are mapped to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud.

[0075] Among them, cascaded rendering computer clusters refer to distributed computing architectures composed of multi-node servers, used for parallel processing of neural radiation field training tasks. Neural radiation field models are machine learning models that implicitly represent 3D scenes using multilayer perceptrons, capable of outputting the density and radiance of arbitrary spatial points. Voxel feature space refers to a data structure that discretizes 3D space into raster cells, with each cell storing geometric, semantic, and other feature vectors. Semantic point clouds refer to a set of 3D points that integrates spatial coordinates, radiation attributes, and semantic labels, constituting an explicit semantic representation of the scene.

[0076] In this embodiment, a neural radiation field model is first loaded into a cascaded rendering computer cluster. Then, the model parameters are iteratively optimized using image data with semantic labels. Subsequently, the two-dimensional semantic labels are mapped to the corresponding coordinates in the three-dimensional voxel feature space, and finally, a semantic point cloud is constructed.

[0077] S14. Based on semantic point cloud, identify non-critical regions in multi-view images, perform redundant voxel clipping on non-critical regions, and generate clipped semantic point cloud.

[0078] Non-critical regions refer to scene subspaces with low visual contribution, such as distant skies or walls with monotonous textures, identified through semantic frequency statistics. Redundant voxel pruning refers to the operation of removing invalid voxels based on density thresholds and semantic importance to reduce model complexity. The pruned semantic point cloud refers to the simplified 3D point set after redundant voxel pruning, preserving the complete semantic attributes of key regions.

[0079] In this embodiment, the frequency of occurrence of spatial regions is first statistically analyzed based on the object category distribution of the semantic point cloud. Then, regions with frequencies below a threshold are marked as candidate regions. Subsequently, adjacent candidate regions whose geometric continuity meets preset conditions are merged to form non-critical regions. Finally, redundant voxel clipping is performed on these regions to remove low-density voxels and reorganize the remaining voxels to generate the clipped semantic point cloud.

[0080] S15. Use the cropped semantic point cloud to drive the neural radiation field model to generate arbitrary viewpoints of outdoor scenes.

[0081] Among them, arbitrary viewpoint scene refers to the free-viewpoint rendered image generated by virtual camera parameters to meet the needs of real-time interactive applications.

[0082] In this embodiment, virtual camera parameters are first set according to the target viewpoint, then a set of rays covering the pixel direction is emitted, then a sampling sequence is generated based on the cropped semantic point cloud-driven adaptive sampling, and finally, the color and density attributes of the sampling points along the rays are integrated to generate an arbitrary viewpoint of the outdoor scene.

[0083] Here's a specific example: First, multi-view images of urban roads are acquired using wide-angle lenses mounted on a swarm of drones. A calibration board is then used to align the spatiotemporal parameters of each lens to generate calibrated images. Next, the Deep Lab v3+ model is used to perform semantic segmentation on the calibrated images, labeling them with categories such as vehicles, pedestrians, and traffic signs. Then, in a cascaded rendering computer cluster equipped with four GPU servers, the semantic labels are mapped to a neural radiation field voxel space to construct a semantic point cloud. Subsequently, areas of distant billboards that appear at low frequencies in the point cloud are identified as non-critical regions, and their low-density voxels are cropped to form a simplified point cloud. Finally, the virtual camera's overhead view parameters are set, and the color values ​​of the sampled points are integrated along pixel rays to generate a panoramic overhead rendering of the road.

[0084] By executing S11~S15, this embodiment of the application ensures geometric consistency through multi-camera calibration, injects structured understanding capabilities through semantic segmentation, and constructs a three-dimensional neural field representation with fused semantics under the support of distributed computing power; it dynamically prunes low-value voxels based on semantic importance, thereby reducing the computational load; and finally, through a semantically driven adaptive sampling mechanism, it improves boundary accuracy by encrypting sampling in key areas and optimizes resource allocation by sparse sampling in non-key areas, thereby achieving high-quality real-time rendering of large-scale outdoor scenes.

[0085] In one possible embodiment, S15, using the cropped semantic point cloud to drive a neural radiation field model to generate an arbitrary view of the outdoor scene, including:

[0086] Step 151: Define the virtual camera parameters for the target viewpoint. The virtual camera parameters include the virtual camera center position, pose, and intrinsic parameter matrix.

[0087] The target viewpoint refers to the virtual observation orientation specified by the user or automatically generated by the system, including pitch angle, azimuth angle, and observation distance parameters. The virtual camera center position refers to the coordinates of the virtual camera's optical center on the X, Y, and Z axes of the three-dimensional world coordinate system. Attitude refers to the rotation state of the virtual camera coordinate system relative to the world coordinate system, expressed through a rotation matrix or quaternions. The intrinsic parameter matrix describes the imaging characteristics of the virtual camera, including focal length, principal point offset, and pixel aspect ratio parameters.

[0088] In this embodiment, the target observation angle is first determined according to user interaction instructions or preset path planning. Then, the spatial coordinates of the center position of the virtual camera, the attitude rotation matrix of the lens orientation, and the intrinsic parameter matrices of the focal length and imaging plane are defined, and finally a complete set of virtual camera parameters is formed.

[0089] Step 152: Based on the virtual camera parameters, emit rays from the center of the virtual camera along the direction of each pixel to form a ray set.

[0090] Here, pixel direction refers to the normalized three-dimensional direction vector pointing from the camera center to a specific pixel on the imaging plane. Ray set refers to the spatial ray cluster covering all pixels on the virtual camera's imaging plane, with each ray corresponding to one pixel.

[0091] In this embodiment, the direction vector corresponding to each pixel on the imaging plane is first calculated based on the virtual camera intrinsic parameter matrix. Then, spatial rays are emitted from the center position of the virtual camera along the direction vector, and finally a ray set covering all pixels is generated.

[0092] Step 153: For each ray in the ray set, use the clipped semantic point cloud to drive adaptive sampling along the ray direction to generate a sampling point sequence.

[0093] Adaptive sampling refers to a strategy that dynamically adjusts the density of sampling points based on scene features, driven by the geometric and semantic features of the semantic point cloud. A sampling point sequence refers to a set of three-dimensional points arranged in spatial order along a single ray, used for subsequent attribute queries and integral calculations.

[0094] In this embodiment, firstly, an initial sampling interval with uniform step size is divided along the ray direction and an initial sampling point set is generated. Secondly, the dynamic step size is calculated based on the voxel density distribution and semantic continuity of the clipped semantic point cloud. Candidate sampling points are generated in the neighborhood of the initial point. Then, the semantic label similarity between the candidate point and the neighboring initial point is checked. If it is lower than the threshold, the label difference region is expanded and encrypted sampling points are inserted. Finally, the initial point, the retained candidate point and the encrypted point are merged to form a sampling point sequence.

[0095] Step 154: For each sampling point in the sampling point sequence, obtain the color and density attributes of each sampling point from the voxel feature space of the neural radiation field model.

[0096] Among them, the color attribute refers to the RGB three-channel luminance value output by the neural radiation field, reflecting the reflective properties of the object's surface. The density attribute refers to the scalar value of light absorption rate at a spatial point, determining the object's opacity and light transmission ability.

[0097] In this embodiment, the three-dimensional coordinates of the sampling point in the voxel feature space of the neural radiation field are first located, then the color vector and light absorption density attribute stored at the coordinates are queried, and finally the complete optical attribute data of each sampling point are obtained.

[0098] Step 155: Integrate the color and density attributes of each sampling point along the ray direction to accumulate and generate the final color value of each pixel.

[0099] The integral calculation refers to the process of summing the colors of the sampling points along the ray path according to their transmittance weights, simulating the physical laws of light propagation in reality. The final color value is the result of the integral calculation for a single ray, representing the color of the corresponding pixel on the virtual imaging plane.

[0100] In this embodiment, the sampling point sequence is first sorted by spatial distance along the ray direction, then the density attribute of adjacent sampling points is converted into transmittance weights, and then the cumulative integral of the transmittance weighted color attribute is calculated to finally generate the final color value of the pixel corresponding to the current ray.

[0101] Step 156: Aggregate the final color values ​​of all pixels to form an arbitrary view of the outdoor scene.

[0102] Aggregation refers to the operation of recombining the final color values ​​of all pixels into a two-dimensional image according to the topology of the imaging plane grid.

[0103] In this embodiment, the pixel color values ​​of all rays are first calculated, then the color data are aggregated according to the pixel arrangement order of the imaging plane, and finally a complete visual image of the outdoor scene that conforms to the target viewpoint is generated.

[0104] Here's a specific example: First, a swarm of drones equipped with wide-angle lenses captures multi-angle images of urban roads. Then, a calibration board is used to synchronously calibrate the multiple cameras, correcting lens distortion and shooting time differences to generate geometrically aligned corrected images. Next, an image recognition model is used to label object categories pixel-by-pixel in the corrected images, adding semantic tags to elements such as vehicles, pedestrians, and traffic signs to form structured image data. Then, in a computing cluster consisting of four GPU servers, the two-dimensional semantic tags are fused with a three-dimensional neural radiation field model to construct a semantic point cloud containing spatial coordinates and object attributes. Subsequently, non-critical regions are identified based on the frequency of object occurrence in the point cloud; for example, low-density three-dimensional pixels are removed from distant billboard areas that appear less frequently, forming a simplified point cloud. Finally, based on traffic monitoring requirements, the virtual camera parameters were set: the camera center was located directly above the road, 100 meters above the ground, with the lens pointing vertically downwards, and an imaging parameter of 1920×1080 resolution was used; a detection ray was emitted from the camera center to each pixel on the road surface, and the sampling point density was automatically increased in the vehicle outline area using a simplified point cloud; after querying the color and transparency attributes of each sampling point, high-precision color accumulation was performed on the top of the vehicle, and low-precision calculation was performed on the sky area; finally, a panoramic overhead view of the traffic conditions was synthesized, clearly showing the vehicle position and road markings.

[0105] By executing steps 151 to 156, this embodiment of the application uses a semantic point cloud-driven adaptive sampling mechanism to dynamically increase the reconstruction accuracy in geometrically complex regions and reduce the computational load by sparsely sampling in flat regions; combined with semantic boundary verification, it eliminates object contour artifacts and achieves high-fidelity rendering of key regions; finally, it generates high-quality visual scenes from any perspective through ray integration and pixel aggregation, ensuring visual realism while meeting real-time requirements.

[0106] In one possible embodiment, step 153, for each ray in the ray set, uses the clipped semantic point cloud to drive adaptive sampling along the ray direction to generate a sampling point sequence, including:

[0107] Step a1: Based on the spatial distribution of the clipped semantic point cloud, divide the ray into multiple initial sampling intervals along the ray direction, and generate an initial sampling point set in each initial sampling interval with a uniform step size.

[0108] Spatial distribution refers to the coordinate distribution characteristics of the clipped semantic point cloud in the 3D scene, reflecting the object clustering degree and geometric complexity. The initial sampling interval refers to equal-length sub-segments divided along the ray direction, used to initialize the uniform sampling framework. The uniform step size refers to the fixed sampling interval within the initial sampling interval, ensuring basic sampling coverage. The initial sampling point set refers to the initial 3D point set generated according to the uniform step size, constituting the basic sampling framework.

[0109] In this embodiment, the scene complexity through which the ray passes is first analyzed based on the spatial distribution characteristics of the clipped semantic point cloud. Then, equidistant initial sampling intervals are divided along the ray direction. Subsequently, an initial sampling point set is generated in each interval with a fixed uniform step size.

[0110] Step a2: Within the neighborhood of the initial sampling point set, calculate the dynamic sampling step size based on the voxel density distribution of the clipped semantic point cloud and the spatial continuity of the semantic labels, and generate candidate sampling points around the initial sampling points according to the dynamic sampling step size.

[0111] Here, voxel density distribution refers to the rate of change in the number of voxels per unit volume within a neighborhood, reflecting the richness of local geometric features. Spatial continuity of semantic labels refers to the degree of spatial clustering of voxels with the same semantic label within a neighborhood. Dynamic sampling step size refers to the sampling interval dynamically calculated based on density distribution and semantic continuity, automatically reducing the step size in feature-rich regions. Candidate sampling points refer to the 3D sampling points supplemented in the neighborhood of the initial point according to the dynamic step size.

[0112] In this embodiment, a spherical neighborhood space is first defined with the initial sampling point as the center. Then, the voxel density distribution gradient of the semantic point cloud and the spatial continuity index of the semantic label are calculated. Subsequently, the dynamic sampling step size is derived based on the inverse relationship between the gradient and the continuity index. Finally, candidate sampling points are generated around the initial sampling point according to the dynamic step size.

[0113] Step a3: Perform semantic label consistency verification on candidate sampling points. When the similarity between the predicted semantic label of a candidate sampling point in the voxel feature space of the neural radiation field and the label of the semantic point cloud of the nearest initial sampling point is less than a preset similarity, expand the candidate sampling point by a preset distance in the ray direction to generate a label difference region. Alternatively, if the similarity is greater than the preset similarity, retain the candidate sampling point.

[0114] The semantic label consistency verification process refers to the verification procedure that compares the semantic label similarity between candidate points and neighboring initial points. Predicted semantic labels refer to the semantic category identifiers of sampled points inferred through a neural radiation field model. The nearest initial sampled point refers to the initial sampling frame reference point with the closest spatial Euclidean distance. Similarity refers to the cosine similarity measure of the semantic label vectors of two sampled points. Preset similarity refers to the discrimination threshold set based on scene complexity.

[0115] In this embodiment, the semantic label of the candidate sampling point is first predicted by the neural radiation field model. Then, the similarity between the label and the label of the nearest initial sampling point is calculated. Subsequently, when the similarity is lower than the preset similarity, a label difference region is generated by extending a preset distance along the ray with the candidate point as the center. When the similarity reaches or exceeds the threshold, the candidate sampling point is retained.

[0116] Step a4: Insert encrypted sampling points in the ray direction of the label difference area.

[0117] Among them, the label difference region refers to the ray sub-segment with obvious semantic mutation, which is generated by expanding candidate points with similarity below the threshold.

[0118] In this embodiment, the start and end boundaries of the label difference region are first located, and then encrypted sampling points with a density higher than the initial sampling are inserted within the boundary along the ray direction, ultimately forming a refined sampling cluster for the semantic mutation region.

[0119] Step a5: Merge the initial set of sampling points, the retained candidate sampling points, and the encrypted sampling points to generate a sampling point sequence along the ray direction.

[0120] The preset distance refers to a fixed length parameter that extends the label difference region along the ray direction. The encrypted sampling points refer to high-density three-dimensional sampling points inserted within the label difference region.

[0121] In this embodiment, the initial set of sampling points is first merged with the reserved candidate sampling points, and then the encrypted sampling points are inserted into the merged sequence in spatial order, finally generating the final sampling point sequence arranged in an orderly manner along the ray direction.

[0122] Here's a specific example: First, a swarm of drones equipped with wide-angle lenses captures multi-angle images of urban roads. A calibration board aligns the camera positions and shooting time differences to generate geometrically aligned corrected images. Second, an image recognition model labels object categories pixel-by-pixel in the corrected images, adding semantic tags to elements such as vehicles, pedestrians, and traffic signs. Next, in a computing cluster consisting of four GPU servers, the two-dimensional semantic tags are fused with the three-dimensional scene model to construct point cloud data containing spatial coordinates and object attributes. Subsequently, low-frequency regions in the point cloud are identified, such as distant billboards that appear less frequently, and their low-density three-dimensional pixels are removed to form a simplified point cloud. Finally, the virtual camera's overhead parameters were set, with the camera center positioned directly above the road at a height of 100 meters above the ground. This height was calculated based on the relationship between the road width and the lens's vertical field of view angle. The formula is: monitoring height equals the road width divided by the tangent of half the lens's vertical field of view angle. The lens is positioned vertically downwards, using high-definition resolution imaging parameters. Detection rays are emitted from the camera center towards the road surface. Initial sampling intervals are divided along the rays based on the spatial distribution of a simplified point cloud, and initial sampling points are generated at fixed intervals. When high-density point cloud distribution and abrupt changes in object category characteristics are detected in the vehicle edge area, the sampling interval is automatically shortened to generate candidate points. When the difference in category label between a vehicle candidate point and an adjacent road surface initial point exceeds a set threshold, encrypted sampling points are inserted in the difference area. Initial points, candidate points, and encrypted points are merged to form a sampling sequence. After querying the color and transparency attributes of each point, refined color accumulation is performed on the vehicle outline area, and routine calculations are performed on the road surface area. Finally, a panoramic overhead view of the traffic conditions is synthesized, clearly showing the sharp transition between vehicle edges and road markings.

[0123] By executing steps a1 to a5, this embodiment of the application uses a dynamic sampling mechanism driven by semantic point clouds to automatically increase the reconstruction accuracy in geometrically complex regions and reduce the computational load by sparsely sampling in semantically smooth regions; combined with semantic boundary verification, it accurately identifies abrupt change areas in the object contour and inserts targeted encrypted sampling points to eliminate boundary artifacts; ultimately, it achieves intelligent allocation of computing resources, optimizing rendering efficiency while ensuring the visual quality of key areas.

[0124] In one possible embodiment, S14 involves identifying non-critical regions in a multi-view image based on the semantic point cloud, performing redundant voxel clipping on the non-critical regions, and generating a clipped semantic point cloud, including:

[0125] Step 141: Based on the object categories in the semantic tags, count the frequency of each object category in the spatial region, and use the region where the frequency of all physical categories is lower than the preset frequency threshold as the marked region.

[0126] In this context, frequency of occurrence refers to the proportion of semantic tags for a specific object category within a unit of spatial area, reflecting the importance of that category in the local space. A spatial area refers to a cubic mesh unit that divides a 3D scene into fixed-size units, used for local statistical analysis. The preset frequency threshold is the critical frequency value used to determine whether an area is of low importance; it needs to be dynamically set according to the scene size. A marked area is a spatial mesh unit where the frequency of occurrence of all object categories is below the preset frequency threshold.

[0127] In this embodiment of the application, the three-dimensional space is first divided into regular grid units as spatial regions. Then, the occurrence frequency of semantic labels of different object categories in each grid is counted and converted into frequency values. Subsequently, all grids with all category frequencies lower than a preset frequency threshold are marked as marked regions.

[0128] Step 142: Merge adjacent marked regions that meet the preset continuity conditions to generate non-critical regions. The preset continuity conditions include: the overall similarity of adjacent marked regions exceeds the preset similarity threshold.

[0129] The preset continuity condition refers to the geometric and semantic association rules that must be satisfied to merge adjacent marked regions. Adjacent marked regions refer to sets of marked grid cells that are spatially contiguous and share a boundary. Comprehensive similarity is a weighted evaluation index that integrates semantic composition similarity, boundary geometric similarity, and density distribution similarity. The preset similarity threshold is the comprehensive similarity threshold that triggers the region merging operation.

[0130] In this embodiment, spatially adjacent marked regions are first detected, then the semantic composition similarity, boundary shape similarity, and density distribution similarity between adjacent regions are calculated, and then a weighted sum is obtained to obtain a comprehensive similarity. Finally, adjacent marked regions with a comprehensive similarity exceeding a preset similarity threshold are merged into continuous non-critical regions.

[0131] Step 143: Locate the set of voxels corresponding to non-critical regions in the semantic point cloud and remove voxels with density values ​​below the density threshold.

[0132] In this context, a voxel set refers to all three-dimensional rasterized units belonging to the same non-critical region. The density value is a scalar measure of light absorption rate stored in a voxel unit, reflecting the probability of the existence of a spatially located geometric entity. The density threshold is the critical density value that distinguishes between effective and ineffective voxels.

[0133] In this embodiment, firstly, all voxel sets covered by non-critical regions are located in the semantic point cloud; secondly, the density value of each voxel is read; then, voxels with density values ​​below the density threshold are removed, and high-density voxels are retained.

[0134] Step 144: Recombine the retained voxels with voxels in the semantic point cloud that do not belong to non-critical regions to generate the cropped semantic point cloud.

[0135] In this embodiment, all voxels not marked as non-critical regions in the semantic point cloud are first extracted. Then, the retained high-density voxels are recombined with the voxels outside the non-critical regions according to spatial coordinates, and finally, a structurally complete cropped semantic point cloud is generated.

[0136] Here is a specific example: First, a swarm of drones equipped with wide-angle lenses acquires multi-angle images of urban roads. A calibration board is used to align the camera positions and shooting times to generate corrected images. Second, an image recognition model is used to label object categories pixel-by-pixel in the corrected images, adding semantic tags to elements such as vehicles, pedestrians, and traffic signs. Then, in a computing cluster consisting of four graphics processing units (GPUs), the two-dimensional semantic tags are fused with the three-dimensional scene model to construct a labeled 3D point set. The three-dimensional space was then divided into cubic grid units with sides of five meters. This size was determined based on the minimum building size along the road, and the calculation formula was that the grid side length was equal to one-tenth of the width of a typical building. The proportion of vehicles, pedestrians, and other categories appearing in each grid was counted. When the proportion of all categories was less than three percent, the grid was marked. Adjacent marked grids were detected, and their building surface feature similarity and boundary shape matching degree were calculated. The weighted summation formula was that the comprehensive similarity was equal to the surface similarity multiplied by 0.6 plus the boundary matching degree multiplied by 0.4. When the result exceeded 0.7, they were merged into non-critical areas. The three-dimensional pixel set corresponding to the non-critical areas was located, and the billboard bracket pixels with a density value lower than 0.1 per cubic meter were removed. This density threshold was set based on the air density benchmark value. Finally, the retained high-density pixels of the walls and the pixels of the road area were recombined to form a simplified three-dimensional point set. Finally, the virtual camera parameters were set, with the camera center located directly above the road at a height of 35 meters above the ground. This height was calculated using a road width of 30 meters and a vertical viewing angle of 45 degrees. The calculation formula is: monitoring height equals road width divided by twice the tangent function value multiplied by half the vertical viewing angle of the lens. The simplified point set was sampled and rendered along the pixel ray to generate a panoramic overhead view of the traffic conditions, clearly showing vehicles and road markings.

[0137] By executing steps 141 to 144, this embodiment of the application identifies low-value regions through semantic frequency statistics, expands the clipping range by combining geometric and semantic continuity, accurately removes invalid voxels through density filtering, and compresses the model size while retaining key structures; ultimately, it achieves targeted optimization of computing resources and provides a lightweight data foundation for real-time rendering.

[0138] In one possible embodiment, step 142, merging adjacent marked regions that meet a preset continuity condition to generate a non-critical region, includes:

[0139] Step b1: Abstract each marked region into a graph node. If the Euclidean distance between two adjacent marked regions is less than a preset spatial distance threshold, then establish an undirected edge between the graph nodes to form an adjacency graph.

[0140] In this context, a graph node is an abstract topological unit representing a single marked region, used to construct a network of regional adjacency relationships. Euclidean distance is the straight-line distance between two points in three-dimensional space, used to quantify the spatial proximity of marked regions. The preset spatial distance threshold is the critical value for determining the maximum spatial interval between adjacent regions. An undirected edge is a bidirectional connection line connecting two graph nodes, representing spatial adjacency. An adjacency graph is a topological network composed of nodes and undirected edges, expressing the spatial adjacency relationships of marked regions.

[0141] In this embodiment, each marked region is first abstracted as an independent node in a graph structure. Then, the straight-line distance between the center points of the spatial regions corresponding to any two nodes is detected. Subsequently, when the distance is less than a preset spatial distance threshold, a bidirectional connection edge is established between the corresponding nodes, and finally a connection graph expressing the adjacency relationship of the regions is formed.

[0142] Step b2: Calculate the semantic label similarity, boundary geometric similarity, and density distribution similarity for each undirected edge in the adjacency graph. Then, weight the semantic label similarity, boundary geometric similarity, and density distribution similarity to obtain the corresponding comprehensive similarity.

[0143] Semantic label similarity refers to the cosine similarity measure of the semantic category composition distribution between two regions. Boundary geometric similarity refers to the Hausdorff distance index, which measures the matching degree of the polygonal shapes of the two regions' outlines. Density distribution similarity refers to the Pearson correlation coefficient, which measures the trend of point cloud density changes between two regions. Weighted processing refers to the calculation process of linearly combining multiple indicators according to preset weight coefficients.

[0144] In this embodiment of the application, firstly, the semantic label composition similarity, boundary contour geometric similarity, and point cloud density distribution similarity of the two ends of each connecting edge in the connection graph are calculated. Secondly, the three similarity indicators are weighted and summed according to preset weight coefficients to finally obtain a comprehensive similarity value that represents the overall correlation between regions.

[0145] Step b3: Iteratively merge adjacent labeled regions whose overall similarity is greater than the preset similarity threshold to update the adjacency graph until the overall similarity between all adjacent labeled regions in the adjacency graph is lower than the preset similarity threshold, thus obtaining candidate regions.

[0146] Iterative merging refers to the optimization process of repeatedly performing region fusion and graph structure updates. Candidate regions refer to the set of regions to be screened after similarity-driven merging.

[0147] In this embodiment, the connection graph is first traversed to filter connection edges whose overall similarity exceeds a preset threshold. Then, the marked regions represented by the corresponding nodes are merged and the node attributes are updated. Subsequently, the connection graph structure is reconstructed and the similarity is recalculated. This process is repeated until no connection edges exceeding the threshold exist, and finally, a merged candidate region set is obtained.

[0148] Step b4: Calculate the semantic entropy of the candidate regions, and select the candidate regions whose semantic entropy is less than the preset semantic entropy threshold as valid merged regions.

[0149] Semantic entropy refers to a measure of the uncertainty in the distribution of semantic labels within a region; a lower entropy value indicates a more homogeneous category. The preset semantic entropy threshold is the critical entropy value standard for determining whether semantic consistency is met. An effective merged region refers to a subset of candidate regions whose semantic entropy is below the threshold and meets the merging requirements.

[0150] In this embodiment, the semantic entropy value of the candidate region is first calculated, which reflects the degree of confusion of semantic categories in the region. Then, the candidate regions with entropy values ​​lower than the preset semantic entropy threshold are determined as valid merged regions that meet the semantic consistency standard.

[0151] Step b5: Perform a filtering process on the effective merged areas to generate non-critical areas. The filtering process includes at least one of the following: area threshold filtering and aspect ratio constraint.

[0152] Among these, filtering refers to a secondary refinement of regions based on geometric rules. Area threshold filtering refers to removing invalid small regions with areas smaller than a preset value. Aspect ratio constraint refers to excluding irregular regions with aspect ratios exceeding a preset limit.

[0153] In this embodiment of the application, the area of ​​the effective merged area is first filtered to remove excessively small areas, and then narrow and irregular areas are excluded according to the length-width ratio constraint. Finally, a set of non-critical areas that meet the spatial regularity requirements is generated.

[0154] Here is a specific example: First, a swarm of drones equipped with wide-angle lenses captures multi-angle images of the city's main roads. A calibration board is used to align the positions of each camera and the shooting sequence to generate corrected images. Next, an image recognition model is used to label object categories pixel-by-pixel in the corrected images, adding semantic tags to elements such as vehicles, streetlights, and billboards. Then, in a computing cluster consisting of four GPU servers, the two-dimensional semantic tags are fused with a three-dimensional scene model to construct a 3D point set with category labels. Subsequently, areas of distant, low-frequency billboards are marked as processing units, with each billboard area considered an independent node. When the straight-line distance between the center points of two billboards is less than 10 meters, a connection is established to form a network graph. This distance threshold is derived from statistical values ​​of the minimum billboard spacing on city streets. Then, the similarity of the billboard category composition, the matching degree of the outer contour shape, and the correlation of the point cloud density in connected areas are calculated. These are weighted and summed in a 5:3:2 ratio to obtain a comprehensive similarity value, with the weight allocation determined based on the importance of the billboard structural features. Next, adjacent regions with a comprehensive similarity exceeding 0.85 are cyclically merged and the network graph is updated. This threshold is optimized by testing the merging effect in different scenarios until no excessive connections exist. Subsequently, a semantic confusion index of the merged regions is calculated using a mathematical formula, retaining regions with an index value below 1.2. This threshold is set based on the requirement that the index meets the requirement when the proportion of advertising categories in the region exceeds 85%. Finally, regions with an area smaller than five square meters and a length-to-width ratio exceeding 3:1 are removed. This size limit is determined with reference to the minimum building facade size. Non-critical regions are generated, and their low-density support points are removed. Finally, the virtual camera parameters are set, with the camera center located directly above the road at a height of 50 meters above the ground. This height is calculated using a road width of 40 meters and a vertical viewing angle of 60 degrees. The calculation formula is: monitoring height equals road width divided by twice the tangent function value multiplied by half the vertical viewing angle of the lens. The simplified point set is sampled and rendered along the pixel ray to generate a panoramic top-down view of the road, clearly showing the positional relationship between vehicles and billboards.

[0155] By executing steps b1 to b5, this embodiment of the application accurately expresses the spatial relationships of regions through topological graph modeling, drives progressive region merging by combining multi-dimensional similarity evaluation, uses semantic entropy to quantitatively screen semantically consistent regions to avoid loss of effective information caused by excessive merging, and finally generates structurally regular non-critical regions through geometric rule constraints, laying the foundation for refined voxel clipping.

[0156] In one possible embodiment, S13, updating the neural radiation field model based on image data with semantic labels, and mapping the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud, including:

[0157] Step 131: Extract the geometric and semantic features of each voxel from the image data with semantic labels.

[0158] Geometric features refer to mathematical vectors describing the spatial structural properties of voxels, including parameters reflecting three-dimensional shape such as center point coordinates, boundary dimensions, and surface curvature. Semantic features refer to discrete or continuous vectors characterizing voxel category attributes, such as high-level semantic information like object category probability distributions and material attribute encodings.

[0159] In this embodiment, each three-dimensional voxel unit is first located from image data with semantic labels, then geometric feature vectors describing spatial position and shape are extracted, and semantic feature vectors representing object category attributes are extracted, and finally bimodal feature data of all voxels are obtained.

[0160] Step 132: Update the parameters of the neural radiation field model by jointly optimizing geometric and semantic features.

[0161] Joint optimization refers to a training strategy that simultaneously minimizes geometric reconstruction error and semantic classification error, thereby achieving cross-modal knowledge transfer by sharing the underlying features of the network.

[0162] In this embodiment, a geometric feature reconstruction error function and a semantic feature classification error function are first constructed. Then, the two errors are jointly optimized through a multi-task learning framework. Subsequently, the gradient backpropagation algorithm is used to update the weight parameters of the fully connected layer of the neural radiation field model, and finally, the coordinated expression optimization of geometry and semantics is achieved.

[0163] Step 133: Map the semantic labels to the corresponding voxel positions in the voxel feature space of the neural radiation field model.

[0164] Here, voxel position refers to the integer index value of the voxel unit in the three-dimensional mesh coordinate system, which is used to locate the storage location in the feature space.

[0165] In this embodiment, the three-dimensional index coordinate system of the voxel feature space of the neural radiation field is first determined, and then the semantic labels in the two-dimensional image data are associated with the corresponding voxel index positions through projection transformation, and finally the semantic labels are accurately mapped in three-dimensional space.

[0166] Step 134: Combine the spatial coordinates of voxel locations with semantic labels to construct a semantic point cloud.

[0167] Spatial coordinates refer to the floating-point physical coordinates of the voxel center point in the three-dimensional world coordinate system, used to describe the absolute spatial position.

[0168] In this embodiment, the spatial coordinates of the voxel in the three-dimensional world coordinate system are first read, and then the coordinates are concatenated with the mapped semantic labels to generate a semantic point cloud dataset that contains both spatial location and category attributes.

[0169] Here's a specific example: First, a swarm of drones equipped with wide-angle lenses captures multi-angle images of an urban intersection. A calibration board aligns the camera lens positions and shooting time differences to generate geometrically corrected images. Next, an image recognition model labels each pixel of the corrected images with object categories, adding category labels to traffic elements such as vehicles, pedestrians, and traffic lights. Then, in a computing cluster consisting of four GPU servers, the 3D size features and vehicle type attribute features of vehicle areas are extracted from the labeled images. Through joint training, the accuracy of vehicle shape reconstruction and type recognition is simultaneously optimized, updating the internal parameters of the 3D scene model. Subsequently, the vehicle labels in the 2D images are converted to 3D mesh cell index positions through spatial mapping. This mapping process is based on camera imaging principles, specifically calculating the correspondence between 2D pixels and 3D space using the camera's intrinsic and extrinsic parameter matrices. Finally, the latitude and longitude coordinates and altitude coordinates of the center point of the 3D mesh cell are combined with the vehicle type label to construct a vehicle point set containing spatial location and semantic attributes. Next, distant billboard areas with a frequency of less than 3% in the point set were marked as non-critical areas. This frequency threshold was determined based on the statistical distribution of major objects in the road scene. Low-density areas of billboard supports were identified, and grid cells with a density value of less than 0.05 points per cubic meter were removed. This density threshold was set based on the air density benchmark. Finally, virtual camera parameters were set, with the camera center located 45 meters above the ground directly above the intersection. This height was calculated using the maximum road width of 50 meters and the vertical viewing angle of the lens of 50 degrees. The calculation formula is: monitoring height equals the maximum road width divided by twice the tangent function value multiplied by half the vertical viewing angle of the lens. Ray tracing calculations were performed on the simplified point set along all pixel directions. First, basic sampling points were generated. The sampling density was automatically increased in the vehicle outline area according to the point set distribution, and the number of sampling points was reduced in the sky area. After querying the color and transparency attributes of each sampling point, refined color accumulation was performed on the vehicle surface, and routine calculations were performed on the background area. Finally, a panoramic top-down view of the intersection was synthesized, clearly showing the vehicle positions and traffic light status.

[0170] By executing steps 131 to 134, this embodiment of the application enhances the consistency of 3D representation through joint optimization of geometric and semantic features, improves the model's ability to model complex scenes by utilizing multi-task learning, establishes cross-modal associations through accurate semantic label 3D mapping, and constructs a point cloud model with both spatial accuracy and semantic understanding capabilities, providing a structured data foundation for subsequent scene editing and analysis.

[0171] In one possible embodiment, step 132, updating the parameters of the neural radiation field model by jointly optimizing geometric and semantic features, includes:

[0172] Step c1: Calculate the matching error between the geometric features and the corresponding three-dimensional geometric representation in the neural radiation field model, and generate the geometric loss.

[0173] Here, 3D geometric representation refers to the spatial density field function output by the neural radiation field model, used as an implicit mathematical expression to describe the 3D shape of the scene. Matching error is a measure of the positional deviation between the model-predicted geometry and the actual spatial structure of the scene. Geometric loss is a numerical function constructed based on the matching error, reflecting the optimization goal of 3D reconstruction accuracy.

[0174] In this embodiment, three-dimensional geometric representation data predicted by the neural radiation field model is first obtained, then the spatial position difference between the data and the geometric features extracted from the image is calculated, and finally the difference value is quantized into a geometric loss function output.

[0175] Step c2: Calculate the consistency error between semantic features and semantic labels to generate semantic loss.

[0176] Consistency error refers to the degree of information difference between the model's predicted semantic category and the true label. Semantic loss is a loss function term that quantifies the semantic prediction bias and is used to improve category recognition accuracy.

[0177] In this embodiment, the probability distributions of semantic feature vectors and real semantic labels are first compared, then the category identification deviation value is calculated using the cross-entropy algorithm, and finally a semantic loss function output reflecting the accuracy of semantic prediction is generated.

[0178] Step c3: Jointly optimize the geometric loss and semantic loss to generate the joint optimized loss.

[0179] Among them, joint optimization loss refers to a composite objective function that integrates geometric loss and semantic loss to achieve multi-task collaborative training.

[0180] In this embodiment, the weight coefficients of geometric loss and semantic loss are first set, and then the two weighted losses are added together to form a joint loss function that drives model optimization.

[0181] Step c4: Update the parameters of the neural radiation field model based on the joint optimization loss.

[0182] In this embodiment, the gradient of the joint loss function with respect to the model parameters is first calculated, then an adaptive learning rate optimization algorithm is used to update the weights of the fully connected layer along the negative gradient direction, and finally the iterative optimization of the neural radiation field model parameters is achieved.

[0183] Here is a specific example: First, a swarm of drones equipped with wide-angle lenses captures multi-angle images of an urban intersection. A calibration board is used to align the positions of each camera lens and the time difference between shots to generate geometrically corrected images. Next, an image recognition model is used to label object categories pixel-by-pixel in the corrected images, adding type labels to traffic elements such as vehicles, pedestrians, and traffic lights. Then, the following operations are performed on a computing cluster consisting of four GPU servers: Precise vehicle shape data is obtained using a laser scanning device as a benchmark. The spatial position difference between the predicted vehicle body surface and the actual vehicle shape is calculated to generate a shape error. Simultaneously, the vehicle type prediction result output by the model is compared with the type difference of manually labeled categories to generate a category error. The shape error is multiplied by 0.7 and the category error is multiplied by 0.3, and then added to obtain a comprehensive optimization target value. This weight ratio, after multiple tests, has been found to balance shape accuracy and type recognition requirements. Based on this target value, the 3D scene model is optimized using an internal parameter adjustment algorithm. Finally, the vehicle labels in the 2D images are converted to 3D grid cell positions through spatial mapping. A semantic point set is constructed by combining the latitude and longitude of the grid center point, altitude, and vehicle type. Next, the statistical points focused on distant billboard areas with a frequency of less than 5%, a threshold set based on the minimum occurrence frequency of major traffic elements. Support structures with a density value below 0.1 per cubic meter within the billboard area were removed; this threshold was determined based on air density measurement data. Finally, virtual camera parameters were set, with the camera center positioned 50 meters above the ground directly above the intersection. This height was calculated using the maximum road width of 60 meters and a vertical viewing angle of 55 degrees. The formula was: monitoring height equals the maximum road width divided by twice the tangent function value multiplied by half the vertical viewing angle. Ray calculations were performed on the simplified point set along all pixel directions, automatically increasing the sampling point density in vehicle outline areas. After querying the color and transparency attributes of each point, refined color accumulation was performed on the vehicle surface. Finally, a panoramic overhead view of the intersection was generated, clearly displaying vehicle colors, outline details, and traffic light status.

[0184] By executing steps c1 to c4, this embodiment of the application balances the training objectives of geometric reconstruction and semantic recognition through a joint optimization strategy, and improves spatial accuracy and semantic consistency simultaneously by utilizing gradient updates; the constructed composite loss function effectively coordinates the multi-task learning process and enhances the model's ability to express complex outdoor scenes.

[0185] Figure 2 A schematic diagram of a visual environment generation system based on neural radiation fields provided in this application embodiment is shown below. Figure 2 As shown, the system includes:

[0186] The acquisition module 21 is used to acquire multi-view images of outdoor scenes captured by a wide-angle lens, perform multi-camera synchronous calibration on the multi-view images, and generate corrected images.

[0187] The generation module 22 is used to perform semantic segmentation on the corrected image and generate image data with semantic labels. The semantic labels are used to label the categories of objects in the outdoor scene.

[0188] Module 23 is used to update the neural radiation field model based on image data with semantic labels in a cascaded rendering computer cluster, and to map the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud.

[0189] The recognition module 24 is used to identify non-critical regions in multi-view images based on semantic point clouds, perform redundant voxel clipping on the non-critical regions, and generate clipped semantic point clouds.

[0190] The driving module 25 is used to drive the neural radiation field model using the cropped semantic point cloud to generate arbitrary viewpoint scenes of outdoor scenes.

[0191] Figure 2 The aforementioned visual environment generation system based on neural radiation fields can perform... Figure 1 The implementation principle and technical effects of the visual environment generation method based on neural radiation fields described in the illustrated embodiment will not be repeated here. The specific methods by which each module and unit performs its operations in the visual environment generation system based on neural radiation fields described in the above embodiments have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0192] In one possible design, Figure 2 The visual environment generation system based on neural radiation fields in the illustrated embodiment can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32.

[0193] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0194] The processing component 32 is used to: acquire multi-view images of an outdoor scene captured by a wide-angle lens; perform multi-camera synchronous calibration on the multi-view images to generate corrected images; perform semantic segmentation on the corrected images to generate image data with semantic labels, which are used to label the categories of objects in the outdoor scene; in a cascaded rendering computer cluster, update the neural radiation field model based on the image data with semantic labels, and map the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud; based on the semantic point cloud, identify non-critical regions in the multi-view images, perform redundant voxel clipping on the non-critical regions, and generate clipped semantic point clouds; use the clipped semantic point cloud to drive the neural radiation field model to generate arbitrary viewpoint scenes of the outdoor scene.

[0195] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0196] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Random Access Memory (RAM), Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0197] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0198] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0199] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0200] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0201] This application also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The illustrated embodiment is a method for generating a visual environment based on neural radiation fields.

[0202] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0203] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0204] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating visual environment based on neural radiation fields, characterized in that, include: Acquire multi-view images of an outdoor scene captured by a wide-angle lens, perform multi-camera synchronous calibration on the multi-view images, and generate a corrected image; The corrected image is semantically segmented to generate image data with semantic labels, which are used to label the categories of objects in the outdoor scene; In a cascaded rendering computer cluster, the neural radiation field model is updated based on the image data with semantic labels, and the semantic labels are mapped to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud. Based on the semantic point cloud, non-critical regions in multi-view images are identified, and redundant voxel clipping is performed on the non-critical regions to generate a clipped semantic point cloud. The cropped semantic point cloud is used to drive the neural radiation field model to generate an arbitrary view of an outdoor scene. The step of using the cropped semantic point cloud to drive the neural radiation field model to generate an arbitrary viewpoint of an outdoor scene includes: Define the virtual camera parameters for the target viewpoint, including the virtual camera center position, pose, and intrinsic parameter matrix; Based on the virtual camera parameters, rays are emitted from the center of the virtual camera along the direction of each pixel to form a ray set. For each ray in the ray set, adaptive sampling along the ray direction is driven by the cropped semantic point cloud to generate a sampling point sequence; For each sampling point in the sampling point sequence, the color attribute and density attribute of each sampling point are obtained from the voxel feature space of the neural radiation field model; The color and density attributes of each sampling point are integrated along the ray direction to generate the final color value of each pixel. The final color values ​​of all the pixels are aggregated to form the arbitrary viewpoint of the outdoor scene; For each ray in the ray set, adaptive sampling along the ray direction is driven by the cropped semantic point cloud to generate a sampling point sequence, including: Based on the spatial distribution of the cropped semantic point cloud, the ray is divided into multiple initial sampling intervals along the ray direction, and an initial sampling point set is generated in each initial sampling interval according to a uniform step size. Within the neighborhood of the initial sampling point set, a dynamic sampling step size is calculated based on the voxel density distribution of the cropped semantic point cloud and the spatial continuity of the semantic labels, and candidate sampling points are generated around the initial sampling points according to the dynamic sampling step size. The candidate sampling points are subjected to semantic label consistency verification. When the similarity between the predicted semantic label of the candidate sampling point in the voxel feature space of the neural radiation field and the label of the semantic point cloud of the nearest initial sampling point is less than a preset similarity, a preset distance is extended in the ray direction with the candidate sampling point as the center to generate a label difference region; or, when the similarity is greater than the preset similarity, the candidate sampling point is retained. Insert encrypted sampling points along the ray direction of the label difference region; The initial set of sampling points, the retained candidate sampling points, and the encrypted sampling points are merged to generate a sampling point sequence along the ray direction.

2. The method for generating visual environment based on neural radiation field according to claim 1, characterized in that, The step of identifying non-critical regions in multi-view images based on the semantic point cloud, performing redundant voxel clipping on the non-critical regions, and generating a clipped semantic point cloud includes: Based on the object categories in the semantic tags, the frequency of each object category in the spatial region is counted, and the region where the frequency of all physical categories is lower than a preset frequency threshold is designated as the marked region. Merge adjacent marked regions that meet a preset continuity condition to generate a non-critical region. The preset continuity condition includes: the overall similarity of adjacent marked regions exceeds a preset similarity threshold. Locate the set of voxels corresponding to the non-critical region in the semantic point cloud, and remove voxels whose density values ​​are below the density threshold. The retained voxels are recombined with voxels in the semantic point cloud that do not belong to the non-critical regions to generate a cropped semantic point cloud.

3. The method for generating visual environment based on neural radiation field according to claim 2, characterized in that, The process of merging adjacent marked regions that meet a preset continuity condition to generate a non-critical region includes: Each marked region is abstracted as a graph node. If the Euclidean distance between two adjacent marked regions is less than a preset spatial distance threshold, an undirected edge is established between the graph nodes to form an adjacency graph. For each undirected edge in the adjacency graph, calculate semantic label similarity, boundary geometric similarity, and density distribution similarity. Then, weight the semantic label similarity, boundary geometric similarity, and density distribution similarity to obtain the corresponding comprehensive similarity. Iteratively merge adjacent marked regions whose overall similarity is greater than a preset similarity threshold to update the adjacency graph until the overall similarity between all adjacent marked regions in the adjacency graph is lower than the preset similarity threshold to obtain candidate regions; Calculate the semantic entropy of the candidate regions, and take the candidate regions whose semantic entropy is less than a preset semantic entropy threshold as valid merged regions; A filtering process is performed on the effective merged region to generate non-critical regions. The filtering process includes at least one of the following: area threshold filtering and aspect ratio constraint.

4. The method for generating visual environment based on neural radiation field according to claim 1, characterized in that, The step of updating the neural radiation field model based on the image data with semantic labels, and mapping the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud includes: Extract the geometric and semantic features of each voxel from the image data with semantic labels; The parameters of the neural radiation field model are updated by jointly optimizing the geometric and semantic features. In the voxel feature space of the neural radiation field model, the semantic labels are mapped to the corresponding voxel positions; By fusing the spatial coordinates of the voxel positions with the semantic labels, a semantic point cloud is constructed.

5. The visual environment generation method based on neural radiation field according to claim 4, characterized in that, The method of updating the parameters of the neural radiation field model by jointly optimizing the geometric and semantic features includes: Calculate the matching error between the geometric features and the corresponding three-dimensional geometric representation in the neural radiation field model, and generate a geometric loss. Calculate the consistency error between the semantic features and the semantic labels to generate semantic loss; The geometric loss and semantic loss are jointly optimized to generate a joint optimized loss. The parameters of the neural radiation field model are updated based on the joint optimization loss.

6. A visual environment generation system based on neural radiation fields, characterized in that, include: The acquisition module is used to acquire multi-view images of outdoor scenes captured by a wide-angle lens, perform multi-camera synchronous calibration on the multi-view images, and generate corrected images. The generation module is used to perform semantic segmentation on the corrected image to generate image data with semantic labels, which are used to label the categories of objects in the outdoor scene; The module is used to update the neural radiation field model based on the image data with semantic labels in a cascaded rendering computer cluster, and to map the semantic labels to the voxel feature space corresponding to the neural radiation field model to construct a semantic point cloud. The recognition module is used to identify non-critical regions in multi-view images based on the semantic point cloud, perform redundant voxel clipping on the non-critical regions, and generate a clipped semantic point cloud. The driving module is used to drive the neural radiation field model using the cropped semantic point cloud to generate an arbitrary view of the outdoor scene. The step of using the cropped semantic point cloud to drive the neural radiation field model to generate an arbitrary viewpoint of an outdoor scene includes: Define the virtual camera parameters for the target viewpoint, including the virtual camera center position, pose, and intrinsic parameter matrix; Based on the virtual camera parameters, rays are emitted from the center of the virtual camera along the direction of each pixel to form a ray set. For each ray in the ray set, adaptive sampling along the ray direction is driven by the cropped semantic point cloud to generate a sampling point sequence; For each sampling point in the sampling point sequence, the color attribute and density attribute of each sampling point are obtained from the voxel feature space of the neural radiation field model; The color and density attributes of each sampling point are integrated along the ray direction to generate the final color value of each pixel. The final color values ​​of all the pixels are aggregated to form the arbitrary viewpoint of the outdoor scene; For each ray in the ray set, adaptive sampling along the ray direction is driven by the cropped semantic point cloud to generate a sampling point sequence, including: Based on the spatial distribution of the cropped semantic point cloud, the ray is divided into multiple initial sampling intervals along the ray direction, and an initial sampling point set is generated in each initial sampling interval according to a uniform step size. Within the neighborhood of the initial sampling point set, a dynamic sampling step size is calculated based on the voxel density distribution of the cropped semantic point cloud and the spatial continuity of the semantic labels, and candidate sampling points are generated around the initial sampling points according to the dynamic sampling step size. The candidate sampling points are subjected to semantic label consistency verification. When the similarity between the predicted semantic label of the candidate sampling point in the voxel feature space of the neural radiation field and the label of the semantic point cloud of the nearest initial sampling point is less than a preset similarity, a preset distance is extended in the ray direction with the candidate sampling point as the center to generate a label difference region; or, when the similarity is greater than the preset similarity, the candidate sampling point is retained. Insert encrypted sampling points along the ray direction of the label difference region; The initial set of sampling points, the retained candidate sampling points, and the encrypted sampling points are merged to generate a sampling point sequence along the ray direction.

7. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement a visual environment generation method based on neural radiation fields as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The system contains a computer program that, when executed by a computer, implements a visual environment generation method based on neural radiation fields as described in any one of claims 1-5.

Images