Double-flow NeRF method for real-time reconstruction of dynamic environment

Abstract

The invention records a double-flow NeRF method for real-time reconstruction of a dynamic environment, and the method comprises the steps: S1, obtaining the dynamic environment data of a vehicle, including a four-way polarization image of the dynamic environment, a near-infrared image of the dynamic environment and the like, carrying out the multispectral fusion of the four-way polarization image and the near-infrared image, and outputting a feature map after fusion; s2, generating a dynamic mask based on the theoretical optical flow of the fused feature map; fusing the transient field and the dynamic field and outputting a final dynamic mask of the fused feature map; s3, constructing a three-dimensional reconstruction space taking the vehicle as the center, and dispersing the three-dimensional reconstruction space into a plurality of blocks; block feature vectors are loaded for the blocks needing to be updated; implicit scene representation of the three-dimensional reconstruction space is constructed, and graph-free semantic representation is output; s4, converting the three-dimensional reconstruction space into semantic understanding, segmenting a drivable domain of the three-dimensional reconstruction space, and generating a skeleton path of the three-dimensional reconstruction space; and optimizing a skeleton path through an energy functional minimization framework, and identifying and classifying a lane center line.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving environment perception, and particularly relates to a double-flow NeRF method for real-time reconstruction of a dynamic environment. BACKGROUND

[0002] An environment perception system integrates devices such as lidar, camera, and ultrasonic sensor, aiming to capture and process the environmental data around the vehicle, and is a key component of an L3-level automatic driving system. At present, the environment perception system mainly has the following problems or defects in practical application:

[0003] Poor dynamic interference robustness: a pure visual perception scheme based on monocular RGB is limited by optical physical characteristics, lacks multi-spectral physical modeling capability and rigid body motion constraint mechanism, and the dynamic target recall rate is only about 67.2±3.8% in rain, fog scattering and strong light polarization scenes;

[0004] Low real-time calculation efficiency: although the NeRF neural radiance field three-dimensional reconstruction technology can generate a scene model with centimeter-level precision, the inference delay of the existing architecture (such as Instant-NGP) on a vehicle embedded platform reaches about 52.7±4.3ms, which is far more than the ≤20ms requirement of the automatic driving control cycle, and also does not solve the problem of radiance field degradation caused by dynamic objects;

[0005] Geographical information is not in compliance: a high-precision map-dependent positioning scheme (such as Huawei MDC) violates the mandatory provisions of Article 5 of the Regulations on the Qualification Management of Surveying and Mapping of the Ministry of Natural Resources (Natural Resources Regulations (2023) No. 1) on geographical information collection. The statistical results of the ApolloScape dataset show that it also faces a city scene mismatch rate of up to about 18.6%.

[0006] In order to solve the above problems or defects, researchers at present try to break through the above limitations through multi-modal fusion, dynamic NeRF and visual SLAM schemes, but still have the following defects:

[0007] The computing power requirement of the multi-modal fusion system is >200TOPS, IEEE Trans.ITS 24:1123, 2023, which forms a generational gap with domestic vehicle-grade chips (such as Horizon J5, 48TOPS);

[0008] The dynamic NeRF scene reconstruction method needs to know the 3D label of the object, which is difficult to adapt to unknown motion patterns on open roads;

[0009] The topological error rate of the visual SLAM mapping scheme at a non-marked intersection is 15.4±2.1%.

[0010] For the three problems of dynamic distortion, real-time bottleneck and map dependence, the current research blank mainly focuses on:

[0011] Lack of dynamic interference filtering mechanism combining optical physics and kinematics, leading to non-rigid object false detection rate of about >40%;

[0012] There are few records in existing literature about the real-time NeRF architecture supported by the embedded platform of the vehicle, and the existing scheme has a reconstruction delay of about >50ms;

[0013] The high-precision map replacement scheme has a high error rate at complex intersections (such as about 15% in Baidu Apollo Lite), and risks of surveying and mapping policies. SUMMARY

[0014] The present application aims to solve the problems existing in the prior art to some extent, and provides a technical scheme for providing autonomous and controllable perception ability for map-free autonomous driving, and gives a dynamic separation method with physical interpretability, a lightweight incremental reconstruction engine and a policy-compliant instantaneous environment representation technology. Therefore, the present application discloses a dual-flow NeRF method for real-time reconstruction of dynamic environment, and the technical scheme is as follows.

[0015] A dual-flow NeRF method for real-time reconstruction of dynamic environment, comprising the following steps:

[0016] S1, acquiring dynamic environment data of a vehicle, including four-way polarization images of the dynamic environment, near-infrared images of the dynamic environment, and IMU six-axis data of the vehicle in the dynamic environment, performing multispectral fusion on the four-way polarization images and the near-infrared images, and outputting a fused feature map;

[0017] S2, constructing a rigid body motion field of the vehicle, generating a dynamic mask based on the theoretical optical flow of the fused feature map; fusing the transient field and the dynamic field based on an optimization objective function and a constraint function, and outputting a final dynamic mask of the fused feature map;

[0018] S3, based on the fused feature map and the dynamic mask, constructing a three-dimensional reconstruction space centered on the vehicle, and discretizing it into multiple blocks;

[0019] Weighted sum of geometric changes and appearance changes of the blocks, setting an update threshold, marking blocks with a weighted sum greater than the update threshold as blocks that need to be updated, and loading block feature vectors for the blocks that need to be updated;

[0020] Constructing an implicit scene representation of the three-dimensional reconstruction space, coupling the dynamic area and the static area of the three-dimensional reconstruction space based on artifact suppression, and outputting a map-free semantic representation;

[0021] S4, converting the three-dimensional reconstruction space into semantic understanding through the generated map-free semantic representation, and segmenting the drivable domain of the three-dimensional reconstruction space;

[0022] input the bird's eye view semantic segmentation binary lane pixel results into the drivable area, generate the skeleton path of the three-dimensional reconstruction space based on the morphological thinning algorithm;

[0023] construct the objective function of the energy functional, and optimize the skeleton path through the energy functional minimization framework to identify and classify the lane centerline.

[0024] Further, the four-way polarization image and the near-infrared image are quality evaluated through the following expression:

[0025] Q = 0.5 · R valid + 0.3 · e -σDoP + 0.2 · SNR NIR

[0026] In the formula, Q is the quality evaluation score; R valid is the proportion of effective pixels; -σDop is the standard deviation of the polarization degree; DoP is the polarization degree of the polarization image; e -σDop is the exponential decay term of the standard deviation of the polarization degree; SNR NIR is the signal-to-noise ratio of the near-infrared band;

[0027] The four-way polarization image and the near-infrared image with a quality evaluation score >0.65 are multispectral fused through the following formula:

[0028]

[0029] In the formula, F fused is the fused feature map; DoP is the polarization degree of the four-way polarization image; AoP is the polarization angle of the four-way polarization image; min max is a normalization function; is the near-infrared image; is a gradient operator.

[0030] Further, the following formula is used to suppress glare of the fused feature map:

[0031]

[0032] In the formula, I recovered is the real intensity of the reconstructed fused feature map; I obs is the observed intensity of the fused feature map; φ opt is the optimized polarization angle of the minimum saturation area intensity of the fused feature map; DoP is the polarization degree of the polarization image; AoP is the polarization angle of the polarization image;

[0033] The following formula is used to correct rain and fog of the fused feature map:

[0034]

[0035] where I object is the target light intensity; I obs is the observed post-fusion feature map intensity; I atm is the light intensity generated by atmospheric scattering; e -βd is the degree of attenuation of light in the atmosphere; β is the scattering coefficient; d is the distance of light propagation in the atmosphere.

[0036] Further, the SE(3) pose transformation matrix is constructed by the following expression to describe the rigid body motion field of the vehicle:

[0037]

[0038] where T t→t+1 is the pose transformation matrix of the vehicle from time "t" to "t+1"; ΔR is the rotation increment; Δs is the translation increment;

[0039] The theoretical optical flow of the post-fusion feature map is calculated by the following expression:

[0040]

[0041] where V rigid is the theoretical optical flow of the post-fusion feature map; π is the projection function; K is the camera intrinsic matrix; d is the depth estimate; T t→t+1 is the pose transformation matrix of the vehicle from time "t" to "t+1"; P is the original pixel coordinate of the four-direction polarization image and / or near-infrared image, i.e., P = (u, v).

[0042] Further, the dynamic mask is generated by the following expression to identify the binary image of the dynamic region in the post-fusion feature map:

[0043]

[0044] where M dynamic is the dynamic mask; ∏ is the indicator function, which takes the value of 1 when the condition is met, otherwise 0; is the divergence threshold condition; is the divergence of the optical flow residual; δV is the difference between the observed optical flow and the theoretical optical flow.

[0045] Further, the optimization objective function of the transient field is constructed by the following expression:

[0046] L τ =∑ r (KL(τ(r)‖M dynamic (r))+λ tv TV(τ(r)))

[0047] where L τis the optimization objective function of transient field; KL(τ(r)‖M dynamic is KL divergence; λ tv is the weight parameter of TV regularization; TV(τ(r)) is total variation;

[0048] The constraint function of static field is constructed by the following expression:

[0049] L static =||τ(r)·M static ||1

[0050] In the formula, L static is the constraint function of static field; ||τ(r)·M static ||1 is L1 norm, which is used to force the transient component of static region to be zero, wherein the transient component is introduced by the following expression:

[0051]

[0052] In the formula, is the estimated color value at position r; T(t) is a time function; σ(r(t)) is the absorption coefficient of the scene point r(t) at time t; c static (r(t)) is static component; τ(r(t)) is transient radiation component.

[0053] Further, the transient field and dynamic field fusion is carried out by the following steps, and the final dynamic mask of the fused feature map is output:

[0054] I, set the total loss function equation to fuse the transient field and dynamic field, the expression is as follows:

[0055] L total = L rgb +λ1L τ +λ2L static

[0056] In the formula, L total is the total loss function; L rgb is RGB photometric loss; L τ is the optimization objective function of transient field; L static is the constraint function of static field; λ1 and λ2 are weight parameters;

[0057] II, set the alternating optimization process and decision threshold, and output the final dynamic mask by the following expression:

[0058]

[0059] In the formula, is the final dynamic mask, and the dynamic region is determined by setting the threshold value 0.65; P dynamicis the probability of dynamic region, used to generate dynamic mask, the expression is:

[0060]

[0061] In the formula, σ is the Sigmoid function; α and β are weight parameters; is the divergence of optical flow residual; τ(r) is the transient radiation component.

[0062] Further, the implicit scene representation of the three-dimensional reconstruction space is constructed by the following steps:

[0063] I. The space point coordinates of the three-dimensional reconstruction space are spliced with the block feature vector through the position encoder to form the input feature;

[0064] II. Set the multi-layer perception, set the network structure to two full connection layers, and the hidden layer has 128 neurons; set the activation function to ReLU hidden layer / Sigmoid output layer, output the body density and RGB color value;

[0065] III. Sample 128 points along the light ray, and calculate the final pixel color by the following formula:

[0066]

[0067] In the formula, is the final pixel color value at position r; T_i is the transmittance; α_i is the opacity; c_i is the color information of the i th medium.

[0068] Further, the drivable area of the three-dimensional reconstruction space is segmented by the following formula:

[0069] M drivable ={p||k(p)|<0.08}∩M recon

[0070] In the formula, M drivable is the set of drivable areas; p is a point in space; k(p) is the Gaussian curvature at point p; |k(p)| is the absolute value of the Gaussian curvature at point p; M recon is the effective area of the reconstructed scene.

[0071] Further, the objective function of the energy functional is constructed by the following steps, and the skeleton path is optimized through the energy functional minimization framework:

[0072] I. Set the objective function expression of the energy functional as:

[0073]

[0074] In the formula, ξ is the skeleton path to be optimized; k(s) is the path curvature; is a semantic classification loss, c i is a predicted class, is a real class; is a number of non-zero segments of the skeleton path; η1, η2, η3 are weight coefficients;

[0075] II, the path curvature of each point on the skeleton path is calculated, the square gradient of the path curvature is solved, the path point coordinates are updated along the negative gradient direction, and ∫‖k(s)‖ 2 is minimized;

[0076] III, the lane type semantic features at the path points are calculated synchronously, the difference between the predicted class and the real class is measured through a cross-entropy loss function, and the semantic classification error is back propagated to the path position adjustment;

[0077] IV, a maximum branch number threshold is set, when the number of branches of the skeleton path exceeds the threshold, the average curvature change rate Δk of each branch segment is calculated, the branch with the smallest Δk is preferentially removed, and the L0 norm constraint is realized to realize the sparsification of the topological structure;

[0078] V, the node is defined as an end point or an intersection point of a lane line, and automatic detection is realized by finding a local extreme point |k|>k thresh of the path curvature; each node stores its two-dimensional coordinates and connection degree information; an edge represents a skeleton path segment connecting the nodes, and each edge is attached with lane type attributes, geometric attributes and connection relationships.

[0079] Based on the above technical solutions, the method disclosed by the application has the following beneficial effects:

[0080] 1. The test platform is selected as the NIO ET7, equipped with a full-vision sensor configuration, and the test scene is selected at the early peak period of the Shanghai Yan'an East Road tunnel. The method disclosed by the application is verified in terms of indicators, including end-to-end delay, dynamic target recall rate, reconstruction geometric error, safe mis-triggering rate and radar dependency. The data summary results are shown in the following table:

[0081] Indicator Measured value Target value End-to-end latency 11.7 ± 0.8 ms ≤ 20 ms Dynamic target recall rate 95.7％ ≥95％ Reconstruction geometry error 13.8 ± 3.1 cm ≤ 15 cm False positive safety rate 0.02 per km ≤ 0.1 per km Radar dependency 0％ --

[0082] The actual measurement results show that:

[0083] ① The system end-to-end delay is 11.7±0.8 milliseconds, which meets the requirement that the target value is less than or equal to 20 milliseconds;

[0084] ② The dynamic target recall rate reaches 95.7%, which exceeds the target value of ≥95%;

[0085] ③ The reconstruction geometric error is 13.8±3.1 centimeters, which is slightly higher than the target value of ≤15 centimeters;

[0086] 4. The false trigger rate is 0.02 times per 100 kilometers, which is much lower than the target value of 0.1 times per 100 kilometers;

[0087] 5. The radar dependence is 0%, indicating that the system does not rely on radar data during testing.

[0088] 2. The method described in the application can maintain a dynamic recall rate of about 93.2% under extreme working conditions, such as a rainstorm environment with visibility below 50 meters; the positioning drift is controlled within about 0.25 meters within 5 seconds; and the polarization glare suppression function ensures the continuous operation of topology generation under strong glare conditions with brightness exceeding 105 lux. BRIEF DESCRIPTION OF DRAWINGS

[0089] Figure 1 The figure is a schematic diagram of the overall process of the method of the application. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in combination with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without carrying out creative work are within the scope of protection of the present disclosure.

[0091] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood as the usual meaning understood by those skilled in the art in the field to which the present disclosure belongs.

[0092] The technology, methods and devices known to those skilled in the relevant art can not be discussed in detail, but in appropriate cases, the technology, methods and devices should be considered as part of the specification.

[0093] In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, and not as a limitation, so that other examples of exemplary embodiments can have different values.

[0094] The vehicle in the specification and claims refers to an autonomous vehicle; the dynamic environment refers to the comprehensive environment that the autonomous vehicle perceives and responds to in real time during driving, such as other vehicles, pedestrians, road conditions, etc.

[0095] In the present specification, the road curvature feature extraction method based on differential geometry is briefly described as follows:

[0096] Through the first order basic form "E = ||sigma u ||2 , F = <σ u , σ v >, G = ||σ v ||2 2 , where u, v are texture coordinates; E is the square length of the tangent vector σ u at a point on the surface in the u direction; F is the dot product of two tangent vectors σ u and σ v ; and G is the square length of the tangent vector σ v at a point on the surface in the v direction.

[0097] The degree of curvature of the surface is described by the second fundamental form "L = <σ uu , n>, M = <σ uv , n>, N = <σ vv , n>", where L is the dot product of the second partial derivative σ uu in the u direction and the surface normal n, representing the degree of curvature in the u direction; M is the dot product of the mixed second partial derivative σ uv and the surface normal n, representing the degree of twist of the surface; and N is the dot product of the second partial derivative σ vv in the v direction and the surface normal n, representing the degree of curvature in the v direction.

[0098] The core feature Gaussian curvature K of the road structure is calculated by the formula When K > 0, it represents an elliptic point, i.e. a convex curve; when K < 0, it represents a hyperbolic point, i.e. a transition area of a road intersection.

[0099] In combination with the flowchart shown in Figure 1 , the embodiment describes a dual-flow NeRF method for real-time reconstruction of dynamic environment, with the following steps.

[0100] S1, acquire dynamic environment data of the vehicle, including but not limited to nanosecond timestamp, four-way polarization image of the dynamic environment, near-infrared image of the dynamic environment, and IMU six-axis data of the vehicle in the dynamic environment, and perform multispectral fusion on the polarization image and the near-infrared image to output a feature map after fusion.

[0101] The specific steps are as follows:

[0102] S101, collect dynamic environment data of the autonomous vehicle, including but not limited to nanosecond timestamp, four-way polarization image of the dynamic environment, near-infrared image of the dynamic environment, and IMU six-axis data of the vehicle in the dynamic environment.

[0103] For the convenience of description, the "nanosecond timestamp", "four-way polarization image of dynamic environment", "near-infrared image of dynamic environment", and "IMU six-axis data of vehicle in dynamic environment" are sequentially referred to as "timestamp", "four-way polarization image", "near-infrared image", and "six-axis data" in this embodiment.

[0104] wherein,

[0105] ①The acquisition mode of the four-way polarization image is as follows: four polarization cameras with a resolution of 1280x1024, a frame rate of 30Hz, and an optical axis angle of 15°±0.5° are uniformly arranged transversely about 20cm above the center line of the front bumper of the vehicle to construct a multi-camera system, and the polarization angles of the four polarization cameras of the multi-camera system are set to 0°, 45°, 90°, and 135° respectively, and the polarization camera serving as the reference in the multi-camera system is defined as the main polarization camera.

[0106] ②The near-infrared camera and the main polarization camera are coaxially fixedly installed on the same rigid support to ensure that the optical centers of the two cameras deviate by less than 0.1 pixel, and the light paths are overlapped through mechanical calibration (such as laser alignment) or optical lens (such as a beam splitter prism).

[0107] ③The six-axis IMU inertial measurement instrument is fixedly arranged at the center of mass of the vehicle, and the model ADIS16470 is adopted to obtain the IMU six-axis data of the vehicle in the dynamic environment, including but not limited to vehicle rotation data and linear acceleration data.

[0108] ④The FPGA is triggered by the PPS signal to generate a synchronization pulse, so that the exposure time deviation of the polarization camera and the near-infrared camera is less than 1μs, and the timestamp alignment error of the IMU six-axis data and the four-way polarization image and the near-infrared image is less than 200μs.

[0109] S102, a quality evaluation model is established for evaluating the overall quality of the four-way polarization image and the near-infrared image.

[0110] The expression of the quality evaluation model is as follows:

[0111] Q=0.5·R valid +0.3·e -σDoP +0.2·SNR NIR

[0112] In the formula, Q is the quality evaluation score, Q>0.65 indicates that the quality of the image is better; R valid is the effective pixel ratio, which indicates the proportion of effective pixels in the total pixels in the image; -σDop is the standard deviation of the degree of polarization, DoP is the degree of polarization of the polarization image; e -σDop is the exponential decay term of the standard deviation of the degree of polarization, which is used to measure the stability of the degree of polarization; and SNR NIRThe signal-to-noise ratio in the near-infrared band represents the ratio of signal strength to noise strength.

[0113] S103, preferably, the four-way polarization image with a quality evaluation score > 0.65 is multispectral fused with the near-infrared image, and a fused feature map is output, and noise suppression is performed on the fused feature map through anisotropic diffusion filtering.

[0114] The expression for multispectral fusion of the four-way polarization image and the near-infrared image is as follows:

[0115]

[0116] In the formula, F fused is the fused feature map; DoP is the degree of polarization of the four-way polarization image, and the calculation formula is The value range is [0, 1]; AoP is the polarization angle of the four-way polarization image, and the calculation formula is The value range is S0 is the total intensity of incident light, S1 is the intensity difference between 0° and 90° linearly polarized light, S2 is the intensity difference between 45° and 135° linearly polarized light; min max is a normalization function; is the near-infrared image, is a gradient operator.

[0117] S104, in order to better solve the image calibration problem of vehicles in rain and fog scattering and strong light polarization scenes, this step performs glare suppression and rain and fog correction on the fused feature map, so that the fused feature map can realize adaptive calibration in a dynamic environment. The specific method is as follows:

[0118] S104-1, in the saturated region of the fused feature map, i.e. I>0.95·I saturation Based on the polarization physical model, the true intensity of the fused feature map is reconstructed through the following formula, so as to perform glare suppression on the fused feature map:

[0119]

[0120] In the formula, I recovered is the reconstructed true intensity of the fused feature map; I obs is the observed intensity of the fused feature map; φ opt is the optimized polarization angle of the fused feature map minimizing the saturated region intensity; DoP is the degree of polarization of the polarization image; AoP is the polarization angle of the polarization image.

[0121] S104-2, based on the atmospheric scattering model, the target light intensity is inverted through the following formula, and the fused feature map is corrected for rain and fog:

[0122]

[0123] In the formula, I object is the target light intensity, that is, the original light intensity of the four-way polarization image without the influence of atmospheric scattering; I obs is the observed post-fusion feature map intensity; I atm is the light intensity generated by atmospheric scattering; e -βd is the degree of attenuation of light in the atmosphere; β is the scattering coefficient, which describes the degree of light scattering in the atmosphere; d is the distance of light propagation in the atmosphere.

[0124] In the formula, β is the scattering coefficient, which is estimated by the difference between the orthogonal polarization components, and the expression is as follows:

[0125]

[0126] In the formula, β is the scattering coefficient; d is the distance of light propagation in the atmosphere; I perp is the light intensity when the polarization direction of light is perpendicular to the incident plane; I parallel is the light intensity when the polarization direction of light is parallel to the incident plane.

[0127] S2, construct the rigid body motion field of the vehicle, and perform theoretical optical flow calculation on the post-fusion feature map to generate a dynamic mask of a binary image for identifying dynamic regions in the post-fusion feature map;

[0128] Based on the optimization objective function for constructing the transient field and the constraint function for the static field, the transient field and the dynamic field are fused, and the dynamic confidence of the post-fusion feature map is output, thereby solving the distortion problem of scene reconstruction caused by the degradation of the radiation field due to dynamic objects in a dynamic environment.

[0129] The specific steps are as follows:

[0130] S201, describe the rigid body motion field of the vehicle by constructing a SE(3) pose transformation matrix, and the expression is as follows:

[0131]

[0132] In the formula,

[0133] T t→t+1 is the pose transformation matrix of the vehicle from time “t” to “t+1”;

[0134] ΔR is the rotation increment, which represents the rotation change of the vehicle from time “t” to “t+1”, and is obtained based on the IMU six-axis data of the vehicle in step S1, specifically based on the integral of the IMU angular velocity, and the expression is ΔR = exp([ωΔt]×), wherein exp([ωΔt]×) is the exponential mapping based on the angular velocity ω and the time increment Δt;

[0135] As is the translational increment, representing the translational change of the vehicle from "t" to "t+1", obtained based on double integration of the acceleration of the vehicle, expressed as where a is the acceleration and At is the time increment.

[0136] S202, theoretical optical flow calculation is performed on the fused feature map obtained in step S1, expressed as:

[0137] V rigid = P' - P

[0138] In the formula, V rigid is the theoretical optical flow of the fused feature map; P is the original pixel point coordinate of the four-way polarization image and / or near-infrared image; P' is the pixel point coordinate of the four-way polarization image and / or near-infrared image after re-projection.

[0139] wherein the pixel point coordinate P' after re-projection is obtained by the following steps:

[0140] S202-1, the original pixel point coordinate P = (u, v) of the four-way polarization image and / or near-infrared image is back-projected to the three-dimensional space by the formula

[0141] In the formula, P cam is the three-dimensional point under the camera coordinate; K is the camera intrinsic matrix; d is the depth estimation value, representing the distance from the pixel point to the camera.

[0142] S202-2, the three-dimensional point P cam is applied to the pose transformation by the formula P' t→t+1 = T cam · P cam

[0143] In the formula, P' cam is the three-dimensional point after applying the pose transformation; T t→t+1 is the pose transformation matrix from time t to t+1.

[0144] S202-3, the original pixel point coordinate P of the four-way polarization image and / or near-infrared image is re-projected to the image plane by the formula P' = π(KP' cam ),

[0145] In the formula, π is the projection function, projecting the three-dimensional point to the image plane, and K is the camera intrinsic matrix.

[0146] S203, a dynamic mask is generated by the following expression, which is used to identify the binary image of the dynamic region in the fused feature map:

[0147]

[0148] In the formula, M​​dynamic is a dynamic mask; ∏ is an indicator function, which takes value 1 when the condition is satisfied, otherwise 0; is a divergence threshold condition, used to determine the dynamic region;

[0149] is the divergence of the optical flow residual, used to analyze the local variation of the optical flow field, expressed as δv x and δv y are the components of the optical flow residual in x and y directions, respectively;

[0150] δV is the difference between the observed optical flow and the theoretical optical flow, expressed as δV = V obs -V rigid , V obs is the actual optical flow calculated from the image sequence, V rigid is the theoretical optical flow of the fused feature map.

[0151] S204, an optimization objective function of the transient field is constructed, which is used to separate the contribution of the dynamic object to the radiation field, expressed as follows:

[0152] L τ =∑ r (KL(τ(r)||M dynamic (r))+λ tv TV(τ(r)))

[0153] In the formula, L τ is the optimization objective function of the transient field; KL(τ(r)||M dynamic (r)) is the KL divergence, which measures the consistency between the transient component and the dynamic mask; λ tv is the weight parameter of the TV regularization, which is used to balance the influence of the regularization term; TV(τ(r)) is the total variation, which is used to suppress noise and maintain the image edges of the four-way polarization image and / or near-infrared image.

[0154] S205, a constraint function of the static field is constructed, which is used to force the transient component of the static region to be zero, expressed as follows:

[0155] L static =||τ(r)·M static ||1

[0156] In the formula, L static is the constraint function of the static field; ||τ(r)·M static ||1 is the L1 norm, which is used to force the transient component of the static region to be zero, i.e., the transient effect is not calculated in the static region.

[0157] Wherein, the transient component is introduced by the following expression:

[0158]

[0159] wherein, is the estimated color value at position r, is the result of the extended standard body rendering equation; T(t) is a time function representing the change of illumination over time; σ(r(t)) is the absorption coefficient of the point r(t) in the scene at time t; c static (r(t)) is the static component representing the color of static objects in the scene; τ(r(t)) is the transient radiation component representing the contribution of dynamic objects to the radiation field.

[0160] S206, based on steps S203-S205, the transient field and dynamic field fusion is carried out by the following steps, and the dynamic confidence of the fused feature map is output, the specific steps are as follows:

[0161] S206-1, set the total loss function equation for transient field and dynamic field fusion, the expression is as follows:

[0162] L total = L rgb + λ1L τ + λ2L static

[0163] wherein, L total is the total loss function, used for joint optimization; L rgb is the RGB photometric loss, which measures the difference between the predicted color and the true color; L τ is the optimization objective function of the transient field; L static is the constraint function of the static field; λ1 and λ2 are weight parameters for balancing the influence of different loss terms, λ1=0.5 and λ2=0.3.

[0164] S206-2, set the alternating optimization process: phase one, fix the transient field, optimize the static radiation field parameters; phase two, fix the static field, optimize the transient field parameters; each round of iteration executes 3 times of alternating optimization.

[0165] S206-3, set the decision threshold to output the final dynamic mask, the expression is as follows:

[0166]

[0167] wherein, is the final dynamic mask, the dynamic region is determined by setting the threshold value 0.65; P dynamic is the probability of the dynamic region, used for generating the dynamic mask, the expression is:

[0168]

[0169] where σ is a Sigmoid function that maps input values to the interval (0, 1) for calculating probability; and α and β are weight parameters for adjusting the influence of divergence and radiance component on dynamic probability. is the divergence of optical flow residual, used to measure the consistency of dynamic region; and τ(r) is the transient radiance component, representing the contribution of dynamic object to the radiance field.

[0170] Based on steps S201 to S206, as preferred:

[0171] ① This embodiment adopts the HS optical flow algorithm and is implemented on FPGA to improve the real-time performance of optical flow calculation. The specific implementation parameter setting is resolution 1280x1024 and frame rate 30fps. Through this setting, the delay of this embodiment on the Xilinx Zynq UltraScale+ platform is only 2.1 milliseconds, significantly improving the processing speed and meeting the real-time requirements in the vehicle-mounted environment.

[0172] ② This embodiment adopts the following strategies to further optimize the inference process of the radiance field: dual-network shared encoder for transient / static field, which reduces the consumption of computing resources while maintaining the accuracy of inference; operator fusion, which combines volume rendering and KL loss for calculation to reduce the delay caused by separate calculation; on the NVIDIA Orin-X platform, this optimization strategy reduces the inference delay to 3.8 milliseconds, further improving the real-time processing capability of the vehicle-mounted system.

[0173] ③ This embodiment adopts a dynamic allocation strategy for memory management of the optical flow field and the transient field, which specifically includes: for the optical flow field, 120MB of memory is fixedly allocated to ensure the stability and continuity of optical flow calculation; for the transient field, a sparse storage method is used, with a peak memory usage of 45MB, effectively utilizing memory resources and reducing unnecessary memory usage.

[0174] ④ This embodiment further improves the efficiency of memory usage through a data multiplexing mechanism, which specifically includes: pose transformation matrix multiplexing period, set to 5 frames, i.e., multiplexing the pose transformation matrix in consecutive 5 frames to reduce repeated calculations; theoretical optical flow field cache update threshold, when the rotation change exceeds 0.5° or the translation exceeds 0.2 meters, update the theoretical optical flow field cache to ensure the accuracy and real-time performance of the data.

[0175] S3, based on the fused feature map of S1 and the dynamic mask of S2, a three-dimensional reconstruction space centered on the vehicle is constructed to generate a hash code with spatial position information;

[0176] A fusion update decision is set to mark the blocks in the three-dimensional reconstruction space that need to be updated, and load the block feature vectors for the blocks that need to be updated for block incremental optimization;

[0177] An implicit scene representation of the three-dimensional reconstruction space is constructed, and an artifact suppression technique is used in dynamic-static coupling processing to ensure effective coupling processing between dynamic and static regions, outputting a semantic representation without a graph, and achieving efficient scene reconstruction of the three-dimensional reconstruction space.

[0178] Specifically, the following steps are included:

[0179] S301, a three-dimensional reconstruction space centered on the vehicle is constructed, which is discretized into a plurality of blocks, including active blocks and inactive blocks;

[0180] The three-dimensional integer coordinates of each block are subjected to XOR operation and prime number mapping, and are mapped to hash slot bits through modulo operation to generate hash codes with spatial position information.

[0181] For example, the range of the three-dimensional reconstruction space is set to be centered on the vehicle, with a forward range of 50 meters, a rearward range of 10 meters, and a lateral range of ±15 meters; the three-dimensional reconstruction space is divided into standard cubic blocks with a size of 1 cubic meter, totaling 48,750 blocks; the three-dimensional integer coordinates of each block are subjected to XOR operation and prime number mapping, and are then mapped to 524,288 hash slot bits through modulo operation, i.e., modulo 2 19 .

[0182] As preferred:

[0183] ①Each block is associated with an 8-dimensional feature vector, and these feature vectors are stored in an open addressing hash table with a load factor controlled below 0.7 to ensure the efficiency and performance of the hash table.

[0184] ②During backpropagation, only the gradients of the active blocks are calculated, and the gradients of the inactive blocks are set to zero, thereby realizing the differentiable property.

[0185] ③Supporting optimization of feature values through gradient descent, the learning rate is set to 10 -3 , so as to update the feature vectors during the training process and establish a feature vector update interface.

[0186] S302, setting a fusion update decision marker to mark the blocks in the three-dimensional reconstruction space that need to be updated, and loading the block feature vectors of the blocks that need to be updated for block incremental optimization.

[0187] As preferred:

[0188] ①The method of fusion update decision is to weight-sum the geometric changes and appearance changes of the blocks, and set an update threshold, if the weighted sum > threshold, then mark the block for update. For example, the geometric change weight of the block is set to 0.6, the appearance change weight is set to 0.4, and the update threshold is set to 0.2.

[0189] wherein,

[0190] The geometric change metric of the block is calculated by calculating the difference in volume density between the current frame and the previous frame in the static area, i.e., the L1 norm;

[0191] The appearance change metric of the block is calculated by the formula The L2 norm difference of the RGB color gradient of the block is calculated, where ΔC is the appearance change metric, indicating the degree of color change between the two frames of images; is the RGB color gradient, indicating the rate of change of image color in space; c t is the color value of the current frame; c t-1 is the color value of the previous frame; ||·||2 is the L2 norm, also known as the Euclidean norm, used to calculate the length or size of a vector.

[0192] ②The incremental optimization of the blocks that need to be updated can be performed by the following method: traverse all blocks, detect the block update flag; load the block feature vector for the blocks that need to be updated, uniformly sample 256 spatial points in the block, optimize the feature vector based on the photometric loss function, and write the updated feature vector back to the hash table; the optimization strategy can use the stochastic gradient descent method with momentum, and the momentum coefficient is set to 0.9.

[0193] S303, constructing an implicit scene representation of the three-dimensional reconstruction space, specifically including the following steps:

[0194] S303-1, concatenating the spatial point coordinates of the three-dimensional reconstruction space through the position encoder with the block feature vector to form the input feature;

[0195] S303-2, setting the MLP, i.e., the multi-layer perceptron, to two fully connected layers with 128 neurons in the hidden layer; setting the activation function to ReLU hidden layer / Sigmoid output layer; based on the input feature of step S303-1, outputting the volume density and RGB color value;

[0196] S303-3, sampling 128 points along the light ray for volume rendering or light transmission calculation, estimating the final observed color by considering the absorption and color contribution of different media to light, and calculating the final pixel color by the following formula:

[0197]

[0198] wherein, is the final pixel color value at position r; T_i is the transmittance; α_i is the opacity; c_i is the color information of the i-th medium.

[0199] S304, to further improve the efficiency and accuracy of scene reconstruction in the vehicle-mounted system, a step in the dynamic-static coupling process adopts artifact suppression technology, including static region constraint, dynamic region constraint, and space-time consistency guarantee, to ensure effective coupling processing between dynamic and static regions, and output no-graph semantic representation.

[0200] The specific steps are as follows:

[0201] S304-1, an L1 regularization term of the transient component is added in the loss function to force the transient component of the static region to be zero, ensuring that the contribution of the transient component in the static region is suppressed, thereby reducing the generation of artifacts, achieving static region constraint, and the expression is as follows:

[0202] L static =∑||τ(r)·σ(r)||1

[0203] In the formula, L static is the constraint function of the static field; τ(r) is the transient radiation component; σ is the indicator function of the static region.

[0204] S304-2, dynamic consistency loss is applied to achieve dynamic region constraint, ensuring that dynamic objects do not contribute to static geometry, ensuring consistency of dynamic objects in static geometry, and further reducing artifacts. The expression of dynamic consistency loss is as follows:

[0205] L dynamic =||M dynamic ·(1-e -σ )|| 2

[0206] In the formula, L dynamic is the loss function of the dynamic region; M dynamic is the dynamic mask; σ is the indicator function of the static region.

[0207] S304-3, space-time consistency is guaranteed by block state machine, specifically: the block state machine includes new block, stable block and failed block; in the new block state, the initial optimization iteration number is doubled to quickly converge; in the stable block state, the learning rate is reduced to refine the optimization result; in the failed block state, the feature vector is reset to restore the effectiveness of the block. The state transition is based on the continuous change detection of the block, ensuring the adaptability and robustness of the system.

[0208] In step S303, as preferred:

[0209] ①In order to realize gradient approximation acceleration, the embodiment adopts Straight-Through Estimator technology to solve the problem of non-differentiable hash function in neural network training. This technology allows the network to perform standard hash lookup during forward propagation, and during back propagation, the output gradient is directly copied to the input, thereby bypassing the non-differentiability of the hash function and realizing end-to-end trainability.

[0210] ②In order to improve the deployment efficiency and performance of the vehicle-mounted system, the embodiment designs the memory hierarchy through a three-level cache structure. Specifically, L0 cache is used to store hotspot block features, has a capacity of 1024 blocks, and uses LRU replacement strategy to optimize cache hit rate; L1 storage is used to store active block features, and uses GDDR6 video memory to provide 8GB of fast data access capacity; L2 storage is used to store the full hash table, and uses LPDDR5 memory to provide 16GB of larger storage space to store the complete data set.

[0211] ③In order to improve the deployment efficiency and performance of the vehicle-mounted system, the embodiment also accelerates the calculation through kernel fusion technology. Specifically, the hash lookup and MLP inference are combined into a single CUDA kernel to reduce global memory access times and improve calculation efficiency; the TVM compiler is used to generate custom operators, and is optimized for NVIDIA CUDA sm_87 architecture to fully utilize hardware features; a configuration of up to 1024 threads / block is used to optimize thread parallelism and improve parallel computing efficiency; FP16 arithmetic operations are enabled to reduce computational resource consumption while maintaining accuracy, and tensor cores are used to accelerate matrix multiplication to improve the inference speed of deep learning models.

[0212] S4, by generating the graph-free semantic representation, the three-dimensional reconstruction space established in step S3 is converted into semantic understanding, based on the differential geometry theory and lightweight engineering design, realizing the real-time compliance output of autonomous driving environment cognition. The specific steps are as follows:

[0213] S401, define semantic mapping rules based on curvature features of differential geometry theory, and divide the drivable domain of the three-dimensional reconstruction space through the following formula:

[0214] M drivable ={p||k(p)|<0.08}∩M recon

[0215] In the formula, M drivable is the set of drivable areas, i.e. the part of the road where vehicles can safely drive; p is a point in space, which can be any position on the road; k(p) is the Gaussian curvature at point p, which is a measure of the curvature of a surface at a certain point; |k(p)| is the absolute value of the Gaussian curvature at point p; Mrecon To reconstruct the effective area of the scene; the structured element recognition includes lane lines, curves, and kerbs.

[0216] S402, input the binarized lane pixel result of the bird's eye view semantic segmentation to the drivable area, and generate a skeleton path of the three-dimensional reconstruction space by using a morphological thinning algorithm.

[0217] The morphological thinning algorithm retains the center pixels of the lane area through iterative erosion operations, ensures that the generated skeleton path maintains the topological connectivity of the original lane, and at the same time reduces the lane width to a single pixel level, thereby providing a good initial condition for subsequent optimization. In a specific implementation, an 8-neighbor connection mode is used to process complex topological structures such as intersections, thereby avoiding skeleton breaking.

[0218] S403, construct a target function of an energy functional, and optimize the skeleton path by using an energy functional minimization framework, so as to realize accurate extraction and classification of the lane center line, and further realize accurate recognition and classification of the lane.

[0219] The steps are as follows:

[0220] S403-1, the target function expression of the energy functional is:

[0221]

[0222] In the formula, ξ is the skeleton path to be optimized; k(s) is the path curvature, and the geometric smoothing term constrains the minimization of the lane curvature variation; is a semantic classification loss, that is, a cross-entropy function; c i is a predicted category, is a real category; is the number of non-zero segments of the path, which forces sparse topological connection; η1, η2, and η3 are weight coefficients, and the default values are 0.6, 0.3, and 0.1, respectively.

[0223] S403-2, geometric smoothing optimization: calculate the curvature k(s) of each point on the skeleton path, and solve the square gradient of the curvature by using a numerical differentiation method such as the central difference method Update the path point coordinates in the negative gradient direction to minimize the term 2 ds, thereby ensuring the smoothness of the lane curvature variation.

[0224] S403-3, semantic constraint optimization: in the process of updating the path point coordinates, the lane type semantic features at the point are calculated synchronously, and the difference between the predicted category c and the real category i is measured by using a cross-entropy loss function , and the semantic classification error is back propagated to the path position adjustment.

[0225] S403-4, topology pruning: set a maximum branch number threshold N_max, typical value N_max = 5, when the path branch number exceeds the threshold, calculate the average curvature change rate Δk of each branch segment, preferentially remove the branch with the smallest Δk, and constrain by L0 norm Realize the sparsification of the topology structure. The entire optimization process adopts the gradient descent method with adaptive step size until the change ΔE of the energy functional E(P) is less than the convergence threshold ε, and generally set ε = 10 -4 .

[0226] S403-5, convert the final path after optimization into a structured topology graph representation, the method is: define the node as the end point or intersection point of the lane line, find the local extreme point |k| > k thresh Automatic detection, each node stores its two-dimensional coordinates and connection degree information; the edge represents the skeleton path segment connecting the nodes, each edge is attached with three types of attributes, such as lane type attribute, which is marked as solid line / dotted line / double yellow line based on semantic classification result; geometric attribute, calculate the average curvature radius of the path segment Connection relationship, record the unique identifier and connection direction of the adjacent nodes.

[0227] The final output topology graph adopts the adjacency list structure for storage, which supports the path query operation with complex degree. In specific implementation, for complex intersection scenarios, virtual nodes are introduced to handle multi-level connection relationship, ensuring the completeness of the topology expression.

[0228] As preferred,

[0229] ① Through the differential chain code encoding process, it is used for lightweight compression of the skeleton path, the method is:

[0230] Through path discretization processing, taking continuous Bezier curve B(t), t ∈ [0, 1] as input, N represents the number of curve segments, calculate the sampling points And get the discrete point set {p k =B(t k )};

[0231] Carry out directional differential quantization, calculate the differential vector Δp k =p k -p k-1 , and get the angle θ k =arctan2(Δy k , Δx k ); Quantize the angle To reduce the storage space.

[0232] ② In the entropy encoding optimization stage, a probability model to estimate the frequency of angle occurrence and calculate the Range Coding compression rate In the formula, H represents information entropy. This method realizes efficient compression of path data through probability models and entropy coding techniques.

[0233] ③The output interface design includes a data frame structure, which contains a 4-byte Header, i.e., a timestamp + version number; a variable-length RopoData, i.e., compressed chain code data; a variable-length DrivableArea, i.e., boundary point difference encoding; and a 4-byte CRC32 check code. In terms of bandwidth control, a single frame of data does not exceed 1 KB, supporting 10 Hz updates with a bandwidth of no more than 80 kbps, ensuring the real-time and reliability of data transmission.

[0234] ④The spatiotemporal consistency of dynamic and static areas in the vehicle-mounted system is ensured through an incremental update strategy, improving the stability and reliability of the vehicle-mounted system. The method is as follows:

[0235] A change detection index ΔT is defined to quantify the degree of path change, and its calculation formula is as follows:

[0236]

[0237] In the formula, p' k represents a point on the current path; n is the total number of sampling points; k is an index variable; B -1 (t k ) represents the inverse mapping of the last frame of fitting curve function, which is used to calculate the deviation of the path point from the last frame of path; when ΔT < 0.3, the update of the local optimization control point P i is triggered; when ΔT ≥ 0.3, the topological path is regenerated.

[0238] State equations and observation equations are defined to realize Kalman filter fusion, where:

[0239] The state equation describes the evolution of the system state, and its expression is as follows:

[0240] x k = F k x k-1 + w k , F k = I 16

[0241] In the formula, is a 16-dimensional vector representing the positions of the four control points on the path; F k = I 16 is the identity matrix, representing the uniform speed model.

[0242] The observation equation describes the relationship between the observation value and the system state, and its expression is as follows:

[0243] z k = Hx k + v k , H = I 16

[0244] where z k is the observation; v k is the observation noise, representing the sensor measurement error; H = I 16 is the observation matrix.

[0245] The noise covariance matrices are defined as Q = diag(0.1 2 ), R = diag(0.05 2 ), representing the covariance of the process noise and observation noise, respectively.

[0246] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application.

Claims

1. A two-stream NeRF method for real-time reconstruction in dynamic environments, characterized in that, Includes the following steps: S1, acquire dynamic environmental data of the vehicle, including four-axis polarization image of the dynamic environment, near-infrared image of the dynamic environment, and six-axis IMU data of the vehicle under dynamic environment. Perform multispectral fusion of the four-axis polarization image and near-infrared image and output the fused feature map. S2, construct the rigid motion field of the vehicle, and generate a dynamic mask based on the theoretical optical flow of the fused feature map; The transient and dynamic fields are fused based on the optimization objective function and constraint function, and the final dynamic mask of the fused feature map is output. S3, based on fused feature maps and dynamic masks, constructs a vehicle-centric 3D reconstruction space and discretizes it into multiple blocks; The geometric and appearance changes of the blocks are weighted and summed. An update threshold is set. Blocks whose weighted sum is greater than the update threshold are marked as blocks that need to be updated. The block feature vector is then loaded onto the blocks that need to be updated. Construct an implicit scene representation of the 3D reconstruction space, and output a graph-free semantic representation based on artifact suppression to couple the dynamic and static regions of the 3D reconstruction space. S4 transforms the 3D reconstructed space into semantic understanding through the generated graph-free semantic representation, and segments the drivable domain of the 3D reconstructed space. Input the binarized lane pixel results of the bird's-eye view semantic segmentation into the drivable domain, and generate the skeleton path of the three-dimensional reconstruction space based on the morphological thinning algorithm; We construct the objective function of the energy functional and optimize the skeleton path through the energy functional minimization framework to identify and classify the lane centerline.

2. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, The quality of four-axis polarized images and near-infrared images is assessed using the following expression: Q=0.5·R valid +0.3 e -σDoP +0.2 SNR NIR In the formula, Q is the quality assessment score; R valid -σDop is the effective pixel ratio; -σDop is the standard deviation of the polarization degree; DoP is the polarization degree of the polarized image; e -σDop The exponential decay term of the standard deviation of polarization; SNR NIR The signal-to-noise ratio in the near-infrared band; Four-dimensional polarization images and near-infrared images with a quality assessment score >0.65 are fused using the following formula: In the formula, F fused The fused feature map is shown; DoP represents the degree of polarization of the four-way polarization image; AoP represents the polarization angle of the four-way polarization image; min and max are normalization functions. Near-infrared image; This is the gradient operator.

3. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, Glare suppression of the fused feature map is achieved using the following formula: In the formula, I recovered The true intensity of the reconstructed and fused feature map; I obs The observed intensity of the fused feature map; φ opt The optimized polarization angle minimizes the intensity of the saturated region in the fused feature map; DoP is the degree of polarization of the polarization image; AoP is the polarization angle of the polarization image. Rain and fog correction is performed on the fused feature map using the following formula: In the formula, I object For target light intensity; I obs I represents the observed intensity of the fused feature map. atm The intensity of light produced by atmospheric scattering; e -βd β is the attenuation of light as it travels through the atmosphere; d is the scattering coefficient; and d is the distance light travels through the atmosphere.

4. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, The rigid body motion field of the vehicle is described by constructing the SE(3) pose transformation matrix using the following expression: In the formula, T t→t+1 Let be the vehicle's pose transformation matrix from time "t" to "t+1"; ΔR is the rotation increment; Δs is the translation increment; The theoretical optical flow of the fused feature map is calculated using the following expression; In the formula, V rigid π is the theoretical optical flow of the fused feature map; K is the projection function; d is the camera intrinsic parameter matrix; T is the depth estimate. t→t+1 P is the pose transformation matrix of the vehicle from time "t" to "t+1"; P is the original pixel coordinates of the four-way polarization image and / or near-infrared image, i.e., P = (u, v).

5. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, A dynamic mask is generated using the following expression to identify binary images of dynamic regions in the fused feature map: In the formula, M dynamic is a dynamic mask; ∏ is an indicator function that takes the value 1 when the condition is met, and 0 otherwise. This is the divergence threshold condition; δV represents the divergence of the optical flow residual; δV represents the difference between the observed optical flow and the theoretical optical flow.

6. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, The objective function for optimizing the transient field is constructed using the following expression: L τ =∑ r (KL(τ(r)||M dynamic (r))+λ tv TV(τ(r))) In the formula, L τ Let KL(τ(r)||M be the objective function for optimizing the transient field. dynamic (r) is the KL divergence; λ tv τ is the weight parameter for TV regularization; TV(τ(r)) is the total variation. The constraint function of the static field is constructed using the following expression: L static =||τ(r)·M static ||1 In the formula, L static Let ||τ(r)·M be the constraint function for the static field; static ||1 represents the L1 norm, used to force the transient components in the static region to be zero. The transient components are introduced by the following expression: In the formula, Let be the estimated color value at position r; T(t) is the time function; σ(r(t)) is the absorption coefficient of point r(t) in the scene at time t; c static (r(t)) is the static component; τ(r(t)) is the transient radiation component.

7. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 6, characterized in that, The transient and dynamic fields are fused using the following steps to output the final dynamic mask of the fused feature map: I. Set up the total loss function equation to fuse the transient and dynamic fields, as shown in the following expression: L total =L rgb +λ1Lτ+λ2L static In the formula, L total L is the total loss function; rgb For RGB luminance loss; L τ Let L be the objective function for optimizing the transient field; static λ1 and λ2 are the constraint functions for the static field; λ1 and λ2 are the weight parameters. II. Set up an alternating optimization process and decision threshold, and output the final dynamic mask using the following expression: In the formula, For the final dynamic mask, the dynamic region is determined by setting a threshold of 0.65; P dynamic The probability of the dynamic region is used to generate the dynamic mask, and its expression is: In the formula, σ is the Sigmoid function; α and β are weighting parameters; τ is the divergence of the optical flow residual; τ(r) is the transient radiation component.

8. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, The implicit scene representation of the 3D reconstructed space is constructed through the following steps: Ⅰ. The spatial coordinates of the three-dimensional reconstructed space are concatenated with the block feature vector through a position encoder to form the input features; II. Set up a multilayer perceptron with a network structure of two fully connected layers and 128 neurons in the hidden layer; set the activation function to ReLU hidden layer / Sigmoid output layer, and output volume density and RGB color value; III. Sample 128 points along the ray and calculate the final pixel color using the following formula: In the formula, This represents the final pixel color value at position r; T_i represents transmittance; α_i represents opacity; and c_i represents the color information of the i-th medium.

9. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, The feasible region of the 3D reconstruction space is segmented using the following formula: M drivable ={p||k(p)|<0.08}∩M recon In the formula, M drivable Let M be the set of drivable regions; p be a point in space; k(p) be the Gaussian curvature at point p; |k(p)| be the absolute value of the Gaussian curvature at point p; M recon The effective area for reconstructing the scene.

10. The dual-stream NeRF method for real-time reconstruction of dynamic environments according to claim 1, characterized in that, The objective function of the energy functional is constructed through the following steps, and the skeleton path is optimized using the energy functional minimization framework: I. Set the objective function expression of the energy functional as follows: In the formula, ξ is the skeleton path to be optimized; k(s) is the path curvature; For semantic classification loss, c i For predicting categories, For the true category; ||p|| L0 This represents the number of non-zero segments in the skeleton path. η1, η2, and η3 are weighting coefficients; II. Calculate the path curvature at each point on the skeleton path, solve for the squared gradient of the path curvature, and update the coordinates of the path points along the negative gradient direction, such that ∫‖k(s)‖ 2 Minimize the ds term; III. Simultaneously calculate the lane type semantic features at the path points, measure the difference between the predicted category and the true category using the cross-entropy loss function, and backpropagate the semantic classification error to the path position adjustment; IV. Set a maximum branch number threshold. When the number of branches in the skeleton path exceeds the threshold, calculate the average rate of curvature change Δk of each branch segment, and prioritize removing the branch with the smallest Δk, using L0 norm constraints. To achieve sparsity in the topology; V. Define nodes as endpoints or intersections of lane lines, and find local extrema of the path curvature |k|>k. thresh Automatic detection is performed, and each node stores its two-dimensional coordinates and connectivity information; edges represent the skeleton path segments connecting nodes, and each edge is accompanied by lane type attributes, geometric attributes, and connection relationships.

Images