An Echo-State Multiplane Feature Reconstruction Method for Complex Scene Perception

Abstract

An echo-state multi-plane feature reconstruction method for complex scene perception comprises the following steps: First, a video image sequence is input. After preprocessing, a four-dimensional surgical scene presented by six orthogonal two-dimensional planes is modeled, resulting in static and dynamic feature planes. These static and dynamic feature planes are then fused to output the original fused features. Next, the static and dynamic feature planes are input to a depth uncertainty filter. The differences between the static and dynamic feature planes are modeled and filtered by a threshold, outputting a dynamic filter. This dynamic filter is used for dynamic feature enhancement, and dynamic weights are generated and fused. Iterative correction of color values ​​in dynamic regions is achieved by cyclically updating color residual information. Forward inference and modeling of the dynamic scene are performed, outputting continuous rendered image frames and dynamic 3D reconstruction results. This invention can alleviate problems such as depth distortion and discontinuous dynamic blurring in occluded areas, and can significantly improve modeling accuracy.

Description

Technical Field

[0001] This invention relates to a multi-plane feature reconstruction method, specifically an echo-state multi-plane feature reconstruction method for complex scene perception, belonging to the field of computer vision and image rendering processing technology. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, medical imaging, augmented reality, and intelligent robotics, the demand for 3D visual modeling and dynamic reconstruction in fields such as medical diagnosis, surgical navigation, digital twins, and complex environment perception continues to increase. Especially in medical scenarios and complex environments, systems not only need to perform detailed modeling of static anatomical structures or backgrounds, but also must be able to capture and represent dynamic changes such as tissue deformation and instrument movement. Constructing dynamic reconstruction methods with spatiotemporal consistency and high robustness has become an important research direction in the fields of computer vision and medical image processing.

[0003] In recent years, multi-planar 3D reconstruction methods based on neural rendering have made some progress, achieving temporal representation of some scenes through joint modeling of static and dynamic features. However, current 3D dynamic scene modeling methods still face significant challenges in medical imaging and complex scene perception tasks. On the one hand, most existing methods rely solely on static modeling mechanisms, failing to effectively adapt to tissue deformation, instrument occlusion, and multi-object dynamic interactions, leading to depth distortion and temporal inconsistency in occluded areas. On the other hand, some neural rendering methods often treat dynamic and static features equally during dynamic modeling, lacking differentiated representation and adaptive fusion capabilities, resulting in dynamic blurring and texture distortion in soft tissue regions. Furthermore, existing methods lack effective mechanisms to suppress uncertainty when processing temporal information, making it difficult to maintain model stability and robustness under weak supervision or low sensor dependence conditions, severely limiting their practical deployment in medical, robotic perception, and virtual interaction scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an echo-state multiplane feature reconstruction method for complex scene perception. This method does not rely on external global geometric priors and can complete high-fidelity modeling of dynamic 3D scenes based solely on the input image sequence. It can not only alleviate the problems of depth distortion and discontinuous dynamic blurring in occluded areas in existing methods, but also has good generalization and low-cost adaptability, which can significantly improve the applicability and modeling accuracy in medical image reconstruction, complex scene perception and virtual simulation.

[0005] To achieve the above objectives, the present invention provides a method for reconstructing echo-state multiplane features for complex scene perception, comprising the following steps:

[0006] S1: Image Acquisition and Multiplanar Feature Construction; First, a sequence of video images from the target scene is acquired, preprocessed after adding an instrument mask, and then the processed video image sequence is input into the dynamic-static feature separation network for multiplanar modeling, obtaining the static field... With dynamic field Two fields, then bilinear interpolation is used to convert the static field... Generated static feature plane With dynamic field Generated dynamic feature plane Perform feature fusion and finally output the original fused features. ;

[0007] S2: Construct a deep uncertainty filter; filter Z-axis information based on data characteristics, and select 3 reliable feature planes. With dynamic feature plane and Jointly construct dynamic representation By quantifying its relationship with The feature differences are used to construct a deep uncertainty filter; the static feature plane and dynamic feature plane generated in step S1 are input into the constructed deep uncertainty filter to model the differences between the static feature plane and the dynamic feature plane and filter them through a threshold, and finally output the dynamic filter;

[0008] S3: Construct a dynamic feature enhancement and weight fusion mechanism; use the dynamic filter output from step S2 for dynamic feature enhancement, and perform dynamic weight generation and fusion to highlight active dynamic information and achieve adaptive integration of dynamic and static features;

[0009] S4: Construct a color cycle refinement mechanism; achieve iterative correction of color values ​​in dynamic regions by cyclically updating color residual information;

[0010] S5: Construct a training and optimization mechanism for dynamic scene reconstruction; After completing steps S1, S2, S3, and S4, use a network model trained and optimized based on a dual loss constraint mechanism to perform forward inference and modeling of the dynamic scene, and finally output continuous rendered image frames and dynamic 3D reconstruction results through volume rendering.

[0011] Step S1 of the present invention specifically includes:

[0012] S11: First, acquire a sequence of video images from the target scene. After adding an instrument mask, preprocess the video images. Then, jointly encode the pixels and their timestamps in the processed video image sequence to form a 4D spatiotemporal mesh. ;

[0013] S12: Then, the dynamic and static feature separation network is used to decompose it into six different planes;

[0014] Three spatial planes Used to describe static fields The other three spacetime planes Used to characterize dynamic fields ;

[0015] S13: Then use bilinear interpolation to refine the obtained 4D spatiotemporal mesh. Projected onto the decomposed dynamic field and static field Above, dynamic feature planes are generated respectively. and static feature plane Finally, the dynamic feature plane and static feature plane The original fused features are output after feature fusion. :

[0016] (1).

[0017] Step S2 of the present invention specifically includes:

[0018] S21: Filter Z-axis information based on data characteristics and select 3 reliable feature planes. With dynamic feature plane and Jointly construct dynamic representation By quantifying its relationship with Feature differences are used to construct a deep uncertainty filter;

[0019] S22: The dynamic feature plane generated in step S1 A dynamic representation is constructed after input. ,use To evaluate the dynamic representation of a pixel and static feature plane The difference between them;

[0020] S23: Subsequently, the dynamic characterization of step S21 is described. With static feature plane Perform a global average calculation:

[0021] (2)

[0022] Obtain the average change and Used as a threshold hyperparameters Identify dynamic features with higher activation levels in the time dimension;

[0023] By each value With threshold A dynamic-static difference comparison is performed, and the result of the comparison determines the value of the dynamic characteristic index.

[0024] When comparing dynamic and static differences When the dynamic characteristic index is 1, it is set to 0 otherwise.

[0025] The dynamic feature index is then processed by average pooling and max pooling to generate a time-dimension related index. Dynamic filters of the same size.

[0026] Step S3 of the present invention specifically includes:

[0027] S31: Dynamic Feature Enhancement; The temporal dimension is corrected using a dynamic filter and then fused with the original features. Element-wise multiplication generates dynamic augmented features This allows for the filtering and enhancement of active and dynamic elements;

[0028] The dynamic filter output from step S2 is applied to the time dimension. Perform filtering to generate the corrected time dimension. Then, the original fused features output in step S1 are... With the time dimension Element-wise multiplication is performed between them to obtain enhanced features that more closely reflect the dynamic characteristics over time. This process retains the fusion feature components closely related to temporal dynamics, thereby achieving feature selection; finally, this enhanced feature... The original fusion features output from step S1 Perform element-wise multiplication again to generate dynamically enhanced features. ;

[0029] S32: Dynamic weight generation and fusion; adaptively generating dynamic and static weights based on the activation ratio of the dynamic filter, and enhancing the features. With static feature plane Weighted fusion is performed to obtain the final fusion features. ;

[0030] First, calculate the total sequence length l of the dynamic filter and the number of eigenvalues ​​of 1. To determine dynamic characteristics The weights of static features are set to [weights]. And the final fusion features Through weighted dynamic enhancement features and weighted original fusion features Adding them together yields:

[0031] (3).

[0032] Step S4 of the present invention specifically includes:

[0033] S41: Final fused features output from step S3 After passing through a lightweight decoder, the initial value of the output color is obtained. and opacity In the color cycle refinement mechanism, set initial values Let the dynamic mask of the dynamic filter output in step S2 be... The hidden state is obtained by multiplying each channel element by element. :

[0034] (4)

[0035] S42: Hide state With initial value The data is stitched together and used as input to a multilayer perceptron (MLP) to obtain reset gates. With the update gate :

[0036] (5)

[0037] Based on reset gate With hidden state Calculate candidate hidden states :

[0038] (6)

[0039] S43: Calculate the color residual And obtain the updated color value Complete the iterative update of color values ​​in areas of significant dynamic change in soft tissue:

[0040] (7).

[0041] Step S5 of the present invention specifically includes:

[0042] S51: Utilize the rendered color values The true color value of the image To simultaneously optimize the micro multilayer perceptron and sampling features, the color loss function is defined as follows:

[0043] (8)

[0044] Rendering depth value and depth information generated through stereo matching To assist in optimizing neural networks, the depth loss is defined as follows:

[0045] (9)

[0046] Total variation regularization is used to optimize static features, defined as follows:

[0047] (10)

[0048] in: This represents the squared difference between adjacent eigenvalues ​​along the H direction;

[0049] It means The squared difference between adjacent eigenvalues ​​in the direction;

[0050] To robustly reconstruct the plastic organization from a limited perspective, a temporal smoothing regularization is further applied to all dynamic features, defined as follows:

[0051] (11)

[0052] The purpose is to calculate the squared difference between adjacent frames;

[0053] In addition, a histogram calculation function was introduced. To construct a histogram loss function, we train the sampling network to improve the accuracy of sampling points and enhance volumetric rendering. The specific definition is as follows:

[0054] (12)

[0055] in: It is the number of sampling layers;

[0056] express layer;

[0057] Indicates in In the layer The weights of the sampling points;

[0058] It is a constant used for numerical stability;

[0059] The total loss for each iteration of optimization is defined as:

[0060] (13)

[0061] In all experiments, parameters were set. .

[0062] Compared with existing technologies, this invention constructs a multi-plane feature representation with dynamic and static decoupling and introduces an echo-state temporal feedback mechanism to achieve differentiated modeling and joint expression of dynamic and static features. This can effectively improve the temporal consistency and structural fidelity in the dynamic modeling process and is widely applicable to fields such as medical image reconstruction, surgical scene perception, virtual simulation and complex environment modeling.

[0063] The multi-plane feature modeling framework constructed in this invention includes an orthogonal decomposition structure of static and dynamic planes. Combined with a deep uncertainty filtering module, it filters and enhances active dynamic regions in the time dimension, thereby significantly improving the model's ability to model complex dynamic environments such as soft tissue deformation and instrument occlusion. This feature decomposition and echo-state response mechanism enables the system to correct for time dimension differences like echo feedback, effectively mitigating dynamic blurring and detail loss problems.

[0064] The dynamic feature enhancement and weight fusion mechanism designed in this invention generates adaptive weights, highlighting the contribution of dynamic regions while maintaining the stability of static structures during the integration of dynamic and static features. This ensures that the final fused features possess both high dynamic expressiveness and spatial continuity. Simultaneously, combined with a color cycle refinement module, it uses a residual iterative update method to cyclically correct the color values ​​of dynamic regions, achieving high-precision texture restoration for soft tissues and complex dynamic surfaces.

[0065] This invention introduces inter-frame constraint loss and incremental motion consistency loss during training, forming a dual-loss joint optimization framework. Without relying on external labels or global geometric priors, supervision can be achieved using the input image sequence and the model's own structural information, significantly improving the algorithm's transferability and adaptability to weak supervision, and enhancing the stability of temporal modeling.

[0066] The echo-state multiplane feature reconstruction method constructed in this invention has the following advantages in dynamic scene reconstruction tasks: it can achieve high-fidelity dynamic modeling without relying on external global point clouds; it can achieve adaptive decomposition and fusion of dynamic and static features, reducing modeling ambiguity; it has the characteristics of strong structural continuity, high texture restoration accuracy, and excellent temporal generalization ability, and is suitable for various tasks such as medical imaging, surgical navigation and complex scene perception, and has good engineering implementation and promotion value. Attached Figure Description

[0067] Figure 1 This forms the overall operational framework of the multi-plane feature reconstruction method of the present invention;

[0068] Figure 2 This is an overall flowchart of the dynamic feature enhancement, dynamic weight generation and fusion, and color cycle refinement modules in this invention;

[0069] Figure 3This is an overall flowchart of the multi-plane feature reconstruction method of the present invention;

[0070] Figure 4 This is a qualitative comparison of different methods using the method of the present invention in typical frames of the EndoNerf dataset;

[0071] Figure 5 This is a qualitative comparison of different methods using the method of this invention in typical frames of the StereoMIS dataset. Detailed Implementation

[0072] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0073] To achieve high-fidelity modeling and temporal consistency in 3D reconstruction within complex dynamic scenes, this invention proposes an echo-state multi-plane feature reconstruction method for complex scene perception. Its overall operational framework is as follows: Figures 1-3 As shown, the system includes a deep uncertainty filtering module, a dynamic feature enhancement module, a dynamic weight generation and fusion module, and a color cycle refinement module. The deep uncertainty filtering module uses a deep uncertainty filtering mechanism to model the differences between static and dynamic feature planes, generate a dynamic filter, and filter uncertain regions to retain high-confidence dynamic responses.

[0074] To achieve both high-fidelity modeling and temporal consistency in 3D reconstruction within complex dynamic scenes, this invention proposes an echo-state multi-plane feature reconstruction method for complex scene perception. The specific implementation steps are as follows:

[0075] S1: Image Acquisition and Multiplanar Feature Construction; First, a sequence of video images from the target scene is acquired, preprocessed after adding an instrument mask, and then the processed video image sequence is input into the dynamic-static feature separation network for multiplanar modeling, obtaining the static field... With dynamic field Two fields, then bilinear interpolation is used to convert the static field... Generated static feature plane With dynamic field Generated dynamic feature plane Perform feature fusion and finally output the original fused features. ;

[0076] Step S1 is as follows:

[0077] S11: First, acquire a sequence of video images from the target scene. After adding an instrument mask, preprocess the video images. Then, jointly encode the pixels and their timestamps in the processed video image sequence to form a 4D spatiotemporal mesh. ;

[0078] S12: Then, the dynamic and static feature separation network is used to decompose it into six different planes;

[0079] Three spatial planes Used to describe static fields The other three spacetime planes Used to characterize dynamic fields ;

[0080] S13: Then use bilinear interpolation to refine the obtained 4D spatiotemporal mesh. Projected onto the decomposed dynamic field and static field Above, dynamic feature planes are generated respectively. and static feature plane Finally, the dynamic feature plane and static feature plane The original fused features are output after feature fusion. :

[0081] (1).

[0082] S2: Construct a deep uncertainty filter; filter Z-axis information based on data characteristics, and select 3 reliable feature planes. With dynamic feature plane and Jointly construct dynamic representation By quantifying its relationship with The feature differences are used to construct a deep uncertainty filter; the static feature plane and dynamic feature plane generated in step S1 are input into the constructed deep uncertainty filter to model the differences between the static feature plane and the dynamic feature plane and filter them through a threshold, and finally output the dynamic filter;

[0083] Step S2 is as follows:

[0084] S21: Filter Z-axis information based on data characteristics and select 3 reliable feature planes. With dynamic feature plane and Jointly construct dynamic representation By quantifying its relationship with Feature differences are used to construct a deep uncertainty filter;

[0085] S22: Combine the dynamic feature plane generated in step S1 with... A dynamic representation is constructed after input. ,use To evaluate the dynamic representation of a pixel and static feature plane The difference between them;

[0086] S23: Subsequently, the dynamic characterization of step S21 is described. With static feature plane Perform a global average calculation:

[0087] (2)

[0088] Obtain the average change and Used as a threshold hyperparameters Identify dynamic features with higher activation levels in the time dimension;

[0089] By each value With threshold A dynamic-static difference comparison is performed, and the result of the comparison determines the value of the dynamic characteristic index.

[0090] When comparing dynamic and static differences When the dynamic characteristic index is 1, it is set to 0 otherwise.

[0091] The dynamic feature index is then processed by average pooling and max pooling to generate a time-dimension related index. Dynamic filters of the same size.

[0092] S3: Construct a dynamic feature enhancement and weight fusion mechanism; use the dynamic filter output from step S2 for dynamic feature enhancement, and perform dynamic weight generation and fusion to highlight active dynamic information and achieve adaptive integration of dynamic and static features, thereby improving the expressive power and reconstruction accuracy of dynamic scenes.

[0093] Step S3 is as follows:

[0094] S31: Dynamic Feature Enhancement; The temporal dimension is corrected using a dynamic filter and then fused with the original features. Element-wise multiplication generates dynamic augmented features This allows for the filtering and enhancement of active and dynamic elements;

[0095] The dynamic filter output from step S2 is applied to the time dimension. Perform filtering to generate the corrected time dimension. Then, the original fused features output in step S1 are... With the time dimension Element-wise multiplication is performed between them to obtain enhanced features that more closely reflect the dynamic characteristics over time. This process retains the fusion feature components closely related to temporal dynamics, thereby achieving feature selection; finally, this enhanced feature... The original fusion features output from step S1 Perform element-wise multiplication again to generate dynamically enhanced features. ;

[0096] S32: Dynamic weight generation and fusion; adaptively generating dynamic and static weights based on the activation ratio of the dynamic filter, and enhancing the features. With static feature plane Weighted fusion is performed to obtain the final fusion features. ;

[0097] First, calculate the total sequence length l of the dynamic filter and the number of eigenvalues ​​of 1. To determine dynamic characteristics The weights of static features are set to [weights]. And the final fusion features Through weighted dynamic enhancement features and weighted original fusion features Adding them together yields:

[0098] (3).

[0099] S4: Construct a color cycle refinement mechanism; achieve iterative correction of color values ​​in dynamic regions by cyclically updating color residual information, thereby improving texture consistency and reconstruction accuracy in highly dynamic regions such as soft tissue;

[0100] Step S4 is as follows:

[0101] S41: Final fused features output from step S3 After passing through a lightweight decoder, the initial value of the output color is obtained. and opacity In the color cycle refinement mechanism, set initial values Let the dynamic mask of the dynamic filter output in step S2 be... The hidden state is obtained by multiplying each channel element by element. :

[0102] (4)

[0103] S42: Hide state With initial value The data is stitched together and used as input to a multilayer perceptron (MLP) to obtain reset gates. With the update gate :

[0104] (5)

[0105] Based on reset gate With hidden state Calculate candidate hidden states :

[0106] (6)

[0107] S43: Calculate the color residual And obtain the updated color value Complete the iterative update of color values ​​in areas of significant dynamic change in soft tissue:

[0108] (7).

[0109] S5: Construct a training and optimization mechanism for dynamic scene reconstruction; After completing steps S1, S2, S3, and S4, use a network model trained and optimized based on a dual loss constraint mechanism to perform forward inference and modeling of the dynamic scene, and finally output continuous rendered image frames and dynamic 3D reconstruction results through volume rendering.

[0110] Step S5 is as follows:

[0111] S51: Utilizing the rendered color values ​​and The true color value of the image To simultaneously optimize the micro multilayer perceptron and sampling features, the color loss function is defined as follows:

[0112] (8)

[0113] Rendering depth value and depth information generated through stereo matching To assist in optimizing neural networks, the depth loss is defined as follows:

[0114] (9)

[0115] Total variation regularization is used to optimize static features, defined as follows:

[0116] (10)

[0117] in: This represents the squared difference between adjacent eigenvalues ​​along the H direction;

[0118] It means The squared difference between adjacent eigenvalues ​​in the direction;

[0119] To robustly reconstruct the plastic organization from a limited perspective, a temporal smoothing regularization is further applied to all dynamic features, defined as follows:

[0120] (11)

[0121] The purpose is to calculate the squared difference between adjacent frames;

[0122] In addition, a histogram calculation function was introduced. To construct a histogram loss function, we train the sampling network to improve the accuracy of sampling points and enhance volumetric rendering. The specific definition is as follows:

[0123] (12)

[0124] in: It is the number of sampling layers;

[0125] express layer;

[0126] Indicates in In the layer The weights of the sampling points;

[0127] It is a constant used for numerical stability;

[0128] The total loss for each iteration of optimization is defined as:

[0129] (13)

[0130] In all experiments, parameters were set. . Example

[0131] In this embodiment, the system is based on the Nerf dynamic-static separation neural implicit expression framework, combined with the dynamic enhancement and echo-state color cycling mechanism proposed in this invention, to achieve differentiated modeling of dynamic and static features in dynamic surgical scenarios. The model is trained using the Adam optimizer, and experiments are performed iteratively on a single GPU for convergence.

[0132] In this embodiment, to verify the effectiveness of the echo-state multiplane feature reconstruction method proposed in this invention, a complete experimental system was constructed, and functional verification and performance evaluation were performed on a medical surgical scene dataset. Two public datasets, EndoNerf and StereoMIS, were selected: the EndoNerf dataset contains continuous dynamic image sequences in laparoscopic scenes, covering various soft tissue deformation and instrument interaction scenarios, while the StereoMIS dataset contains more complex binocular surgical image sequences, with strong dynamic blur and occlusion in the scenes.

[0133] Specifically, video image sequences from the EndoNerf dataset are first acquired, and instrument masks are added during preprocessing to remove occlusion interference. Then, the sequences are spatiotemporally co-encoded, and a dynamic-static feature separation network is used to model them into six orthogonal two-dimensional planes, thereby obtaining representations of the static and dynamic fields. Interpolation mapping is then used to achieve initial feature fusion. Next, a depth uncertainty filter is introduced to filter the differences between dynamic and static features, suppressing unreliable depth information caused by tissue deformation or instrument occlusion, and generating a dynamic filter. Based on this filter, dynamic feature enhancement and weight fusion are further performed, adaptively highlighting active dynamic parts while maintaining structural consistency with the static field. To improve the color precision of dynamic regions in soft tissue, this embodiment designs a color cyclic refinement mechanism, iteratively updating the color and opacity obtained from the initial decoding to achieve residual correction of the color in dynamic regions. Finally, in the training and optimization phase, a joint optimization framework based on color loss, depth loss, total variation regularization, temporal smoothing regularization, and histogram loss is constructed to constrain and guide the entire network. Through this process, the system can dynamically model and render high-fidelity volumes of continuous medical surgical scenes, ultimately outputting continuous image sequences and three-dimensional dynamic reconstruction results, effectively improving its adaptability to soft tissue deformation and surgical instrument interference.

[0134] In the experiments, the method of this invention was compared with several mainstream dynamic modeling and rendering methods, including EndoNerf, EndoSurf, Forplane, and DnFPlane. The evaluation metrics used were three image quality assessment standards: PSNR, SSIM, and LPIPS. The comparison results are presented in [the table / document / etc.]. Figure 4 and Figure 5 Experimental results show that the present invention achieves excellent performance on all indicators: it significantly improves PSNR and SSIM, and performs better on the LPIPS perception quality indicator, proving the effectiveness of the proposed method in reducing dynamic ambiguity and enhancing spatiotemporal consistency.

[0135] Furthermore, in the ablation experiments, the core modules proposed in this invention, including the depth uncertainty filtering module, dynamic feature enhancement module, dynamic weight generation and fusion module, and color cycle refinement mechanism, were removed one by one for comparison. The results showed that each module was a key factor in improving the overall performance. Among them, the depth uncertainty filtering module played a significant role in dynamic region enhancement, solving the problem of depth distortion in areas obscured by instruments and under tissue deformation. Depth distortion and artifacts are prone to occur under instrument obscuration and tissue deformation. The color cycle mechanism effectively alleviated the dynamic blurring problem in soft tissue contact deformation areas.

[0136] In summary, this embodiment verifies the feasibility and advancement of the present invention in medical imaging and complex dynamic scenes. The present invention not only achieves high-fidelity dynamic 3D modeling under complex conditions such as occlusion and self-deformation, but also significantly improves modeling accuracy and visual quality while reducing reliance on external depth supervision and prior knowledge, demonstrating excellent potential for medical imaging applications and promising prospects for widespread adoption.

[0137] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for echo-state multiplane feature reconstruction in complex scene perception, characterized in that, Includes the following steps: S1: Acquire images and construct multiplane features; First, a sequence of video images from the target scene is acquired. After adding an instrument mask, the images are preprocessed. Then, the processed video image sequence is input into a dynamic-static feature separation network for multi-plane modeling, yielding static field results. With dynamic field Two fields, then bilinear interpolation is used to convert the static field... Generated static feature plane With dynamic field Generated dynamic feature plane Perform feature fusion and finally output the original fused features. ; S2: Construct a deep uncertainty filter; filter Z-axis information based on data characteristics, and select 3 reliable feature planes. With dynamic feature plane and Jointly construct dynamic representation By quantifying its relationship with The feature differences are used to construct a deep uncertainty filter; the static feature plane and dynamic feature plane generated in step S1 are input into the constructed deep uncertainty filter to model the differences between the static feature plane and the dynamic feature plane and filter them through a threshold, and finally output the dynamic filter; S3: Construct a dynamic feature enhancement and weight fusion mechanism; use the dynamic filter output from step S2 for dynamic feature enhancement, and perform dynamic weight generation and fusion to highlight active dynamic information and achieve adaptive integration of dynamic and static features; S4: Construct a color cycle refinement mechanism; achieve iterative correction of color values ​​in dynamic regions by cyclically updating color residual information; S5: Construct a training and optimization mechanism for dynamic scene reconstruction; After completing steps S1, S2, S3, and S4, use a network model trained and optimized based on a dual loss constraint mechanism to perform forward inference and modeling of the dynamic scene, and finally output continuous rendered image frames and dynamic 3D reconstruction results through volume rendering.

2. The method for reconstructing echo-state multiplane features for complex scene perception according to claim 1, characterized in that, Step S1 is as follows: S11: First, acquire a sequence of video images from the target scene. After adding an instrument mask, preprocess the video images. Then, jointly encode the pixels and their timestamps in the processed video image sequence to form a 4D spatiotemporal mesh. ; S12: Then, the dynamic and static feature separation network is used to decompose it into six different planes; Three spatial planes Used to describe static fields The other three spacetime planes Used to characterize dynamic fields ; S13: Then use bilinear interpolation to refine the obtained 4D spatiotemporal mesh. Projected onto the decomposed dynamic field and static field Above, dynamic feature planes are generated respectively. and static feature plane Finally, the dynamic feature plane and static feature plane The original fused features are output after feature fusion. : （1）。 3. The method for reconstructing echo-state multiplane features for complex scene perception according to claim 2, characterized in that, Step S2 is as follows: S21: Filter Z-axis information based on data characteristics and select 3 reliable feature planes. With dynamic feature plane and Jointly construct dynamic representation By quantifying its relationship with Feature differences are used to construct a deep uncertainty filter; S22: The dynamic feature plane generated in step S1 A dynamic representation is constructed after input. ,use To evaluate the dynamic representation of a pixel and static feature plane The difference between them; S23: Subsequently, the dynamic characterization of step S21 is described. With static feature plane Perform a global average calculation: （2） Obtain the average change and Used as a threshold hyperparameters Identify dynamic features with higher activation levels in the time dimension; By each value With threshold A dynamic-static difference comparison is performed, and the result of the comparison determines the value of the dynamic characteristic index. When comparing dynamic and static differences When the dynamic characteristic index is 1, it is set to 0 otherwise. The dynamic feature index is then processed by average pooling and max pooling to generate a time-dimension related index. Dynamic filters of the same size.

4. The method for reconstructing echo-state multiplane features for complex scene perception according to claim 1, characterized in that, Step S3 is as follows: S31: Dynamic Feature Enhancement; The temporal dimension is corrected using a dynamic filter and then fused with the original features. Element-wise multiplication generates dynamic augmented features This allows for the filtering and enhancement of active and dynamic elements; The dynamic filter output from step S2 is applied to the time dimension. Perform filtering to generate the corrected time dimension. ; Then, the original fused features output in step S1 are... With the time dimension Element-wise multiplication is performed between them to obtain enhanced features that more closely reflect the dynamic characteristics over time. This process retains the fusion feature components closely related to temporal dynamics, thereby achieving feature selection; finally, this enhanced feature... The original fusion features output from step S1 Perform element-wise multiplication again to generate dynamically enhanced features. ; S32: Dynamic weight generation and fusion; adaptively generating dynamic and static weights based on the activation ratio of the dynamic filter, and enhancing the features. With static feature plane Weighted fusion is performed to obtain the final fusion features. ; First, calculate the total sequence length l of the dynamic filter and the number of eigenvalues ​​of 1. To determine dynamic characteristics The weights of static features are set to [weights]. And the final fusion features Through weighted dynamic enhancement features and weighted original fusion features Adding them together yields: （3）。 5. The method for reconstructing echo-state multiplane features for complex scene perception according to claim 1, characterized in that, Step S4 is as follows: S41: Final fused features output from step S3 After passing through a lightweight decoder, the initial value of the output color is obtained. and opacity In the color cycle refinement mechanism, set initial values. Let the dynamic mask of the dynamic filter output in step S2 be... The hidden state is obtained by multiplying each channel element by element. : （4） S42: Hide state With initial value The data is stitched together and used as input to a multilayer perceptron (MLP) to obtain reset gates. With the update gate : （5） Based on reset gate With hidden state Calculate candidate hidden states : （6） S43: Calculate the color residual And obtain the updated color value Complete the iterative update of color values ​​in areas of significant dynamic change in soft tissue: （7）。 6. The method for reconstructing echo-state multiplane features for complex scene perception according to claim 1, characterized in that, Step S5 is as follows: S51: Utilize the rendered color values The true color value of the image To simultaneously optimize the micro multilayer perceptron and sampling features, the color loss function is defined as follows: （8） Rendering depth value and depth information generated through stereo matching To assist in optimizing neural networks, the depth loss is defined as follows: （9） Total variation regularization is used to optimize static features, defined as follows: （10） in: This represents the squared difference between adjacent eigenvalues ​​along the H direction; It means The squared difference between adjacent eigenvalues ​​in the direction; To robustly reconstruct the plastic organization from a limited perspective, a temporal smoothing regularization is further applied to all dynamic features, defined as follows: （11） The purpose is to calculate the squared difference between adjacent frames; In addition, a histogram calculation function was introduced. To construct a histogram loss function, we train the sampling network to improve the accuracy of sampling points and enhance volumetric rendering. The specific definition is as follows: （12） in: It is the number of sampling layers; express layer; Indicates in In the layer The weights of the sampling points; It is a constant used for numerical stability; The total loss for each iteration of optimization is defined as: （13） In all experiments, parameters were set. .

Images